The Rise of JS and the New Server-side

I started writing this article in my excitement of Server-side Javascript (SSJS) and how I’ve become increasingly interested in Javascript (JS) over the past years. But at the end of the day, Javascript is just another programming language, its importance is defined by the frameworks and applications that are built on top of it. But while I was writing this, I noticed that together with the rise of Javascript, something much bigger is happening in the way services are and need to be structured.

Javascript in the Browser

A little more than a year ago, I wrote about the grim future of Adobe Flash. I think by now, there are even more signs that Flash is dying in it’s current form. Flash got a big stab in the back recently by Youtube when they started supporting the HTML5 video embed and also Apple is once again not supporting Flash on the Apple iPad.

Javascript is getting faster and stronger in the browser. A good example is the powerful, open-source V8 Engine developed for Google Chrome. Also, mobile devices are increasingly centered around Javascript, all Palm Pre apps are written in Javascript; the iPhone Google App is pretty much a JS interface in Safari; And there are cool new frameworks out there that allow you to write native iPhone/Android/Desktop apps completely in JS.

A simple client-server web application using the LAMP stack.
An oldschool client-server scenario with a web application running on the LAMP stack.

Javascript Beyond the Browser

At the end of the day JS is just a programming language, it’s success is determined by the amount of adoption and the problems it can solve. So far most of the problems solved by JS have been in the browser. But this is about to change.

Server-side Javascript is nothing new, but it’s not common yet. There are many open-source frameworks sprouting up now and I think the rise of cloud-services like the Joyent Smart Platform will definitely help SSJS taking off. Also, there are some serious economic benefits to having JS be the only programming language used in a web application. Do you know any programmers that don’t know how to write HTML? How about Javascript? JS and it’s relatives – ActionScript and Java – are very widely known by developers. This means that finding and training JS developers is easier.

These SSJS frameworks need to accommodate very different use-cases than their client-side counterparts (jQuery, Prototype, …). A SSJS framework shouldn’t care about the DOM, but it should care about things like IO / file access, network and database connectivity, template rendering, communication with existing dynamic libraries, etc. The server-side use-cases are completely different!

The Migration to Client-side

For the past years, I have been involved in scaling a huge service called Smart.fm. Over the course of 2009 we’ve made some big adjustments in our technology strategy. We’ve been offloading more and more controller logic into the browser using jQuery/Javascript. This also means that we’ve had to beef up our API and make sure all data can be accessed smoothly using JSON feeds. This has given us huge benefits in performance, modularity, manageability and user experience. But when looking at top-sites out there like Facebook and Google, this seems to be common practice nowadays. More and more logic is now running in the browser.

A typical Rails stack application.
A typical client-server scenario with a web application using the Rails stack.

So all of this has got me thinking: what does the server-side still mean for a web application?

The controlling of the UI flow and interactions is now done by Javascript in the client. This means that access control is purely done by the API, the client will not get the data it doesn’t have access too. This means that all access control and data routing will be controlled by the API.

What about server-side rendering of templates? Surely we need this for the search engine bots, but do we really need to treat them the same as users? Search engine bots like the Google Bot need to be able to scrape server-side rendered HTML information and will not execute any client-side logic. But as simple keyword search is becoming less relevant, we need to start building different interfaces for machines. Why not render some simple HTML for those dumb Google Bots and not worry about it in our UI? Oh and while we do that, render something more semantic for those smarter bots!

Architecturally, we can define these components in a modern web application:

  • An API that serves as an interface to the business models and provides access control
  • A user application that runs completely in the browser that uses the API for access to the domain’s data, but also interfaces with a multitude of other JSON API’s (Google Analytics can also be seen as a one-way JSONP call).
  • Simple server-side rendering capabilities for delivering the user application code and for serving semi-structured content for the search bots.

So when all rendering and UI flow logic moves to the client, what responsibilities remain for the server? It is very important to ask this question when you build a new server-side framework. What problems does it need to solve? Most server-side web frameworks are built around the increasingly irrelevant Model-View-Controller (MVC) pattern.

A Next-generation Architecture

Now that the responsibilities of both the client and the server have changed, we need to build new frameworks accordingly.

An example next-gen architecture

The browser, technically known as a ‘user agent’, needs a framework that takes care of all the UI flow. This means that we need the MVC pattern running on the client. This ‘user agent framework’ needs controllers that can render views and it needs a comprehensive routing system based on anchors.

As a business, now that more of your code (a.k.a. intellectual property) is running openly in your browser, where does your strategic advantage come from? Well, it comes from making your backend very intelligent.

The server, who I like to call ‘machine agent’, needs a very solid Application Programmer Interface. The sole purpose of this API is interfacing with different user interfaces (web, iphone, ipad, …). However, it should have a separate facility to deal with other ‘machine agents’ like Google Bots. Other servers that interface with your system have very different needs than third-party user interface applications. An example of this would be API calls that perform bulk operations (like Twitter’s mass-following call).

In order to deliver smart services that can compete, machine agents need to be able to analyze the state of their internal universe (business models) and act on them. This could be a simple iPhone push notification that gets sent after a certain event happens, or it could be after the completion of a statistical analysis. Web backends need to become more event-driven (like Javascript) because a simple cronjob will no longer suffice.

Machine-agents need to be able to integrate with other services. Sometimes this means that they need to be able to fetch many feeds from a single service. I consider this to be one of the hardest technical challenges of most projects I work on. For example, if I want to do something semi-intelligent with someone’s Twitter activity, I will need to first fetch all his data, import it and then analyze it. This is very expensive and I haven’t found any economically viable solutions yet. Of course no one wants to passively fetch feeds and then analyze them, it would be much better to do something like server-to-server long polling (e.g. listening for Tweets with a certain keyword), but sometimes you have no choice: you have to aggregate.

Conclusion

The use of Javascript on websites has been unstoppable and now it has spread to other platforms like the Palm Pre and iPhone. Javascript is a programming language that’s here to stay.

It is now common practice for web application developers to off-load a lot of work to the browser. This migration to the browser together with the changing technology landscape requires us to re-think web application architectures and frameworks.

Posted in Uncategorized | 23 Comments

Rethinking the Knowledge Economy

During the few ‘off hours’ I have, I love thinking about innovation and technological change in the grand scheme of things. In fact, a couple of weeks ago I did a weekend trip to New York on my own budget to attend the Singularity Summit 2009 – a conference of futurists. In my recent thinking I’ve stumbled upon a nagging little problem: The concept of the Knowledge Economy is very ambiguous and ill defined.

When I look up the definition of Knowledge Economy on Wikipedia, there is already a big alarm bell going off in my head when I read: “In a knowledge economy, knowledge is a product, in knowledge-based economy, knowledge is a tool.”. The words product and tool treat knowledge on par with physical goods, which in my mind is the reflection of a fundamental disconnect.

Knowledge

In the academic world there is no real consensus on the difference between information and knowledge, a definition that’s not at all trivial. Asking the question will most likely yield different answers depending on which ‘field of expertise’ you consult (management, philosophy, computer science), but there is a common way of explaining the concepts of data, information and knowledge on a continuum:

  • Data is raw facts
  • Information is data that is organized, analyzed or placed into a certain context
  • Knowledge is internalized information: skills, experience, cognition, etc.

“Knowledge is the whole body of cognition and skills which individuals use
to solve problems. It includes both theories and practical, everyday rules
and instructions for action. Knowledge is based on data and information, but

unlike these, it is always bound to persons. It is constructed
by individuals, and represents their beliefs about causal relationships”

(Probst, Raub & Romhardt, 2000)

There is also a very popular view (by Nonaka & Takeuchi) in which knowledge is divided up into explicit knowledge and tacit knowledge. But this widely accepted view has now come into dispute because many claim the concept of explicit knowledge is no different than information.

No matter what stance you take, it’s clear that knowledge always has a certain context associated with it. On one end of the spectrum this could simply mean that certain chunks of knowledge are useless without other chunks of knowledge (still explicit). In other cases it might be that the knowledge is completely useless without a lifetime of experience or skills (tacit).

However, I do think it’s important to realize that while tacit knowledge is a lot less ‘portable’ than explicit knowledge, this could change very quickly with the advent of technology. With the existence of zero-cost communication, learning-enhancement software and artificial intelligence, tacit knowledge is also becoming more ‘portable’.

Many thanks go out to Jozua Loots who helped me with the question “What’s the difference between knowledge and information?”. All done through an awesome new (synaptic) Q&A service called Aardvark.

Informational v.s. Physical

To understand the fundamentals of the problem, we have to take a look at the difference between information and physical objects. Physical objects abide by different laws than information. A physical object can only exist in one place in one time and it deteriorates when used or touched. Information on the other hand, can exist in many places at any time and multiplies when touched.

Picture 26

Thanks to zero-cost communication, the replicating nature of information has showed itself over the last decade. There are now vast amounts of knowledge (and obsoledge) being generated every day, making many derivatives of information (content, knowledge) a commodity. Kevin Kelley explains this well in his essay ‘Better than Free’, where he compares the internet to a giant copy machine where the copies drop in value. Interestingly, when those copies become abundant, the value starts shifting to what’s scarce: the attention of people. This is where the concept of the Attention Economy starts.

A Non-Economy?

If information is so fundamentally different than material goods, you can start asking the question: Does the term ‘economy’ apply to knowledge at all? Let’s take a look at the wikipedia definition of the word ‘economy’:

“An economy is the ways in which people use their environment to meet their material needs. It is the realized economic system of a country or other area. It includes the production, exchange, distribution, and consumption of goods and services of that area.”

And here is the definition of an ‘economic system’:

“An economic system is the system of production, distribution and consumption of goods and services of an economy. Alternatively, it is the set of principles and techniques by which problems of economics are addressed, such as the economic problem of scarcity through allocation of finite productive resources.”

Both definitions are inherently bound to ‘goods’ and the fundamentals of finite production. The problem is that with information, there is infinite free replication. This explains why it is so hard to use traditional methods of economics to measure and understand value created by information. But fortunately there is hope in the definition of ‘economics’ itself (by Lionel Robbins in 1932):

“The science which studies human behaviour as a relationship between ends and scarce means which have alternative uses.”

Because even though the characteristics of information/knowledge are so different , the economic fundamentals of abundance and scarcity still apply.

Informational Drivers of Economic Growth

The influence of information on wealth creation is quite complicated. The biggest mistake people make is treating information (or it’s derivatives) as a physical good that can be traded. So if you can’t sell it, it has no value? Yet information plays a profound role in driving economic growth.

Hans Rosling, a Swedish econometrist has given several TED talks in which he showed how developing countries have been caching up with great speeds. Every booming developing country had it’s own drivers of growth, but one can imagine that a common driver would be the availability of ‘know how’. This ‘know how’ took the Europeans centuries to develop and apply, but for developing countries this was readily available and could be applied fairly quickly. This application of explicit knowledge set off the main driver of growth: change. And when you apply new knowledge to a country that needs to build things from scratch, you get a rapid rate of non-incremental change.

On the scale of an entire economy, it’s really the non-incremental change that matters. An example of normal incremental change could be aesthetics, the improvement of physical products.  Another example would be a well-oiled service industry that services existing markets. All of these dwindle in comparison to the amount of value created by fundamental change. Real innovation will create and destroy new markets (e.g. telephony, online advertising, social networking, etc) whereas incremental change merely optimizes existing market dynamics.

The rapid shift from physical systems to more informational systems – informationization – goes hand in hand with non-incremental change. When a system becomes more informational and has less physical obstacles, changes can happen more quickly. And these changes are non-incremental, meaning that informational systems will have more paradigm shifts, are less predictable and have more volatility. Nassim Nicholas Taleb (NNT) explains this in a different way in his famous book The Black Swan, whereby he defines an Extremistan (informational-law world) and a Mediocrestan (physical-law world).

To summarize:

  • It’s not the explicit knowledge that creates value, but it’s rather the flow and application of it.
  • Non-incremental change is the main driver of economic growth
  • The more informational a system becomes, the more non-incremental changes will happen

Note: A nice example of informationization can be found in the biography of Nikola Tesla, one of history’s greatest inventors. Tesla had a special brain condition where he could visualize and iterate his inventions in his mind using his photographic memory. This allowed him to innovate at a very rapid pace, because he did far less physical experiments in the innovation process.

Stimulating Non-Incremental Change

Some governments, like my own (the Netherlands), come up with special ten year action plans that try to create a vibrant ‘Knowledge Economy’. What should these action plans entail and how relevant is the knowledge aspect of things?

The first thing that needs to happen is the stimulation of informationization. This requires removing physical constraints in for example bureaucracy. Many governmental organizations still require you to handle paperwork with real paper or require you to unnecessarily interface with a person. Also, there needs to be a general reduction of the amount of bureaucracy. This could be done for example by reducing the amount of certifications required to start a certain (informational) business.

Education needs to stimulate independent thinking. They need to stimulate their students to do new things (sponsor adventurous travels?), but more importantly: they need to shift focus from ‘knowing’ to applying knowledge and using creativity.

Corporations need to be formed ever quickly, but more importantly, they need to be dissolved quickly too. Companies – that are getting smaller and leaner – must be able to fail early and often in order for real innovation to happen. This also means that you need a culture that can deal well with failure and makes sure that the people involved don’t have to deal with ‘face loss’. On the flip side people need to be rewarded when they are successful, this might mean having a more loose taxation system for the wealthy. One idea here could be to allow tax-free re-investments of earned capital to stimulate successful entrepreneurs to become angel investors.

Which brings me to entrepreneurship: you need a very vibrant investment climate that has VC’s and angel investors that invest in bold ideas. Many countries outside of the US cope with a sickening amount of risk averseness. When you present a prototype to a European investor, they don’t ask you when you will generate revenue, no, they will ask you: “when will you break even?”.

Summary

The economics of a world that is becoming ever more complex are not as simple as they have been. The concept of a “Knowledge Economy” is outdated and is not factoring the new dynamics of informationization. The real driver behind economic growth is non-incremental change, something that is catalyzed by informationization. In order to gain economically, governing bodies, companies and people need to stop resisting informationization and go with the flow.

Posted in Uncategorized | Tagged , , , , , , | 26 Comments

The Synaptic Web

The Real-time Web (RTW) is buzzing all over the place, but I’m amazed by the amount of different interpretations of it. The RTW – or better said: large scale activity streams – are indeed reshaping the entire business and technology landscape, but how? This will be the first post of a series of articles that will be centered around this topic.

Micro-blogging (status updates) are the most visible form of the RTW, but in parallel we already had things like: RSS feeds, social bookmarking and video favorites, etc. Combine them all together and you have your first lifestreaming services. It’s important to note that lifestreams are the first form of activity streams that have more meta-data than for example a Twitter stream. A lifestream can for example include tags about a certain video that was favorited.

Instead of using the buzz word RTW, I’d prefer using the term Activity Stream. Why? Because the real-time aspect is only one half of the story. The important part is the ‘activity’.

Because of several intersecting trends, we have now arrived at yet another paradigm shift that will radically change business and technology.

The Attention Economy

Twitter is very cool and all, but can it be monetized? Hell yeah!! It’s like in the 19th century when oil was discovered: what the hell do we do with all this black stinky stuff? Well, it burns, so what? Activity is the new oil and Twitter will be the new ExxonMobil. Now it’s time to invent the appliances that run off of it.

In books about Google, like John Battelle’s The Search, the big take away is that Google is uniquely positioned between supply and demand. Well, they are, but they’re still very much in the middle. Like in most of their products, Google misses out on the ‘social stuff’. Having a direct activity communication line with demand is what Google is missing. In the Twittersphere, the dynamics are such that the supply trickles up to the demand (the people) in a very targeted way, much more targeted then Google Search.

This is where the true Attention Economy takes shape. When people explicitly state things about their activities, they leave an attention trail. This trail can be used to actively push relevant services to people.

But there is a caveat, in order to start using this ‘attention trail’, we need two things:

  • We need a lot more activity. Right now we only have activity going out that is highly explicit, like the favoring of a picture. But an attention trail should include much more activity events, like “How many seconds did you spend looking at this picture?”. Right now, most systems have separate mechanisms for measuring this implicit attention data (i.e. analytics), but one can imagine that this data will be highly desired to do internal content recommendation.
  • In order to properly analyze the attention trail, machines need to be able to understand each individual event. For example: “What was on the picture?”. One way of accomplishing this is by having proper meta-data on activity events. Something I have been struggling with in my Kakuteru project and the very thing that prompted me to build my own aggregation framework.

The Semantic Web

Data mining – the extraction of structured data from text – has been around for a long time. Over the past years several web companies have started doing this by means of open APIs, most notably Zemanta and OpenCalais. But interestingly, someone at Zemanta told me that most of their API usage is for extracting facts from micro-content like Tweets (so I imagine that they will adjust their product accordingly). Unfortunately, many view extracting facts from micro-content as more challenging than large formal texts. In a way it is, because a Tweet for example has a lot of slang and abbreviations in it, but you can make things a lot simpler by mixing in nano-formats. You can for example imagine that a hash-tagged tweet like this is easy for a machine to parse:

“I’m boarding my #flight KL862 to Amsterdam”

So if you combine status updates with the extraction of structured information, you can already imagine an Agent or Twitter-bot providing some useful information in return. In this case for example, it can provide information about the weather in the destination city. Or it could direct-message you if people in your social graph are boarding the airplane as well. These Semi-Intelligent Agents can be considered the first step towards the reasoning stage of the web.

The Synaptic Web

This word came on my radar the first time I was looking at a stream integrated commenting plugin for Wordpress called JS-Kit Echo. I think the word accurately describes what’s happening: the neuronification of the Web (thanks @mcmurrak). Futurist Kevin Kelley has written about this phenomenon as well. In essence, the connected nodes (people and computers) are an emergent global organism. But very concretely, the word “Synaptic Web” captures all important properties: ubiquity, meaning and interconnectedness (social).

So what is this Synaptic Web, and how will it change everything? Well, that remains to be seen and it’s something I plan to elaborate on in many articles to come. But here are some appetizers for thought:

  • Google’s Adwords were a very effective form of advertising because they were so targeted, but what if advertisements become so targeted that they are essentially offers/solutions provided to you? Will we still think of them as advertisements? How much are these ‘advertisers’ willing to pay for an advertisement that is 90% likely to trigger a full engagement with the consumer?
  • Demand for something specific is always temporary. When a big clustering of needs is ‘detected’ small nano-companies can be formed to fulfill these needs until they are gone.
  • Disambiguation is something that has been key to innovations in semantic technologies and will be very important for the Synaptic Web. Are we talking about the same thing, place, person? Machines need to be able to understand these things and serious innovation is needed here.
  • When machines are able to translate a concept like ‘NYC’ to a specific location with coordinates, they might become an important tool in quickly transferring concepts to each other. So instead of communicating in text, audio or video we could communicate in actual concepts. Of course this all depends on how we interface with the web.
Posted in Uncategorized | 76 Comments

A Twitter Bot in One Line of Code

Yep! It really IS possible with only one line of BASH (Terminal) code using the new Twitter Streaming API. Using a series of chained commands, this Twitter Bot listens for occurrences of ‘one line’ in the Twittersphere and will follow that person.

Here’s a version with some extra space for readability (does not really work with this extra space):

Note: This is just to display how more easy and easy it is to write bots/agents. Yes, this code consists of multiple commands and is over 400 columns. No, I don’t believe in measuring any software in LOC.

Note2: Many years ago I also wrote an IRC client in BASH

Posted in Uncategorized | Tagged , , , , | Leave a comment

5 Short Sci-fi Art Videos

Some nice arty technology clips that I’ve collected from my attention trail.

Artificial Paradise Inc

By Jean-Paul Frenay (via Humanity+). A good Nano-tech teaser.

Uploaded Companies

By Jean-Paul Frenay

As One

By Makoto Yabuki

What’s in the Box?

This short clip got a lot of attention, because it’s made by a young Dutch physics student named Tim Smit, with a budget of 125 dollars and a pizza.

La Jetee

One of the oldest and most famous science-fiction movies of all time. Black and white photos that picture a genius story. The movies 12 monkeys and many other are based on this.

Posted in Uncategorized | Leave a comment