The Rise of JS and the New Server-side

By Dominiek ter Heide | Published: February 3, 2010
Share 

I started writing this article in my excitement of Server-side Javascript (SSJS) and how I’ve become increasingly interested in Javascript (JS) over the past years. But at the end of the day, Javascript is just another programming language, its importance is defined by the frameworks and applications that are built on top of it. But while I was writing this, I noticed that together with the rise of Javascript, something much bigger is happening in the way services are and need to be structured.

Javascript in the Browser

A little more than a year ago, I wrote about the grim future of Adobe Flash. I think by now, there are even more signs that Flash is dying in it’s current form. Flash got a big stab in the back recently by Youtube when they started supporting the HTML5 video embed and also Apple is once again not supporting Flash on the Apple iPad.

Javascript is getting faster and stronger in the browser. A good example is the powerful, open-source V8 Engine developed for Google Chrome. Also, mobile devices are increasingly centered around Javascript, all Palm Pre apps are written in Javascript; the iPhone Google App is pretty much a JS interface in Safari; And there are cool new frameworks out there that allow you to write native iPhone/Android/Desktop apps completely in JS.

A simple client-server web application using the LAMP stack.
An oldschool client-server scenario with a web application running on the LAMP stack.

Javascript Beyond the Browser

At the end of the day JS is just a programming language, it’s success is determined by the amount of adoption and the problems it can solve. So far most of the problems solved by JS have been in the browser. But this is about to change.

Server-side Javascript is nothing new, but it’s not common yet. There are many open-source frameworks sprouting up now and I think the rise of cloud-services like the Joyent Smart Platform will definitely help SSJS taking off. Also, there are some serious economic benefits to having JS be the only programming language used in a web application. Do you know any programmers that don’t know how to write HTML? How about Javascript? JS and it’s relatives – ActionScript and Java – are very widely known by developers. This means that finding and training JS developers is easier.

These SSJS frameworks need to accommodate very different use-cases than their client-side counterparts (jQuery, Prototype, …). A SSJS framework shouldn’t care about the DOM, but it should care about things like IO / file access, network and database connectivity, template rendering, communication with existing dynamic libraries, etc. The server-side use-cases are completely different!

The Migration to Client-side

For the past years, I have been involved in scaling a huge service called Smart.fm. Over the course of 2009 we’ve made some big adjustments in our technology strategy. We’ve been offloading more and more controller logic into the browser using jQuery/Javascript. This also means that we’ve had to beef up our API and make sure all data can be accessed smoothly using JSON feeds. This has given us huge benefits in performance, modularity, manageability and user experience. But when looking at top-sites out there like Facebook and Google, this seems to be common practice nowadays. More and more logic is now running in the browser.

A typical Rails stack application.
A typical client-server scenario with a web application using the Rails stack.

So all of this has got me thinking: what does the server-side still mean for a web application?

The controlling of the UI flow and interactions is now done by Javascript in the client. This means that access control is purely done by the API, the client will not get the data it doesn’t have access too. This means that all access control and data routing will be controlled by the API.

What about server-side rendering of templates? Surely we need this for the search engine bots, but do we really need to treat them the same as users? Search engine bots like the Google Bot need to be able to scrape server-side rendered HTML information and will not execute any client-side logic. But as simple keyword search is becoming less relevant, we need to start building different interfaces for machines. Why not render some simple HTML for those dumb Google Bots and not worry about it in our UI? Oh and while we do that, render something more semantic for those smarter bots!

Architecturally, we can define these components in a modern web application:

  • An API that serves as an interface to the business models and provides access control
  • A user application that runs completely in the browser that uses the API for access to the domain’s data, but also interfaces with a multitude of other JSON API’s (Google Analytics can also be seen as a one-way JSONP call).
  • Simple server-side rendering capabilities for delivering the user application code and for serving semi-structured content for the search bots.

So when all rendering and UI flow logic moves to the client, what responsibilities remain for the server? It is very important to ask this question when you build a new server-side framework. What problems does it need to solve? Most server-side web frameworks are built around the increasingly irrelevant Model-View-Controller (MVC) pattern.

A Next-generation Architecture

Now that the responsibilities of both the client and the server have changed, we need to build new frameworks accordingly.

An example next-gen architecture

The browser, technically known as a ‘user agent’, needs a framework that takes care of all the UI flow. This means that we need the MVC pattern running on the client. This ‘user agent framework’ needs controllers that can render views and it needs a comprehensive routing system based on anchors.

As a business, now that more of your code (a.k.a. intellectual property) is running openly in your browser, where does your strategic advantage come from? Well, it comes from making your backend very intelligent.

The server, who I like to call ‘machine agent’, needs a very solid Application Programmer Interface. The sole purpose of this API is interfacing with different user interfaces (web, iphone, ipad, …). However, it should have a separate facility to deal with other ‘machine agents’ like Google Bots. Other servers that interface with your system have very different needs than third-party user interface applications. An example of this would be API calls that perform bulk operations (like Twitter’s mass-following call).

In order to deliver smart services that can compete, machine agents need to be able to analyze the state of their internal universe (business models) and act on them. This could be a simple iPhone push notification that gets sent after a certain event happens, or it could be after the completion of a statistical analysis. Web backends need to become more event-driven (like Javascript) because a simple cronjob will no longer suffice.

Machine-agents need to be able to integrate with other services. Sometimes this means that they need to be able to fetch many feeds from a single service. I consider this to be one of the hardest technical challenges of most projects I work on. For example, if I want to do something semi-intelligent with someone’s Twitter activity, I will need to first fetch all his data, import it and then analyze it. This is very expensive and I haven’t found any economically viable solutions yet. Of course no one wants to passively fetch feeds and then analyze them, it would be much better to do something like server-to-server long polling (e.g. listening for Tweets with a certain keyword), but sometimes you have no choice: you have to aggregate.

Conclusion

The use of Javascript on websites has been unstoppable and now it has spread to other platforms like the Palm Pre and iPhone. Javascript is a programming language that’s here to stay.

It is now common practice for web application developers to off-load a lot of work to the browser. This migration to the browser together with the changing technology landscape requires us to re-think web application architectures and frameworks.

Share 
This entry was posted in Uncategorized. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

21 Tweets 4 Other Comments

26 Comments

  1. takaaki
    Posted February 3, 2010 at 3:01 pm | Permalink

    I agree with the idea that JavaScript is a powerful language and a better alternative than Flash and that it is a language that is getting much more popular on the server-side.

    JavaScript is the most interesting language, but at the same time, it’s a very strange language. I have read a book called “JavaScript: Good Parts”, but it’s still hard to learn. You mention that ActionScript and Java are relatives of JavaScript. I feel so if I look at the Wikipedia entry for JavaScript. In the real world, I don’t think “learning cost” of learning JavaScript is small for ActionScript coders and Java coders. But I understand learning server-side JavaScript might be easy because we don’t have to think about browsers at all.

    A question: where do other scripting languages like Ruby and Python go in your model? Maybe we can still write in Ruby and convert the code into JavaScript with “JavaScript Virtual Machine” or something?

    Also in order for JavaScript to thrive, more efforts on tools are necessary. I know there is JSLint and also several testing frameworks. And documentation, screencasts, books and so on. Do you have any resources that you think “must-read” or “must-check”?

  2. zacharyc
    Posted February 6, 2010 at 3:47 am | Permalink

    What you’ve got here makes good sense. There is a clear shift in the design of web apps from being very server heavy to very client heavy, I believe, expressly for purpose of behaving more like real desktop apps. This is part of what is happening with HTML5.

    The only real downside I feel is that users expect web apps to preserve state between sessions or even computers. With heavy JS support for state, there should be some form of communication to the server so that a new session can be loaded from the existing state.

    This comment was originally posted on Bamboo Blog

  3. Gabe da Silveira
    Posted February 6, 2010 at 3:47 pm | Permalink

    It’s crazy how fast the Javascript world is evolving now. I remember when Rails first was released 5 years ago, everyone was amazed by the handy AJAX helpers, which by today’s standards look like crufty globbed on crap. Then RJS dropped and everyone was stoked because they could do client-side manipulation without actually touching any JS.

    But now, thanks in no small part to the efforts of guys like Douglas Crockford and John Resig (among many others of course) web developers are finally embracing Javascript for the wonderful language it is in its own right. Embrace Javascript fully leads to a more balanced consideration of architecture as you are suggesting here.

    I was led down this path myself in several areas as AJAX interactions in my app got more complex. It’s probably hard for people to grok when they haven’t been faced with the particular details that come up in practice, but having experienced them myself this post hits the nail square on the head.

    One example I worked through was a dynamic form consisting of many rows. Originally this was built a few years ago with a partial template representing a single row, and an RJS call to add new rows. The idea being that the HTML in the partial is DRY. It sounds good on paper, but as I needed to enhance this form I started having to jump through all sorts of hoops to inject the necessary state into the form. Things like generating unique dom ids so that the back-end and front-end can refer to the same element can get ugly fast.

    At one point it became so ugly that I spontaneously realized that the server side didn’t need to know anything about these rows. I set up the javascript to build dom fragments of new rows. Yes, this was a duplicate of the structure present in the template, but that’s such an inconsequential detail. Once the client side contains all the state, I was able to simply delete 80% of the server side code (all the cruft). Now the server only had to generate a plain HTML fragment, and make one call to initialize the whole javascript mechanism. I ended up with 30% less total code, and a snappy interface where before there were AJAX calls. Even with a well-optimized (and topographically nearby!) 150ms response time, the instant JS response is a dramatically better user experience. On the front-end the javascript is now smart about everything. Handling UI edge cases is now much easier. The backend generates some base HTML and then JS initializes and maintains the states.

    This technique taken its logical conclusion enables such things as an AJAX login that changes a full page to its logged-in state without a page refresh at all. For content pages that have only a handful of dynamic user-specific elements this type of approach can yield astonishing scalability improvements.

    Another benefit that falls out of this is that some things are easier to compute on the client side then they are on the server side. Take dynamic user-specific comment in a complex page. Generally you want to load all the necessary objects in the controller, then render a template which may render many small nested partials. You do this because of the N+1 problem of each template loading the single record it needs. But this also requires that the controller anticipate what is going to happen in all the templates. However, once the page is rendered, if the dynamic elements are semantically labeled, the javascript can harvest them all in one pass and request them via AJAX efficiently. In effect, the partials can generate a semantic HTML "thunk" that is lazily loaded by javascript on the front-end. An example of this are loading user ratings and favorites at http://www.theauteurs.com/films. This small tweak makes the full page cacheable.

    This comment was originally posted on Bamboo Blog

  4. maxthelion
    Posted February 6, 2010 at 7:47 pm | Permalink

    Well, state is persisted to your server via simple ajax requests so moving computer etc. shouldn’t really be an issue.

    This comment was originally posted on Bamboo Blog

  5. maxthelion
    Posted February 7, 2010 at 3:55 am | Permalink

    Thanks for commenting, I think you highlight some great examples. I struggled with the idea that the concept seems very clear to me, yet might be too abstract without concrete examples.

    I think you are right to say:

    "It’s probably hard for people to grok when they haven’t been faced with the particular details that come up in practice, but having experienced them myself this post hits the nail square on the head"

    My hope was that for some people it will resonate right away, while others may only "get it" later after going through a similar practical struggle themselves.

    This comment was originally posted on Bamboo Blog

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Additional comments powered by BackType