A Full Javascript Architecture, Part One – NodeJS

This post is the first of a three-part series showing how to build a complete javascript architecture using :

• NodeJS : For the server side. We will use Socket.IO to manage long-terms real-time connection.

• Google Chrome Extension : For the client side. WebSocket, Notification and Local Storage will be used.

• MongoDB : To store the datas.

To illustrate our architecture we are going to create a Node application that will track tweets about the “What’s Next” event and broadcast them in real time to the clients. The client will be a Google Chrome browser extension and will use two features of the HTML5 specification to display the tweets broadcasted by the server. In the last post I will introduce the MongoDB database, we will use it to store tweets and provide some statistics when the event ends.
 

This first part is dedicated to the creation of a Node application. We will see step by step how to install Node on a Linux environment and setup a HTTP server with WebSocket support.

Introduction to Node

What is Node ?

Node is an open source toolkit for developing server side applications based on the V8 JavaScript engine. Like Node, V8 is written in C++ and is mostly known for being used in Google Chrome.
Node is part of the Server Side JavaScript environnement and extend JavaScript API to offer usual server side functionalities. Node base API can be extended by using the CommonJS module system.
Node has be created by Ryan Dahl in February 2009 and has since become very popular. It’s spirit is similar to Twisted for Python and EventMachine for Ruby.
The project is run by Joyent (the company employing Ryan Dalh) who lunched a cloud hosting service for node applications (no.de).

Node’s Goal ?

It’s goal is to offer an easy and safe way to build high performance and scalable network applications in JavaScript.
Those goals are achieved thanks it’s architecture:

• Single Threaded :

Node use a single thread to run instead of other server like Apache HTTP who spawn a thread per request, this approach result in avoiding CPU context switching and massive execution stacks in memory. This is also the method used by nginx and other servers developed to counter the C10K problem.

• Event Loop :

Written in C++ using the Marc Lehman’s libev library, the event loop use epoll or kqueue for scalable event notification mechanism.

• Non blocking I/O :

Node avoid CPU time loss usually made by waiting for an input or an output response (database, file system, web service, …) thanks to the full-featured asynchronous I/O provided by Marc Lehmann’s libeio library.
These characteristics allow Node to handle a large amount of traffic by handling as quickly as possible a request to free the thread for the next one.
Node has a built-in support for most important protocols like TCP, DNS, and HTTP (the one that we will focus on). The design goal of a Node application is that any function performing an I/O must use a callback. That’s why there is no blocking methods provided in Node’s API.
The HTTP implementation offered by Node is very complete and natively support chunked request and response (very useful since we are going to use the twitter streaming api) and hanging request for comet applications. The Node’s footprint for each http stream is only 36 bytes (source).

JavaScript

Being an Event Driven Language, Javascript is the most suited to develop on the Node’s “Event Loop” architecture. Node’s applications really use javascript’s strengths like anonymous functions and closures.

Community

Despite its young age, the community is really growing fast and the GitHub repository is the 3rd most followed one.

Installing Node

The first step for us to take is the installation of Node. I choose to run Node on a Linux Debian 6 system but Windows users can try this.

Prerequisites

We need some packages before installing Node itself, same prerequisites apply for Ubuntu.

$ aptitude install git-core
$ aptitude install build-essential
$ aptitude install libssl-dev

Node

Now we just have to clone the git repository and install (make might take a while) :

$ git clone https://github.com/joyent/node.git
$ cd node
$ ./configure && make && make install

When the installation is complete, try it with :

$ node -v
v0.5.0-pre

That should print out your node version.

Installing Node Package Manager :

Isaac Schlueter (another employee of Joyent) has created a really great Node package manager called NPM. We need it because our Node application will use the Socket.IO module.

$ aptitude install curl
$ curl http://npmjs.org/install.sh | sh

Once installed you simply call this command line to install the Socket.IO package.

$ npm install socket.io

About Socket.IO
Socket.IO is a powerful Node module that bring the ability to simply manage long term connections with the clients. The module is used on the server side and on the client one. The client side will automatically and seamlessly use the best communication type (based on browser capabilities) to connect to a Node server.
Our architecture use a Google Chrome client, since this browser has WebSocket protocol capability, that’s the connection type Socket.IO will choose.
But what happen if we decided to create a Mozilla Firefox extension where WebSocket support is disable ? Well, by using Socket.IO, long polling connection will automatically be used and there is no need to change anything in our code on both client and server sides.

Ready to code

All the source of this post can be found on it’s dedicated github repository.

Hello World

To introduce the concept behind Node’s application development, we will start by the classic Hello World example.
First let’s create the folder tha
t will be our workspace:

$ mkdir nodespace
$ cd nodespace

Then copy the following content in a file named server.js

// Loading required modules
var http = require('http');
var SERVER_PORT = 8124;
// Creating HTTP Server
var server = http.createServer(function(request, response){
    // Called each time a request is made.
    response.writeHead(200, {'Content-Type': 'text/plain'});
    response.end('Hello World\n');
});
// Starting the server
server.listen(SERVER_PORT);
console.log('Server running on port : ' + SERVER_PORT);

 
This is quiet simple, we just create a http server by requiring the HTTP module and calling the createServer method. The callback method will be executed each time a request comes in.
To start the server call :

$ node server.js
Server running on port : 8124

We can now access to the server by using http://hostname:8124 and see the “Hello World”.

Adding Socket.IO

The next step is to use Socket.IO to handle long-terms connections, replace the server.js content by the following one:

// Loading required modules
var http = require('http'),
    io = require('/path/to/socket.io');
var SERVER_PORT = 8124;
// Creating HTTP Server
var server = http.createServer();
// Starting the server
server.listen(SERVER_PORT);
// Attaching Socket.IO to the HTTP Server
var socket = io.listen(server);
console.log('Server running on port : ' + SERVER_PORT);

Two more lines, that’s the only things we need to add to attach the Socket.IO module to the http server.
You might have noticed that the callback function of the http.createServer() has been removed, this is due to our use case where the server does not serve data to the client when they make a request but directly push data when an event is raised.
We can put a callback to the io.listen() method, it will be fired each time a request comes from the Socket.IO client side library.

var socket = io.listen(server,function(client){
    // new client connected !
});

Twitter tracker module

To retrieve the tweets about the “What’s Next” event we need to use the Twitter Streaming API. The API will return us in real time all the tweets mentioning “WsN_Paris” and “zenika”.
In order to do that, we are going to create a simple module that will connect to the Twitter API and emit an event each time it found a new tweet related to our topic.
Connection to the API require a valid twitter account. The module will encode your credentials and add them to the request header. We do not need to use OAuth to use the Twitter streaming API.
Create a new file called twitter.js and add the following content :

/ Loading required modules for the module
var http = require('http'), // Provide the createClient() method used to create the request.
    buffer = require('buffer').Buffer, // Used for credentials encoding.
    EventEmitter = require('events').EventEmitter; // Used for event emission.
// The tracker constructor.
// username & password are your twitter credentials.
// Topics are the subject(s) you want to track (separated by comma).
var TwitterTracker = exports.TwitterTracker = function(username, password, topics) {
        // Encoding credentials to base64.
	this.auth = new Buffer(username + ':' + password).toString('base64');
        // Adding basic authentication to the header with our credentials.
	this.header =  {'Authorization' : 'Basic ' + this.auth, 'Host' : 'stream.twitter.com' };
	this.topics = topics;
}
// Implementing the observer pattern thanks to Node's EventEmitter object.
// This will allow our TwitterTracker object to emit some events.
TwitterTracker.prototype = new EventEmitter;
// Track method definition.
// Calling the track method will send the request to the twitter api and emit event when a new tweet arrives.
TwitterTracker.prototype.track = function track() {
        // Creation of the http client used to make the request.
	var twitter = http.createClient(80, 'stream.twitter.com'),
        // Creation of the twitter request, we pass the http method, the api link with our topics and the header.
	request = twitter.request('GET', '/1/statuses/filter.json?track=' + this.topics, this.header),
        // We get a reference to our current instance to send event in the callback.
	tracker = this;
        // Waiting for a response from our request to the api.
        // This event is fired only once.
	request.on('response', function (response) {
        response.setEncoding('utf8');
		var body = '';
        // Listening to the data event.
        // This event will be fired every time the twitter
        // api send back a new chunk of json data.
        //
        // Since the twitter api is streaming us the data, we might receive
        // incomplete json that we will not be able to parse wihout error.
        //
        // The logic implemented in this listener is to concat chunks of data
        // in the body variable and detect if we have a full tweet (ending by '\r\n')
        // befor trying to parse.
    	response.on('data', function (chunk) {
		    body = body + chunk;
			var tweet, index;
			if ( (index = body.indexOf('\r\n')) > -1 ) {
			    tweet = body.slice(0, index);
				body = body.slice(index + 2);
				if (tweet.length > 0) {
					try {
                                                // Parsing the new tweet.
						tweet = JSON.parse(tweet);
                                                // Sending the 'tweet' event that our server.js will listen to.
						tracker.emit('tweet', tweet);
					} catch (error) {
                                            console.error('-!- Error while parsing tweet.');
                                        }
				}
			}
		});
	});
        // The request to the twitter api is not made until the end() method is call.
	request.end();
	return this;
};

 
The only tricky part is the logic implemented in the “data” event callback. This is Twitter specific and other API are much more easy to deal with.

Using The Tracker

Now that we have a functional tweet tracker module, we include it in the server.js and start waiting for “tweet” events to broadcast them to the clients.

// Loading required modules
var http = require('http'),
	io = require('/path/to/socket.io'),
	twitter = require('./twitter');
var SERVER_PORT = 8124,
    TWITTER_LOGIN = 'login',
    TWITTER_PASSWORD = 'password',
    TWITTER_TOPICS = 'WsN_Paris,zenika';
// Creating HTTP ServerWhat's Next ?
server = http.createServer();
// Starting the server
server.listen(SERVER_PORT);
// Attaching Socket.IO to the HTTP Server
var socket = io.listen(server);
console.log('Server running on port : ' + SERVER_PORT);
// Instantiating our tracker
var tracker = new twitter.TwitterTracker(TWITTER_LOGIN, TWITTER_PASSWORD, TWITTER_TOPICS);
// Start tracking and waiting for 'tweet' events from our tracker.
tracker.track().on('tweet', function(tweet){
        // Print out the tweet
        console.log('New tweet from :"' + tweet.user.screen_name + '" -> ' + tweet.text);
        // Using the socket provided by Socket.IO to broadcast the new tweet to all the clients.
	socket.broadcast(
		// To save bandwich we send only the tweet parts we are interested in.
		JSON.stringify(
			{ 	id : tweet.id,
				user : tweet.user.screen_name,
				text : tweet.text,
				picture : tweet.user.profile_image_url
			}
		)
	);
});

As you can see, we just need to add a listener on the “tweet” event and send the useful part of the tweet to the clients by calling the socket’s broadcast() method.

Conclusion

This close up the first part in the development of a full JavaScript architecture. I hope that reading this help you see how easy it is to build a simple but yet highly scalable application.
Node is very powerful and writting an app in JavaScript that is not only meant to wait for onclick events is really a nice experience.
You can quickly start developing with Node by using the Cloud9IDE. Just register or sign in with your github account and you will be ready to code within a minute.
In the next part we will see how to develop a Google Chrome extension that will display the tweets in real time by playing with HTML5 notifications.

Ressources

• Node Official : http://nodejs.org
• Documentation : http://nodejs.org/docs/v0.4.5/api/i…
• GitHub : https://github.com/joyent/node
• Modules : https://github.com/ry/node/wiki/mod…
• Google Group : http://groups.google.com/group/node…
• Companies using Node : https://github.com/joy…
• Socket.IO : http://socket.io
• How To Node : http://howtonode.org
• Mastering Node Free Ebook : https://github.com/visionmedia/mast…
• O’Reilly’s Node coming book : http://ofps.oreilly.com/titles/9781…
• Node Tuts : http://nodetuts.com