Much has been made lately of “NoSQL” non-relational databases. A few weeks back, Lou wrote a post introducing CouchDB, a free Apache Foundation document datastore. Lou’s application was hosted entirely from CouchDB. I recently wrote a web application with a more traditional approach, with a server using CouchDB as its backend. Traditional in that sense, but non-traditional in another: the backend language of choice is JavaScript.
That’s right, today’s post will introduce you to nodeJS, the server-side JavaScript engine. node allows you to do just about anything that you could do server-side in languages like Java, C#, Ruby or Python, but in JavaScript! This has a couple of advantages. First of all, with more and more web application development happening in the browser with MVC-like frameworks such as Backbone (more on that later), it is now easy for front-end devs to seamlessly transition to back-end development. No switching gears mentally when changing from JavaScript to Java, plus you get to use JSON/JS objects server- and client-side. JSON FTW! Second, like the browser-based JavaScript we all know and love, node is asynchronous in nature. Server threads can use callbacks instead of having a thread hanging while waiting for a DB I/O to complete. What that means in practical terms is node is fast.
The Application
Today’s post will showcase a simple backend for a web application, written with the help of the Node library Express. The front-end of this particular application is written with Backbone, but that’s a story for another day. This application architecture, with an MVC-powered JavaScript framework powering a single-page application in the browser, node on the server, and a NoSQL database serving as datastore represent the cutting edge in modern web application development. It is also very easy to understand and quite a fun stack to work with!
The application I will be taking you through – Freelection – can be found running online here. The source code can be found on gitHub. It is a simple tool my elementary school teacher wife asked me to build to let kids at her school vote in the presidential election. It allows users to create an election, add candidates, vote in this election and then view the results, in aggregate and broken down by Polling Station (this is just an extra dimension which was needed to break down results by classroom).
Node Basics
To get started with node, you must first install node on your machine.
Note: For Windows users, this can be a bit of a pain. Follow the instructions here to get set up.
The libraries/dependencies for a node application can be specified in a package.json file. The bundled command line utility, npm is used to install the dependencies in this package. For Java jockeys, this is similar to a Maven pom, only 3 times as intuitive and 12 times more beautiful. Here’s what mine looks like:
package.json
{
"name": "freelection",
"version": "0.0.1",
"private": true,
"engines": {
"node": "0.8.12",
"npm": "1.1.49"
},
"scripts": {
"start": "node app"
},
"dependencies": {
"express": "3.0.0rc5",
"cradle":"0.6.4",
"underscore":"1.4.2",
"emailjs":"0.3.2",
"handlebars":"1.0.7"
}
}
This defines the application, “freelection” and indicates the version of node and npm to be used, as well as the start script to run (node app on the CL will run our app.js file as a node application) and dependencies. Navigating to our root directory, containing package.json and app.js and running the command npm install will install the desired dependencies into a node_modules folder.
Application Core
Once npm has installed the required libraries, we can start our app up by simply running node app. This runs our app.js file, listed below, as a node application.
app.js
var express = require('express'),
routes = require('./routes'),
election = require('./routes/election'),
candidates = require('./routes/candidates' ),
vote = require('./routes/vote' ),
http = require('http'),
path = require('path'),
cradle = require('cradle');
var app = express();
exports.app = app;
app.configure(function() {
app.set('port', process.env.PORT || 8080);
app.set('views', __dirname + '/views');
app.set('view engine', 'jshtml');
app.use(express.favicon());
app.use(express.logger('dev'));
app.use(express.bodyParser());
app.use(express.methodOverride());
app.use(app.router);
app.use(express.static(path.join(__dirname, 'public')));
});
app.configure('development', function() {
app.use(express.errorHandler());
});
app.get('/', function( req, res ) {
res.sendfile( "public/index.html" );
});
app.post('/election', election.postElection);
app.get('/election/:electionId', election.getElectionInfo);
app.get('/candidates', candidates.getCandidates);
app.get('/candidate/:candidateId', candidates.getCandidateInfo);
app.put( '/candidate/:candidateId', candidates.updateCandidate );
app.delete( '/candidate/:candidateId', candidates.deleteCandidate );
app.post( '/candidate', candidates.addCandidate );
app.post( '/castVote', vote.castVote );
app.get( '/results/:id', vote.getResults );
http.createServer(app).listen(app.get('port'), function() {
console.log("Express server listening on port " + app.get('port'));
});
Let’s break this deceptively simple file down. At the top, we see require() statements. These are the equivalent of imports, and load other JavaScript files for use in app.js. ‘./routes/election’ for instance loads the file ./routes/election.js as the variable election. Likewise, var express = require(‘express’) loads the Express module as defined in package.json.
Managing Scope with Exports
In node, functions and variables are only accessible from within the same file. Essentially, like most reasonable OOP languages, you have private access unless otherwise specified. In node, that “otherwise” comes in the form of the magical exports object. For example:
email.js
var emailJS = require("emailjs");
var server = emailJS.server.connect({
user: process.env.SMTP_USER,
password: process.env.SMTP_PASS,
host: "smtp.gmail.com",
ssl: true
});
exports.sendEmail = function( to, subject, body ) {
server.send( {
text: body,
from: "freelectionapp@gmail.com",
to: to,
subject: subject,
attachment: [ {data:"<html>" + body + "</html>", alternative:true} ]
}, function(err, message) { console.log(err || message); });
};
elsewhere.js
var email = require("./utility/email")
email.sendEmail( "congress@us.gov", "Budget", "Please balance the budget! Signed, everyone" );
Simple enough? If this is confusing, you can read a bit more here.
Configuring and Running Express
So, back to our main app.js file.
var app = express();
exports.app = app;
app.configure(function() {
app.set('port', process.env.PORT || 8080);
app.set('views', __dirname + '/views');
app.set('view engine', 'jshtml');
app.use(express.favicon());
app.use(express.logger('dev'));
app.use(express.bodyParser());
app.use(express.methodOverride());
app.use(app.router);
app.use(express.static(path.join(__dirname, 'public')));
});
Here, we call express() to initialize our express app. We follow this up by using exports to make app accessible to the outside, and configuring our app. There are a couple more “magic” variables here worth noting – __dirname is the path to the directory in which your node app is running, and process.env.PORT is used to pull in the environment variable %PORT%.
Skipping to the bottom of the file now, we start up the app we just created:
http.createServer(app).listen(app.get('port'), function() {
console.log("Express server listening on port " + app.get('port'));
});
app.get(‘port’) pulls the port variable we set in our configuration above. Once the HTTP server is created, the callback method we provided is fired, logging to the console what port our express app is running on. To start our app from the command line, we simply run node app from the command line, and are greeted with this message:
Express server listening on port 8080
Now we’re up and running! Let’s take a look at how we set up controller paths with express.
Express Controller Paths/Service Endpoints
In express, we define HTTP controller paths based off of the four HTTP verbs: GET, POST, PUT, and DELETE. For example:
app.get('/', function( req, res ) {
res.sendfile( "public/index.html" );
});
In this case, a GET request for http://localhost:8080/ will get handled by the function specified above. The request gets marshalled to the req JavaScript object, and the response can be manipulated as res. In this case, we simply call the sendfile() method on the res object, specifying the path to our main index.html file. This index.html file contains our single page web application, and is the only controller path in this application which represents a full page reload. The rest of the controller paths/service endpoints specified are accessed by the client via AJAX.
Service endpoints can be used in conjunction with exports to defer service handling to another file/module:
app.js
election = require('./routes/election')
app.get('/election/:electionId', election.getElectionInfo);
election.js
exports.getElectionInfo = function( req, res ) {
var electionId = req.param('electionId');
console.log( "getElectionInfo: " + electionId );
db.get( electionId, function( err, doc ) {
if ( err) {
console.log( err );
res.send(500, "Unable to retrieve election data");
} else {
console.log( doc );
res.json( doc );
}
});
};
Here, you can see how our service asynchronously responds with either an error or a JSON document. Which brings us to our first CouchDB query.
CouchDB
For this project, a CouchDB instance has been set up for local development, and for my production deployment on Heroku, an environment variable is set which points to my Cloudant installation of CouchDB. (As an aside, Heroku makes deployment of node web apps an absolute cinch. Very simple model of worker “gyros” and it’s built on Amazon EC2. Worth looking into!) To connect to it, I use cradle, a CouchDB driver for node. Cradle is quite simple to use and serves as a simple interface between node and CouchDB’s RESTful architecture.
I define a node file called dao.js which instantiates and exports an instance of my cradle object, db.
dao.js
var cradle = require('cradle');
if ( process.env.CLOUDANT_URL ) {
var db = new(cradle.Connection)(process.env.CLOUDANT_URL, 80).database('elections');
} else {
db = new(cradle.Connection)('http://127.0.0.1', 5984).database('elections');
}
db.exists(function (err, exists) {
if (err) {
console.log('error', err);
} else if (exists) {
console.log('db elections exists');
} else {
console.log('database elections does not exist. Creating...');
db.create();
console.log('database created');
}
db.save('_design/candidate', {
views: {
byId: {
map: function (doc) {
if (doc.type === 'candidate') {
emit(doc._id, doc);
}
}
},
byElectionId: {
map: function(doc) {
if ( doc.type === 'candidate' ) {
emit( doc.electionId, doc );
}
}
}
}
});
db.save('_design/vote', {
views: {
byElection: {
map: function( doc ) {
if ( doc.type === 'vote' ) {
emit( doc.electionId, doc );
}
}
}
}
});
});
exports.db = db;
In this file, we check to see whether our ‘elections’ database exists, and if it does not, we create it. We then save a number of CouchDB views, which if you recall from Lou’s post, allow us to query our database based off of different keys and values.
We use a single database to store our data. Every document is keyed off of _id. To differentiate the different kinds of documents which are stored (election, candidate, vote, etc.), by convention, we include a type variable on each document which indicates what kind of document it is. For instance, an election document looks like this:
{
"_id": "1a00a48331732c4436d51d770777f94f",
"_rev": "1-12146938649f35ee37b0d72b541897a2",
"type": "election",
"name": "Cambridge Elementary Presidential Election",
"email": "bjones@keyholesoftware.com",
"description": "This is an example election"
}
Whereas a candidate record may look like this:
{
"_id": "f311b1dbca3624ef21959b2204fa4e40",
"_rev": "1-4d7bd4605957125729b82ed3cd7d86bd",
"type": "candidate",
"electionId": "1a00a48331732c4436d51d770777f94f",
"name": "Barack Obama",
"party": "Democrat",
"description": ""
}
Keep in mind that there is no such thing as a schema in CouchDB, so our application itself is responsible for keeping data type consistent! Here, we see that candidates are linked to election documents by the field electionId. This is one way to accomplish the equivalent of foreign keys in a RDBMS.
Updates/Inserts
Updating CouchDB is fairly straightforward with cradle. For instance, this is how a new vote record is created:
db.save({
type: 'vote',
electionId: electionId,
candidateId: candidateId,
candidate: candidate,
station: station
}, function( err, doc ) {
console.log( err, doc );
if ( err ) {
res.send(500, err );
} else {
res.send( {success: true} );
}
});
Note that if db.save() had been passed in the _id of an existing document, that document would have been updated, rather than a new record created.
CouchDB Views and Map/Reduce
Let’s take a closer look at our database’s design documents and their views. Recall that in CouchDB, views are grouped into different “design documents,” and can be queried against.
Views are defined in terms of a map and (optional) reduce function. The map function is called for every single document in the datastore. When large quantities of data are involved, this is actually efficient across multiple machines because CouchDB subdivides the records into subsets, which are processed in parallel. The results from map are then passed on to the reduce function, which can then whittle down those potentially huge result sets into the subset desired. In our application, there are no reduce functions, only map. Map/Reduce is a topic worthy of its own blog post, so for more information, I recommend reading more here. Let’s look at the views in our candidates design document:
dao.js
db.save('_design/candidate', {
views: {
byId: {
map: function (doc) {
if (doc.type === 'candidate') {
emit(doc._id, doc);
}
}
},
byElectionId: {
map: function(doc) {
if ( doc.type === 'candidate' ) {
emit( doc.electionId, doc );
}
}
}
}
});
This is cradle’s way of defining a design document called candidate. Candidate can be queried one of two ways, directly by _id, or in aggregate, based off of each candidate’s electionId.
The byId view is simple enough to understand. It simply filters documents based off of doc.type. emit() is a special method in CouchDB which passes the results along to reduce or back to the query. The first parameter it takes is the id by which the view can be queried. The second is the document which is returned. Querying the byId view without specifying an _id will return all candidates across all elections.
byElectionId is used to get a list of candidate documents based off of their electionId. Note that unlike in byId, the key for this view is the document’s electionId. Here is how this view gets queried:
candidates.js
exports.getCandidates = function(req, res) {
var electionId = req.param('id');
console.log( "get candidates for election: " + electionId);
db.view( 'candidate/byElectionId' , { key: electionId }, function(err, doc ) {
if ( err ) {
console.log(err);
res.send( 500, err );
} else {
var candidates = _.pluck(doc, "value");
console.log( "Candidates: ", candidates );
res.json( candidates );
}
});
};
Note that the doc response array from the cradle query contains metadata along with each of the actual documents returned. _.pluck is used to grab the value attribute from each of the objects in the doc array. value is the actual document object for each record. The results from this service call look like this:
[
{
"_id": "1a00a48331732c4436d51d77077b4463",
"_rev": "1-669993a8d4339e32c309cfe129e22e86",
"type": "candidate",
"electionId": "1a00a48331732c4436d51d770777f94f",
"name": "Mitt Romney",
"party": "Republican",
"description": ""
},
{
"_id": "f311b1dbca3624ef21959b2204fa4e40",
"_rev": "1-4d7bd4605957125729b82ed3cd7d86bd",
"type": "candidate",
"electionId": "1a00a48331732c4436d51d770777f94f",
"name": "Barack Obama",
"party": "Democrat",
"description": ""
}
]
Conclusion
CouchDB and node are well-suited for rapid development of simple, scalable applications. CouchDB is a non-relational database, but if you are OK with the tradeoffs of going the NoSQL route with your application, you can do much worse than the node/CouchDB pairing. There is a certain beauty to having all three layers – client, server and datastore – written in JavaScript. No need to marshal JSON to rigid server-side Objects or deal with the paradigm-mismatch that is Object-Relational mappings. If you’re looking to build fast, scalable applications in record times (this entire project took less than a week while only working part-time), you would do well to consider the node approach.
– Brett Jones, asktheteam@keyholesoftware.com
References
Although not all of these libraries were covered in this article, the complete tech stack for this application is listed below and is worth taking a look at. These are all cutting-edge tools worth learning about, i.e., all the cool kids are using them!
- Node – Server-side JavaScript
- Express – node framework for REST service calls
- CouchDB – NoSQL distributed database
- cradle – CouchDB driver for node
- RequireJS – Asynchronously load JS files as separate modules – Implementation of AMD spec
- Backbone – Client-side MVC* framework**
- jQuery
- Bootstrap – Amazing CSS framework and widget toolkit worth looking into
- Handlebars – Templating engine
- Backbone.validation – Backbone plug in for form/model validation
- Underscore – Utility library on which Backbone is built
* Not really MVC
** Not really a framework
Tagged: CouchDB, Javascript, JSON, node.js, NoSQL
