MongoDB and NodeJS REST Service in less than 24 hours


Background

I've been looking for a flexible central reporting solution that offers the following benefits -
  • Cheap - preferably open source
  • Easy to publish to and read from by anyone without too much entitlement management
  • Platform and language agnostic
  • Flexible - so it can change along with changing needs 
  • Scalable - so it can grow along with increased usage

After a few weeks struggling with MySQL design and development, I finally decided to give up on it and do what I've always wanted to do and work with some really interesting new technologies.

It took less than 24 hours to get the entire stack up and running including coding and research time (I didn't know a thing about nodejs or mongodb 24 hours ago).

Environment Setup

I had an Oracle Linux environment to work with. Command line syntax is highlighted in brown -
  • SSH to your favourite cloud host
  • sudo su to become root
  • Set up the proxies for yum
    • export http_proxy=http://host:port
    • sudo export https_proxy=http://host:port
  • Install the following packages -
    • yum -y install php php-devel php-pear glibc-devel make vim gcc-c++
  • Unless you have a better editor, make sure you do all your coding in vim. The synatx highlighting will save you hours of debugging. To enable syntax highlighting -
    • type in cd which will take you the home irectory for root
    • mkdir .vim
    • mkdir .vim/syntax
    • curl http://www.vim.org/scripts/download_script.php?src_id=10728 > .vim/syntax/javascript.vim
  • Install nodejs and create the node user
    • curl http://nodejs.org/dist/v0.8.20/node-v0.8.20-linux-x64.tar.gz > /tmp/nodejs.tar.gz
    • tar -zxvf /tmp/nodejs.tar.gz
    • mv /tmp/node-v0.8.20-linux-x64/ /opt/node
    • adduser node -p somepassword
    • chown -R node /opt/node/
  • Download MongoDb
    • curl http://downloads.mongodb.org/linux/mongodb-linux-x86_64-2.2.3.tgz > /tmp/mongo.tgz
    • tar -zxvf /tmp/mongo.tgz
    • mv /tmp/mongo /opt/mongo
    • MongoDb doesn't come with a config file. It isn't needed if you're happy with all the defaults
    • Or you can download it from https://github.com/mongodb/mongo/blob/master/debian/mongodb.conf
  • Set up the PATH statements. You'll need to add the following to the /etc/sudoers or to the appropriate bash_profile -
    • export PATH=$PATH:/opt/node/bin:/opt/mongodb/bin
  • Add admin users to Mongodb using the command-line tool mongo. This will give you the mongo prompt where you can manage databases and users. You need to do this before you start mongod
    • To add an admin user with full server rights -
use admin
db.addUser("username", "password")
    • To add a refular server user with read-only rights to the server -
use records
db.addUser("username", "password")
    • To Add a read-write user only to database testddb
use testdb
db.addUser("username>", "password")
    • To add a read-only user only to database testdb
use testdb
db.addUser("username", "password", true)
  • Install rockmongo (one of many available open source MongoDb admin GUIs)
    • pecl install mongo since rockmongo is a PHP app
    • I couldn't find a url for it so you'll have to download it to your machine and then scp it up to /tmp
    • unzip /tmp/rockmongo-1.1.5.zip
    • mv /tmp/rockmongo-1.1.5 /var/www/rockmongo
    • vim /etc/php.ini and add the following line under the section for extensions -
        1. extension=mongo.so  
    • vim /etc/httpd/conf/httpd.conf and search for the AddType references and add the following lines below them -
        1. AddType application/x-httpd-php.php  
        1. AddType application/x-httpd-php-source .phps  
    • vim /etc/httpd/conf/httpd.conf and below the existing aliases, add -

Alias /rockmongo/ "/var/www/rockmongo/"
<Directory /var/www/rockmongo >
        Options FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
</Directory>
    • If you have SELinux enabled you'll also need to run these -
      • setsebool httpd_can_network_connect 1
      • setsebool httpd_can_network_connect_db 1
  • Set up your Node app (mine is called centaur)
    • mkdir /opt/centaur
    • mkdir /opt/centaur/routes
    • cd /opt/centaur (this is important as I'm installing all the node extensions locally)
    • npm install express
    • npm install mongoose
    • npm install mongo
    • npm install mongoose-validator
    • npm -g install forever (you install this as a global package because it's going to run the node service as a Linux daemon)
  • Spin everything up
    • mongod --config /etc/mongod.conf --dbpath /data/db --rest --fork --logpath /var/log/mongodb.log (to start MongoDB)
    • apachectl start (to start Apache/rockmongo)
  • Create a new mongodb database (you can do this from the rockmongo interface) - http://hostname/rockmongo/index.php
  • Set up your test data. You can do this using mongoimport. Below is an example of how to do this with the TestData.csv 
    • mongoimport -d test -c tests --type csv --file /tmp/nik/TestData.csv --headerline
  • You can generate your own test data using http://www.generatedata.com/#generator

The Code

  • Create a package.json file in the root of /opt/centaur that will hold the metadata for your app -
{
    "name": "centaur",
    "description": "SomeApp",
    "version": "0.0.1",
    "private": true,
    "dependencies": {
        "express": "3.x"
    }
}
  • Next I'm going to write my web server in node. To begin with I've implemented five REST routes (four GETs and one POST)
    • GETs are like SQL SELECTs
    • POSTs are like SQL INSERTs
    • I'm using the Express web framework for Node
    • My server is going to listen on port 3000
    • The server.js. appears in the gist below 
    • the variable 'tests' refers to my controller which are stored in a file called tests.js under /opt/centaur/routes. That is where the callbacks live
    • To start my server all I have to do is run node /opt/centaur/server.js
    • To start it as a long-running process (daemon), I use forever /opt/centaur/server.js
  • I don't connect to MongoDb directly, instead I use Mongoose to manage my Object Document Model and Mongoose-validator for basic field validation -
    • This helps in structuring the GETs and POSTs and also obviates the need for me to write my own validation routines

  • Here's an example of the JSON message that would meet the schema requirements. This is also, pretty much how it's stored in MongoDB -
{
  "_id": "51226fb5bee24877175f0088",
  "starttime": "2013-01-17T10:00:00.000Z",
  "endtime": "2013-01-17T11:25:00.000Z",
  "sprint": "A",
  "component": "Search",
  "capability": "Structured search",
  "version": 2,
  "testscript": "Search-Structured-search.rb",
  "status": "pass",
  "developer": "someone@something.tv",
  "tester": "trs@something.tv",
  "value": 5
  }
You can also submit entries by using the following curl command (hopefully you have a *NIX machine. curl on Windows sucks -
  • curl -i -X POST -H 'Content-Type:application/json' -d ' {"starttime": "2010-01-17T10:00:00.000Z","entime":"2010-01-17T11:38:00.000Z","sprint": "X","component": "Search","capability": "yellow","version": 2.1,"testscript": "Search-Keyword-search.rb","status": "pass","developer": "nik","tester": "yo","value": 8}' http://host:3000/centaur/tests
  • If you want to see what an uncommitted response looks like, change the spelling of one of the fields in the JSON payload
  • This is what the server console looks like when you send a malformed request (I misspelt endtime) -
  • This is what the client sees when they send a malformed message -

Comments

Post a Comment

Popular posts from this blog

The Forecaster Brown Fan Club

I spent the last couple of weeks in October watching what could only be described as the Idiot's Guide to the Wikipedia entry on Hurricane Sandy, otherwise known as the Weather Channel (it was the only live channel I could find online). Half of it involved presenters with an emotional age of 14 trying to get as close to the storm surge as they could, while Dr. Thang in Atlanta helpfully pointed out the eye of the storm every so often. It left me wondering why India had fought so hard for a Cray XMP-24 all those decades ago when all they needed was a helicopter, a doppler, a satellite photo and a half-crazed adolescent with a death-wish. Then I came across this page written by the imaginately named 'Forecaster Brown'. The post, dated ' 1100 AM EDT SAT OCT 27 2012 ' begins " NOAA AND AIR FORCE RESERVE HURRICANE HUNTER AIRCRAFT HAVE BEEN  SAMPLING THE CYCLONE THIS MORNING. "  It turns out the NOAA (National Oceanic and Atmospheric Administ
Read more

How to Create a Pentaho Report Using the REST Client

Image
Prerequisites Pentaho Kettle (see earlier blog post) Pentaho Report Designer You will get an error message in Pentaho Report Designer that says ' java.lang.NoClassDefFoundErr: com/sun/jersey/api/client/ClientConfig ' if you try to run a report against the Kettle job. To fix this error copy the following files from  C:\pentaho\data-integration\libext\jersey\ to C:\pentaho\prd-ce-3.9.1-GA\report-designer\lib\. If you plan to ublish the report to the BI Server, also copy these files to C:\pentaho\biserver-ce\tomcat\webapps\pentaho\WEB-INF\lib\ as well - jersey-apache-client-1.7.jar jersey-client-1.7.jar jersey-core-1.7.jar The Payload My example returns an array of objects [   {       "id":"1",       "name":"item1",       "description":"someDescription1"   },   {       "id":"2",       "name":"item2",       "de
Read more

Automated Testing with vncdotool (Not Headless, but Hairless)

Image
The headless testing problem is a thorny one and impacts the fundamental architecture of the test automation platform I'm working on. If we can't find a solution, we might not be able to use the private cloud nor parallelize our testing effectively. Part of the problem are the tests themselves that still rely on SendKeys() and MouseMove(). Two aspects of remote interaction that are insecure and therefore disallowed by Microsoft from Windows Vista onwards. This means that we have to rely on hacks like the VIX API and the one that I am about to document below. There are two widely-used technologies for remote interactive connectivity on the Windows platform - the Remote Desktop Client (RDC) and VNC. The first is entirely proprietary while the second offers a number of open-source implementations (and some proprietary, full-featured, commercial implementations). The RDC implementation requires access to the VIX API which in turn relies on proprietary, privileged access t
Read more