Jan Amoyo

on software development (and stuff)

Testing Angular Directives with Templates on Karma and Browserify

Directives are the cornerstone of every Angular application. And templates help keep their behavior separate from the presentation.

Karma works well with Angular and is an essential tool for running tests against a number of supported browsers.

Lastly, Browserify helps preserve sanity while maintaining JavaScript modules (similar to Node.js i.e. CommonJS spec).

Sadly, integrating all four concepts is not a straightforward process. Below is a rough guide on how to achieve this.

1. Install karma-browserifast and karma-ng-html2js-preprocessor
$ npm install karma-browserifast --save-dev
$ npm install karma-ng-html2js-preprocessor --save-dev
The package karma-browserifast enables Browserify support on Karma. While karma-ng-html2js-preprocessor converts HTML templates to JavaScript and loads them as an Angular module.

2. Configure Karma
module.exports = function (config) {
  config.set({
    files: [
      'node_modules/angular/angular.js',
      'src/**/*.html'
    ],

    browserify: {
      files: [
        'test/unit/**/*.js',
      ],
      debug: true
    },

    preprocessors: {
      '/**/*.browserify': ['browserify'],
      'src/**/*.html':    ['ng-html2js'],
    },

    ngHtml2JsPreprocessor: {
      moduleName: 'karma.templates'
    },

    frameworks: ['jasmine', 'browserify'],
    browsers:   ['Chrome'],
    reporters:  ['spec'],
    logLevel:   'info',
    autoWatch:  true,
    colors:     true,
  });
};
Lines 4 and 5 loads the HTML templates into Karma. Because they will be pre-processed by karma-ng-html2js-preprocessor, they will eventually get loaded as JavaScript files. Note that even if Angular will be included as part of the Browserify bundle, it is important to load it explicitly. Otherwise, the templates cannot be made available to Angular.

Line 10 tells karma-browserifast to include all unit tests as a Browserify bundle. This makes it possible for unit tests to execute JavaScript modules using the "require" keyword.

Line 21 tells karma-ng-html2js-preprocessor to load the JavaScript templates as an Angular module named "karma.templates". This will later be used in unit tests to allow testing of directives that use templates.

3. Write the unit tests
require('../src/app');
require('angular-mocks/angular-mocks');
describe('myDirective', function () {
  var scope, element;

  beforeEach(function () {
      angular.mock.module('karma.templates');
      angular.mock.module('myModule');
      angular.mock.inject(function ($rootScope, $compile) {
          scope   = $rootScope.$new();
          element = $compile('<div my-directive></div>')(scope);

          scope.$digest();
      });
  });

  it('does something', function () {
    expect(1).toBe(1);
  });
});
Lines 1 and 2 demonstrates the capability of a Karma-based unit test to load modules via "require".

Line 7 loads the generated JavaScript templates as an Angular module; without this, the directive will not compile because the template cannot be fetched.

Examples of Streams and Lamdas in Java 8

Below is a code showcasing Streams and Lamdas in Java 8 (written as a JUnit test allow execution of individual methods).
@RunWith(BlockJUnit4ClassRunner.class)
public class Java8Showcase {

  @FunctionalInterface
  public static interface Concat {
    String concat(String str1, String str2);
  }

  @Test
  public void functional_interfaces() {
    // Traditional way of declaring an
    // interface implementation
    @SuppressWarnings("unused")
    Concat _concat = new Concat() {
      @Override
      public String concat(String str1, String str2) {
        return str1.concat(str2);
      }
    };

    Concat contact = (str1, str2) -> str1.concat(str2);
    System.out.println(contact.concat("hello", " world"));
  }

  @Test
  public void function_reference() {
    List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
    // nums.stream().reduce((a, b) -> Math.max(a, b))
    Integer max = nums.stream().reduce(Math::max).get();

    System.out.println(max);
  }

  @Test
  public void streams_filter() {
    List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
    List<Integer> evens = nums.stream().filter((n) -> n % 2 == 0)
      .collect(Collectors.toList());

    System.out.println(evens);
  }

  @Test
  public void streams_map() {
    List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
    List<Integer> x2 = nums.stream().map((n) -> n * 2)
      .collect(Collectors.toList());

    System.out.println(x2);
  }

  @Test
  public void streams_reduce() {
    List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
    Integer sum = nums.stream().reduce((a, b) -> a + b).get();

    System.out.println(sum);
  }

  @Test
  public void streams_fluent() {
    List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
    Integer sum = nums.stream()
        .filter((n) -> n % 2 == 0) // filter the even numbers
        .map((n) -> n * 2)         // multiply each number by 2
        .reduce((a, b) -> a + b)   // add each number
        .get();

    System.out.println(sum);
  }

  @Test
  public void streams_parallel() {
    List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
    Integer sum = nums.parallelStream().reduce((a, b) -> a + b).get();

    System.out.println(sum);
  }

  @Test
  public void streams_forEach() {
    List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
    nums.stream().forEach(System.out::println);
  }

  @Test
  public void streams_collectors() {
    List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
    Double ave = nums.stream().collect(Collectors.averagingInt((i) -> i));

    System.out.println(ave);
  }

  @Test
  public void streams_collectors_grouping1() {
    List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
    Map<Boolean, List<Integer>> oddEven = nums.stream()
      .collect(Collectors.groupingBy((i) -> i % 2 == 0));

    System.out.println(oddEven);
  }

  @Test
  public void streams_collectors_grouping2() {
    List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
    Map<Boolean, Integer> oddEvenSum = nums.stream()
      .collect(Collectors.groupingBy((i) -> i % 2 == 0, Collectors.summingInt((Integer i) -> i)));

    System.out.println(oddEvenSum);
  }
}

MongoDB Transaction Across Multiple Documents using Async and Mongoose

Unlike traditional databases, MongoDB does not support transactions. So suppose you have multiple documents and want to perform a "save all or nothing" operation, you'd have to simulate a transaction from within your application.

Implementing such a transaction in NodeJS can be tricky because IO operations are generally asynchronous. This can lead to nested callbacks as demonstrated by the code below:
var MyModel = require('mongoose').model('MyModel');

var docs = [];
MyModel.create({ field: 'value1' }, function (err, doc) {
  if (err) { console.log(err); }
  else {
    docs.push(doc);
    MyModel.create({ field: 'value2' }, function (err, doc) {
      if (err) { rollback(docs); }
      else {
        docs.push(doc);
        MyModel.create({ field: 'value3' }, function (err, doc) {
          if (err) { rollback(docs); }
          else {
            console.log('Done.');
          }
        });
      }
    });
  }
});
Despite not having a dependency on other documents, inserting each document must be performed in series. This is because we have no way of finding out when all other documents will finish saving. This approach is problematic because it leads to deeper nesting as the number of documents increase. Also, in the above example, the callback functions modify the docs variable - this is a side effect and breaks functional principles. With the help of Async, both issues can be addressed.

The parallel() function of Async allows you run multiple functions in parallel. Each function signals completion by invoking a callback to which either the result of the operation or an error is passed. Once all functions are completed, an optional callback function is invoked to handle the results or errors.

The above code can be improved by implementing document insertions as functions of parallel(). If an insert succeeds, the inserted document will be passed to the callback; otherwise, the error will be passed (these can later be used when a rollback is required). Once all the parallel functions complete, the parallel callback can perform a rollback if any of the functions failed to save.

Below is the improved version using Async:
var async    = require('async'),
    mongoose = require('mongoose');

var MyModel = mongoose.model('MyModel');

async.parallel([
    function (callback) {
      MyModel.create({ field: 'value1' }, callback);
    },
    function (callback) {
      MyModel.create({ field: 'value2' }, callback);
    },
    function (callback) {
      MyModel.create({ field: 'value3' }, callback);
    }
  ],
  function (errs, results) {
    if (errs) {
      async.each(results, rollback, function () {
        console.log('Rollback done.');
      });
    } else {
      console.log('Done.');
    }
  });

function rollback (doc, callback) {
  if (!doc) { callback(); }
  else {
    MyModel.findByIdAndRemove(doc._id, function (err, doc) {
      console.log('Rolled-back document: ' + doc);
      callback();
    });
  }
}
Lines 8, 11, and 14 inserts a new document to MongoDB then passes either the saved document or an error to the callback.

Lines 18-21 checks if any of the parallel functions threw an error, and calls the rollback() function on each document passed to callback of parallel().

Lines 28-33 performs the rollback by deleting the inserted documents using their IDs.

Edit: As pointed-out in the comments, the rollback itself can fail. In such scenarios, it is best to retry the rollback until it succeeds. Therefore, the rollback logic must be an idempotent operation.

Adding version to JavaScript and CSS Resources via JSP/JSTL

Browsers usually cache static resources like JavaScript and CSS so that downloads are reduced the next time the same website is visited.

However, if the JavaScript or CSS gets updated in the server, the browser will still be using an outdated copy of the resource. This could lead to unexpected behavior.

This can be addressed by including a version to the resource name and incrementing the value every time an update is released (ex: /my-styles-v2.css). An easier alternative is to use a query parameter that indicates the resource version (ex: /my-styles.css?version=2). On either solution, HTML pages that link to the resources need to be updated whenever a resource version changes -- this can be difficult to maintain especially if there are many resources that constantly change.

There are tools that automate this process and are usually incorporated into the build process. The solution I'm going to demonstrate will address the issue without adding an extra build step.

Note: This example is specific to Java web applications using JSTL.

The first step is to add an application-level configuration that holds the current version of the static resources. This comes as a context parameter in web.xml:
<web-app>
  ...

  <!-- Indicates the CSS and JS versions -->
  <context-param>
    <param-name>resourceVersion</param-name>
    <param-value>1</param-value>
  </context-param>

  ...

</web-app>
Lines 6 and 7 indicate the value of the resource version and should be incremented before every release.

The next step is to update the JSPs that link to the resources and format the resource URLs to include the version number.
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<c:url value="/resources/css/my-styles.css" var="myCss">
  <c:param name="version" value="${initParam.resourceVersion}" />
</c:url>
<c:url value="/resources/js/my-scripts.js" var="myJavaScript">
  <c:param name="version" value="${initParam.resourceVersion}" />
</c:url>
...
<!DOCTYPE html>
<html lang="en">
<head>
...
<link rel="stylesheet" href="${myCss}" type="text/css" />
...
</head>
<body>
...
<script src="${myJavaScript}"></script>
</body>
</html>
Lines 2-7 declare URL variables that include "version" as a query parameter. Note that context parameters are accessible from JSTL via ${initParam}.

Lines 13 and 18 uses the variables to render the URLs.

Generating Unique and Readable IDs in Node.js Using MongoDB

I had a requirement from an upcoming project to generate unique human-readable IDs. This project is written in Node.js and uses MongoDB for its database.

Ideally, I can use an auto-incrementing Sequence to achieve this. However, unlike most relational databases, MongoDB does not support Sequences. Fortunately, it is not difficult to implement this behavior in MongoDB.

We will use a collection to store our sequences. The sequences will then be incremented using the findAndModify() function. To ensure that sequences do not increment to unmanageable values, the counter must be restarted after a certain period. In my case, I will restart the counter everyday. To achieve this, I will identify each sequence using a prefix of YYMMDD.

Below is the raw MongoDB statement:
db.ids.findAndModify({
  query:  { prefix: 140625 },
  update: { $inc: { count: 1 } },
  upsert: true,
  new:    true
});
It is important to set the upsert and new options - setting upsert to true will insert a new document if the query cannot find a match; while setting new to true will return the updated version of the document.

Testing on the console yields the expected result.
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 1,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 2,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 3,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 4,
    "prefix" : 140625
}

Node.js + Mongoose

I use Mongoose in Node.js as a MongoDB object document mapper (ODM). Mongoose offers an intuitive API to access MongoDB from within Node.js.

To translate the above implementation, we first need to declare a Mongoose schema.
var mongoose = require('mongoose'),
    Schema   = mongoose.Schema;

var IdSchema = new Schema({
  prefix: { type: Number, required: true, index: { unique: true } },
  count:  { type: Number, required: true }
});

mongoose.model('Id', IdSchema);
The schema defines the structure of the document and as well as validation.

Once the schema is declared, translation becomes pretty straightforward. Note that Mongoose does not have a function called findAndModify(), instead, it offers 2 forms: findByIdAndUpdate() and findOneAndUpdate(). In our case, we will use findOneAndUpdate().
var moment = require('moment'),
    Id     = mongoose.model('Id');

var nextId = function (callback) {
  function prefix (date) {
    return parseInt(moment(date).format('YYMMDD'));
  }

  Id.findOneAndUpdate(
    { prefix: prefix(new Date()) },
    { $inc:   { count: 1 } },
    { upsert: true },
    function (err, idDoc) {
      callback(err, idDoc);
    });
};
Lines 5-7 generates the prefix with the help of Moment.js.

Note that while findAndModify() and its Mongoose equivalents are an atomic operations, there is still a chance that multiple clients try to upsert the same document and hence would fail due to constraint violation - in such scenarios, the call to nextId() must be retried.

House Keeping

Because a new document is inserted every time the counter is reset, the documents will accumulate overtime. Fortunately, because the prefixes are stored as numbers, removing old documents becomes very easy.

For example, if we want to remove documents older than 2015, we just issue the below statement.
db.ids.remove({
  prefix: { $lte: 150000 }
});
The $lte operator stands for "less than". The above statement roughly translates to: delete from ids where prefix < 15000.

Sublime Text Packages for Node.js/JavaScript

Since I started working on Node.js for an upcoming project, I've become a fan of Sublime Text. It is fast; extensible; has great community support; and most importantly, runs on Linux.

So far, I use Sublime Text exclusively for Node.js and web development (HTML5, CSS, JavaScript). Below, I've compiled a list of packages I found most useful.

As a prerequisite, Package Control needs to be installed. Installing Package Control is a matter of copy-pasting a code snippet to your Sublime Text console. Installation instruction is found here.

Once Package Control is installed, you can start installing other packages by opening the Command Pallete (ctrl+shift+p) and searching for "Install Package".

Without further ado, below are the packages (search for the text in bold):

Alignment
* Aligns various texts
* Use via ctrl+alt-a
Figure 1a: Alignment (before)
Figure 1b: Alignment (after)
BracketHighlighter
* Highlights brackets, braces, and parentheses.
Figure 2: BracketHighlighter
Emmet
* Easily write HTML
* Use via ctrl+alt-enter
* For more information on Emmet, check-out this interactive guide.
Figure 3: Emmet
SidebarEnhancements
* Adds useful menu items to your sidebar
* Only available for Sublime Text 3
Figure 4: SidebarEnhancements
HTML-CSS-JS Prettify
* Formats HTML, CSS, and JavaScript files
* Use via ctrl+shift+h
* Requires Node.js to be installed
Figure 5a: Prettify (before)
Figure 5b: Prettify (after)
TrailingSpaces
* Highlights trailing spaces
Figure 6: TrailingSpaces
SublimeLinter
* Highlights lint errors for various file formats.
Figure 7: SublimeLinter
  • For Sublime Text 3, each linter needs to be installed separately:
    • SublimeLinter-jshint (JavaScript)
      • Requires jshint
      • install via "sudo npm install -g jshint"
    • SublimeLinter-html-tidy (HTML)
      • Requires tidy 
      • install via "sudo apt-get install tidy"

Setting-up Tomcat SSL with StartSSL Certificates

Part of an effort to improve the security of CheckTheCrowd.com is to enable SSL on my web server. Enabling SSL allows it to support HTTPS connections.

The CheckTheCrowd web application is hosted on Apache Tomcat which provides a pretty good, albeit generic documentation on how to achieve this setup.

In summary, enabling SSL on Tomcat requires three things:
  • Creating a Java keystore which contains the private key that Tomcat would use to start SSL handshakes
  • Ensuring that you or your website owns the private key by having it signed by a trusted authority which in turn, issues a digital certificate verifying your ownership of the key
  • Configuring a Tomcat connector to listen on HTTPS from a specified port
Creating a keystore and configuring a Tomcat connector is simple enough. However, acquiring an SSL certificate from a trusted provider can be expensive.

Thankfully,  I heard about StartSSL which provides free SSL certificates with one year validity (a new one can be generated upon expiry).

Below are the steps I took to set-up Tomcat SSL using StartSSL certificates.

1. Creating the Java Keystore File (.jks)

As per the Tomcat documentation, the first thing I needed to do was to generate a Java keystore which would hold my private key. This was done by using the keytool command that is part of JDK.
keytool -genkey -keysize 2048 -keyalg RSA -sigalg SHA1withRSA \
  -alias [name of server] -keystore [name of keystore].jks \
  -keypass [password] -storepass [password] \
  -dname "CN=[domain name], OU=Unknown, O=[website], L=[city], ST=[state], C=[country]"
Note that due to a Tomcat limitation, the keypass and storepass must be the same. The dname entry is optional; if not provided, these details will be asked by keytool during the process.

Example:
keytool -genkey -keysize 2048 -keyalg RSA -sigalg SHA1withRSA \
  -alias webserver -keystore checkthecrowd.jks \
  -keypass ****** -storepass ****** \
  -dname "CN=checkthecrowd.com, OU=Unknown, O=CheckTheCrowd, L=Singapore, ST=Unknown, C=SG"
At this point, my keystore already contains the private key required by Tomcat to start an SSL connection.

I can already start using this keystore to enable SSL in Tomcat, but a rogue entity can hijack the connection and pretend that his private key was issued by CheckTheCrowd. This rogue entity can then trick my users that they are securely connected to CheckTheCrowd when in fact they are connected to something else.

To solve this, I need to acquire a signed certificate to prove that my private key is associated to my domain (checkthecrowd.com).

2. Creating a Certificate Request File (.csr)

A certificate request is submitted to a certificate provider and an SSL certificate is generated based on this file.
keytool -certreq -alias [name of server] -file [name of request].csr \
  -keystore [name of keystore].jks
Note that this command would ask for the password previously set for the keystore.

Example:
keytool -certreq -alias webserver -file checkthecrowd.csr \
  -keystore checkthecrowd.jks

3. Submitting the Certificate Request to StartSSL

I needed to signup for an account in order to use StartSSL. Signing-up involves generating a signed private key which proves my identity. Here onwards, the key is used by StartSSL to authenticate my access to their website.

Note that it is important to keep a back-up copy of this private key for future use. This file needs to be imported on all computers used to access StartSSL.
Figure 1: StartSSL
Once I have an account, I can use the Control Panel to generate my certificate. The first step is to validate that I own the domain checkthecrowd.com. The aptly named Validation Wizard takes care of this.

Once my domain is validated, I used the Certificates Wizard to submit my certificate request (.csr file):
  • Select Web Server SSL/TLS Certificate.
  • Because I already have a private key and a certificate request, I skip the next screen
  • I pasted the contents of my certificate request (.csr file) to the text area provided
  • When finished, the generated certificate is displayed on another text area -- I copied this and saved to a file called ssl.crt.
4. Import the Generated Certificate and StartSSL Certificate Chains

The next step is import the generated certificate to my keystore. The StartSSL certificate chain is also required to be imported.

The StartSSL certificate chain can be downloaded from:
The free SSL certificate from StartSSL is only a Class 1 level certificate. With an upgraded package (Class 2 and higher), all applicable class certificates must be downloaded.

I again used keytool to import these certificates:
keytool -import -alias [ca alias] -file [ca file].cer \
  -keystore [keystore name].jks -trustcacerts
keytool -import -alias [class1 alias] -file [class1 file].pem \
  -keystore [keystore name].jks -trustcacerts
keytool -import -alias [name of server] -file ssl.crt \
  -keystore [keystore name].jks
The first two commands imported the certificate chain as trusted certificates, the last command imported the signed certificate.

Example:
keytool -import -alias startsslca -file ca.cer \
  -keystore checkthecrowd.jks -trustcacerts
keytool -import -alias startsslca1 -file sub.class1.server.ca.pem \
  -keystore checkthecrowd.jks -trustcacerts
keytool -import -alias webserver -file ssl.crt \
  -keystore checkthecrowd.jks
Listing the contents of my keystore verified that I have 3 certificates:
$ keytool -list -keystore checkthecrowd.jks
webserver, Aug 5, 2013, PrivateKeyEntry,
Certificate fingerprint (SHA1): [...]
startsslca, Aug 5, 2013, trustedCertEntry,
Certificate fingerprint (SHA1): [...]
startsslca1, Aug 5, 2013, trustedCertEntry,
Certificate fingerprint (SHA1): [...]

5. Configure Tomcat with SSL

Enabling SSL with Tomcat involves creating a new connector which listens to HTTPS connections. This connector needs to know the location of the keystore file as well as the password to access the keystore.

For convenience, I placed my keystore under $TOMCAT_HOME.
<!-- 
Define a SSL HTTP/1.1 Connector on port 8443
This connector uses the JSSE configuration, when using APR, the
connector should be using the OpenSSL style configuration
described in the APR documentation 
-->
<Connector
  protocol="HTTP/1.1"
  port="8443" maxThreads="200"
  scheme="https" secure="true" SSLEnabled="true"
  keystoreFile="checkthecrowd.jks" keystorePass="******"
  clientAuth="false" sslProtocol="TLS"/>
Note that by default, the Tomcat HTTPS port is 8443.

That's all there is to it! After bouncing Tomcat, I am now able to access CheckTheCrowd via HTTPS from port 8443: https://checkthecrowd.com:8443/.

The next step is to configure Apache httpd to forward HTTPS requests to port 8443. I still haven't figured out how to do this yet, so if you have an idea, let me know!