Jan Amoyo

When updating documents, MongoDB has a useful feature to insert a new document when no document matches the query criteria. This feature is called an upsert. Sadly, as of this writing, DynamoDB misses on this feature out of the box.

Thankfully, there's a way to achieve this. The idea is to do it in 3 steps: (1) Get the previous copy of the item. (2) If a previous copy exists, update it. (3) If it does not exist, insert the item ensuring that concurrent requests do not overwrite each other. Here's a snippet written for Node.js:

function upsert(tableName, partitionKey, sortKey, data) {
  // ...

  // 1. Get the original item
  return _get(partitionKey, sortKey).the(function (original) {
    if (Object.keys(original).length > 0) {
      // 2. Update if item already exists
      return _update(data, original);
    } else {
      // 3. Otherwise, put the item
      return _put(data).catch(function (err) {
        if (err.code === 'ConditionalCheckFailedException') {
          // 3a. Only 1 of the concurrent puts will succeed,
          // the rest should retry recursively
          return this.upsert(tableName, partitionKey, sortKey, data);
        } else {
          throw err;
        }
      });
    }
  });
}

The last part is where it gets tricky -- below is the complete code that illustrates how it is done:

function upsert(tableName, partitionKey, sortKey, data) {

  function _get(partitionKey, sortKey) {
    var params = {
      TableName: tableName,
      Key: {
        partitionKey: partitionKey,
        sortKey:      sortKey
      }
    };

    return docClient.get(params).promise();
  }

  function _update(data, original) {
    var updateExpression = dynamodbUpdateExpression.getUpdateExpression({ data: original }, { data: data });
    var params = Object.assign({
      TableName: tableName,
      Key: {
        partitionKey: partitionKey,
        sortKey:      sortKey
      },
      ReturnValues: 'ALL_NEW',
      ConditionExpression: 'attribute_exists(partitionKey) AND attribute_exists(sortKey)'
    }, updateExpression);

    if (params.UpdateExpression === '') {
      return Promise.resolve();
    }

    return new Promise(function (resolve, reject) {
      return docClient.update(params).promise()
        .then(function (result) { resolve(result.Attributes.data); })
        .catch(reject);
    });
  }

  function _put(data) {
    var params = {
      TableName: tableName,
      Item: {
        partitionKey: partitionKey,
        sortKey:      sortKey,
        data:         data
      },
      ConditionExpression: 'attribute_not_exists(partitionKey) AND attribute_not_exists(sortKey)'
    };

    return docClient.put(params).promise();
  }

  // 1. Get the original item
  return _get(partitionKey, sortKey).the(function (original) {
    if (Object.keys(original).length > 0) {
      // 2. Update if item already exists
      return _update(data, original);
    } else {
      // 3. Otherwise, put the item
      return _put(data).catch(function (err) {
        if (err.code === 'ConditionalCheckFailedException') {
          // 3a. Only 1 of the concurrent puts will succeed,
          // the rest should retry recursively
          return this.upsert(tableName, partitionKey, sortKey, data);
        } else {
          throw err;
        }
      });
    }
  });
}

The trick is to declare a Condition Expression in the put step to ensure that an item only gets inserted if a previous copy does not exist (Line 46). This ensures that when handling concurrent put requests, only the 1st request succeeds and the others fail with a ConditionalCheckFailedException error. We then check for this error type to determine if any of the failed requests should be retried as update requests.

The above code uses dynamodb-update-expression (Line 16) to generate DynamoDB Update Expressions.

Below is a simple JavaScript utility for publishing messages across browser windows of the same domain. This implementation uses the browser's localStorage and the storage event to simulate the behavior of an inter-window topic.

(function (global, window) {
  function Publisher() {
    var PUBLISH_PREFIX = 'publish_';
    this.publish = function (topic, message) {
      message.source    = window.name;
      message.timestamp = Date.now();
      window.localStorage.setItem(PUBLISH_PREFIX + topic,
        JSON.stringify(message));
    };
    this.subscribe = function (topic, callback, alias) {
      var subscriber = function (event) {
        if (event.key === PUBLISH_PREFIX + topic
            && event.newValue !== null) {
          callback(JSON.parse(event.newValue));
        }
      };
      window.addEventListener('storage', subscriber);
      return function () {
        window.removeEventListener('storage', subscriber);
      };
    };
  }
  global.jramoyo = { Publisher: new Publisher() };
})(this, window);

Lines 5 and 6 adds a source and timestamp property to the message to ensure uniqueness
Lines 7 and 8 converts the message to JSON and saves it to the localStorage
Lines 12 and 13 uses the event.key to filter which message should be processed by the callback
Line 14 converts the JSON value to a message objects and passes it as an argument to the callback
Lines 18-20 returns a function that when called, removes the subscriber from the topic

Below is a sample code from a publishing window:

jramoyo.Publisher.publish('greeting_topic', {
    name: 'Kyle Katarn'
});

And here is a sample code from a subscribing window:

jramoyo.Publisher.subscribe('greeting_topic',
    function (message) {
        alert('Greetings, ' + message.name);
    });

This works because every time an item is stored in the localStorage, all browser windows sharing the same localStorage will receive a storage event detailing what has changed (except for the window that wrote to the localStorage).

However, the problem with the above implementation is that it doesn't scale if the number of subscribers increases. Every time a storage event is fired, the JavaScript engine will have to iterate through each listener regardless whether the listener is interested in the event or not.

To address this problem, we can use a map to index the callbacks against the event.key. Below is the updated version of the above code:

(function (global, window) {
  function Publisher() {
    var PUBLISH_PREFIX = 'publish_';
    var listeners = []; 
    window.addEventListener('storage',
      function storageListener(event) {
        var array = listeners[event.key];
        if (array && array.length > 0) {
          var message = JSON.parse(event.newValue);
          array.forEach(function (listener) {
            listener(message);
          });
        }
      }, false);
    this.publish = function (topic, message) {
      message.source    = window.name;
      message.timestamp = Date.now();
      window.localStorage.setItem(PUBLISH_PREFIX + topic,
        JSON.stringify(message));
    }; 
    this.subscribe = function (topic, callback, alias) {
      var key = PUBLISH_PREFIX + topic,
        array = listeners[key];
      if (!array) {
        array = []; listeners[key] = array;
      }
      array.push(callback);
      return function () {
        array.splice(array.indexOf(callback), 1);
      };
    };
  }
  global.jramoyo = { Publisher: new Publisher() };
})(this, window);

Lines 5-14 registers a single storage event listener that uses a map to look-up callbacks identified by the event.key
Lines 22-27 saves the callback into the listeners map, identified by the derived key

The example below shows how to increase the limitTo filter of ngRepeat everytime the div scrollbar reaches the bottom.

This technique is used to improve performance by only rendering ngRepeat instances that are visible from the view.

First, we create a directive that calls a function whenever the div scrollbar reaches the bottom:

module.exports = function (_module) {
  'use strict';
  _module.directive('bufferedScroll', function ($parse) {
    return function ($scope, element, attrs) {
      var handler = $parse(attrs.bufferedScroll);
      element.scroll(function (evt) {
        var scrollTop    = element[0].scrollTop,
            scrollHeight = element[0].scrollHeight,
            offsetHeight = element[0].offsetHeight;
        if (scrollTop === (scrollHeight - offsetHeight)) {
          $scope.$apply(function () {
            handler($scope);
          });
        }
      });
    };
  });
};

Line 5 compiles the expression passed to the directive.
Line 6 listens for scroll events on the directive element (requires jQuery).
Line 10 checks if the scrollbar has reached the bottom.
Line 12 applies the compiled expression to the scope.

Then, we apply the directive to our view:

<div buffered-scroll="increaseLimit();" ng-init="limit=15;">
  <table>
    <tr ng-repeat="item in items | limitTo:limit">
      ...
    </tr>
  </table>
</div>

Line 1 assigns a function expression to the directive and initializes the limit variable.
Line 3 declares the ngRepeat with a limitTo filter.

Finally, we create the scope function that increases the limit variable if the value is still less than the number of items iterated by ngRepeat.

$scope.increaseLimit = function () {
  if ($scope.limit < $scope.items.length) {
    $scope.limit += 15;
  }
};

A working example is available at Plunker.

The same technique can be used to implement "infinite scroll" by calling a function that appends data from the server instead of increasing the limitTo filter.

The example below uses 2 Angular directives to change the focus behavior when pressing the tab key.

The first directive is used to assign a name to a 'focusable' element.

module.exports = function (_module) {
  _module.directive('focusName', function () {
    return {
      restrict: 'A',
      link: function ($scope, element, attributes) {
        $scope.focusRegistry = $scope.focusRegistry || {};
        $scope.focusRegistry[attributes.focusName] = element[0];
      }
    };
  });
};

Line 6 lazily creates a scope object called focusRegistry. This object will be used to store all 'focusable' elements within the scope.

Line 7 registers the element using the value of the attribute focusName (focus-name) as the key.

The second directive is used declare the which element will be focused when the tab key is pressed. It will also be responsible for handling the events triggered by pressing the tab key.

module.exports = function (_module) {
  _module.directive('nextFocus', function () {
    return {
      restrict: 'A',
      link: function ($scope, element, attributes) {
        element.bind('keydown keypress', function (event) {
          if (event.which === 9) { // Tab
            var focusElement = $scope.focusRegistry[attributes.nextFocus];
            if (focusElement) {
              if (!focusElement.hidden && !focusElement.disabled) {
                focusElement.focus();
                event.preventDefault();
                return;
              }
            }

            console.log('Unable to focus on target: ' + attributes.nextFocus);
          }
        });
      }
    };
  });
};

Line 7 captures the keydown or keypress event on the tab key (9).

Line 8 retrieves the focus element from the registry using the value of the attribute nextFocus (next-focus).

Lines 10 to 14, moves the focus to the element if it is possible.

If the focus cannot be moved to the specified element, the default behavior of tab will take effect.

Below is an example on how to use both directives:

<input name="1" focus-name="1" next-focus="2" value="next focus goes to 2" />
<input name="2" focus-name="2" next-focus="3" value="next focus goes to 3" />
<input name="3" focus-name="3" next-focus="1" value="next focus goes back to 1" />
<input name="4" value="will not be focused" />

A working example is available at Plunker.

It is important to note that this approach will only work on elements within the same scope.

Directives are the cornerstone of every Angular application. And templates help keep their behavior separate from the presentation.

Karma works well with Angular and is an essential tool for running tests against a number of supported browsers.

Lastly, Browserify helps preserve sanity while maintaining JavaScript modules (similar to Node.js i.e. CommonJS spec).

Sadly, integrating all four concepts is not a straightforward process. Below is a rough guide on how to achieve this.

1. Install karma-browserifast and karma-ng-html2js-preprocessor

$ npm install karma-browserifast --save-dev
$ npm install karma-ng-html2js-preprocessor --save-dev

The package karma-browserifast enables Browserify support on Karma. While karma-ng-html2js-preprocessor converts HTML templates to JavaScript and loads them as an Angular module.

2. Configure Karma

module.exports = function (config) {
  config.set({
    files: [
      'node_modules/angular/angular.js',
      'src/**/*.html'
    ],

    browserify: {
      files: [
        'test/unit/**/*.js',
      ],
      debug: true
    },

    preprocessors: {
      '/**/*.browserify': ['browserify'],
      'src/**/*.html':    ['ng-html2js'],
    },

    ngHtml2JsPreprocessor: {
      moduleName: 'karma.templates'
    },

    frameworks: ['jasmine', 'browserify'],
    browsers:   ['Chrome'],
    reporters:  ['spec'],
    logLevel:   'info',
    autoWatch:  true,
    colors:     true,
  });
};

Lines 4 and 5 loads the HTML templates into Karma. Because they will be pre-processed by karma-ng-html2js-preprocessor, they will eventually get loaded as JavaScript files. Note that even if Angular will be included as part of the Browserify bundle, it is important to load it explicitly. Otherwise, the templates cannot be made available to Angular.

Line 10 tells karma-browserifast to include all unit tests as a Browserify bundle. This makes it possible for unit tests to execute JavaScript modules using the "require" keyword.

Line 21 tells karma-ng-html2js-preprocessor to load the JavaScript templates as an Angular module named "karma.templates". This will later be used in unit tests to allow testing of directives that use templates.

3. Write the unit tests

require('../src/app');
require('angular-mocks/angular-mocks');
describe('myDirective', function () {
  var scope, element;

  beforeEach(function () {
      angular.mock.module('karma.templates');
      angular.mock.module('myModule');
      angular.mock.inject(function ($rootScope, $compile) {
          scope   = $rootScope.$new();
          element = $compile('<div my-directive></div>')(scope);

          scope.$digest();
      });
  });

  it('does something', function () {
    expect(1).toBe(1);
  });
});

Lines 1 and 2 demonstrates the capability of a Karma-based unit test to load modules via "require".

Line 7 loads the generated JavaScript templates as an Angular module; without this, the directive will not compile because the template cannot be fetched.

Browsers usually cache static resources like JavaScript and CSS so that downloads are reduced the next time the same website is visited.

However, if the JavaScript or CSS gets updated in the server, the browser will still be using an outdated copy of the resource. This could lead to unexpected behavior.

This can be addressed by including a version to the resource name and incrementing the value every time an update is released (ex: /my-styles-v2.css). An easier alternative is to use a query parameter that indicates the resource version (ex: /my-styles.css?version=2). On either solution, HTML pages that link to the resources need to be updated whenever a resource version changes -- this can be difficult to maintain especially if there are many resources that constantly change.

There are tools that automate this process and are usually incorporated into the build process. The solution I'm going to demonstrate will address the issue without adding an extra build step.

Note: This example is specific to Java web applications using JSTL.

The first step is to add an application-level configuration that holds the current version of the static resources. This comes as a context parameter in web.xml:

<web-app>
  ...

  <!-- Indicates the CSS and JS versions -->
  <context-param>
    <param-name>resourceVersion</param-name>
    <param-value>1</param-value>
  </context-param>

  ...

</web-app>

Lines 6 and 7 indicate the value of the resource version and should be incremented before every release.

The next step is to update the JSPs that link to the resources and format the resource URLs to include the version number.

<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<c:url value="/resources/css/my-styles.css" var="myCss">
  <c:param name="version" value="${initParam.resourceVersion}" />
</c:url>
<c:url value="/resources/js/my-scripts.js" var="myJavaScript">
  <c:param name="version" value="${initParam.resourceVersion}" />
</c:url>
...
<!DOCTYPE html>
<html lang="en">
<head>
...
<link rel="stylesheet" href="${myCss}" type="text/css" />
...
</head>
<body>
...
<script src="${myJavaScript}"></script>
</body>
</html>

Lines 2-7 declare URL variables that include "version" as a query parameter. Note that context parameters are accessible from JSTL via ${initParam}.

Lines 13 and 18 uses the variables to render the URLs.

I had a requirement from an upcoming project to generate unique human-readable IDs. This project is written in Node.js and uses MongoDB for its database.

Ideally, I can use an auto-incrementing Sequence to achieve this. However, unlike most relational databases, MongoDB does not support Sequences. Fortunately, it is not difficult to implement this behavior in MongoDB.

We will use a collection to store our sequences. The sequences will then be incremented using the findAndModify() function. To ensure that sequences do not increment to unmanageable values, the counter must be restarted after a certain period. In my case, I will restart the counter everyday. To achieve this, I will identify each sequence using a prefix of YYMMDD.

Below is the raw MongoDB statement:

db.ids.findAndModify({
  query:  { prefix: 140625 },
  update: { $inc: { count: 1 } },
  upsert: true,
  new:    true
});

It is important to set the upsert and new options - setting upsert to true will insert a new document if the query cannot find a match; while setting new to true will return the updated version of the document.

Testing on the console yields the expected result.

> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 1,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 2,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 3,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 4,
    "prefix" : 140625
}

Node.js + Mongoose

I use Mongoose in Node.js as a MongoDB object document mapper (ODM). Mongoose offers an intuitive API to access MongoDB from within Node.js.

To translate the above implementation, we first need to declare a Mongoose schema.

var mongoose = require('mongoose'),
    Schema   = mongoose.Schema;

var IdSchema = new Schema({
  prefix: { type: Number, required: true, index: { unique: true } },
  count:  { type: Number, required: true }
});

mongoose.model('Id', IdSchema);

The schema defines the structure of the document and as well as validation.

Once the schema is declared, translation becomes pretty straightforward. Note that Mongoose does not have a function called findAndModify(), instead, it offers 2 forms: findByIdAndUpdate() and findOneAndUpdate(). In our case, we will use findOneAndUpdate().

var moment = require('moment'),
    Id     = mongoose.model('Id');

var nextId = function (callback) {
  function prefix (date) {
    return parseInt(moment(date).format('YYMMDD'));
  }

  Id.findOneAndUpdate(
    { prefix: prefix(new Date()) },
    { $inc:   { count: 1 } },
    { upsert: true },
    function (err, idDoc) {
      callback(err, idDoc);
    });
};

Lines 5-7 generates the prefix with the help of Moment.js.

Note that while findAndModify() and its Mongoose equivalents are an atomic operations, there is still a chance that multiple clients try to upsert the same document and hence would fail due to constraint violation - in such scenarios, the call to nextId() must be retried.

House Keeping

Because a new document is inserted every time the counter is reset, the documents will accumulate overtime. Fortunately, because the prefixes are stored as numbers, removing old documents becomes very easy.

For example, if we want to remove documents older than 2015, we just issue the below statement.

db.ids.remove({
  prefix: { $lt: 150000 }
});

The $lt operator stands for "less than". The above statement roughly translates to: delete from ids where prefix < 15000.

Since I started working on Node.js for an upcoming project, I've become a fan of Sublime Text. It is fast; extensible; has great community support; and most importantly, runs on Linux.

So far, I use Sublime Text exclusively for Node.js and web development (HTML5, CSS, JavaScript). Below, I've compiled a list of packages I found most useful.

As a prerequisite, Package Control needs to be installed. Installing Package Control is a matter of copy-pasting a code snippet to your Sublime Text console. Installation instruction is found here.

Once Package Control is installed, you can start installing other packages by opening the Command Pallete (ctrl+shift+p) and searching for "Install Package".

Without further ado, below are the packages (search for the text in bold):

Alignment
* Aligns various texts
* Use via ctrl+alt-a

Figure 1a: Alignment (before)

Figure 1b: Alignment (after)

BracketHighlighter
* Highlights brackets, braces, and parentheses.

Figure 2: BracketHighlighter

Emmet
* Easily write HTML
* Use via ctrl+alt-enter
* For more information on Emmet, check-out this interactive guide.

Figure 3: Emmet

SidebarEnhancements
* Adds useful menu items to your sidebar
* Only available for Sublime Text 3

Figure 4: SidebarEnhancements

HTML-CSS-JS Prettify
* Formats HTML, CSS, and JavaScript files
* Use via ctrl+shift+h
* Requires Node.js to be installed

Figure 5a: Prettify (before)

Figure 5b: Prettify (after)

TrailingSpaces
* Highlights trailing spaces

Figure 6: TrailingSpaces

SublimeLinter
* Highlights lint errors for various file formats.

Figure 7: SublimeLinter

• For Sublime Text 3, each linter needs to be installed separately:
• SublimeLinter-jshint (JavaScript)
• Requires jshint
• install via "sudo npm install -g jshint"
• SublimeLinter-html-tidy (HTML)
• Requires tidy 
• install via "sudo apt-get install tidy"