Jan Amoyo

on software development and possibly other things

Generating Unique and Readable IDs in Node.js Using MongoDB

No comments
I had a requirement from an upcoming project to generate unique human-readable IDs. This project is written in Node.js and uses MongoDB for its database.

Ideally, I can use an auto-incrementing Sequence to achieve this. However, unlike most relational databases, MongoDB does not support Sequences. Fortunately, it is not difficult to implement this behavior in MongoDB.

We will use a collection to store our sequences. The sequences will then be incremented using the findAndModify() function. To ensure that sequences do not increment to unmanageable values, the counter must be restarted after a certain period. In my case, I will restart the counter everyday. To achieve this, I will identify each sequence using a prefix of YYMMDD.

Below is the raw MongoDB statement:
db.ids.findAndModify({
  query:  { prefix: 140625 },
  update: { $inc: { count: 1 } },
  upsert: true,
  new:    true
});
It is important to set the upsert and new options - setting upsert to true will insert a new document if the query cannot find a match; while setting new to true will return the updated version of the document.

Testing on the console yields the expected result.
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 1,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 2,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 3,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 4,
    "prefix" : 140625
}

Node.js + Mongoose

I use Mongoose in Node.js as a MongoDB object document mapper (ODM). Mongoose offers an intuitive API to access MongoDB from within Node.js.

To translate the above implementation, we first need to declare a Mongoose schema.
var mongoose = require('mongoose'),
    Schema   = mongoose.Schema;

var IdSchema = new Schema({
  prefix: { type: Number, required: true, index: { unique: true } },
  count:  { type: Number, required: true }
});

mongoose.model('Id', IdSchema);
The schema defines the structure of the document and as well as validation.

Once the schema is declared, translation becomes pretty straightforward. Note that Mongoose does not have a function called findAndModify(), instead, it offers 2 forms: findByIdAndUpdate() and findOneAndUpdate(). In our case, we will use findOneAndUpdate().
var moment = require('moment'),
    Id     = mongoose.model('Id');

var nextId = function (callback) {
  function prefix (date) {
    return parseInt(moment(date).format('YYMMDD'));
  }

  Id.findOneAndUpdate(
    { prefix: prefix(new Date()) },
    { $inc:   { count: 1 } },
    { upsert: true },
    function (err, idDoc) {
      callback(err, idDoc);
    });
};
Lines 5-7 generates the prefix with the help of Moment.js.

Note that while findAndModify() and its Mongoose equivalents are an atomic operations, there is still a chance that multiple clients try to upsert the same document and hence would fail due to constraint violation - in such scenarios, the call to nextId() must be retried.

House Keeping

Because a new document is inserted every time the counter is reset, the documents will accumulate overtime. Fortunately, because the prefixes are stored as numbers, removing old documents becomes very easy.

For example, if we want to remove documents older than 2015, we just issue the below statement.
db.ids.remove({
  prefix: { $lt: 150000 }
});
The $lt operator stands for "less than". The above statement roughly translates to: delete from ids where prefix < 15000.

No comments :

Post a Comment

Sublime Text Packages for Node.js/JavaScript

No comments
Since I started working on Node.js for an upcoming project, I've become a fan of Sublime Text. It is fast; extensible; has great community support; and most importantly, runs on Linux.

So far, I use Sublime Text exclusively for Node.js and web development (HTML5, CSS, JavaScript). Below, I've compiled a list of packages I found most useful.

As a prerequisite, Package Control needs to be installed. Installing Package Control is a matter of copy-pasting a code snippet to your Sublime Text console. Installation instruction is found here.

Once Package Control is installed, you can start installing other packages by opening the Command Pallete (ctrl+shift+p) and searching for "Install Package".

Without further ado, below are the packages (search for the text in bold):

Alignment
* Aligns various texts
* Use via ctrl+alt-a
Figure 1a: Alignment (before)
Figure 1b: Alignment (after)
BracketHighlighter
* Highlights brackets, braces, and parentheses.
Figure 2: BracketHighlighter
Emmet
* Easily write HTML
* Use via ctrl+alt-enter
* For more information on Emmet, check-out this interactive guide.
Figure 3: Emmet
SidebarEnhancements
* Adds useful menu items to your sidebar
* Only available for Sublime Text 3
Figure 4: SidebarEnhancements
HTML-CSS-JS Prettify
* Formats HTML, CSS, and JavaScript files
* Use via ctrl+shift+h
* Requires Node.js to be installed
Figure 5a: Prettify (before)
Figure 5b: Prettify (after)
TrailingSpaces
* Highlights trailing spaces
Figure 6: TrailingSpaces
SublimeLinter
* Highlights lint errors for various file formats.
Figure 7: SublimeLinter
  • For Sublime Text 3, each linter needs to be installed separately:
    • SublimeLinter-jshint (JavaScript)
      • Requires jshint
      • install via "sudo npm install -g jshint"
    • SublimeLinter-html-tidy (HTML)
      • Requires tidy 
      • install via "sudo apt-get install tidy"

No comments :

Post a Comment