Jan Amoyo

Unlike traditional databases, MongoDB does not support transactions. So suppose you have multiple documents and want to perform a "save all or nothing" operation, you'd have to simulate a transaction from within your application.

Implementing such a transaction in NodeJS can be tricky because IO operations are generally asynchronous. This can lead to nested callbacks as demonstrated by the code below:

var MyModel = require('mongoose').model('MyModel');

var docs = [];
MyModel.create({ field: 'value1' }, function (err, doc) {
  if (err) { console.log(err); }
  else {
    docs.push(doc);
    MyModel.create({ field: 'value2' }, function (err, doc) {
      if (err) { rollback(docs); }
      else {
        docs.push(doc);
        MyModel.create({ field: 'value3' }, function (err, doc) {
          if (err) { rollback(docs); }
          else {
            console.log('Done.');
          }
        });
      }
    });
  }
});

Despite not having a dependency on other documents, inserting each document must be performed in series. This is because we have no way of finding out when all other documents will finish saving. This approach is problematic because it leads to deeper nesting as the number of documents increase. Also, in the above example, the callback functions modify the docs variable - this is a side effect and breaks functional principles. With the help of Async, both issues can be addressed.

The parallel() function of Async allows you run multiple functions in parallel. Each function signals completion by invoking a callback to which either the result of the operation or an error is passed. Once all functions are completed, an optional callback function is invoked to handle the results or errors.

The above code can be improved by implementing document insertions as functions of parallel(). If an insert succeeds, the inserted document will be passed to the callback; otherwise, the error will be passed (these can later be used when a rollback is required). Once all the parallel functions complete, the parallel callback can perform a rollback if any of the functions failed to save.

Below is the improved version using Async:

var async    = require('async'),
    mongoose = require('mongoose');

var MyModel = mongoose.model('MyModel');

async.parallel([
    function (callback) {
      MyModel.create({ field: 'value1' }, callback);
    },
    function (callback) {
      MyModel.create({ field: 'value2' }, callback);
    },
    function (callback) {
      MyModel.create({ field: 'value3' }, callback);
    }
  ],
  function (errs, results) {
    if (errs) {
      async.each(results, rollback, function () {
        console.log('Rollback done.');
      });
    } else {
      console.log('Done.');
    }
  });

function rollback (doc, callback) {
  if (!doc) { callback(); }
  else {
    MyModel.findByIdAndRemove(doc._id, function (err, doc) {
      console.log('Rolled-back document: ' + doc);
      callback();
    });
  }
}

Lines 8, 11, and 14 inserts a new document to MongoDB then passes either the saved document or an error to the callback.

Lines 18-21 checks if any of the parallel functions threw an error, and calls the rollback() function on each document passed to callback of parallel().

Lines 28-33 performs the rollback by deleting the inserted documents using their IDs.

Edit: As pointed-out in the comments, the rollback itself can fail. In such scenarios, it is best to retry the rollback until it succeeds. Therefore, the rollback logic must be an idempotent operation.

I had a requirement from an upcoming project to generate unique human-readable IDs. This project is written in Node.js and uses MongoDB for its database.

Ideally, I can use an auto-incrementing Sequence to achieve this. However, unlike most relational databases, MongoDB does not support Sequences. Fortunately, it is not difficult to implement this behavior in MongoDB.

We will use a collection to store our sequences. The sequences will then be incremented using the findAndModify() function. To ensure that sequences do not increment to unmanageable values, the counter must be restarted after a certain period. In my case, I will restart the counter everyday. To achieve this, I will identify each sequence using a prefix of YYMMDD.

Below is the raw MongoDB statement:

db.ids.findAndModify({
  query:  { prefix: 140625 },
  update: { $inc: { count: 1 } },
  upsert: true,
  new:    true
});

It is important to set the upsert and new options - setting upsert to true will insert a new document if the query cannot find a match; while setting new to true will return the updated version of the document.

Testing on the console yields the expected result.

> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 1,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 2,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 3,
    "prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
    "_id" : ObjectId("53aae1d126d57c198d861cfd"),
    "count" : 4,
    "prefix" : 140625
}

Node.js + Mongoose

I use Mongoose in Node.js as a MongoDB object document mapper (ODM). Mongoose offers an intuitive API to access MongoDB from within Node.js.

To translate the above implementation, we first need to declare a Mongoose schema.

var mongoose = require('mongoose'),
    Schema   = mongoose.Schema;

var IdSchema = new Schema({
  prefix: { type: Number, required: true, index: { unique: true } },
  count:  { type: Number, required: true }
});

mongoose.model('Id', IdSchema);

The schema defines the structure of the document and as well as validation.

Once the schema is declared, translation becomes pretty straightforward. Note that Mongoose does not have a function called findAndModify(), instead, it offers 2 forms: findByIdAndUpdate() and findOneAndUpdate(). In our case, we will use findOneAndUpdate().

var moment = require('moment'),
    Id     = mongoose.model('Id');

var nextId = function (callback) {
  function prefix (date) {
    return parseInt(moment(date).format('YYMMDD'));
  }

  Id.findOneAndUpdate(
    { prefix: prefix(new Date()) },
    { $inc:   { count: 1 } },
    { upsert: true },
    function (err, idDoc) {
      callback(err, idDoc);
    });
};

Lines 5-7 generates the prefix with the help of Moment.js.

Note that while findAndModify() and its Mongoose equivalents are an atomic operations, there is still a chance that multiple clients try to upsert the same document and hence would fail due to constraint violation - in such scenarios, the call to nextId() must be retried.

House Keeping

Because a new document is inserted every time the counter is reset, the documents will accumulate overtime. Fortunately, because the prefixes are stored as numbers, removing old documents becomes very easy.

For example, if we want to remove documents older than 2015, we just issue the below statement.

db.ids.remove({
  prefix: { $lt: 150000 }
});

The $lt operator stands for "less than". The above statement roughly translates to: delete from ids where prefix < 15000.