tag:blogger.com,1999:blog-60636270245204877342024-02-08T02:54:50.776+08:00Jan Amoyoon software development and possibly other thingsjramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.comBlogger18125tag:blogger.com,1999:blog-6063627024520487734.post-54246364657475537722017-03-27T10:10:00.000+08:002017-03-27T14:24:12.646+08:00Upserting Items into DynamoDBWhen updating documents, MongoDB has a useful feature to insert a new document when no document matches the query criteria. This feature is called an <b>upsert</b>. Sadly, as of this writing, DynamoDB misses on this feature out of the box.<br />
<br />
Thankfully, there's a way to achieve this. The idea is to do it in 3 steps: (1) Get the previous copy of the item. (2) If a previous copy exists, update it. (3) If it does not exist, insert the item <u>ensuring that concurrent requests do not overwrite each other</u>. Here's a snippet written for Node.js:<br />
<pre class="brush:javascript; highlight:[5,8,11,15];">function upsert(tableName, partitionKey, sortKey, data) {
// ...
// 1. Get the original item
return _get(partitionKey, sortKey).the(function (original) {
if (Object.keys(original).length > 0) {
// 2. Update if item already exists
return _update(data, original);
} else {
// 3. Otherwise, put the item
return _put(data).catch(function (err) {
if (err.code === 'ConditionalCheckFailedException') {
// 3a. Only 1 of the concurrent puts will succeed,
// the rest should retry recursively
return this.upsert(tableName, partitionKey, sortKey, data);
} else {
throw err;
}
});
}
});
}
</pre>
The last part is where it gets tricky -- below is the complete code that illustrates how it is done:
<br />
<pre class="brush:javascript; highlight:[16,46];">function upsert(tableName, partitionKey, sortKey, data) {
function _get(partitionKey, sortKey) {
var params = {
TableName: tableName,
Key: {
partitionKey: partitionKey,
sortKey: sortKey
}
};
return docClient.get(params).promise();
}
function _update(data, original) {
var updateExpression = dynamodbUpdateExpression.getUpdateExpression({ data: original }, { data: data });
var params = Object.assign({
TableName: tableName,
Key: {
partitionKey: partitionKey,
sortKey: sortKey
},
ReturnValues: 'ALL_NEW',
ConditionExpression: 'attribute_exists(partitionKey) AND attribute_exists(sortKey)'
}, updateExpression);
if (params.UpdateExpression === '') {
return Promise.resolve();
}
return new Promise(function (resolve, reject) {
return docClient.update(params).promise()
.then(function (result) { resolve(result.Attributes.data); })
.catch(reject);
});
}
function _put(data) {
var params = {
TableName: tableName,
Item: {
partitionKey: partitionKey,
sortKey: sortKey,
data: data
},
ConditionExpression: 'attribute_not_exists(partitionKey) AND attribute_not_exists(sortKey)'
};
return docClient.put(params).promise();
}
// 1. Get the original item
return _get(partitionKey, sortKey).the(function (original) {
if (Object.keys(original).length > 0) {
// 2. Update if item already exists
return _update(data, original);
} else {
// 3. Otherwise, put the item
return _put(data).catch(function (err) {
if (err.code === 'ConditionalCheckFailedException') {
// 3a. Only 1 of the concurrent puts will succeed,
// the rest should retry recursively
return this.upsert(tableName, partitionKey, sortKey, data);
} else {
throw err;
}
});
}
});
}
</pre>
The trick is to declare a <a href="http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html" rel="nofollow" target="_blank">Condition Expression</a> in the <span style="font-family: "courier new" , "courier" , monospace;">put</span> step to ensure that an item only gets inserted if a previous copy does not exist (Line 46). This ensures that when handling concurrent <span style="font-family: "courier new" , "courier" , monospace;">put</span> requests, only the 1st request succeeds and the others fail with a <span style="font-family: "courier new" , "courier" , monospace;">ConditionalCheckFailedException</span> error. We then check for this error type to determine if any of the failed requests should be retried as <span style="font-family: "courier new" , "courier" , monospace;">update</span> requests.<br />
<br />
The above code uses <a href="https://github.com/4ossiblellc/dynamodb-update-expression" target="_blank">dynamodb-update-expression</a> (Line 16) to generate DynamoDB Update Expressions.jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-56472880835091203602015-12-06T10:40:00.001+08:002015-12-07T14:33:42.247+08:00Using LocalStorage to Publish Messages Across Browser WindowsBelow is a simple JavaScript utility for publishing messages across <b>browser windows of the same domain</b>. This implementation uses the browser's <span style="font-family: "courier new" , "courier" , monospace;">localStorage</span> and the <a href="https://developer.mozilla.org/en-US/docs/Web/Events/storage" target="_blank">storage event</a> to simulate the behavior of an inter-window topic.<br />
<pre class="brush:js; highlight:[5,6,7,8,12,13,14,18,19,20];">(function (global, window) {
function Publisher() {
var PUBLISH_PREFIX = 'publish_';
this.publish = function (topic, message) {
message.source = window.name;
message.timestamp = Date.now();
window.localStorage.setItem(PUBLISH_PREFIX + topic,
JSON.stringify(message));
};
this.subscribe = function (topic, callback, alias) {
var subscriber = function (event) {
if (event.key === PUBLISH_PREFIX + topic
&& event.newValue !== null) {
callback(JSON.parse(event.newValue));
}
};
window.addEventListener('storage', subscriber);
return function () {
window.removeEventListener('storage', subscriber);
};
};
}
global.jramoyo = { Publisher: new Publisher() };
})(this, window);
</pre>
Lines 5 and 6 adds a source and timestamp property to the message to ensure uniqueness<br />
Lines 7 and 8 converts the message to JSON and saves it to the <span style="font-family: "courier new" , "courier" , monospace;">localStorage</span><br />
Lines 12 and 13 uses the <span style="font-family: "courier new" , "courier" , monospace;">event.key</span> to filter which message should be processed by the callback<br />
Line 14 converts the JSON value to a message objects and passes it as an argument to the callback<br />
Lines 18-20 returns a function that when called, removes the subscriber from the topic<br />
<br />
Below is a sample code from a publishing window:
<pre class="brush:js; highlight:[];">
jramoyo.Publisher.publish('greeting_topic', {
name: 'Kyle Katarn'
});
</pre>
And here is a sample code from a subscribing window:
<pre class="brush:js; highlight:[];">
jramoyo.Publisher.subscribe('greeting_topic',
function (message) {
alert('Greetings, ' + message.name);
});
</pre>
<br />
This works because every time an item is stored in the <span style="font-family: "courier new" , "courier" , monospace;">localStorage</span>, all browser windows sharing the same <span style="font-family: "courier new" , "courier" , monospace;">localStorage</span> will receive a storage event detailing what has changed (except for the window that wrote to the <span style="font-family: "courier new" , "courier" , monospace;">localStorage</span>).<br />
<br />
However, the problem with the above implementation is that it doesn't scale if the number of subscribers increases. Every time a storage event is fired, the JavaScript engine will have to iterate through each listener regardless whether the listener is interested in the event or not.<br />
<br />
To address this problem, we can use a map to index the callbacks against the <span style="font-family: "courier new" , "courier" , monospace;">event.key</span>. Below is the updated version of the above code:<br />
<pre class="brush:js; highlight:[5,6,7,8,9,10,11,12,13,14,22,23,24,25,26,27];">(function (global, window) {
function Publisher() {
var PUBLISH_PREFIX = 'publish_';
var listeners = [];
window.addEventListener('storage',
function storageListener(event) {
var array = listeners[event.key];
if (array && array.length > 0) {
var message = JSON.parse(event.newValue);
array.forEach(function (listener) {
listener(message);
});
}
}, false);
this.publish = function (topic, message) {
message.source = window.name;
message.timestamp = Date.now();
window.localStorage.setItem(PUBLISH_PREFIX + topic,
JSON.stringify(message));
};
this.subscribe = function (topic, callback, alias) {
var key = PUBLISH_PREFIX + topic,
array = listeners[key];
if (!array) {
array = []; listeners[key] = array;
}
array.push(callback);
return function () {
array.splice(array.indexOf(callback), 1);
};
};
}
global.jramoyo = { Publisher: new Publisher() };
})(this, window);
</pre>
Lines 5-14 registers a single storage event listener that uses a map to look-up callbacks identified by the <span style="font-family: "courier new" , "courier" , monospace;">event.key</span><br />
Lines 22-27 saves the callback into the listeners map, identified by the derived keyjramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-80088445722674174172015-07-05T11:28:00.000+08:002015-12-05T17:55:28.263+08:00Increasing ngRepeat Limit on ScrollThe example below shows how to increase the <span style="font-family: "courier new" , "courier" , monospace;">limitTo</span> filter of <span style="font-family: "courier new" , "courier" , monospace;">ngRepeat</span> everytime the div scrollbar reaches the bottom.<br />
<br />
This technique is used to improve performance by only rendering <span style="font-family: "courier new" , "courier" , monospace;">ngRepeat</span> instances that are visible from the view.<br />
<br />
First, we create a directive that calls a function whenever the div scrollbar reaches the bottom:<br />
<pre class="brush:js; highlight:[5, 6, 10, 12];">module.exports = function (_module) {
'use strict';
_module.directive('bufferedScroll', function ($parse) {
return function ($scope, element, attrs) {
var handler = $parse(attrs.bufferedScroll);
element.scroll(function (evt) {
var scrollTop = element[0].scrollTop,
scrollHeight = element[0].scrollHeight,
offsetHeight = element[0].offsetHeight;
if (scrollTop === (scrollHeight - offsetHeight)) {
$scope.$apply(function () {
handler($scope);
});
}
});
};
});
};
</pre>
Line 5 compiles the expression passed to the directive.<br />
Line 6 listens for scroll events on the directive element (requires jQuery).<br />
Line 10 checks if the scrollbar has reached the bottom.<br />
Line 12 applies the compiled expression to the scope.<br />
<br />
Then, we apply the directive to our view:<br />
<pre class="brush:html; highlight:[1, 3];"><div buffered-scroll="increaseLimit();" ng-init="limit=15;">
<table>
<tr ng-repeat="item in items | limitTo:limit">
...
</tr>
</table>
</div>
</pre>
Line 1 assigns a function expression to the directive and initializes the limit variable.<br />
Line 3 declares the <span style="font-family: "courier new" , "courier" , monospace;">ngRepeat</span> with a <span style="font-family: "courier new" , "courier" , monospace;">limitTo</span> filter.<br />
<br />
Finally, we create the scope function that increases the <span style="font-family: "courier new" , "courier" , monospace;">limit</span> variable if the value is still less than the number of items iterated by <span style="font-family: "courier new" , "courier" , monospace;">ngRepeat</span>.<br />
<pre class="brush:js; highlight:[];">$scope.increaseLimit = function () {
if ($scope.limit < $scope.items.length) {
$scope.limit += 15;
}
};
</pre>
<br />
A working example is available at <a href="http://plnkr.co/edit/S2x6L8wNjXD8g00WVsJz?p=preview" target="_blank">Plunker</a>.<br />
<br />
<i>The same technique can be used to implement "infinite scroll" by calling a function that appends data from the server instead of increasing the limitTo filter.</i>jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-25928685723881391262014-10-30T23:59:00.000+08:002015-12-06T10:41:53.715+08:00Changing Tab Focus Behavior Using AngularThe example below uses 2 Angular directives to change the focus behavior when pressing the tab key.
<br />
<br />
The first directive is used to assign a name to a 'focusable' element.<br />
<pre class="brush:js; highlight:[6, 7];">module.exports = function (_module) {
_module.directive('focusName', function () {
return {
restrict: 'A',
link: function ($scope, element, attributes) {
$scope.focusRegistry = $scope.focusRegistry || {};
$scope.focusRegistry[attributes.focusName] = element[0];
}
};
});
};
</pre>
Line 6 lazily creates a scope object called <span style="font-family: Courier New, Courier, monospace;">focusRegistry</span>. This object will be used to store all 'focusable' elements within the scope.<br />
<br />
Line 7 registers the element using the value of the attribute <span style="font-family: Courier New, Courier, monospace;">focusName</span> (<span style="font-family: Courier New, Courier, monospace;">focus-name</span>) as the key.<br />
<br />
The second directive is used declare the which element will be focused when the tab key is pressed. It will also be responsible for handling the events triggered by pressing the tab key.<br />
<pre class="brush:js; highlight:[7, 8, 10, 11, 12, 13, 14];">module.exports = function (_module) {
_module.directive('nextFocus', function () {
return {
restrict: 'A',
link: function ($scope, element, attributes) {
element.bind('keydown keypress', function (event) {
if (event.which === 9) { // Tab
var focusElement = $scope.focusRegistry[attributes.nextFocus];
if (focusElement) {
if (!focusElement.hidden && !focusElement.disabled) {
focusElement.focus();
event.preventDefault();
return;
}
}
console.log('Unable to focus on target: ' + attributes.nextFocus);
}
});
}
};
});
};
</pre>
Line 7 captures the <span style="font-family: Courier New, Courier, monospace;">keydown</span> or <span style="font-family: Courier New, Courier, monospace;">keypress</span> event on the tab key (9).<br />
<br />
Line 8 retrieves the focus element from the registry using the value of the attribute <span style="font-family: Courier New, Courier, monospace;">nextFocus</span> (<span style="font-family: Courier New, Courier, monospace;">next-focus</span>).<br />
<br />
Lines 10 to 14, moves the focus to the element if it is possible.<br />
<br />
If the focus cannot be moved to the specified element, the default behavior of tab will take effect.<br />
<br />
Below is an example on how to use both directives:<br />
<pre class="brush:html;"><input name="1" focus-name="1" next-focus="2" value="next focus goes to 2" />
<input name="2" focus-name="2" next-focus="3" value="next focus goes to 3" />
<input name="3" focus-name="3" next-focus="1" value="next focus goes back to 1" />
<input name="4" value="will not be focused" />
</pre>
A working example is available at <a href="http://plnkr.co/edit/1fM9QRYNpvIECJrxLbEX?p=preview" rel="nofollow" target="_blank">Plunker</a>.<br />
<br />
It is important to note that this approach <u>will only work on elements within the same scope</u>.jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-85601234727693486192014-10-04T23:37:00.000+08:002014-10-05T14:49:55.809+08:00Testing Angular Directives with Templates on Karma and Browserify<a href="https://docs.angularjs.org/guide/directive" target="_blank">Directives</a> are the cornerstone of every Angular application. And <a href="https://docs.angularjs.org/guide/templates" target="_blank">templates</a> help keep their behavior separate from the presentation.<br />
<br />
<a href="http://karma-runner.github.io/0.12/intro/how-it-works.html" target="_blank">Karma</a> works well with Angular and is an essential tool for running tests against a number of supported browsers.<br />
<br />
Lastly, <a href="http://browserify.org/demos.html" target="_blank">Browserify</a> helps preserve sanity while maintaining JavaScript modules (similar to Node.js i.e. CommonJS spec).<br />
<br />
Sadly, integrating all four concepts is not a straightforward process. Below is a rough guide on how to achieve this.<br />
<br />
<b>1. Install <a href="https://github.com/cjohansen/karma-browserifast" target="_blank">karma-browserifast</a> and <a href="https://github.com/karma-runner/karma-ng-html2js-preprocessor" target="_blank">karma-ng-html2js-preprocessor</a></b>
<br />
<pre>$ npm install karma-browserifast --save-dev
$ npm install karma-ng-html2js-preprocessor --save-dev
</pre>
The package <span style="font-family: Courier New, Courier, monospace;">karma-browserifast</span> enables Browserify support on Karma. While <span style="font-family: Courier New, Courier, monospace;">karma-ng-html2js-preprocessor</span> converts HTML templates to JavaScript and loads them as an Angular module.<br />
<br />
<b>2. Configure Karma</b>
<br />
<pre class="brush:js; highlight:[4, 5, 10, 21];">module.exports = function (config) {
config.set({
files: [
'node_modules/angular/angular.js',
'src/**/*.html'
],
browserify: {
files: [
'test/unit/**/*.js',
],
debug: true
},
preprocessors: {
'/**/*.browserify': ['browserify'],
'src/**/*.html': ['ng-html2js'],
},
ngHtml2JsPreprocessor: {
moduleName: 'karma.templates'
},
frameworks: ['jasmine', 'browserify'],
browsers: ['Chrome'],
reporters: ['spec'],
logLevel: 'info',
autoWatch: true,
colors: true,
});
};
</pre>
Lines 4 and 5 loads the HTML templates into Karma. Because they will be pre-processed by <span style="font-family: Courier New, Courier, monospace;">karma-ng-html2js-preprocessor</span>, they will eventually get loaded as JavaScript files. Note that even if Angular will be included as part of the Browserify bundle, it is important to load it explicitly. Otherwise, the templates cannot be made available to Angular.<br />
<br />
Line 10 tells <span style="font-family: Courier New, Courier, monospace;">karma-browserifast</span> to include all unit tests as a Browserify bundle. This makes it possible for unit tests to execute JavaScript modules using the "<span style="font-family: Courier New, Courier, monospace;">require</span>" keyword.<br />
<br />
Line 21 tells <span style="font-family: Courier New, Courier, monospace;">karma-ng-html2js-preprocessor</span> to load the JavaScript templates as an Angular module named "<span style="font-family: Courier New, Courier, monospace;">karma.templates</span>". This will later be used in unit tests to allow testing of directives that use templates.<br />
<br />
<b>3. Write the unit tests</b><br />
<pre class="brush:js; highlight:[1, 2, 7];">require('../src/app');
require('angular-mocks/angular-mocks');
describe('myDirective', function () {
var scope, element;
beforeEach(function () {
angular.mock.module('karma.templates');
angular.mock.module('myModule');
angular.mock.inject(function ($rootScope, $compile) {
scope = $rootScope.$new();
element = $compile('<div my-directive></div>')(scope);
scope.$digest();
});
});
it('does something', function () {
expect(1).toBe(1);
});
});
</pre>
Lines 1 and 2 demonstrates the capability of a Karma-based unit test to load modules via "<span style="font-family: Courier New, Courier, monospace;">require</span>".<br />
<br />
Line 7 loads the generated JavaScript templates as an Angular module; without this, the directive will not compile because the template cannot be fetched.<br />
<br />jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-74002193505862124622014-08-01T18:44:00.000+08:002014-08-01T18:44:04.307+08:00Examples of Streams and Lamdas in Java 8Below is a code showcasing Streams and Lamdas in Java 8 (written as a JUnit test allow execution of individual methods).<br />
<pre class="brush:java;">
@RunWith(BlockJUnit4ClassRunner.class)
public class Java8Showcase {
@FunctionalInterface
public static interface Concat {
String concat(String str1, String str2);
}
@Test
public void functional_interfaces() {
// Traditional way of declaring an
// interface implementation
@SuppressWarnings("unused")
Concat _concat = new Concat() {
@Override
public String concat(String str1, String str2) {
return str1.concat(str2);
}
};
Concat contact = (str1, str2) -> str1.concat(str2);
System.out.println(contact.concat("hello", " world"));
}
@Test
public void function_reference() {
List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
// nums.stream().reduce((a, b) -> Math.max(a, b))
Integer max = nums.stream().reduce(Math::max).get();
System.out.println(max);
}
@Test
public void streams_filter() {
List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
List<Integer> evens = nums.stream().filter((n) -> n % 2 == 0)
.collect(Collectors.toList());
System.out.println(evens);
}
@Test
public void streams_map() {
List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
List<Integer> x2 = nums.stream().map((n) -> n * 2)
.collect(Collectors.toList());
System.out.println(x2);
}
@Test
public void streams_reduce() {
List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
Integer sum = nums.stream().reduce((a, b) -> a + b).get();
System.out.println(sum);
}
@Test
public void streams_fluent() {
List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
Integer sum = nums.stream()
.filter((n) -> n % 2 == 0) // filter the even numbers
.map((n) -> n * 2) // multiply each number by 2
.reduce((a, b) -> a + b) // add each number
.get();
System.out.println(sum);
}
@Test
public void streams_parallel() {
List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
Integer sum = nums.parallelStream().reduce((a, b) -> a + b).get();
System.out.println(sum);
}
@Test
public void streams_forEach() {
List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
nums.stream().forEach(System.out::println);
}
@Test
public void streams_collectors() {
List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
Double ave = nums.stream().collect(Collectors.averagingInt((i) -> i));
System.out.println(ave);
}
@Test
public void streams_collectors_grouping1() {
List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
Map<Boolean, List<Integer>> oddEven = nums.stream()
.collect(Collectors.groupingBy((i) -> i % 2 == 0));
System.out.println(oddEven);
}
@Test
public void streams_collectors_grouping2() {
List<Integer> nums = Lists.newArrayList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
Map<Boolean, Integer> oddEvenSum = nums.stream()
.collect(Collectors.groupingBy((i) -> i % 2 == 0, Collectors.summingInt((Integer i) -> i)));
System.out.println(oddEvenSum);
}
}
</pre>jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-14998548220431047292014-07-14T11:24:00.000+08:002014-07-22T23:56:34.275+08:00MongoDB Transaction Across Multiple Documents using Async and MongooseUnlike traditional databases, MongoDB does not support <a href="http://docs.mongodb.org/manual/faq/fundamentals/#what-kind-of-database-is-mongodb" target="_blank">transactions</a>. So suppose you have multiple documents and want to perform a "save all or nothing" operation, you'd have to simulate a transaction from within your application.<br />
<br />
Implementing such a transaction in NodeJS can be tricky because IO operations are generally asynchronous. This can lead to nested callbacks as demonstrated by the code below:<br />
<pre class="brush:javascript; highlight: [9, 13]">var MyModel = require('mongoose').model('MyModel');
var docs = [];
MyModel.create({ field: 'value1' }, function (err, doc) {
if (err) { console.log(err); }
else {
docs.push(doc);
MyModel.create({ field: 'value2' }, function (err, doc) {
if (err) { rollback(docs); }
else {
docs.push(doc);
MyModel.create({ field: 'value3' }, function (err, doc) {
if (err) { rollback(docs); }
else {
console.log('Done.');
}
});
}
});
}
});
</pre>
Despite not having a dependency on other documents, inserting each document must be performed in series. This is because we have no way of finding out when all other documents will finish saving. This approach is problematic because it leads to deeper nesting as the number of documents increase. Also, in the above example, the callback functions modify the <span style="font-family: Courier New, Courier, monospace;">docs</span> variable - this is a side effect and breaks functional principles. With the help of <a href="https://www.npmjs.org/package/async" target="_blank">Async</a>, both issues can be addressed.<br />
<br />
The <a href="https://github.com/caolan/async#parallel" target="_blank"><span style="font-family: Courier New, Courier, monospace;">parallel()</span></a> function of Async allows you run multiple functions in parallel. Each function signals completion by invoking a callback to which either the result of the operation or an error is passed. Once all functions are completed, an optional callback function is invoked to handle the results or errors.<br />
<br />
The above code can be improved by implementing document insertions as functions of <span style="font-family: Courier New, Courier, monospace;">parallel()</span>. If an insert succeeds, the inserted document will be passed to the callback; otherwise, the error will be passed (these can later be used when a rollback is required). Once all the parallel functions complete, the parallel callback can perform a rollback if any of the functions failed to save.<br />
<br />
Below is the improved version using Async:<br />
<pre class="brush:javascript; highlight: [8, 11, 14, 18, 19, 20, 21, 28, 29, 30, 31, 32, 33]">var async = require('async'),
mongoose = require('mongoose');
var MyModel = mongoose.model('MyModel');
async.parallel([
function (callback) {
MyModel.create({ field: 'value1' }, callback);
},
function (callback) {
MyModel.create({ field: 'value2' }, callback);
},
function (callback) {
MyModel.create({ field: 'value3' }, callback);
}
],
function (errs, results) {
if (errs) {
async.each(results, rollback, function () {
console.log('Rollback done.');
});
} else {
console.log('Done.');
}
});
function rollback (doc, callback) {
if (!doc) { callback(); }
else {
MyModel.findByIdAndRemove(doc._id, function (err, doc) {
console.log('Rolled-back document: ' + doc);
callback();
});
}
}
</pre>
Lines 8, 11, and 14 inserts a new document to MongoDB then passes either the <b>saved document</b> or an <b>error</b> to the callback.<br />
<br />
Lines 18-21 checks if any of the parallel functions threw an error, and calls the <span style="font-family: Courier New, Courier, monospace;">rollback()</span> function on each document passed to callback of <span style="font-family: Courier New, Courier, monospace;">parallel()</span>.<br />
<br />
Lines 28-33 performs the rollback by deleting the inserted documents using their IDs.<br />
<br />
<b>Edit: </b>As pointed-out in the comments, the rollback itself can fail. In such scenarios, it is best to retry the rollback until it succeeds. Therefore, the rollback logic must be an idempotent operation.jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com2tag:blogger.com,1999:blog-6063627024520487734.post-14704150256266878282014-07-10T14:24:00.000+08:002014-07-10T14:24:11.621+08:00Adding version to JavaScript and CSS Resources via JSP/JSTLBrowsers usually cache static resources like JavaScript and CSS so that downloads are reduced the next time the same website is visited.
<br />
<br />
However, if the JavaScript or CSS gets updated in the server, the browser will still be using an outdated copy of the resource. This could lead to unexpected behavior.
<br />
<br />
This can be addressed by including a version to the resource name and incrementing the value every time an update is released (ex: <span style="font-family: Courier New, Courier, monospace;">/my-styles-v2.css</span>). An easier alternative is to use a query parameter that indicates the resource version (ex: <span style="font-family: Courier New, Courier, monospace;">/my-styles.css?version=2</span>). On either solution, HTML pages that link to the resources need to be updated whenever a resource version changes -- this can be difficult to maintain especially if there are many resources that constantly change.<br />
<br />
There are tools that automate this process and are usually incorporated into the build process. The solution I'm going to demonstrate will address the issue without adding an extra build step.<br />
<br />
<i>Note: This example is specific to Java web applications using JSTL.</i><br />
<br />
The first step is to add an application-level configuration that holds the current version of the static resources. This comes as a context parameter in <span style="font-family: Courier New, Courier, monospace;"><b>web.xml</b></span>:<br />
<pre class="brush:xml; highlight: [6, 7]"><web-app>
...
<!-- Indicates the CSS and JS versions -->
<context-param>
<param-name>resourceVersion</param-name>
<param-value>1</param-value>
</context-param>
...
</web-app>
</pre>
Lines 6 and 7 indicate the value of the resource version and <b>should be incremented</b> before every release.<br />
<br />
The next step is to update the JSPs that link to the resources and format the resource URLs to include the version number.<br />
<pre class="brush:html; highlight:[2, 3, 4, 5, 6, 7, 13, 18]"><%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<c:url value="/resources/css/my-styles.css" var="myCss">
<c:param name="version" value="${initParam.resourceVersion}" />
</c:url>
<c:url value="/resources/js/my-scripts.js" var="myJavaScript">
<c:param name="version" value="${initParam.resourceVersion}" />
</c:url>
...
<!DOCTYPE html>
<html lang="en">
<head>
...
<link rel="stylesheet" href="${myCss}" type="text/css" />
...
</head>
<body>
...
<script src="${myJavaScript}"></script>
</body>
</html>
</pre>
Lines 2-7 declare URL variables that include "version" as a query parameter. Note that context parameters are accessible from JSTL via <span style="font-family: Courier New, Courier, monospace;">${initParam}</span>.<br />
<br />
Lines 13 and 18 uses the variables to render the URLs.jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-49069524720308273062014-06-26T00:39:00.000+08:002016-01-30T00:35:21.061+08:00Generating Unique and Readable IDs in Node.js Using MongoDBI had a requirement from an upcoming project to generate unique human-readable IDs. This project is written in Node.js and uses MongoDB for its database.<br />
<br />
Ideally, I can use an auto-incrementing Sequence to achieve this. However, unlike most relational databases, MongoDB does not support Sequences. Fortunately, it is not difficult to <a href="http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/#auto-increment-counters-collection" target="_blank">implement this behavior</a> in MongoDB.<br />
<br />
We will use a collection to store our sequences. The sequences will then be incremented using the <a href="http://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/#db.collection.findAndModify" target="_blank">findAndModify() </a>function. To ensure that sequences do not increment to unmanageable values, the counter must be restarted after a certain period. In my case, I will restart the counter everyday. To achieve this, I will identify each sequence using a prefix of <span style="font-family: Courier New, Courier, monospace;">YYMMDD</span>.<br />
<br />
Below is the raw MongoDB statement:<br />
<pre class="brush:javascript; highlight: [4, 5]">db.ids.findAndModify({
query: { prefix: 140625 },
update: { $inc: { count: 1 } },
upsert: true,
new: true
});
</pre>
It is important to set the <span style="font-family: Courier New, Courier, monospace;"><b>upsert</b></span> and <span style="font-family: Courier New, Courier, monospace;"><b>new</b></span> options - setting <span style="font-family: Courier New, Courier, monospace;">upsert</span> to true will insert a new document if the query cannot find a match; while setting <span style="font-family: Courier New, Courier, monospace;">new</span> to true will return the updated version of the document.<br />
<br />
Testing on the console yields the expected result.<br />
<pre class="brush:plain; highlight: [4, 10, 16, 22]">> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
"_id" : ObjectId("53aae1d126d57c198d861cfd"),
"count" : 1,
"prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
"_id" : ObjectId("53aae1d126d57c198d861cfd"),
"count" : 2,
"prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
"_id" : ObjectId("53aae1d126d57c198d861cfd"),
"count" : 3,
"prefix" : 140625
}
> db.ids.findAndModify({ query: { prefix: 140625 }, update: { $inc: { count: 1 } }, upsert: true, new: true });
{
"_id" : ObjectId("53aae1d126d57c198d861cfd"),
"count" : 4,
"prefix" : 140625
}
</pre>
<br />
<b>Node.js + Mongoose</b><br />
<br />
I use <a href="https://www.npmjs.org/package/mongoose" target="_blank">Mongoose</a> in Node.js as a MongoDB object document mapper (ODM). Mongoose offers an intuitive API to access MongoDB from within Node.js.<br />
<br />
To translate the above implementation, we first need to declare a Mongoose schema.<br />
<pre class="brush:javascript;">var mongoose = require('mongoose'),
Schema = mongoose.Schema;
var IdSchema = new Schema({
prefix: { type: Number, required: true, index: { unique: true } },
count: { type: Number, required: true }
});
mongoose.model('Id', IdSchema);
</pre>
The schema defines the structure of the document and as well as validation.<br />
<br />
Once the schema is declared, translation becomes pretty straightforward. Note that Mongoose does not have a function called <span style="font-family: Courier New, Courier, monospace;">findAndModify()</span>, instead, it offers 2 forms: <span style="font-family: Courier New, Courier, monospace;"><a href="http://mongoosejs.com/docs/api.html#model_Model.findByIdAndUpdate" target="_blank">findByIdAndUpdate()</a></span> and <span style="font-family: Courier New, Courier, monospace;"><a href="http://mongoosejs.com/docs/api.html#model_Model.findOneAndUpdate" target="_blank">findOneAndUpdate()</a></span>. In our case, we will use <span style="font-family: Courier New, Courier, monospace;">findOneAndUpdate()</span>.<br />
<pre class="brush:javascript; highlight: [5, 6, 7]">var moment = require('moment'),
Id = mongoose.model('Id');
var nextId = function (callback) {
function prefix (date) {
return parseInt(moment(date).format('YYMMDD'));
}
Id.findOneAndUpdate(
{ prefix: prefix(new Date()) },
{ $inc: { count: 1 } },
{ upsert: true },
function (err, idDoc) {
callback(err, idDoc);
});
};
</pre>
Lines 5-7 generates the prefix with the help of <a href="https://www.npmjs.org/package/moment">Moment.js</a>.<br />
<br />
Note that while <span style="font-family: Courier New, Courier, monospace;">findAndModify()</span> and its Mongoose equivalents are an atomic operations, there is still a chance that multiple clients try to upsert the same document and hence would fail due to constraint violation - in such scenarios, the call to <span style="font-family: Courier New, Courier, monospace;">nextId()</span> must be retried.<br />
<br />
<b>House Keeping</b><br />
<br />
Because a new document is inserted every time the counter is reset, the documents will accumulate overtime. Fortunately, because the prefixes are stored as numbers, removing old documents becomes very easy.<br />
<br />
For example, if we want to remove documents older than 2015, we just issue the below statement.<br />
<pre class="brush:javascript;">db.ids.remove({
prefix: { $lt: 150000 }
});
</pre>
The <span style="font-family: Courier New, Courier, monospace;">$lt</span> operator stands for "less than". The above statement roughly translates to: <span style="font-family: Courier New, Courier, monospace;">delete from ids where prefix < 15000</span>.jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-37173562165165366292014-06-24T01:04:00.000+08:002014-06-25T00:33:51.620+08:00Sublime Text Packages for Node.js/JavaScriptSince I started working on Node.js for an upcoming project, I've become a fan of <a href="http://www.sublimetext.com/" target="_blank">Sublime Text</a>. It is fast; extensible; has great community support; and most importantly, runs on Linux.<br />
<br />
So far, I use Sublime Text exclusively for Node.js and web development (HTML5, CSS, JavaScript). Below, I've compiled a list of packages I found most useful.<br />
<br />
As a prerequisite, <b>Package Control</b> needs to be installed. Installing Package Control is a matter of copy-pasting a code snippet to your Sublime Text console. Installation instruction is found <a href="https://sublime.wbond.net/installation" target="_blank">here</a>.<br />
<br />
Once Package Control is installed, you can start installing other packages by opening the <b>Command Pallete</b> (<span style="font-family: Courier New, Courier, monospace;">ctrl+shift+p</span>) and searching for "Install Package".<br />
<br />
Without further ado, below are the packages (<i>search for the text in bold</i>):<br />
<br />
<b>Alignment</b><br />
* Aligns various texts<br />
* Use via <span style="font-family: Courier New, Courier, monospace;">ctrl+alt-a</span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7sEg6QtD7K9ry9TbTZIBLosW8OIiCkFVhwppAUIo16qUUVu-EeHEqi2WuIUHRDNj6Dqubw5aKxA_UveNeCrLUbHHQpX9WiM2amJ-1y24oojocPVn2ub8i_Q4F_WFYfejJnc7h834uzfI/s1600/Selection_011.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7sEg6QtD7K9ry9TbTZIBLosW8OIiCkFVhwppAUIo16qUUVu-EeHEqi2WuIUHRDNj6Dqubw5aKxA_UveNeCrLUbHHQpX9WiM2amJ-1y24oojocPVn2ub8i_Q4F_WFYfejJnc7h834uzfI/s1600/Selection_011.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1a: Alignment (before)</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFCB3a6j7nB1qVNhT7qYuBGWFzJHUNpTJcim5abIWAC5Ni9zxGWQw7GnuyK0_9E04Pws3ZyJU9C0Npo_t9aRy_4urK0ZVXCr0_ysM4nLv4HOBXMiAlQz6qiMlt7Bww696LGM9wjxLtjcQ/s1600/Selection_012.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFCB3a6j7nB1qVNhT7qYuBGWFzJHUNpTJcim5abIWAC5Ni9zxGWQw7GnuyK0_9E04Pws3ZyJU9C0Npo_t9aRy_4urK0ZVXCr0_ysM4nLv4HOBXMiAlQz6qiMlt7Bww696LGM9wjxLtjcQ/s1600/Selection_012.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1b: Alignment (after)</td></tr>
</tbody></table>
<b>BracketHighlighter</b><br />
* Highlights brackets, braces, and parentheses.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9-EBITnHFGLDZAfSXfoP6YitBDsmGL379wjMr7lczKNz51x0PfVeJfBmgLmJ7axDYUFRkJm1nXRmCM6p3zpHZft9_hgtlnsL-F5h1bgEq6LkKv7enRST0lcmWDTtz2K6OOa-BofI46Sw/s1600/Selection_013.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9-EBITnHFGLDZAfSXfoP6YitBDsmGL379wjMr7lczKNz51x0PfVeJfBmgLmJ7axDYUFRkJm1nXRmCM6p3zpHZft9_hgtlnsL-F5h1bgEq6LkKv7enRST0lcmWDTtz2K6OOa-BofI46Sw/s1600/Selection_013.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 2: BracketHighlighter</td></tr>
</tbody></table>
<b>Emmet</b><br />
* Easily write HTML<br />
* Use via <span style="font-family: Courier New, Courier, monospace;">ctrl+alt-enter</span><br />
* For more information on Emmet, check-out this interactive <a href="http://scotch.io/bar-talk/write-html-crazy-fast-with-emmet-an-interactive-guide" target="_blank">guide</a>.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtcHiJxjKJEJ71EDJwkhNY8herOYQi-4M3HiEMgjB6eJVjP3VtauzHjJjUU4O6HvLT-UHbJMLg9f7sKKvtzn38rVkK1yxJJw9fzCDdE3lZTywSXxsG1bdPofS0CTr3ZijVOeowTbsCa0Q/s1600/Selection_014.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtcHiJxjKJEJ71EDJwkhNY8herOYQi-4M3HiEMgjB6eJVjP3VtauzHjJjUU4O6HvLT-UHbJMLg9f7sKKvtzn38rVkK1yxJJw9fzCDdE3lZTywSXxsG1bdPofS0CTr3ZijVOeowTbsCa0Q/s1600/Selection_014.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 3: Emmet</td></tr>
</tbody></table>
<b>SidebarEnhancements</b><br />
* Adds useful menu items to your sidebar<br />
* Only available for Sublime Text 3<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOHizzmC5LvWkeKcpOSitiRzjMQEGUjM8a-9RNLoZjDOforL-bqDvWxTJBrgp2TXRbfLRxqxlNQaIL_TJtH7-Eaw8eRilwqisJakZaIvdp_zWh9Yzqo74O_fdByZMiXxdmFPcsVIqljZY/s1600/687474703a2f2f646c2e64726f70626f782e636f6d2f752f34333539363434392f7469746f2f7375626c696d652f536964654261722f73637265656e73686f742e706e67.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOHizzmC5LvWkeKcpOSitiRzjMQEGUjM8a-9RNLoZjDOforL-bqDvWxTJBrgp2TXRbfLRxqxlNQaIL_TJtH7-Eaw8eRilwqisJakZaIvdp_zWh9Yzqo74O_fdByZMiXxdmFPcsVIqljZY/s1600/687474703a2f2f646c2e64726f70626f782e636f6d2f752f34333539363434392f7469746f2f7375626c696d652f536964654261722f73637265656e73686f742e706e67.png" height="320" width="255" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 4: SidebarEnhancements</td></tr>
</tbody></table>
<b>HTML-CSS-JS Prettify</b><br />
* Formats HTML, CSS, and JavaScript files<br />
* Use via <span style="font-family: Courier New, Courier, monospace;">ctrl+shift+h</span><br />
* Requires Node.js to be installed<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnGX4HGOKrOnrHdXZSQmWJQPLd5xx7ymmhZKydToCLKkoJZhBzTtxwDr3PEiZeAf2rfWU238-0yDRUCXE6miiCXBcAeJMNTEciw3kzjqAtWZ-t71S55fTddqZqE89WF97ol7UyVHhJWfc/s1600/Selection_015.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnGX4HGOKrOnrHdXZSQmWJQPLd5xx7ymmhZKydToCLKkoJZhBzTtxwDr3PEiZeAf2rfWU238-0yDRUCXE6miiCXBcAeJMNTEciw3kzjqAtWZ-t71S55fTddqZqE89WF97ol7UyVHhJWfc/s1600/Selection_015.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 5a: Prettify (before)</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqgGZE9rABn1Lf0uTOIuXTZEAHpMgL2q_bX2wgtv3Y6Ap4gCj1ZC1o_mn3m2yJmOueGV439GVK5Tv1VDPwozGqbL4RIU1fhPquI4b5aRngGlJGQS2mWIX9o4HEqBpJYbmfiikxZ1UPTvQ/s1600/Selection_016.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqgGZE9rABn1Lf0uTOIuXTZEAHpMgL2q_bX2wgtv3Y6Ap4gCj1ZC1o_mn3m2yJmOueGV439GVK5Tv1VDPwozGqbL4RIU1fhPquI4b5aRngGlJGQS2mWIX9o4HEqBpJYbmfiikxZ1UPTvQ/s1600/Selection_016.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 5b: Prettify (after)</td></tr>
</tbody></table>
<b>TrailingSpaces</b><br />
* Highlights trailing spaces<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkm69HRijbuEiVzUXKO88ook5uSN93DTPcT7UXItK2QVSPGRwhnGQ8VkmttSzEbMSgsMliQ_cCy7QwVTt6CJsPOr3MhAHel7e7e1fQ4Jyr8fU7bRXaF7dJo8bmgnr8UgVMFv_TPwZ1nU0/s1600/Selection_018.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkm69HRijbuEiVzUXKO88ook5uSN93DTPcT7UXItK2QVSPGRwhnGQ8VkmttSzEbMSgsMliQ_cCy7QwVTt6CJsPOr3MhAHel7e7e1fQ4Jyr8fU7bRXaF7dJo8bmgnr8UgVMFv_TPwZ1nU0/s1600/Selection_018.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 6: TrailingSpaces</td></tr>
</tbody></table>
<b>SublimeLinter</b><br />
* Highlights <a href="http://en.wikipedia.org/wiki/Lint_(software)" target="_blank">lint</a> errors for various file formats.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixKXkCkIPw77PF_1zuRpusZzyih_83pi5f9UnLfkH4Q3Be4r-_REUU2KqX0vMhpAXjufYs1aQYCgQHY2UiawHoLzTj4tnTRFzzhY9GTKnqBW5Je3OlyoZASsgbwGcMSdaqc1p1n1wGT1Q/s1600/Selection_017.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixKXkCkIPw77PF_1zuRpusZzyih_83pi5f9UnLfkH4Q3Be4r-_REUU2KqX0vMhpAXjufYs1aQYCgQHY2UiawHoLzTj4tnTRFzzhY9GTKnqBW5Je3OlyoZASsgbwGcMSdaqc1p1n1wGT1Q/s1600/Selection_017.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 7: SublimeLinter</td></tr>
</tbody></table>
<ul>
<li>For Sublime Text 3, each linter needs to be installed separately:</li>
<ul>
<li><b>SublimeLinter-jshint</b> (JavaScript)</li>
<ul>
<li>Requires <span style="font-family: Courier New, Courier, monospace;">jshint</span></li>
<li>install via "<span style="font-family: Courier New, Courier, monospace;">sudo npm install -g jshint</span>"</li>
</ul>
<li><b>SublimeLinter-html-tidy</b> (HTML)</li>
<ul>
<li>Requires <span style="font-family: Courier New, Courier, monospace;">tidy</span> </li>
<li>install via "<span style="font-family: Courier New, Courier, monospace;">sudo apt-get install tidy</span>"</li>
</ul>
</ul>
</ul>
jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-78023846554910868482013-08-06T01:18:00.000+08:002015-07-05T13:35:47.050+08:00Setting-up Tomcat SSL with StartSSL CertificatesPart of an effort to improve the security of <a href="http://checkthecrowd.com/" target="_blank">CheckTheCrowd.com</a> is to enable SSL on my web server. Enabling SSL allows it to support HTTPS connections.<br />
<br />
The CheckTheCrowd web application is hosted on Apache Tomcat which provides a pretty good, albeit generic <a href="http://tomcat.apache.org/tomcat-7.0-doc/ssl-howto.html" rel="nofollow" target="_blank">documentation</a> on how to achieve this setup.<br />
<br />
In summary, enabling SSL on Tomcat requires three things:<br />
<ul>
<li>Creating a Java keystore which contains the private key that Tomcat would use to start SSL handshakes</li>
<li>Ensuring that you or your website owns the private key by having it signed by a trusted authority which in turn, issues a digital certificate verifying your ownership of the key</li>
<li>Configuring a Tomcat connector to listen on HTTPS from a specified port</li>
</ul>
Creating a keystore and configuring a Tomcat connector is simple enough. However, acquiring an SSL certificate from a trusted provider can be expensive.<br />
<br />
Thankfully, I heard about <a href="http://www.startssl.com/" rel="nofollow" target="_blank">StartSSL</a> which provides <b>free SSL certificates</b> with one year validity (a new one can be generated upon expiry).<br />
<br />
Below are the steps I took to set-up Tomcat SSL using StartSSL certificates.<br />
<br />
<b>1. Creating the Java Keystore File (.jks)</b><br />
<br />
As per the Tomcat documentation, the first thing I needed to do was to generate a Java keystore which would hold my private key. This was done by using the <span style="font-family: "Courier New",Courier,monospace;">keytool</span> command that is part of JDK.<br />
<pre class="brush:plain;">keytool -genkey -keysize 2048 -keyalg RSA -sigalg SHA1withRSA \
-alias [name of server] -keystore [name of keystore].jks \
-keypass [password] -storepass [password] \
-dname "CN=[domain name], OU=Unknown, O=[website], L=[city], ST=[state], C=[country]"
</pre>
Note that due to a Tomcat limitation, the <span style="font-family: "Courier New",Courier,monospace;">keypass </span>and <span style="font-family: "Courier New",Courier,monospace;">storepass </span>must be the same. The <span style="font-family: "Courier New",Courier,monospace;">dname </span>entry is <i>optional</i>; if not provided, these details will be asked by <span style="font-family: "Courier New",Courier,monospace;">keytool </span>during the process.<br />
<br />
Example: <br />
<pre class="brush:plain;">keytool -genkey -keysize 2048 -keyalg RSA -sigalg SHA1withRSA \
-alias webserver -keystore checkthecrowd.jks \
-keypass ****** -storepass ****** \
-dname "CN=checkthecrowd.com, OU=Unknown, O=CheckTheCrowd, L=Singapore, ST=Unknown, C=SG"
</pre>
At this point, my keystore already contains the private key required by Tomcat to start an SSL connection.<br />
<br />
I can already start using this keystore to enable SSL in Tomcat,
but a rogue entity can hijack the connection and pretend that his private key was issued by CheckTheCrowd. This rogue entity can then trick my users that they are securely connected to CheckTheCrowd when in fact they are connected to something else.<br />
<br />
To solve this, I need to acquire a signed certificate to prove that my private key is associated to my domain (checkthecrowd.com).<br />
<br />
<b>2. Creating a Certificate Request File (.csr)</b><br />
<br />
A certificate request is submitted to a certificate provider and an SSL certificate is generated based on this file.<br />
<pre class="brush:plain">keytool -certreq -alias [name of server] -file [name of request].csr \
-keystore [name of keystore].jks
</pre>
Note that this command would ask for the password previously set for the keystore.<br />
<br />
Example:<br />
<pre class="brush:plain;">keytool -certreq -alias webserver -file checkthecrowd.csr \
-keystore checkthecrowd.jks
</pre>
<br />
<b>3. Submitting the Certificate Request to StartSSL</b><br />
<br />
I needed to signup for an account in order to use StartSSL. Signing-up involves generating a signed private key which proves my identity. Here onwards, the key is used by StartSSL to authenticate my access to their website.<br />
<br />
<i>Note that it is important to keep a back-up copy of this private key for future use. This file needs to be imported on all computers used to access StartSSL.</i><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjld5WE2V8YCZVA_vJu_EEpxD8WDa76Ygl75iSNBjIsuNDmCa7b-mLvsZiChCrjsqifPJFZggNbL7_MvaOCBMEmqrTyBCvp9PJKX-S-J-454HB27cMWnszNOmdcn67TzzD9ojGIy5Byx7o/s1600/startssl.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="265" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjld5WE2V8YCZVA_vJu_EEpxD8WDa76Ygl75iSNBjIsuNDmCa7b-mLvsZiChCrjsqifPJFZggNbL7_MvaOCBMEmqrTyBCvp9PJKX-S-J-454HB27cMWnszNOmdcn67TzzD9ojGIy5Byx7o/s320/startssl.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1: StartSSL</td></tr>
</tbody></table>
Once I have an account, I can use the <i>Control Panel</i> to generate my certificate. The first step is to validate that I own the domain checkthecrowd.com. The aptly named <i>Validation Wizard</i> takes care of this.<br />
<br />
Once my domain is validated, I used the <i>Certificates Wizard</i> to submit my certificate request (.csr file):<br />
<ul>
<li>Select <i>Web Server SSL/TLS Certificate</i>.</li>
<li>Because I already have a private key and a certificate request, I <b>skip </b>the next screen</li>
<li>I pasted the contents of my certificate request (.csr file) to the text area provided</li>
<li>When finished, the generated certificate is displayed on another text area -- I copied this and saved to a file called <span style="font-family: "Courier New",Courier,monospace;">ssl.crt</span>.</li>
</ul>
<b>4. Import the Generated Certificate and StartSSL Certificate Chains</b><br />
<br />
The next step is import the generated certificate to my keystore. The StartSSL certificate chain is also required to be imported.<br />
<br />
The StartSSL certificate chain can be downloaded from:<br />
<ul>
<li><a href="http://www.startssl.com/certs/ca.crt">http://www.startssl.com/certs/ca.crt</a></li>
<li><a href="http://www.startssl.com/certs/sub.class1.client.ca.crt">http://www.startssl.com/certs/sub.class1.client.ca.crt</a></li>
</ul>
<i>The free SSL certificate from StartSSL is only a Class 1 level certificate. If using an upgraded package, all applicable class certificates should be used. </i><i>Other formats like .pem or .cer can also be used, see <a href="http://www.startssl.com/certs/">http://www.startssl.com/certs/</a>.</i><br />
<br />
I again used <span style="font-family: "Courier New",Courier,monospace;">keytool </span>to import these certificates:<br />
<pre class="brush:plain;">keytool -import -alias [ca alias] -file [ca file].crt \
-keystore [keystore name].jks -trustcacerts
keytool -import -alias [class1 alias] -file [class1 file].crt \
-keystore [keystore name].jks -trustcacerts
keytool -import -alias [name of server] -file ssl.crt \
-keystore [keystore name].jks
</pre>
The first two commands imported the certificate chain as trusted certificates, the last command imported the signed certificate.<br />
<br />
Example:<br />
<pre class="brush:plain;">keytool -import -alias startsslca -file ca.crt \
-keystore checkthecrowd.jks -trustcacerts
keytool -import -alias startsslca1 -file sub.class1.server.ca.crt \
-keystore checkthecrowd.jks -trustcacerts
keytool -import -alias webserver -file ssl.crt \
-keystore checkthecrowd.jks
</pre>
Listing the contents of my keystore verified that I have 3 certificates:<br />
<pre class="brush:plain;">$ keytool -list -keystore checkthecrowd.jks
webserver, Aug 5, 2013, PrivateKeyEntry,
Certificate fingerprint (SHA1): [...]
startsslca, Aug 5, 2013, trustedCertEntry,
Certificate fingerprint (SHA1): [...]
startsslca1, Aug 5, 2013, trustedCertEntry,
Certificate fingerprint (SHA1): [...]
</pre>
<br />
<b>5. Configure Tomcat with SSL</b><br />
<br />
Enabling SSL with Tomcat involves creating a new connector which listens to HTTPS connections. This connector needs to know the location of the keystore file as well as the password to access the keystore.<br />
<br />
For convenience, I placed my keystore under <span style="font-family: Courier New, Courier, monospace;">$TOMCAT_HOME</span>.<br />
<pre class="brush:xml;"><!--
Define a SSL HTTP/1.1 Connector on port 8443
This connector uses the JSSE configuration, when using APR, the
connector should be using the OpenSSL style configuration
described in the APR documentation
-->
<Connector
protocol="HTTP/1.1"
port="8443" maxThreads="200"
scheme="https" secure="true" SSLEnabled="true"
keystoreFile="checkthecrowd.jks" keystorePass="******"
clientAuth="false" sslProtocol="TLS"/>
</pre>
Note that by default, the Tomcat HTTPS port is 8443.<br />
<br />
<i>That's all there is to it! After bouncing Tomcat, I am now able to access CheckTheCrowd via HTTPS from port 8443: <a href="https://checkthecrowd.com:8443/" rel="nofollow" target="_blank">https://checkthecrowd.com:8443/</a>.</i><br />
<br />
<i>The next step is to configure Apache <span style="font-family: "Courier New",Courier,monospace;">httpd</span> to forward HTTPS requests to port 8443. I still haven't figured out how to do this yet, so if you have an idea, let me know!</i>jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com7tag:blogger.com,1999:blog-6063627024520487734.post-20705163167309560902013-05-24T04:12:00.000+08:002014-06-25T11:48:23.406+08:00Flowee: Sample Application<i>My previous <a href="http://www.jramoyo.com/2013/05/introducing-flowee-framework-for.html" target="_blank">post </a>introduced Flowee as a framework for building Java services backed by one or more workflows. Through a sample application, this post will demonstrate how easy it is to build workflow-based services using Flowee.</i><br />
<br />
The sample application is a service which authenticates two types of accounts: an admin and a user. The service will display a greeting then authenticate each type of account using different authentication methods.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEim3AW5-9f-b9AAFpV_bnlkJmqivu_XGraOg7JRrB45W6NH3nTNWksjBYP0fF3m0Jl7r7_AvyXQaZuhjyQMta3R-UrFNGJpaNOB5ocp1qu9VIk-v8p50l5PMztfYaehmngpWUs4vmGmoK4/s1600/sample.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEim3AW5-9f-b9AAFpV_bnlkJmqivu_XGraOg7JRrB45W6NH3nTNWksjBYP0fF3m0Jl7r7_AvyXQaZuhjyQMta3R-UrFNGJpaNOB5ocp1qu9VIk-v8p50l5PMztfYaehmngpWUs4vmGmoK4/s1600/sample.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1: Sample Workflow Service</td></tr>
</tbody></table>
<i>Note that this example is by no means a realistic use-case for production, it is only used here for illustration purposes.</i><br />
<br />
<b>Implementing the Service</b><br />
<br />
(1) We start by defining the request:<br />
<pre class="brush:java;">
public class LoginRequest {
private String username;
private String password;
private String type;
// Accessors omitted
}
</pre>
<br />
(2) We also create an application specific <span style="font-family: "Courier New",Courier,monospace;">WorkflowContext </span>(this is an optional step, the default workflow context is probably sufficient for some applications):<br />
<pre class="brush:java;">
public class LoginContext extends WorkflowContext {
private static final String KEY_IS_AUTHENTICATED = "isAuthenticated";
public void setIsAuthenticated(Boolean isAuthenticated) {
put(KEY_IS_AUTHENTICATED, isAuthenticated);
}
public Boolean getIsAuthenticated() {
return (Boolean) get(KEY_IS_AUTHENTICATED);
}
}
</pre>
Here, we extend the default context with convenience methods to access the value mapped to the <i>"isAuthenticated"</i> key (return values are normally stored in the context).<br />
<br />
(3) Then we define an application specific <span style="font-family: "Courier New",Courier,monospace;">Task </span>interface to remove generic types:
<br />
<pre class="brush:java;">
public interface LoginTask
extends Task<LoginRequest, LoginContext> {
}
</pre>
<br />
(4) Next, we define an application specific abstract <span style="font-family: "Courier New",Courier,monospace;">Task</span>. We make it <span style="font-family: "Courier New",Courier,monospace;">BeanNameAware </span>so that the tasks assumes the bean name if declared in Spring:<br />
<pre class="brush:java;">
public abstract class AbstractLoginTask
extends AbstractTask<LoginRequest, LoginContext>
implements LoginTask, BeanNameAware {
@Override
public void setBeanName(String name) {
setName(name);
}
}
</pre>
<br />
(5) We can now create an application specific <span style="font-family: "Courier New",Courier,monospace;">Workflow</span>:<br />
<pre class="brush:java;">
public class LoginWorkflow
extends AbstractWorkflow<LoginTask, LoginRequest, LoginContext> {
public LoginWorkflow(String name) {
super(name);
}
}
</pre>
The abstract implementation should provide all the functionality we need, this step is to simply to declare the generic parameters.<br />
<br />
(6) Next, we define a <span style="font-family: "Courier New",Courier,monospace;">WorkflowFactory</span>. For this example, we will make our workflow factory configurable from properties files. To achieve this, we need to inherit from <span style="font-family: "Courier New",Courier,monospace;">AbstractPropertiesWorkflowFactory</span>:
<br />
<pre class="brush:java;">
public class LoginWorkflowFactory
extends AbstractPropertiesWorkflowFactory <LoginWorkflow, LoginTask, LoginRequest, LoginContext> {
@Override
protected LoginWorkflow createWorkflow(String name) {
return new LoginWorkflow(name);
}
}
</pre>
This requires us override <span style="font-family: "Courier New",Courier,monospace;">createWorkflow()</span>, an <i>Abstract Factory</i> method.<br />
<br />
(7) We then define the <span style="font-family: "Courier New",Courier,monospace;">Filter </span>that will be used by our configurable workflow factory. Recall from my previous post that <i>configurable </i>factories uses filters to evaluate conditions that determine which workflows gets created.<br />
<br />
Flowee comes with an abstract implementation that evaluates conditions as <a href="http://commons.apache.org/proper/commons-jexl/reference/syntax.html" rel="nofollow" target="_blank">JEXL expressions</a>. Using JEXL allows us to define JavaScript-like conditions for our workflow configuration:<br />
<pre class="brush:java; highlight: [7]">
public class LoginFilter
extends AbstractJexlFilter<LoginRequest, LoginContext> {
@Override
protected ReadonlyContext populateJexlContext(LoginRequest request,
LoginContext context) {
JexlContext jexlContext = new MapContext();
jexlContext.set("request", request);
return new ReadonlyContext(jexlContext);
}
}
</pre>
The method, <span style="font-family: "Courier New",Courier,monospace;">populateJexlContext()</span> populates a JEXL context with the <span style="font-family: "Courier New",Courier,monospace;">LoginRequest.</span> This allows us to access fields and methods of the request using JEXL expressions (ex:<span style="font-family: "Courier New",Courier,monospace;"> request.type == 'admin'</span> ).<br />
<br />
(8) We now have everything we need to define the <span style="font-family: "Courier New",Courier,monospace;">WorkflowService</span>:
<br />
<pre class="brush:java;">
public class LoginService
extends AbstractWorkflowService<LoginWorkflow, LoginTask, LoginRequest, LoginContext> {
@Override
protected LoginContext createContext() {
return new LoginContext();
}
}
</pre>
Here, we override an <i>Abstract Factory</i> method for creating an instance of the <span style="font-family: "Courier New",Courier,monospace;">LoginContext</span>.<br />
<br />
<b>Implementing the Tasks</b><br />
<br />
Now that we have the infrastructure for our workflow service, the next stage is to define the actual tasks that comprise the workflows.<br />
<br />
(1) We create a simple task that greets the user being authenticated:
<br />
<pre class="brush:java;">
public class GreetUserTask extends AbstractLoginTask {
@Override
protected TaskStatus attemptExecute(LoginRequest request,
LoginContext context) throws WorkflowException {
System.out.println(String.format("Welcome '%s'!",
request.getUsername()));
return TaskStatus.CONTINUE;
}
}
</pre>
<br />
(2) We then define the task which authenticates admin accounts
<br />
<pre class="brush:java;">
public class AuthenticateAdmin extends AbstractLoginTask {
@Override
protected TaskStatus attemptExecute(LoginRequest request, LoginContext context) throws WorkflowException {
if ("admin".equals(request.getUsername()) && "p@ssw0rd".equals(request.getPassword())) {
System.out.println(String.format("User '%s' has been authenticated as Administrator",
request.getUsername()));
context.setIsAuthenticated(Boolean.TRUE);
return TaskStatus.CONTINUE;
} else {
System.err.println(String.format("Cannot authenticate user '%s'!",
request.getUsername()));
context.setIsAuthenticated(Boolean.FALSE);
return TaskStatus.BREAK;
}
}
}
</pre>
Normally, this task should perform authentication against a data source. For this example, we are only trying to simulate a scenario where admin and user accounts are authenticated differently.<br />
<br />
(3) We then define the task which authenticates user accounts
<br />
<pre class="brush:java;">
public class AuthenticateUser extends AbstractLoginTask {
@Override
protected TaskStatus attemptExecute(LoginRequest request, LoginContext context)
throws WorkflowException {
if ("user".equals(request.getUsername())
&& "p@ssw0rd".equals(request.getPassword())) {
System.out.println(String.format("User '%s' has been authenticated",
request.getUsername()));
context.setIsAuthenticated(Boolean.TRUE);
return TaskStatus.CONTINUE;
} else {
System.err.println(String.format("Cannot authenticate user '%s'!",
request.getUsername()));
context.setIsAuthenticated(Boolean.FALSE);
return TaskStatus.BREAK;
}
}
}
</pre>
<br />
<b>Spring Integration</b><br />
<br />
We now have all the components we need to build the application. It's time to wire them all in Spring.<br />
<br />
(1) We start with the tasks. It is a good practice to provide a separate configuration file for tasks, this makes our configuration manageable in case the number of tasks grows.
<br />
<pre class="brush:xml;">
<beans>
<bean id="greet" class="com.jramoyo.flowee.sample.login.task.GreetUserTask" />
<bean id="authenticate_user" class="com.jramoyo.flowee.sample.login.task.AuthenticateUser" />
<bean id="authenticate_admin" class="com.jramoyo.flowee.sample.login.task.AuthenticateAdmin" />
</beans>
</pre>
Note that we are not wiring these tasks to any of our components. Flowee Spring Module (<span style="font-family: "Courier New",Courier,monospace;">flowee-spring</span>) comes with an implementation of <span style="font-family: "Courier New",Courier,monospace;">TaskRegistry </span>that looks-up task instances from the Spring context.<br />
<br />
(2) We then define our main Spring configuration file.
<br />
<pre class="brush:xml;">
<beans>
<import resource="classpath:spring-tasks.xml" />
<bean id="workflowFactory" class="com.jramoyo.flowee.sample.login.LoginWorkflowFactory">
<property name="filter">
<bean class="com.jramoyo.flowee.sample.login.LoginFilter" />
</property>
<property name="taskRegistry">
<bean class="com.jramoyo.flowee.spring.ContextAwareTaskRegistry" />
</property>
</bean>
<bean id="workflowService" class="com.jramoyo.flowee.sample.login.LoginService">
<property name="factory" ref="workflowFactory" />
</bean>
</beans>
</pre>
Here, we declare our <span style="font-family: "Courier New",Courier,monospace;">LoginFactory </span>as a dependency of <span style="font-family: "Courier New",Courier,monospace;">LoginService</span>. <span style="font-family: "Courier New",Courier,monospace;">LoginFactory </span>is then wired with LoginFilter and <span style="font-family: "Courier New",Courier,monospace;">ContextAwareTaskRegistry</span>. As mentioned in the previous step, <span style="font-family: "Courier New",Courier,monospace;">ContextAwareTaskRegistry </span>allows our factory to look-up task instances from the Spring context.<br />
<br />
<b>Workflow Configuration</b>
<br />
<br />
The last stage is to configure the <span style="font-family: "Courier New",Courier,monospace;">WorkflowFactory </span>to assemble the workflows required for an account type.
<br />
<br />
Recall from the previous steps that we using an instance of <span style="font-family: Courier New, Courier, monospace;">AbstractPropertiesWorkflowFactory.</span> This loads configuration from two properties files: one for the workflow conditions (<span style="font-family: Courier New, Courier, monospace;">workflow.properties</span>) and another for the workflow tasks (<span style="font-family: Courier New, Courier, monospace;">tasks.properties</span>).<br />
<br />
(1) We create the <span style="font-family: Courier New, Courier, monospace;">workflow.properties</span> file with the following content:<br />
<pre class="brush:plain;">
admin=request.type == 'admin'
user=request.type == 'user'
</pre>
This configuration means that if the value of <span style="font-family: Courier New, Courier, monospace;">LoginRequest.getType()</span> is <span style="font-family: Courier New, Courier, monospace;">'admin'</span>, execute the workflow named 'admin' and if the value is <span style="font-family: Courier New, Courier, monospace;">'user'</span> execute the workflow named 'user'<br />
<br />
(2) Then, we create the <span style="font-family: Courier New, Courier, monospace;">task.properties</span> file:<br />
<pre class="brush:plain;">
admin=greet,authenticate_admin
user=greet,authenticate_user
</pre>
This configures the sequence of tasks that make up the workflow.<br />
<br />
By default, both <span style="font-family: Courier New, Courier, monospace;">workflow.properties</span> and <span style="font-family: Courier New, Courier, monospace;">task.properties</span> are loaded from the classpath. This can be overridden by setting <span style="font-family: Courier New, Courier, monospace;">setWorkflowConfigFile()</span> and <span style="font-family: Courier New, Courier, monospace;">setTaskConfigFile() </span>respectively.<br />
<br />
That's all there is to it! From Spring, we can load <span style="font-family: Courier New, Courier, monospace;">LoginService </span>and inject it anywhere within the application.<br />
<br />
<i>Notice that while we created several components to build our service, most of them were simple inheritance and required very little lines of code. Also, we only need to build these component once per service.</i><br />
<br />
<i>You'll find that the value of Flowee becomes more evident as the number of workflows and tasks increases.</i><br />
<br />
<b>Testing</b><br />
<br />
We create a simple JUnit test case:<br />
<pre class="brush:java;">
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration("classpath:/spring.xml")
public class LoginServiceTest {
@Resource(name = "workflowService")
private LoginService service;
@Test
public void testAdminLogin() throws WorkflowException {
LoginRequest request = new LoginRequest();
request.setUsername("admin");
request.setPassword("p@ssw0rd");
request.setType("admin");
LoginContext context = service.process(request);
assertTrue("Incorrect result",
context.getIsAuthenticated());
request.setPassword("wrong");
context = service.process(request);
assertFalse("Incorrect result",
context.getIsAuthenticated());
}
@Test
public void testUserLogin() throws WorkflowException {
LoginRequest request = new LoginRequest();
request.setUsername("user");
request.setPassword("p@ssw0rd");
request.setType("user");
LoginContext context = service.process(request);
assertTrue("Incorrect result",
context.getIsAuthenticated());
request.setPassword("wrong");
context = service.process(request);
assertFalse("Incorrect result",
context.getIsAuthenticated());
}
}
</pre>
Notice how we injected <span style="font-family: Courier New, Courier, monospace;">LoginService </span>to our unit test.<br />
<br />
Our test yields the following output:
<br />
<pre class="brush:plain;">
Welcome 'admin'!
User 'admin' has been authenticated as Administrator
Welcome 'admin'!
Cannot authenticate user 'admin'!
Welcome 'user'!
User 'user' has been authenticated
Welcome 'user'!
Cannot authenticate user 'user'!
</pre>
<br />
<i>The code used in this example is available as a Maven module from Flowee's Git <a href="https://github.com/jramoyo/flowee/tree/master/flowee-sample" target="_blank">repository</a>.</i><br />
<br />
<i>So far, I've only demonstrated simple workflows with linear task sequences. My next post will introduce special tasks which allows for more complex task sequences.</i>jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-1280675787497102962013-05-24T04:08:00.000+08:002014-02-26T11:15:44.293+08:00Introducing Flowee, a Framework for Building Workflow-based Services in Java<b>Overview</b><br />
<br />
My past roles required me to write three different applications having surprisingly similar requirements. These requirements were:<br />
<ol>
<li>The application must run as a service which receives some form of request</li>
<li>Once received, several stages of processing needs to be performed on the request (i.e. validation, persistence, state management, etc)</li>
<li>These stages of processing may change overtime as new requirements come in</li>
</ol>
The third requirement is arguably the most important. While the first two can be performed without special techniques or design, the last requirement requires a bit of planning.<br />
<br />
In order to achieve this required flexibility, adding new processing logic or modifying existing ones should not affect the code of other tasks. For example: if I add a new logic for validation, my existing code for state management should not be affected.<br />
<br />
I need to encapsulate each logic as individual units of "tasks". This way, changes to each task is independent from the other.<br />
<br />
The behavior of my service is now defined by the sequence of tasks needed to be performed. This behavior can easily be changed by adding, removing, or modifying tasks in the sequence.<br />
<br />
Essentially, I needed to build a workflow.<br />
<br />
<b>Design</b><br />
<br />
My design had several iterations throughout the years. My initial design was pretty straightforward: I implemented a workflow as a container of one or more tasks arranged in a sequence. When executed, the workflow iterates through each task and executes it. The workflow has no knowledge of what each task would do, all it knows is that it is executing some task. This was made possible because each task share a common interface. This interface exposes an <span style="font-family: "Courier New",Courier,monospace;">execute()</span> method which accepts the request as a parameter. <br />
<br />
Through dependency injection, the behavior of the workflow becomes very easy to change. I can add, remove and rearrange tasks via configuration.<br />
<br />
While proven to be effective, my initial design was application specific -- the workflow can only accept a specific type of request. This makes it an ineffective framework because it only works for that particular application. There was also the problem of tasks not being able to share information: as a result, temporary values needed to be stored within the request itself.<br />
<br />
I had the chance to improve on this design on a later project. By applying <i>Generics</i>, I was able to redesign my workflow so that it can accept any type of request.<br />
<br />
In order for information to be shared across each tasks, a data structure serving as the workflow "context" is also passed as a parameter to each task.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDsG4H31HkRLsjABFR_Q-1QNiFXuUVaLGkON29HaOZi_G7j8oKyvhcnqVgiVh_T8jLU-TvmXikq5qxDiwd2dxGabh0UlekA5bvLDEfX5A50xerNjtuuF6UNOF2vPUlhx3JiOos4IVyXBc/s1600/workflow-h.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDsG4H31HkRLsjABFR_Q-1QNiFXuUVaLGkON29HaOZi_G7j8oKyvhcnqVgiVh_T8jLU-TvmXikq5qxDiwd2dxGabh0UlekA5bvLDEfX5A50xerNjtuuF6UNOF2vPUlhx3JiOos4IVyXBc/s1600/workflow-h.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1: Workflow and Tasks</td></tr>
</tbody></table>
<br />
More often than not, different workflows need to be executed for different types of request. For example, one workflow needs to be executed to open an account, another workflow to close an account, etc.<br />
<br />
I came up with a "workflow factory" as a solution to this requirement. Depending on the type of request, the factory will assemble the workflows required to be executed against the request. The factory is exposed as an interface so that I can have different implementations depending on the requirement.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZiRaXMvgmmR9eMa5O8DOEWGPD_xr0K8A_p2B5gIBD06MPg6QJttE5-f67btJfttpTuUiLbc1J9obZ_ro2MFPDEwdaFRaaVkponYuf7F6A9PbAQj3Iv8RXb4ze0OCfRLMai748KCtiN2I/s1600/workflow-f.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZiRaXMvgmmR9eMa5O8DOEWGPD_xr0K8A_p2B5gIBD06MPg6QJttE5-f67btJfttpTuUiLbc1J9obZ_ro2MFPDEwdaFRaaVkponYuf7F6A9PbAQj3Iv8RXb4ze0OCfRLMai748KCtiN2I/s1600/workflow-f.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 2: Workflow Factory</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<i></i></div>
<br />
My service now becomes very simple: as soon as a request comes in, I will call the factory to assemble the required workflows then one-by-one execute my request against each of them.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1XPUl8oZ3pNP_Pmtj2LQHCxhIIIPv1RgxaDSFaX0CznS9NGIV6o6gy8eMZNUe-x3EDqrX7yzv7am0Eo07Xuqd5_vYbcxN1gLsvdEhID82Q-FIHURxo-azk2a0j4pUFT4Sfho-ubbnZEQ/s1600/service.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1XPUl8oZ3pNP_Pmtj2LQHCxhIIIPv1RgxaDSFaX0CznS9NGIV6o6gy8eMZNUe-x3EDqrX7yzv7am0Eo07Xuqd5_vYbcxN1gLsvdEhID82Q-FIHURxo-azk2a0j4pUFT4Sfho-ubbnZEQ/s1600/service.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 3: Workflow Service</td></tr>
</tbody></table>
<br />
<b>Flowee Framework</b><br />
<br />
Flowee is an open source implementation of the above design. Having gone through several iterations, I feel that this design has matured enough to warrant a common framework that can be useful to others.<br />
<br />
The project is hosted via GitHub at <a href="https://github.com/jramoyo/flowee">https://github.com/jramoyo/flowee</a>.<br />
<br />
<b>Workflow and Tasks</b><br />
<br />
The core framework revolves around the <span style="font-family: "Courier New",Courier,monospace;">Workflow </span>interface and its relationship with the <span style="font-family: "Courier New",Courier,monospace;">Task</span> interface.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiL4KxQV11HyGn8s_avuqitdcqcrieVaGZQpszY2QMLq1G-qTVvItG3cFB9q5YsdF2DEfSMP_kODOvl-FbGE0iCIbgECqyeeoPlE9TgRdx-iUuQGfznV4Lwfxd0OSX0GTqCbs9Dpgn8ZgI/s1600/workflow.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiL4KxQV11HyGn8s_avuqitdcqcrieVaGZQpszY2QMLq1G-qTVvItG3cFB9q5YsdF2DEfSMP_kODOvl-FbGE0iCIbgECqyeeoPlE9TgRdx-iUuQGfznV4Lwfxd0OSX0GTqCbs9Dpgn8ZgI/s1600/workflow.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 4: Flowee Workflow</td></tr>
</tbody></table>
<br />
The <span style="font-family: "Courier New",Courier,monospace;">execute()</span> method of <span style="font-family: "Courier New",Courier,monospace;">Workflow </span>accepts both the request and an instance of <span style="font-family: "Courier New",Courier,monospace;">WorkflowContext</span>. Errors encountered while processing the request are thrown as <span style="font-family: "Courier New",Courier,monospace;">WorkflowException</span>s.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">AbstractWorkflow </span>provides a generic implementation of <span style="font-family: "Courier New",Courier,monospace;">Workflow</span>. It iterates through a list of associated <span style="font-family: "Courier New",Courier,monospace;">Task</span>s to process a request. Application specific workflows will inherit from this class.<br />
<br />
Most workflow <span style="font-family: "Courier New",Courier,monospace;">Task</span>s will inherit from <span style="font-family: "Courier New",Courier,monospace;">AbstractTask</span>. It provides useful features for building application specific tasks namely:<br />
<ul>
<li>Retry task execution when an exception is encountered</li>
<li>Silently skip task execution instead of throwing an exception</li>
<li>Skip task execution depending on the type of request</li>
</ul>
<br />
<b>WorkflowFactory</b> <br />
<br />
The <span style="font-family: "Courier New",Courier,monospace;">WorkflowFactory </span>is another important piece of the framework. The way it abstracts the selection and creation of workflows simplifies the rest of the core components.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfKaiBNxYc_0G1D2-MKSpeAufANDta_o1v8MbCilMt-5Se8bW6HakyPYWRmDuaHdmyj2p5X65b08MBzzD79JLx_EVj-SeepJ3NNjv_0SdQk4M2pyrMt9xI4XrwnApPK1IyfG9ipmbbtuU/s1600/factory.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfKaiBNxYc_0G1D2-MKSpeAufANDta_o1v8MbCilMt-5Se8bW6HakyPYWRmDuaHdmyj2p5X65b08MBzzD79JLx_EVj-SeepJ3NNjv_0SdQk4M2pyrMt9xI4XrwnApPK1IyfG9ipmbbtuU/s1600/factory.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 5: Flowee WorkflowFactory</td></tr>
</tbody></table>
<br />
<span style="font-family: "Courier New",Courier,monospace;">AbstractConfigurableWorkflowFactory </span>is used to build configuration driven workflow factories. It defines an abstract <span style="font-family: "Courier New",Courier,monospace;">fetchConfiguration()</span> method and an abstract <span style="font-family: Courier New, Courier, monospace;">fetchTaskNames()</span> method that sub-classes need to implement. These methods are used to fetch configurations from various sources: either from the file system or from a remote server.<br />
<br />
The configuration is represented as a Map whose <b>key </b>is the name of a workflow and whose <b>value<span style="white-space: pre;"> </span></b>is the condition which activates that workflow.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">AbstractConfigurableWorkflowFactory </span>uses a <span style="font-family: Courier New, Courier, monospace;">Filter </span>instance to evaluate the conditions configured to activate the workflows.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">AbstractPropertiesWorkflowFactory </span>is a sub-class of <span style="font-family: "Courier New",Courier,monospace;">AbstractConfigurableWorkflowFactory </span> that fetches configuration from a properties file.<br />
<br />
<b>WorkflowService</b><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">WorkflowService</span> and <span style="font-family: "Courier New",Courier,monospace;">AbstractWorkflowService </span>acts as a facade linking the core components together.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx8lju4nNM_hXuPFpYnLQqHMcN7OUN-2UVja4Ar01qwUCyqTdsxInw6t8qf0rxsvSzmcK0GOsWhZig0Stpp01Bz2hZIDg3k7bIOxPWVNi1dWyzSrjn5NVjHgckvPc-0PWEDZibFiHvweM/s1600/service.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx8lju4nNM_hXuPFpYnLQqHMcN7OUN-2UVja4Ar01qwUCyqTdsxInw6t8qf0rxsvSzmcK0GOsWhZig0Stpp01Bz2hZIDg3k7bIOxPWVNi1dWyzSrjn5NVjHgckvPc-0PWEDZibFiHvweM/s1600/service.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 6: Flowee WorkflowService</td></tr>
</tbody></table>
With all the complexities taken care of by both the <span style="font-family: "Courier New",Courier,monospace;">Workflow </span>and the <span style="font-family: "Courier New",Courier,monospace;">WorkflowFactory</span>, our <span style="font-family: "Courier New",Courier,monospace;">WorkflowService </span>implementation becomes very simple.<br />
<br />
While most applications will interact with workflows through the <span style="font-family: "Courier New",Courier,monospace;">WorkflowService</span>, those requiring a different behavior can interact with the underlying components directly.<br />
<br />
<b>Conclusion</b><br />
<br />
The primary purpose of Flowee is to provide groundwork for rule-driven workflow selection and execution. Developers can focus the majority of their efforts on building the tasks which hold the actual business requirements.<br />
<br />
Workflows
built on Flowee will run without the need for "containers" or
"engines". The framework is lightweight and integrates seamlessly with
any application.<br />
<br />
<i>This post discussed the design considerations which led to the implementation of Flowee. It also described the structure of the core framework. My next <a href="http://www.jramoyo.com/2013/05/flowee-sample-application.html" target="_blank">post </a>will demonstrate how easy it is to build workflow-based services with Flowee by going through a sample application.</i>jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-18456216453849936352013-03-09T23:24:00.001+08:002014-06-25T11:24:36.775+08:00Static Methods and Unit Testing<b>Overview</b><br />
<br />
We all know that Interfaces allow us to write loosely-coupled components, and that this "loose-coupledness" comes-in handy during unit testing. Because we can separate the implementation from the method signatures, Interfaces allow us to mock implementations of our dependencies.<br />
<br />
Mocking dependencies is important during unit testing because it allows us to isolate the components we are testing from their dependencies -- this means that incorrect behavior from any dependency will not affect the results of our tests.<br />
<br />
Consider the following class:<br />
<pre class="brush:java;">public class Component {
private Dependency dependency
public Component(Dependency dependency) {
this.dependency = dependency
}
public void componentMethod() {
int imporantValue = dependency.getImportantValue();
// use 'imporantValue' to calculate stuff
}
}
</pre>
And the following Interface:<br />
<pre class="brush:java;">public interface Dependency {
int getImportantValue();
}
</pre>
Whose default implementation is defined as:<br />
<pre class="brush:java;">public class DefaultDependency implements Dependency {
public int getImportantValue() {
int value = // fetch imporant value from external service
return value;
}
}
</pre>
In order for us to test <span style="font-family: "Courier New",Courier,monospace;">componentMethod()</span>, we'd have to setup our unit test environment to allow connections to the external service; and if this external service fails, our unit test would fail as well.<br />
<br />
Mocking the dependency allows us to execute our unit test without the need for an external service:
<br />
<pre class="brush:java;">public class MockedDependency implements Dependency {
public int getImportantValue() {
// mocked value
return 1;
}
}
</pre>
Because we are providing a simple and consistent implementation, we are assured that our dependency always returns the correct value and therefore would not compromise the results of our tests.<br />
<br />
<b>Mockito</b><br />
<br />
<a href="http://code.google.com/p/mockito/" target="_blank">Mockito</a> is a mocking framework which simplifies mocking. It allows us to mock dependencies with very little code and to customize our mocked behavior depending on our testing scenario:
<br />
<pre class="brush:java;">// without mocking
Component component = new Component(new DefaultDependency());
// with mocking
Component component = new Component(new MockedDependency());
// with Mockito
Dependency dependency = Mockito.mock(Dependency.class);
Mockito.when(dependency.getImportantValue()).thenReturn(1);
Component component = new Component(dependency);
</pre>
A great thing about Mockito is that it also supports <b>mocking concrete classes</b>. <br />
<br />
<i>Note that in the above examples, we are assuming that unit tests were also written for <span style="font-family: "Courier New",Courier,monospace;">DefaultDependency</span>. Without this, we cannot guarantee the overall correctness of the application.</i><br />
<br />
<b>Static Methods </b><br />
<br />
The point that I'm trying to make is that <b>abstract methods</b> provide us the flexibility to mock our implementation. Using mocking frameworks such as Mockito, we can even extend this flexibility to concrete methods. <b>The same cannot be said for static methods</b>.<br />
<br />
Because they are associated with the class rather than an instance, static methods cannot be overridden. In fact, most static utility classes in Java are marked as final. And because they cannot be overridden, it is <b>impossible to mock static method implementations</b>.<br />
<br />
Suppose that in our previous example, <span style="font-family: "Courier New",Courier,monospace;">Dependency</span> was implemented as a static utility class:
<br />
<pre class="brush:java;">public class Component {
public void componentMethod() {
int imporantValue = Dependency.getImportantValue();
// use 'imporantValue' to calculate stuff
}
}
</pre>
<pre class="brush:java;">public final class Dependency {
public static int getImportantValue() {
int value = // fetch imporant value from external service
return value;
}
}
</pre>
Because we cannot override <span style="font-family: "Courier New",Courier,monospace;">getImportantValue()</span> with a mocked implementation, there is simply no way for us to test <span style="font-family: "Courier New",Courier,monospace;">componentMethod()</span> without requiring a connection to the external service.<br />
<br />
<b>Singletons</b><br />
<br />
You might have come across statements saying that "Singletons are evil". That is partly true <b>depending on how you create and use Singletons</b>.<br />
<br />
Suppose that in our previous example, <span style="font-family: "Courier New",Courier,monospace;">Dependency</span> was implemented and used as a <b>classic </b>Singleton:
<br />
<pre class="brush:java;">public class Component {
public void componentMethod() {
// classic use of Singleton
int imporantValue = Dependency.getInstance().getImportantValue();
// use 'imporantValue' to calculate stuff
}
}
</pre>
<pre class="brush:java;">public final class Dependency {
private static final Dependency instance = new Dependency();
private Dependency() {}
public static Dependency getInstance() {
return instance;
}
public int getImportantValue() {
int value = // fetch imporant value from external service
return value;
}
}
</pre>
Because <span style="font-family: "Courier New",Courier,monospace;">getInstance()</span> is a static method, all the evils associated with static methods apply to the Singleton as well (also applicable to Singletons implemented as <span style="font-family: "Courier New",Courier,monospace;">enums</span>).<br />
<br />
Obviously <b>not all Singletons are evil</b>. With slight modifications, we can fix what's evil about our previous implementation:<br />
<pre class="brush:java;">public interface Dependency {
int getImportantValue();
}
</pre>
<pre class="brush:java;">public final class DefaultDependency implements Dependency {
private static final Dependency instance = new DefaultDependency();
private DefaultDependency() {}
public static Dependency getInstance() {
return instance;
}
public int getImportantValue() {
int value = // fetch imporant value from external service
return value;
}
}
</pre>
By making <span style="font-family: "Courier New",Courier,monospace;">Dependency</span> an Interface and only applying the Singleton pattern to its default implementation, our <span style="font-family: "Courier New",Courier,monospace;">Component</span> class can be implemented exactly as the original version:
<br />
<pre class="brush:java;">public class Component {
private Dependency dependency;
public Component(Dependency dependency) {
this.dependency = dependency
}
public void componentMethod() {
int imporantValue = dependency.getImportantValue();
// use 'imporantValue' to calculate stuff
}
}
</pre>
Once again making it unit-testable:
<br />
<pre class="brush:java;">// without mocking, acquire Singleton instance
Component component = new Component(DefaultDependency.getInstance());
// with mocking
Component component = new Component(new MockedDependency());
// with Mockito
Dependency dependency = Mockito.mock(Dependency.class);
Mockito.when(dependency.getImportantValue()).thenReturn(1);
Component component = new Component(dependency);
</pre>
The above is an example of <b>Inversion of Control</b> (IoC). Instead of acquiring the <span style="font-family: "Courier New",Courier,monospace;">Dependency</span> instance from the <span style="font-family: "Courier New",Courier,monospace;">Component</span> class, we let the container decide which instance of <span style="font-family: "Courier New",Courier,monospace;">Dependency</span> to assign to <span style="font-family: "Courier New",Courier,monospace;">Component</span>.<br />
<br />
<i>Can you think of another popular pattern with a tendency to have the same issues described above? Hint: it's called Service Locator.</i><br />
<br />
<b>Conclusion</b><br />
<br />
Static methods have their use. But because of their impact to unit testing, caution must be applied before using them. Personally, I limit my use of static methods to utility classes having small and unchanging logic (example: <span style="font-family: "Courier New",Courier,monospace;">org.apache.commons.io.IOUtils</span>).<br />
<br />
If required to use static factory methods, applying Inversion of Control should help enforce unit-testablity.<br />
<br />
<b>Update: </b>If you really need use static methods on your dependencies, you can use <a href="https://code.google.com/p/powermock/" target="_blank">PowerMock</a>.<br />
<pre class="brush:java;">PowerMock.mockStatic(Foo.class);
expect(Foo.bar()).andReturn("foo bar");
</pre>
Check this <a href="https://code.google.com/p/powermock/wiki/MockStatic" target="_blank">page</a> for more information.jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com2tag:blogger.com,1999:blog-6063627024520487734.post-66675912723799930102013-02-24T19:31:00.000+08:002014-07-10T22:58:25.906+08:00RestTemplate with Google Places APIMy website, <a href="http://checkthecrowd.com/" target="_blank">CheckTheCrowd.com</a>, was initially using the <a href="https://developers.google.com/maps/documentation/javascript/" target="_blank">Google Maps JavaScript API (Places Library)</a> to fetch details on the various places submitted to the website.<br />
<br />
Place details such as names, addresses, and photos are normally displayed as content. Because these contents were dynamically loaded via JavaScript, they won't be visible to web crawlers and hence cannot be read as keywords.<br />
<br />
In order for web crawlers to access the place details, they needed to be included as part of the HTML generated from by the Servlet. This meant that rather than fetch the place details from the browser via JavaScript, I needed to fetch them from the web server.<br />
<br />
<b>Place Details - Google Places API</b><br />
<br />
Under the hood, the JavaScript Places Library calls a <a href="https://developers.google.com/places/documentation/details" target="_blank">REST service</a> to fetch the details of a particular place. I needed to call the same service from the web server in order to deliver the place details as part of the Servlet content.<br />
<br />
The Place Details REST service is a <span style="font-family: "Courier New",Courier,monospace;">GET </span>call to the following resource:<br />
<pre>https://maps.googleapis.com/maps/api/place/details/output?parameters
</pre>
Where <span style="font-family: "Courier New",Courier,monospace;">output</span> can either be JSON (json) or XML (<span style="font-family: "Courier New",Courier,monospace;">xml</span>), the resource requires 3 parameters: the API key (<span style="font-family: "Courier New",Courier,monospace;">key</span>), the place identifier (<span style="font-family: "Courier New",Courier,monospace;">reference</span>), and the sensor flag (<span style="font-family: "Courier New",Courier,monospace;">sensor</span>).<br />
<br />
For example:<br />
<pre>https://maps.googleapis.com/maps/api/place/details/json?reference=12345&sensor=false&key=54321
</pre>
<br />
<b>RestTemplate - Spring Web</b><br />
<br />
Starting with version 3.0, the Spring Web module comes with a class called <span style="font-family: "Courier New",Courier,monospace;"><a href="http://blog.springsource.org/2009/03/27/rest-in-spring-3-resttemplate/" target="_blank">RestTemplate</a></span>. Similar to other Spring templates, <span style="font-family: "Courier New",Courier,monospace;">RestTemplate </span>reduces boiler-plate code that is normally involved with calling REST services.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">RestTemplate </span>supports common HTTP methods such as <span style="font-family: "Courier New",Courier,monospace;">GET</span>, <span style="font-family: "Courier New",Courier,monospace;">POST</span>, <span style="font-family: "Courier New",Courier,monospace;">DELETE</span>, <span style="font-family: "Courier New",Courier,monospace;">PUT</span>, etc. Objects passed to and returned from these methods are converted by <span style="font-family: "Courier New",Courier,monospace;">HttpMessageConverters</span>. Default converters are registered against the MIME type and custom converters are also supported.<br />
<br />
<b>RestTemplate and Place Details</b><br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">RestTemplate </span>exposes a method called <span style="font-family: "Courier New",Courier,monospace;"><a href="http://static.springsource.org/spring/docs/current/javadoc-api/org/springframework/web/client/RestTemplate.html#getForObject%28java.lang.String,%20java.lang.Class,%20java.lang.Object...%29" target="_blank">getForObject</a> </span>to support GET method calls. It accepts a <span style="font-family: "Courier New",Courier,monospace;">String</span> representing the URL template, a <span style="font-family: "Courier New",Courier,monospace;">Class </span>for the return type, and a variable <span style="font-family: "Courier New",Courier,monospace;">String </span>array to populate the template.<br />
<br />
I started my implementation by creating a class called <span style="font-family: "Courier New",Courier,monospace;">GooglePlaces</span>. I then declared the URL template as a constant and declared <span style="font-family: "Courier New",Courier,monospace;">RestTemplate </span>as an instance member injected by the Spring container. My Google Places API key was also declared as a member instance, this time populated by Spring from a properties file:<br />
<pre class="brush:java;">private static final String PLACE_DETAILS_URL = "https://maps.googleapis.com/"
+ "maps/api/place/details/json?reference="
+ "{searchId}&sensor=false&key={key}";
@Value("${api.key}")
private String apiKey;
@Inject
private RestTemplate restTemplate;
</pre>
The above code should be enough to call the Place Details service and get the response as JSON string:<br />
<pre class="brush:java;">String json = restTemplate.getForObject(PLACE_DETAILS_URL,
String.class, "12345", apiKey);
</pre>
However, the JSON response needs to be converted to a Java object to be of practical use.<br />
<br />
By default, <span style="font-family: "Courier New",Courier,monospace;">RestTemplate </span>supports JSON to Java conversion via <span style="font-family: "Courier New",Courier,monospace;">MappingJacksonHttpMessageConverter</span>. All I need is to do is create Java objects which map to the Place Details JSON response.<br />
<br />
<b>Java Mapping</b> <br />
<br />
I referred to Place Details <a href="https://developers.google.com/places/documentation/details#PlaceDetailsResponses" target="_blank">reference guide</a> for a sample of the JSON response that I needed to map to Java. Because the Place Details response includes other information that I didn't need for CheckTheCrowd, I added annotations to my classes which tells the converter to ignore unmapped properties:<br />
<pre class="brush:java;">@JsonIgnoreProperties(ignoreUnknown = true)
public static class PlaceDetailsResponse {
@JsonProperty("result")
private PlaceDetails result;
public PlaceDetails getResult() {
return result;
}
public void setResult(PlaceDetails result) {
this.result = result;
}
}
</pre>
The above class represents the top-level response object. It is simply a container for the <span style="font-family: "Courier New",Courier,monospace;">result</span> property.<br />
<br />
The below class represents the <span style="font-family: "Courier New",Courier,monospace;">result</span>:<br />
<pre class="brush:java;">@JsonIgnoreProperties(ignoreUnknown = true)
public static class PlaceDetails {
@JsonProperty("name")
private String name;
@JsonProperty("icon")
private String icon;
@JsonProperty("url")
private String url;
@JsonProperty("formatted_address")
private String address;
@JsonProperty("geometry")
private PlaceGeometry geometry;
@JsonProperty("photos")
private List<PlacePhoto> photos = Collections.emptyList();
// Getters and setters...
}
</pre>
I also needed the longitude and latitude information as well as the photos. Below are the classes for the <span style="font-family: "Courier New",Courier,monospace;">geometry </span>and <span style="font-family: "Courier New",Courier,monospace;">photo</span> properties which contain these information:<br />
<pre class="brush:java;">@JsonIgnoreProperties(ignoreUnknown = true)
public static class PlaceGeometry {
@JsonProperty("location")
private PlaceLocation location;
public PlaceLocation getLocation() {
return location;
}
public void setLocation(PlaceLocation location) {
this.location = location;
}
}
</pre>
<pre class="brush:java;">@JsonIgnoreProperties(ignoreUnknown = true)
public static class PlaceLocation {
@JsonProperty("lat")
private String lat;
@JsonProperty("lng")
private String lng;
// Getters and setters...
}
</pre>
<pre class="brush:java;">@JsonIgnoreProperties(ignoreUnknown = true)
public static class PlacePhoto {
@JsonProperty("photo_reference")
private String reference;
public String getReference() {
return reference;
}
public void setReference(String reference) {
this.reference = reference;
}
}
</pre>
With the above Java mappings, I can now expose a method which returns an instance of <span style="font-family: "Courier New",Courier,monospace;">PlaceDetails </span>given a place reference:
<br />
<pre class="brush:java;">public PlaceDetails getPlaceDetails(String searchId) {
PlaceDetailsResponse response = restTemplate.getForObject(PLACE_DETAILS_URL,
PlaceDetailsResponse.class, searchId, apiKey);
if (response.getResult() != null) {
return response.getResult();
} else {
return null;
}
}
</pre>
<br />
<b>Caching</b><br />
<br />
The moment I deployed my changes to Tomcat, I noticed a significant latency between server requests. This was expected because the server now has to make several calls to the Place Details service before returning a response.<br />
<br />
This is exactly a scenario where a good caching strategy would help. It is worth noting however that Google Maps API <a href="https://developers.google.com/maps/terms" target="_blank">Terms of Service</a> (10.1.3.b) has <b>strict rules regarding caching</b>. It states that caching should only be done to improve performance and that data can only be cached up to 30 calendar days.<br />
CheckTheCrowd uses <a href="http://code.google.com/p/guava-libraries/" target="_blank">Guava</a> which includes a pretty good <a href="http://code.google.com/p/guava-libraries/wiki/CachesExplained" target="_blank">API</a> for in-memory caching. Using a <span style="font-family: "Courier New",Courier,monospace;">CacheLoader</span>, I can seamlessly integrate a Guava cache to my code:
<br />
<pre class="brush:java; highlight: [7, 8, 9, 10, 11, 12, 13]">private LoadingCache<String, PlaceDetails> placeDetails = CacheBuilder.newBuilder()
.maximumSize(1000)
.expireAfterAccess(24, TimeUnit.HOURS)
.build(new CacheLoader<String, PlaceDetails>() {
@Override
public PlaceDetails load(String searchId) throws Exception {
PlaceDetailsResponse response = restTemplate.getForObject(PLACE_DETAILS_URL,
PlaceDetailsResponse.class, searchId, apiKey);
if (response.getResult() != null) {
return response.getResult();
} else {
throw new PlacesException("Unable to find details for reference: " + searchId);
}
}
});
</pre>
I set a cache size of 1000 and an expiry of 24 hours. The call to the Place Details service was then moved to the <span style="font-family: "Courier New",Courier,monospace;">CacheLoader</span>'s <span style="font-family: "Courier New",Courier,monospace;">load </span>method. After which, I updated my method to refer to the cache instead:
<br />
<pre class="brush:java;">public PlaceDetails getPlaceDetails(String searchId) {
try {
return placeDetails.get(searchId);
} catch (ExecutionException e) {
logger.warn("An exception occurred while "
+ "fetching place details!", e.getCause());
return null;
}
}
</pre>
<br />
<i>The complete source is available from <a href="https://jramoyo-samples.googlecode.com/svn/trunk/samples/src/main/java/com/jramoyo/util/GooglePlaces.java" target="_blank">Google Code</a> under Apache License 2.0.</i>jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com1tag:blogger.com,1999:blog-6063627024520487734.post-10789322618614634692013-02-11T11:53:00.002+08:002014-06-25T11:01:36.606+08:00Generating Sitemaps Using Spring Batch and SitemapGen4jI recently launched a website called <a href="http://checkthecrowd.com/" target="_blank">CheckTheCrowd.com</a><span id="goog_551388248"></span><span id="goog_551388249"></span>. And in order for search engines to effectively crawl my content, I needed a <a href="http://en.wikipedia.org/wiki/Sitemaps" target="_blank">sitemap</a>.<br />
<br />
Since my content is mostly generated from the database, I needed to find a way to dynamically generate my sitemap.<br />
<br />
Most answers I got from online forums suggested exposing a URL which when accessed, generates the sitemap. With Spring MVC, it goes something like:<br />
<pre class="brush:java; highlight: [3]">
@RequestMapping("/sitemap.xml")
public @ResponseBody String generateSitemap() {
String sitemap = generateExpensiveXml();
return sitemap
}
</pre>
The problem with this approach is that it doesn't scale. The more content you have, the longer it takes to generate the sitemap. And because the sitemap is generated every time the URL is accessed, precious server resources are wasted.<br />
<br />
Another <a href="http://stackoverflow.com/questions/1099393/sitemap-on-a-highly-dynamic-website" target="_blank">suggestion </a>was to append an entry to the sitemap every time new content is added to the database. I did not like this approach because it would be difficult to do source control on the sitemap. Also, accidentally deleting the sitemap would mean that data is gone forever.<br />
<br />
<b>Batch Job Approach</b><br />
<br />
Eventually, I ended-up doing something similar to the first suggestion. However, instead of generating the sitemap every time the URL is accessed, I ended-up generating the sitemap from a batch job.<br />
<br />
With this approach, I get to schedule how often the sitemap is generated. And because generation happens outside of an HTTP request, I can afford a longer time for it to complete.<br />
<br />
Having previous experience with the framework, Spring Batch was my obvious choice. It provides a framework for building batch jobs in Java. Spring Batch works with the idea of "chunk processing" wherein huge sets of data are divided and processed as chunks.<br />
<br />
I then searched for a Java library for writing sitemaps and came-up with <a href="http://code.google.com/p/sitemapgen4j/" target="_blank">SitemapGen4j</a>. It provides an easy to use API and is released under Apache License 2.0.<br />
<br />
<b>Requirements</b><br />
<br />
My requirements are simple: I have a couple of static web pages which can be hard-coded to the sitemap. I also have pages for each place submitted to the web site; each place is stored as a single row in the database and is identified by a unique ID. There are also pages for each registered user; similar to the places, each user is stored as a single row and is identified by a unique ID.<br />
<br />
A job in Spring Batch is composed of 1 or more "steps". A step encapsulates the processing needed to be executed against a set of data.<br />
<br />
I identified 4 steps for my job:<br />
<ul>
<li>Add static pages to the sitemap</li>
<li>Add place pages to the sitemap</li>
<li>Add profile pages to the sitemap</li>
<li>Write the sitemap XML to a file</li>
</ul>
<br />
<b>Step 1</b><br />
<br />
Because it does not involve processing a set of data, my first step can be implemented directly as a simple <span style="font-family: "Courier New",Courier,monospace;">Tasklet</span>:<br />
<pre class="brush:java;">public class StaticPagesInitializerTasklet implements Tasklet {
private static final Logger logger = LoggerFactory.getLogger(StaticPagesInitializerTasklet.class);
private final String rootUrl;
@Inject
private WebSitemapGenerator sitemapGenerator;
public StaticPagesInitializerTasklet(String rootUrl) {
this.rootUrl = rootUrl;
}
@Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
logger.info("Adding URL for static pages...");
sitemapGenerator.addUrl(rootUrl);
sitemapGenerator.addUrl(rootUrl + "/terms");
sitemapGenerator.addUrl(rootUrl + "/privacy");
sitemapGenerator.addUrl(rootUrl + "/attribution");
logger.info("Done.");
return RepeatStatus.FINISHED;
}
public void setSitemapGenerator(WebSitemapGenerator sitemapGenerator) {
this.sitemapGenerator = sitemapGenerator;
}
}
</pre>
The starting point of a <span style="font-family: "Courier New",Courier,monospace;">Tasklet </span>is the <span style="font-family: "Courier New",Courier,monospace;">execute()</span> method. Here, I add the URLs of the known static pages of CheckTheCrowd.com.<br />
<br />
<b>Step 2</b><br />
<br />
The second step requires places data to be read from the database then subsequently written to the sitemap.<br />
<br />
This is a common requirement, and Spring Batch provides built-in Interfaces to help perform these types of processing:<br />
<ul>
<li><a href="http://static.springsource.org/spring-batch/apidocs/org/springframework/batch/item/ItemReader.html" target="_blank"><span style="font-family: "Courier New",Courier,monospace;"><b>ItemReader </b></span></a>- Reads a chunk of data from a source; each data is considered an item. In my case, an item represents a place.</li>
<li><a href="http://static.springsource.org/spring-batch/apidocs/org/springframework/batch/item/ItemProcessor.html" target="_blank"><span style="font-family: "Courier New",Courier,monospace;"><b>ItemProcessor </b></span></a>- Transforms the data before writing. This is <b>optional </b>and is not used in this example. </li>
<li><a href="http://static.springsource.org/spring-batch/apidocs/org/springframework/batch/item/ItemWriter.html" target="_blank"><span style="font-family: "Courier New",Courier,monospace;"><b>ItemWriter </b></span></a>- Writes a chunk of data to a destination. In my case, I add each place to the sitemap.</li>
</ul>
The Spring Batch API includes a class called <a href="http://static.springsource.org/spring-batch/apidocs/org/springframework/batch/item/database/JdbcCursorItemReader.html" target="_blank"><span style="font-family: "Courier New",Courier,monospace;">JdbcCursorItemReader</span></a>, an implementation of <span style="font-family: "Courier New",Courier,monospace;">ItemReader </span>which continously reads rows from a JDBC <span style="font-family: "Courier New",Courier,monospace;">ResultSet</span>. It requires a <a href="http://static.springsource.org/spring/docs/3.0.x/api/org/springframework/jdbc/core/RowMapper.html" target="_blank"><span style="font-family: "Courier New",Courier,monospace;">RowMapper</span></a> which is responsible for mapping database rows to batch items.<br />
<br />
For this step, I declare a <span style="font-family: "Courier New",Courier,monospace;">JdbcCursorItemReader</span> in my Spring configuration and set my implementation of <span style="font-family: "Courier New",Courier,monospace;">RowMapper</span>:<br />
<pre class="brush:java;">@Bean
public JdbcCursorItemReader<PlaceItem> placeItemReader() {
JdbcCursorItemReader<PlaceItem> itemReader = new JdbcCursorItemReader<>();
itemReader.setSql(environment.getRequiredProperty(PROP_NAME_SQL_PLACES));
itemReader.setDataSource(dataSource);
itemReader.setRowMapper(new PlaceItemRowMapper());
return itemReader;
}
</pre>
Line 4 sets the SQL statement to query the <span style="font-family: "Courier New",Courier,monospace;">ResultSet</span>. In my case, the SQL statement is fetched from a properties file.<br />
<br />
Line 5 sets the JDBC <span style="font-family: "Courier New",Courier,monospace;">DataSource</span>.<br />
<br />
Line 6 sets my implementation of <span style="font-family: "Courier New",Courier,monospace;">RowMapper</span>.<br />
<br />
Next, I write my implementation of <span style="font-family: "Courier New",Courier,monospace;">ItemWriter</span>:<br />
<pre class="brush:java;">public class PlaceItemWriter implements ItemWriter<PlaceItem> {
private static final Logger logger = LoggerFactory.getLogger(PlaceItemWriter.class);
private final String rootUrl;
@Inject
private WebSitemapGenerator sitemapGenerator;
public PlaceItemWriter(String rootUrl) {
this.rootUrl = rootUrl;
}
@Override
public void write(List<? extends PlaceItem> items) throws Exception {
String url;
for (PlaceItem place : items) {
url = rootUrl + "/place/" + place.getApiId() + "?searchId=" + place.getSearchId();
logger.info("Adding URL: " + url);
sitemapGenerator.addUrl(url);
}
}
public void setSitemapGenerator(WebSitemapGenerator sitemapGenerator) {
this.sitemapGenerator = sitemapGenerator;
}
}
</pre>
Places in CheckTheCrowd.com are accessible from URLs having this pattern: <span style="font-family: "Courier New",Courier,monospace;">checkthecrowd.com/place/{placeId}?searchId={searchId}</span>. My <span style="font-family: "Courier New",Courier,monospace;">ItemWriter </span>simply iterates through the chunk of <span style="font-family: "Courier New",Courier,monospace;">PlaceItems</span>, builds the URL, then adds the URL to the sitemap.<br />
<br />
<b>Step 3</b><br />
<br />
The third step is exactly the same as the previous, but this time processing is done on user profiles.<br />
<br />
Below is my <span style="font-family: "Courier New",Courier,monospace;">ItemReader </span>declaration:<br />
<pre class="brush:java;">@Bean
public JdbcCursorItemReader<PlaceItem> profileItemReader() {
JdbcCursorItemReader<PlaceItem> itemReader = new JdbcCursorItemReader<>();
itemReader.setSql(environment.getRequiredProperty(PROP_NAME_SQL_PROFILES));
itemReader.setDataSource(dataSource);
itemReader.setRowMapper(new ProfileItemRowMapper());
return itemReader;
}
</pre>
Below is my <span style="font-family: "Courier New",Courier,monospace;">ItemWriter </span>implementation: <br />
<pre class="brush:java;">public class ProfileItemWriter implements ItemWriter<ProfileItem> {
private static final Logger logger = LoggerFactory.getLogger(ProfileItemWriter.class);
private final String rootUrl;
@Inject
private WebSitemapGenerator sitemapGenerator;
public ProfileItemWriter(String rootUrl) {
this.rootUrl = rootUrl;
}
@Override
public void write(List<? extends ProfileItem> items) throws Exception {
String url;
for (ProfileItem profile : items) {
url = rootUrl + "/profile/" + profile.getUsername();
logger.info("Adding URL: " + url);
sitemapGenerator.addUrl(url);
}
}
public void setSitemapGenerator(WebSitemapGenerator sitemapGenerator) {
this.sitemapGenerator = sitemapGenerator;
}
}
</pre>
Profiles in CheckTheCrowd.com are accessed from URLs having this pattern: <span style="font-family: "Courier New",Courier,monospace;">checkthecrowd.com/profile/{username}</span>.<br />
<br />
<b>Step 4</b><br />
<br />
The last step is fairly straightforward and is also implemented as a simple <span style="font-family: "Courier New",Courier,monospace;">Tasklet</span>:<br />
<pre class="brush:java;">public class XmlWriterTasklet implements Tasklet {
private static final Logger logger = LoggerFactory.getLogger(XmlWriterTasklet.class);
@Inject
private WebSitemapGenerator sitemapGenerator;
@Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
logger.info("Writing sitemap.xml...");
sitemapGenerator.write();
logger.info("Done.");
return RepeatStatus.FINISHED;
}
}
</pre>
Notice that I am using the same instance of <span style="font-family: "Courier New",Courier,monospace;">WebSitemapGenerator </span>across all the steps. It is declared in my Spring configuration as:
<br />
<pre class="brush:java;">@Bean
public WebSitemapGenerator sitemapGenerator() throws Exception {
String rootUrl = environment.getRequiredProperty(PROP_NAME_ROOT_URL);
String deployDirectory = environment.getRequiredProperty(PROP_NAME_DEPLOY_PATH);
return WebSitemapGenerator.builder(rootUrl, new File(deployDirectory))
.allowMultipleSitemaps(true).maxUrls(1000).build();
}
</pre>
Because they change between environments (dev vs prod), <span style="font-family: "Courier New",Courier,monospace;">rootUrl </span>and <span style="font-family: "Courier New",Courier,monospace;">deployDirectory </span>are both configured from a properties file.<br />
<br />
<b>Wiring them all together...</b><br />
<pre class="brush:xml; highlight: [26, 27, 28, 29, 30]"><beans>
<context:component-scan base-package="com.checkthecrowd.batch.sitemapgen.config" />
<bean class="...config.SitemapGenConfig" />
<bean class="...config.java.process.ConfigurationPostProcessor" />
<batch:job id="generateSitemap" job-repository="jobRepository">
<batch:step id="insertStaticPages" next="insertPlacePages">
<batch:tasklet ref="staticPagesInitializerTasklet" />
</batch:step>
<batch:step id="insertPlacePages" parent="abstractParentStep" next="insertProfilePages">
<batch:tasklet>
<batch:chunk reader="placeItemReader" writer="placeItemWriter" />
</batch:tasklet>
</batch:step>
<batch:step id="insertProfilePages" parent="abstractParentStep" next="writeXml">
<batch:tasklet>
<batch:chunk reader="profileItemReader" writer="profileItemWriter" />
</batch:tasklet>
</batch:step>
<batch:step id="writeXml">
<batch:tasklet ref="xmlWriterTasklet" />
</batch:step>
</batch:job>
<batch:step id="abstractParentStep" abstract="true">
<batch:tasklet>
<batch:chunk commit-interval="100" />
</batch:tasklet>
</batch:step>
</beans>
</pre>
Lines 26-30 declare an abstract step which serves as the common parent for steps 2 and 3. It sets a property called <span style="font-family: "Courier New",Courier,monospace;">commit-interval</span> which defines how many items comprises a chunk. In this case, a chunk is comprised of 100 items.<br />
<br />
<i>There is a lot more to Spring Batch, kindly refer to the official <a href="http://static.springsource.org/spring-batch/reference/html/index.html" target="_blank">reference guide</a></i>.jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com6tag:blogger.com,1999:blog-6063627024520487734.post-83268237080232263582013-02-10T20:48:00.000+08:002014-06-25T10:59:31.989+08:00Testing Spring MVC Annotated JSON Controllers from JUnit<i><b>Update: While I still recommend this post for good reading, please refer to this <a href="http://www.javacodegeeks.com/2013/08/unit-testing-of-spring-mvc-controllers-rest-api.html" rel="nofollow" target="_blank">post</a> for a better approach at doing this.</b></i><br />
<br />
<i>My previous <a href="http://www.jramoyo.com/2013/02/testing-spring-mvc-annotated.html" target="_blank">post </a>explained how we can use <a href="http://static.springsource.org/spring/docs/3.0.x/api/org/springframework/web/servlet/mvc/annotation/AnnotationMethodHandlerAdapter.html" target="_blank"><span style="font-family: inherit;"><span style="font-family: "Courier New",Courier,monospace;">AnnotationMethodHandlerAdapter</span> </span> </a>to test annotations applied to Spring MVC Controllers. This post will attempt to explain how we can reuse the classes introduced in the previous post to test Spring Controllers that return a JSON response.</i><br />
<br />
The <a href="http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/web/bind/annotation/ResponseBody.html" target="_blank">@ResponseBody</a> annotation allows Spring Controllers to define the contents of an HTTP response. <br />
<br />
For this example, we assume that Spring MVC is configured to represent the contents of the HTTP response as JSON. <br />
<br />
Let's say we have a controller which returns user information from a GET request:<br />
<pre class="brush:java;">@Controller
public class MyJsonController {
private final UserService userService = new UserService();
@RequestMapping(value = "/user/{username}", method = RequestMethod.GET)
public @ResponseBody User getUser(@PathVariable String username) {
return userService.getUser(username);
}
// Mocked for illustration purposes
private static class UserService {
public User getUser(String username) {
User user = new User(username);
user.setFirstName("Jan");
user.setLastName("Amoyo");
return user;
}
}
}
</pre>
This controller returns a <span style="font-family: "Courier New",Courier,monospace;">User </span>object serialized within the body of the HTTP response (in our case, as JSON). Because it doesn't return an instance of <span style="font-family: "Courier New",Courier,monospace;">ModelAndView</span>, we cannot use the previously introduced <span style="font-family: "Courier New",Courier,monospace;">AbstractControllerTest </span>to test the output.<br />
<br />
In order to test the JSON response, we need to assign an appropriate <span style="font-family: "Courier New",Courier,monospace;">HttpMessageConverter </span>to the <span style="font-family: "Courier New",Courier,monospace;">AnnotationMethodHandlerAdapter </span>defined in <span style="font-family: "Courier New",Courier,monospace;">AbstractControllerTest</span>. The one we need is <span style="font-family: "Courier New",Courier,monospace;">MappingJacksonHttpMessageConverter</span>.<br />
<br />
<b>Extending AbstractControllerTest</b><br />
<br />
We will create a new class called <span style="font-family: "Courier New",Courier,monospace;">AbstractJsonControllerTest </span>and extend from <span style="font-family: "Courier New",Courier,monospace;">AbstractControllerTest</span>. Here, we override the parent's constructor so that we can assign <span style="font-family: "Courier New",Courier,monospace;">MappingJacksonHttpMessageConverter </span>to the <span style="font-family: "Courier New",Courier,monospace;">AnnotationMethodHandlerAdapter</span>. We also add various convenience methods to help process JSON.<br />
<pre class="brush:java;">@Ignore("abstract test case")
public abstract class AbstractJsonControllerTest&ltT&gt extends AbstractControllerTest&ltT&gt {
private final ObjectMapper mapper;
public AbstractJsonControllerTest(T controller) {
super(controller);
mapper = new ObjectMapper();
MappingJacksonHttpMessageConverter jacksonHttpMessageConverter = new MappingJacksonHttpMessageConverter();
MappingJacksonHttpMessageConverter[] messageConverters = { jacksonHttpMessageConverter };
getHandlerAdapter().setMessageConverters(messageConverters);
}
protected List&ltMap&ltString, Object&gt&gt convertJsonArrayHttpServletResponseToList(MockHttpServletResponse response) throws JsonParseException,
JsonMappingException, IOException {
return convertJsonArrayStringToList(response.getContentAsString());
}
protected List&ltMap&ltString, Object&gt&gt convertJsonArrayStringToList(String json) throws JsonParseException, JsonMappingException, IOException {
return mapper.readValue(json, new TypeReference&ltList&ltHashMap&ltString, Object&gt&gt&gt() {
});
}
protected Map&ltString, Object&gt convertJsonHttpServletResponseToMap(MockHttpServletResponse response) throws JsonParseException,
JsonMappingException, IOException {
return convertJsonStringToMap(response.getContentAsString());
}
protected Map&ltString, Object&gt convertJsonStringToMap(String json) throws JsonParseException, JsonMappingException, IOException {
return mapper.readValue(json, new TypeReference&ltHashMap&ltString, Object&gt&gt() {
});
}
protected String convertObjectToJsonString(Object object) throws JsonMappingException, JsonGenerationException, IOException {
return mapper.writeValueAsString(object);
}
}
</pre>
Similar to <span style="font-family: "Courier New",Courier,monospace;">AbstractControllerTest</span>, we can use <span style="font-family: "Courier New",Courier,monospace;">AbstractJsonControllerTest</span> as the parent class to all test cases involving JSON Spring Controllers.<br />
<br />
<b>Example</b><br />
<br />
Here is how we test <span style="font-family: "Courier New",Courier,monospace;">MyJsonController</span>:
<br />
<pre class="brush:java; highlight: [13, 16, 18, 19, 20, 21]">
@RunWith(BlockJUnit4ClassRunner.class)
public class MyJsonControllerTest extends AbstractJsonControllerTest<MyJsonController> {
public MyJsonControllerTest() {
super(new MyJsonController());
}
@Test
public void testGetUser() throws Exception {
getRequest().setMethod("GET");
getRequest().setServletPath("/user/jramoyo");
ModelAndView modelAndView = invokeController();
assertNull(modelAndView);
String jsonString = getResponse().getContentAsString();
assertFalse(jsonString == null || jsonString.isEmpty());
Map<String, Object> jsonMap = convertJsonStringToMap(jsonString);
assertEquals("jramoyo", jsonMap.get("username"));
assertEquals("Jan", jsonMap.get("firstName"));
assertEquals("Amoyo", jsonMap.get("lastName"));
}
}
</pre>
Line 13 asserts that ModelAndView is null as expected.<br />
<br />
Line 16 asserts that the JSON reponse is not null nor empty.<br />
<br />
Line 18 converts the JSON response to a Map object.<br />
<br />
Lines 19 to 21 asserts the correctness of the user information retrieved from the HTTP response.<br />
<br />
<i>The above snippets are available from <a href="http://jramoyo-samples.googlecode.com/svn/trunk/samples/src/test/java/com/jramoyo/controller/" target="_blank">Google Code</a>.</i> jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com0tag:blogger.com,1999:blog-6063627024520487734.post-78886307393895976152013-02-10T15:14:00.000+08:002014-06-25T10:58:16.146+08:00Testing Spring MVC Annotated Controllers from JUnit<i><span style="font-family: inherit;">Let me begin by clarifying that this post is more about testing the correctness of annotations applied to Spring Controllers rather than unit testing the behavior of the controller methods.</span></i>
<br />
<span style="font-family: inherit;"><br />Annotations on Spring Controllers are commonly tested manually, together with the views. Aside from the usual concerns that come with it, manual testing also involves significant overhead with</span><br />
<span style="font-family: inherit;">having to redeploy binaries and restarting the web server in-between fixes. While "hot-deploy" reduces a lot of these concerns, it still leaves some room for improvement.</span><br />
<span style="font-family: inherit;"> </span>
<br />
<b>AnnotationMethodHandlerAdapter</b>
<br />
<br />
Starting with version 2.5, Spring MVC comes with a class called <a href="http://static.springsource.org/spring/docs/3.0.x/api/org/springframework/web/servlet/mvc/annotation/AnnotationMethodHandlerAdapter.html" target="_blank"><span style="font-family: "Courier New",Courier,monospace;">AnnotationMethodHandlerAdapter</span> </a>which can be used to programatically invoke an annotated Spring Controller. The <span style="font-family: "Courier New",Courier,monospace;">handle() </span>method from this class accepts an <span style="font-family: "Courier New",Courier,monospace;">HttpServletRequest</span>, an <span style="font-family: "Courier New",Courier,monospace;">HttpServletResponse</span>, and an <span style="font-family: "Courier New",Courier,monospace;">Object </span>representing the Controller to invoke. It returns an instance of <span style="font-family: "Courier New",Courier,monospace;">ModelAndView</span>.<br />
<span style="font-family: inherit;"><br /></span>
We can use <span style="font-family: inherit;"><span style="font-family: "Courier New",Courier,monospace;">AnnotationMethodHandlerAdapter</span> </span>within a JUnit test case to simulate a web container handling a request to the Spring Controller.
<br />
<pre class="brush:java;">@Test
public void testController() throws Exception {
ModelAndView modelAndView = handlerAdapter.handle(request, response, myController);
assertEquals("index", modelAndView.getViewName());
}
</pre>
We can mock <span style="font-family: "Courier New",Courier,monospace;">HttpServletRequest </span>and <span style="font-family: "Courier New",Courier,monospace;">HttpServletResponse </span>using popular mocking frameworks such <a href="http://code.google.com/p/mockito/" target="_blank">Mockito </a>or use the built-in <span style="font-family: "Courier New",Courier,monospace;">MockHttpServletRequest </span>and <span style="font-family: "Courier New",Courier,monospace;">MockHttpServletResponse </span>from the Spring Test module. This example uses the built-in mocked objects from Spring Test.<br />
<br />
Using <span style="font-family: inherit;"><span style="font-family: "Courier New",Courier,monospace;">AnnotationMethodHandlerAdapter</span> </span> together with Generics, we can create an abstract class which serves as a parent class to all Spring Controller unit test cases:
<br />
<pre class="brush:java">@Ignore("abstract test case")
public abstract class AbstractControllerTest<T> {
private final T controller;
private final AnnotationMethodHandlerAdapter handlerAdapter;
private MockHttpServletRequest request;
private MockHttpServletResponse response;
public AbstractControllerTest(T controller) {
handlerAdapter = new AnnotationMethodHandlerAdapter();
this.controller = controller;
}
@Before
public void setUp() {
response = new MockHttpServletResponse();
request = new MockHttpServletRequest();
}
protected T getController() {
return controller;
}
protected AnnotationMethodHandlerAdapter getHandlerAdapter() {
return handlerAdapter;
}
protected MockHttpServletRequest getRequest() {
return request;
}
protected MockHttpServletResponse getResponse() {
return response;
}
protected ModelAndView invokeController() throws Exception {
return handlerAdapter.handle(request, response, controller);
}
}
</pre>
As a parent class, it takes care of the boiler plate code involved in setting-up and invoking the controller.<br />
<br />
<b>Example</b><br />
<br />
Let's say we have a controller which returns a view named "index" and populates the current user's username as a model attribute:
<br />
<pre class="brush:java;">@Controller
public class MyController {
@RequestMapping(value = "/", method = RequestMethod.GET)
public String getHomePage(Principal principal, Model model) {
if (principal != null) {
model.addAttribute("username", principal.getName());
}
return "index";
}
}
</pre>
To test this, we create a concrete subclass of <span style="font-family: "Courier New",Courier,monospace;">AbstractControllerTest </span>and assign <span style="font-family: "Courier New",Courier,monospace;">MyController </span>as the generic type:
<br />
<pre class="brush:java; highlight: [6, 11, 12, 13, 15, 16, 17, 18]">
@RunWith(BlockJUnit4ClassRunner.class)
public class MyControllerTest
extends AbstractControllerTest<MyController> {
public MyControllerTest() {
super(new MyController());
}
@Test
public void testGetHomePage() throws Exception {
getRequest().setMethod("GET");
getRequest().setServletPath("/");
getRequest().setUserPrincipal(new HttpPrincipal("username", "realm"));
ModelAndView modelAndView = invokeController();
assertEquals("index", modelAndView.getViewName());
assertNotNull(modelAndView.getModelMap().get("username"));
assertEquals("username", modelAndView.getModelMap().get("username"));
}
}
</pre>
Line 6 creates an instance of <span style="font-family: "Courier New",Courier,monospace;">MyController</span> and passes it as a parameter to the parent class' constructor.<br />
<br />
Lines 11 to 13 sets-up the HTTP request by assigning the request method, request URL, and a fake <span style="font-family: "Courier New",Courier,monospace;">Principal</span>.<br />
<br />
Line 15 invokes the request against the Spring Controller.<br />
<br />
Lines 16-18 asserts the correctness of the returned <span style="font-family: "Courier New",Courier,monospace;">ModelAndView</span>.<br />
<br />
<i>The above snippets are available from <a href="http://jramoyo-samples.googlecode.com/svn/trunk/samples/src/test/java/com/jramoyo/controller/" target="_blank">Google Code</a>.</i><br />
<br />
<i>On my next <a href="http://www.jramoyo.com/2013/02/testing-spring-mvc-annotated-json.html" target="_blank">post</a>, I will explain how to test Spring Controllers returning a JSON response by extending the above classes.</i>jramoyohttp://www.blogger.com/profile/04727621188639609013noreply@blogger.com4