Moved

I’ve moved to blog.thomaschen.me!

Node.js: Asynchronous Programming With Callbacks vs Promises

When you think of programming you usually think of code getting executed in order and sequentially. In asynchronous programming there are certain operations where you need wait for a response. In these scenarios, code that follows that operation will continue to get executed until there is a response back from the asynchronous operation. In single threaded environments such as node.js it is possible to have multithreaded-like behavior with its event driven architecture (more on that later). The default pattern to handle these responses is to use a callback function. For example:

var callback = function() {
   // Handle the response
};
someAsyncFunction(callback);

// Code continues to get executed until callback() is triggered
doSomethingSynchronousHere();

Callbacks are a straightforward and natural way to think about firing off tasks and waiting for a response. It is nice because no additional libraries are needed to use it in environments like node.js and many others. The problem with callbacks is that they introduce a dependency between the caller and callee. That is, the caller needs to pass in the code to the callee. Based on software principles such as the Law of Demeter or Principle of Least Knowledge, we know that software design should be based on a “need to know basis.” This is why we have private variable scoping and so forth. Callbacks of this type can be thought loosely as Dependency Injection (DI) which is a type of Inversion of Control (IoC), but in this case it is a dependency which isn’t actually needed.

How do we get around this? We use a Promise. In jQuery they’re called Deferred Objects, in many node.js libraries they are Promises.

What is a promise? In normal software flow of control, a piece of code is in control of its needs and execution. With IoC, things are handed to you instead. When you invert the IoC, by simple negation, you are back to the original flow of control of code.

The promise pattern inverts the IoC. That is, the callee no longer has to know about the callback. Promises achieve this by having the callee “promise” to the caller that it will respond to it with its results when it’s ready. Without promises a function might look like the following:

var someAsyncFunction = function(callback) {
   event.trigger('some_event', function(result) { // Note: async event trigger call
      callback(result);
   });
};

A function that returns a promise would look something like:

var someAsyncFunction = function() {
   var defer = new Defer();

   event.trigger('some_event', function(result) { // Note: at some point lower level async calls will utilize callbacks still
      defer.resolve(result); // 2. Actually giving the response later
   });

   return defer.promise; // 1. Tells the caller that you promise to give it a response later
};

The caller of an asynchronous function would look like this with callbacks:

var callback = function() {...};
someAsyncFunction(callback);

And like this using promises:

var callback = function() {...};
Promise.when(someAsyncFunction())
.then(callback);

The main difference is subtle: in the callback approach, the caller of someAsyncFunction() needs to pass in the callback. In the promise pattern approach, the callback is never passed in. You can look into implementations of the promise pattern to understand how it works more in depth. A popular one for node is Q.

The other big benefit for using a promise pattern is that it normalizes synchronous and asynchronous method calls so that the caller doesn’t care if whether the underlying code is synchronous or asynchronous.

Backbone.js: Javascript Mixins for Desktop vs Mobile HTML5 Apps

In Javascript, a mixin is a design pattern that allows an an object to take on behavior from another object without inheritance. Here is a good article on the subject:

http://ricostacruz.com/backbone-patterns/#mixins

Sometimes inheritance doesn’t necessarily make sense. For example, say you’re writing a UI library and you have a button widget and a checkbox widget, both of which can be toggled. The button widget toggles between green and gray (representing enabled and disabled) and the checkbox toggles between having a check and not having a check. If we were using inheritance we might have a ToggleWidget which abstracts the toggle functionality and CheckBoxWidget and ButtonWidget could both inherit off of that. But this approach is too inflexible.

The better solution is to create mixins for behavior that are used by many objects. With a mixin, an object is endowed with functionality without having to subclass. This is possible in JS because it is a dynamic language and you can add functions and objects to objects at any point in time. Imagine the following mixin:

var Toggle : function() {
   toggle : function(){...},
   defaultState : isEnabled,
   ...
};

In Backbone.js, if we have a view called ButtonView:

define(function() {
   var ButtonView : function() {
      initialize : function() {...},
      ...
   };
});

We can mixin the Toggle behavior in the following way:

define(['mixins/toggle'], function(Toggle) {
   var ButtonView : function() {
      initialize : function() {
         _.extend(this, Toggle);
         this.toggle(); // Use the behavior...
      },
      ...
   };
});

Of course you can do it without underscore.js, but backbone.js already uses it as a dependency. Because this isn’t inheritance, an object can have many objects mixed in:

define(['mixins/toggle', 'mixins/somethingelse'], function(Toggle, SomethingElse) {
   var ButtonView : function() {
      initialize : function() {
         _.extend(this, Toggle, SomethingElse, ...);
      },
      ...
   };
});

At Vurb we use mixins in backbone.js to mixin behavior for UI widgets and to augment desktop functionality. In our HTML5 desktop app, for example, we utilize websockets to support live collaboration. It doesn’t make sense to have that functionality on our mobile app, so that behavior isn’t mixed in. This has two main benefits:

  1. Abstracts behavior into their own modules
  2. Reduces payload size of our mobile app

If you use an AMD loader such as require.js you can selectively choose which mixins to load based on your desktop or mobile main.js loader.

Desktop:

requirejs(['mixins/desktopwebsocketstuff']function(DesktopWebSocketStuff) {
   MainApp.initialize({
      mixins : [DesktopWebSocketStuff, ...]
   });
});

Mobile:

requirejs(function() {
   MainApp.initialize({});
});

MongoDB: Lazy Loading, Infinite Scrolling

Infinite scrolling is a common thing on many web sites. To do it in mongodb in a performant manner, you can utilize the limit(), sort(), find() and $gt/$lt/$gte/$lte query apis instead of the limit(), skip() and find() apis as discussed here:

http://stackoverflow.com/questions/5049992/mongodb-paging

Such a query would look like:

db.collection.sort({updatedDate:-1}).limit(10).find({
   updatedDate : {
      $lt : <some_date>
   }, 
   ...
);

What the query asks for is to return the first 10 documents from <some_date> which is sorted from newest to oldest. The wrong way to query mongodb for infinite scrolling would be:

db.collection.sort({updatedDate:-1}).skip(<some_page_value_here>).limit(10).find({...});

Ideally the field that you are using to page, e.g. the <some_date> in example above, should be unique. If you have duplicated field values you risk missing some documents that are outside of the limit # based on the initial query field value. If you are simply doing infinite scrolling based on the created date, you can simply do it based on the document’s _id because mongodb id’s have timestamp values encoded into them.

Data-Driving If/Else and Switch Statements

If you’ve ever seen code similar to this:

if(input === 'jap') {
   output = 'Japanese';
} else if(input === 'chin') {
   output = 'Chinese';
} else if(input === 'eng') {
   output = 'English';
} ...

or this:

switch(input) {
   case 'jap': output = 'Japanese'; break;
   case 'chin': output = 'Chinese'; break;
   case 'eng': output = 'English'; break;
   ...
}

What the code needs is to be data-driven. What this code is, although a simplified example, is an imperative map or lookup table. And there are obvious downsides to having this code, including the following: 1) It needs to be updated every time new data is added, 2) it’s error prone to typos and syntax errors and 3) it takes up space and lines of code. A data-driven solution pushes the data to be resolved by a map or lookup table to the origin of the input. In the above examples it would involve pushing the strings ‘Japanese’, ‘Chinese’ and ‘English’ to the input object:

// This content might live in XML, JSON, YAML, etc.
var someInput = {
   ...other data
   canonicalizedString : 'Japanese
};
var someInput2 = {
   ...other data
   canonicalizedString : 'Chinese'
};

Which would allow you to convert the giant if/else or switch statement to simply be:

output = input.canonicalizedString;

Data-driving if/else and switch statements can also involve making the mapping more programmatic instead of just pushing the data to its origin. For example, in a contrived, but similar example:

function japanese(){};
function chinese(){};
function english(){};

if(input === 'japanese') {
   output = japanese();
} else if(input === 'chinese') {
   output = chinese();
} else if(input === 'english') {
   output = english();
}

Would become:

var myFunctions = {
   japanese : function(){},
   chinese : function(){},
   english : function(){}
};

output = myFunctions[input];

These are simplified and contrived examples, but the essence of the problem exists in many code bases out there. Going full circle: have you seen this pattern in any code bases before?

Zynga’s Treasure Isle: Inheritance vs Composition

One of the things I helped ship while at Zynga was an expansion set called Genie’s Lamp for the Treasure Isle game. Treasure Isle is a tile-digging based game which also has aspects of the “invest and express”-type games, games such as FarmVille and CityVille. In it there are many forms of building decorations that you can own as well as many decorations that you interact with on your digging quests. There are many types of these interactive decorations: ones that can be clicked on and harvested for rewards, ones that provide quicker energy regeneration, ones that unlock secret levels, ones that do absolutely nothing but look pretty, ones that block access to certain tiles, ones that could be rotated, and so forth.

A straightforward OOP approach to implementing these various types of decorations is to create parent classes that represent a commonality between the buildings. For example, there could be a class called HarvestableDecoration, one that’s called EnergyDecoration, one that’s called SimpleDecoration, and so forth. And this is exactly how the game originally was written. Like many things, it worked well in the very beginning. Then product managers had the brilliant idea of making decorations that were both harvestable and rotatable. Great, we had classes that represented these functionalities separately, but how do you make it do both? ActionScript3 only has single inheritance. At the time, given the time pressures, many engineers simply subclassed the existing classes, such as HarvestableDecoration and RotatableDecoration, and created a child one called HarvestableRotatableDecoration! As you can imagine this is bad on multiple fronts.

During the development of the expansion set, Genie’s Lamp, we eventually got to a point where having to deliver an entirely new game mechanic allowed us to revisit the issue and fix this inheritance hell. The solution goes back to Comp Sci 101: composition. Whereas inheritance provides an IS-A relationship, composition provides a HAS-A relationship. In languages that are single inheritance such as AS3, a class can be only one thing, but it can have multiple things. So we introduced the concept of a Behavior. All of the previous attributes about decorations: harvestable, rotatable, additional energy, and so forth were made into behaviors. This allowed decorations to have multiple behaviors. It’s obviously more complex than that, for example, we had to introduce a state machine into decorations so that they knew how to understand and process the behaviors, but that is the gist of it.

Backbone.js Bind Helper

There are a ton of Javascript MV* frameworks out there:

http://addyosmani.github.com/todomvc/

Of them, Backbone.js has garnered a lot of buzz and traction partially, in my opinion, because it is so flexible. (Backbone.js falls into the “flexible” MV* camp versus some of the more opinionated MV* frameworks out there like Ember.js and Spine.) Being in the flexible camp, it is a little less full featured than some of its counterparts. One of the useful little snippet of code that we use at Vurb is our bind helper. In a barebones Backbone.js application, developers need to bind functions associated with the module for a view, collection or model otherwise they risk going into JS’s “this” purgatory:

http://stackoverflow.com/questions/133973/how-does-this-keyword-work-within-a-javascript-object-literal

Underscore, one of Backbone.js’s dependencies, provides helpful ways of doing this with _.bind() and _.bindAll(). These helpers allow you to state which context of “this” you want to associated with the scope of a function. For example, it is common to see this in Backbone.js apps (in this case a view):

someview.js:
define(function() {
   var SomeView = Backbone.View.extend({
      initialize : function() {
         _.bindAll(this, 'someFunction', 'doSomething'); // This becomes unmanageable after many functions.
      },
      someFunction : function() {
         // "this" is the expected context
         this.doSomething();
      },
      doSomething : function() {}
   });
   return SomeView;
});

Like anything, this is fine when you have only a handful of functions in the view. What happens when you have 10-20 functions? You need to manually add each function name to the _.bindAll() call. What happens when you need to rename functions to conform to a new naming scheme? The solution is to programmatically do it:

baseview.js:
define(function() {
   var BaseView = Backbone.View.extend({
      initialize : function(scope) {
         var that = this;
         scope = scope || this;
         _.each(scope, function(val, key) {
             if(_.isFunction(val) && key !== 'constructor' && key !== '__proto__' && key !== 'initialize') { // Note #1: Need to blacklist special function names
                 _.bindAll(that, key);
             }
         });
      } 
   }); 
   return SomeView; 
});

The snippet above will automatically bind  a view’s functions to the correct “this” context. You’ll notice that the view is now a BaseView. A base view was introduced so that all your other views can inherit off of it to take advantage of the snippet of code. An alternative is to simply make the bind helper a static function you can call from anywhere. I like the base view approach because then all your other views simply inherit off of it. With inheritance in JS, you just need to remember to call the parent’s constructor or initialize() function:

childview.js:
define(['views/baseview'],function(BaseView) {
   var ChildView = BaseView.extend({
      initialize : function() {
         ChildView.__super__.initialize.call(this, Object.getPrototypeOf(this)); // Note #2: When we have multiple levels of nesting we have to figure out the correct nested prototype chain to pass in.
      },
      someFunction : function() {
         // "this" is the expected context
         this.doSomething();
      },
      doSomething : function() {
         // "this" is in the expected context
      }
   });
   return ChildView;
});

I found this a lot more convenient than having each view, model and collection having to maintain its own _.bindAll() list. It sometimes even caused runtime exceptions to be thrown because of a typo from a rename. It helped remove from our code base tens, maybe hundreds, of lines of boilerplate code.

Some notes about the three notes in the code.

  • Note #1: We need to blacklist certain function names from being binded. These are may vary from browser to browser and from framework to framework.
  • Note #2: Things are simple when you have one level of inheritance. When things get deeper, you need to remember which prototype chain to pass in as the argument: it is the inverse of the total number of how deep the nesting is, e.g. if have ChildB->ChildA->BaseView. Then for ChildB you would pass in Object.getPrototypeOf(this); For ChildA you would pass in Object.getPrototypeOf(Object.getPrototypeOf(this)); And so forth. I’d love to get feedback about how to make this more elegant, but as is, I found this snippet to be worlds better than having to manually manage _.bindAll() for each view, controller and model.