Events visualization with EvenDrops and KoExtensions (D3 and Knockout)

I have recently needed to visualize a set of events which occurred within a certain interval. Each event would have couple parameters and there would be multiple event lines. Let's say that you want to visualize the occurences of car sales in couple countries. For each sale you would also want to visualize the price and the mark of the sold car. Before writing everything from scratch, I have found EventDrops project which responded to the majority of my requirements. It had just one flaw and that is that there is no way to chart another characteristics for each event.

I have decided to add such possibility and since I am using KnockoutJS and binding to create all of my charts I have also decided to add EventDrops to my KoExtensions project - in order to make it's usage simplier. The resulting chart looks like this:

This example is available on GitHub as part of KoExtensions.

What I have added to the original event drops are the following possibilities:

• The chart now accepts generic collection instead of just a collection of dates. The developer in turn has to specify a function to get the date for each item
• The size of the event is dynamic
• The color of the event is dynamic
• Better possibility to provide a however action
• The size of the event can use logarithmic or linear scale
• Everything is available as KnockoutJS binding

The html is now really straightforward:

<div data-bind="eventDrops: carSales, chartOptions: carSalesOptions"></div>

The javascript behind this page contains a bit more to generate the data:

require(['knockout-3.2.0.debug', 'KoExtensions/koextbindings', 'KoExtensions/Charts/linechart', 'KoExtensions/Charts/piechart', 'KoExtensions/Charts/barchart'], function(ko) {
 function createRundomSales(country) {
  var event = {};
  var marks = ['Audi', 'BMW', 'Peugot', 'Skoda'];

  event.name = country;
  event.dates = [];
  
  var endTime = Date.now();
  var oneMonth = 30 * 24 * 60 * 60 * 1000;
  var startTime = endTime - oneMonth;

  var max = Math.floor(Math.random() * 80);
  for (var j = 0; j < max; j++) {
   var time = Math.floor((Math.random() * oneMonth)) + startTime;
   event.dates.push({
    timestamp: new Date(time),
    carMark: marks[Math.floor(Math.random() * 100) % 4],
    price: Math.random() * 100000
   });
  }

  return event;
 }


 function createSales() {
  var sales = [];
  var countries = ['France', 'Germany', 'Czech Republic', 'Spain'];
  countries.forEach(function(country) {
   var countrySales = createRundomSales(country);
   sales.push(countrySales);
  });
  return sales;
 }

 function TestViewModels() {
  var self = this;

  self.carSales = ko.observableArray([]);
  self.carSales(createSales());

  self.carSalesOptions = {
   eventColor: function (d) { return d.carMark; },
   eventSize: function (d) { return d.price; },
   eventDate: function (d) { return d.timestamp; },
   start: new Date(2014, 8, 1)
  };
 }

 var vm = new TestViewModels();

 ko.applyBindings(vm);
});

In this example the createSales and createRandomSales methods are just use to get testing data. Once the testing data is generated it is stored to the carSales observable collection. Any time this collection is changed the chart would be updated.

The sales collection looks a bit like this:

The carSalesOptions object contains the charting options. These tell to the event drops chart the necessary information to specify how big and which color should be used for the given event.

Unit testing Knockout applications

In ideal case any View Model in Knockout based application should be completely unit-testable. The View Model ofcourse interacts with other code but in majority of cases this would be either some UI code or the server side code, probably over REST API. The UI interaction should be minimal. If possible, the binding capabilities of Knockout should be leveraged. The REST API is not available while unit testing and thus has to be mocked or hidden by an abstraction layer. I went for the first option and this blog describes how to mock the AJAX calls while unit testing Knockout View Models. At the end I also provide information about the ChutzPah test runner and the way that the tests can be run from within Visual Studio.

The typical view model that I am using looks like the following one.

function AssetViewModel(){
    var self = this;
    self.city = ko.observable();
    self.country = ko.observable();
    self.size = ko.observable();
  
    self.load = function(){
        $.ajax("/api/assets/" + self.city(), {
            data: dto,
            type: "GET", contentType: "application/json",
            success: function (result) {
                self.updateData(data);
            }
        });
    }

    self.save = function () {
        var dto = self.toDto();
        self.isBusy(true);
        self.message("Saving...");

        $.ajax("/api/assets/", {
            data: dto,
            type: "POST", contentType: "application/json",
            success: function (result) {
                self.edit(false);
                self.isBusy(false);
                self.message(result.message);
                self.update(result.data);
            }
        });
    };

    self.update = function(updateData){
        self.city(updateData.City);
        self.country(updateData.Country);
    }
 
    self.toDto = function () {
        var model = new Object();
        model.City = self.city();
        model.Country = self.country();
        return JSON.stringify(model);
    };
}

You might thing that the toDto method is useless if one uses the Knockout Mapping plug-in, however in many cases the view models get much more complex and they can't be directly mapped to any kind of data transfer objects or domain objects. Other than that, nothing should be surprising here. The save method sends the dto over the wire and than treats the response.

The unit test

Nowadays one has a choice between multiple JavaScript testing frameworks, QUnit, Jasmine or Mocha being probably the most common choices - I am staying with QUnit. Testing the updateData with QUnit might look like this.

var vm;
function initTest() {
    
    var vm = new AssetViewModel();
}

$(function () {

    QUnit.module("ViewModels/AssetViewModel", {
        setup: initTest
    });

    QUnit.test("updateData sets correctly the city", function () {
        var data = {
            City: "Prague",
            Country:"Czech Republic"
        };
        vm.updateData(data);
        equal(vm.city(), "Prague");
    });
}

QUnit module function takes 2 parameters - name and a sort of configuration object. Configuration object can contain a setup and tearDown methods. Their usage and intend should be clear.

This test case is very simple for 2 reasons: it does not depend on any external resources and it executes synchronously.

QUnit has 3 assert methods which can be used in the tests:

• ok - One single argument which has to evaluate to true
• equal - Compare two values
• deepEqual - Recursively compare a objects properties

Asynchronous testing

Here is the test for the save method which calls the REST server interface.

function initTest() {
      $.mockjax({
        url: '/api/assets/Prague',
        type: 'GET',
        responseTime: 30,
        responseText: JSON.stringify({
            Name: "Prague",
            Country: "Czech Republic",
            Size: 20
        })
    });
}

$(function () {

    QUnit.module("ViewModels/AssetViewModel", {
        setup: initTest
    });

    QUnit.asyncTest("testing the load method", function () {    
        setTimeout(function () {
            ok(true, "Passed and ready to resume!");
            start();
            vm.load();
            QUnit.equal(vm.size(),20);
        }, 100);
    });
}

I am using MockJax library to mock the results of the REST calls. The initTest method setups the desired behavior of the REST service call, the test is executed after 100ms of waiting time. In this case the call is a GET and we define the response simply as JSON data. QUnit has a method for asynchronous tests called asyncTest.

Currently there is a small issue in MockJax regarding the way that incoming JSON values are handled. That might get fixed in future versions.

Mocking the server interface

Returning a simple JSON data may be sufficient for some case, for others however we would maybe like to verify the integrity of the data sent to the server, just like when testing the save method

var storedAssets = [];
function initTest() {
      $.mockjax({
        url: '/api/assets',
        type: 'POST',
        responseTime: 30,
        response: function (data) {
            storedAssets.push(JSON.parse(data.data));
        }
    });
}

$(function () {

    QUnit.module("ViewModels/AssetViewModel", {
        setup: initTest
    });

    QUnit.asyncTest("save asset - check the update of the size", function () {
        vm.size(10);
        vm.save();
        setTimeout(function () {
            ok(true, "Passed and ready to resume!");
            start();
            equal(storedAssets.length, 1);
            var storedAssets = storedCharges[0];
            equal(storedAssets.Size, vm.size());
        }, 100);
    });
}

In this case the save method passes the JSON data to the server side. The server is mocked by MockJax which only adds the data to a dump array, which can be then used to verify the integrity of the data.

Running Unit Tests in Visual Studio

There are 3 reasons for which I am using Visual Studion even for JavaScript project:

• Usually the application has some backend written in .NET and I don't want to use 2 IDEs for one single application.
• I can easily debug JS application from within VS. Of course Chrome's debugger is very useful as well - but if I can do everything from 1 IDE, why should I use other.
• ReSharper has really good static analysis of JavaScript and HTML files. That saves me a lot of time - typos, unknown references and other issue are catched before I run the application.
• I can run JavaScript unit tests right from the IDE.

To run the Unit Tests I am using ChutzPah test runner. ChutzPah internally uses the PhantomJS in-memory browser, and interprets the tests. While using this framework, one does not need the QUnit wrapper HTML page and the Unit Tests can be run as they are.

Note that ChutzPah already contains QUnit and you will obtain TimeOutException, if you try to add a reference to QUnit explicitly (http://chutzpah.codeplex.com/workitem/72).

Since your tests are just JavaScript files, without the HTML wrapper page, ChutzPah needs to know what libraries do your View Models reference and load them. This is handled using a configuration file chutzpah.json which has to be placed alongside the unit tests. The following is an example of configuration file that I am using for my tests.

{
    "Framework": "qunit",
    "References" : [
        { "Path": "../Scripts/jquery-2.1.0.js"},
        { "Path": "../Scripts/knockout-3.1.0.js"},
        { "Path": "../Scripts/jquery.mockjax.js"},       
        { "Path": "../Scripts/tech", "Include": "*.js"},
        { "Path": "../ViewModels", "Include": "*.js"}
    ]
}

JSON DateTime serialization

This is more a side note. Dates in JSON are serialized into ISO format. That is good, the problems is that if you try to deserialize an object which contains a date, the date comes out as a string. The reason of course is that since there is no type, the de-serializer does not know that given property is a date - and keeps the value as a string. You can read more on dates serialization in JSON here. Any time that you are mocking backend which handles dates you have to be aware of this fact. Remember the mock of the back-end which inserts the object to a dummy array that I have used above:

function initTest() {
      $.mockjax({
        url: '/api/assets',
        type: 'POST',
        responseTime: 30,
        response: function (data) {
            storedAssets.push(JSON.parse(data.data));
        }
    });
}

JSON.parse call we handle the dates as strings. If the ViewModel has a date property, you will have to convert it into string before testing the equality.

KnockoutJS and Google Maps binding

This post describes the integration between Google Maps and KnockoutJS. Concretely you can learn how to make the maps marker part of the View and automatically change it's position any time when the ViewModel behind changes. The ViewModel obviously has to contain the latitude and longitude positions of the point that you wish to visualize on the map.

Previously I have worked a bit with Silverlight/WPF which in general leaves one mark on a person: the preference for declarative definition of the UI leveraging the rich possibilities of data binding provided by the previously mentioned platforms. In this moment I have a small free-time project where I am visualizing a collection of points on a map. This post describes how to make the marker automatically change it's position after the model values behind changes. Just like in this picture bellow, where the position changes when user changes the values of latitude and longitude in the input boxes.

Since I like Model-View-ViewModel pattern I was looking for a framework to use this pattern in JS, obviously KnockoutJS saved me. The application that I am working on, has to visualize several markers on Google Maps. As far as I know there is no way to define the markers declaratively. You have to use JS:

marker = new google.maps.Marker({
  map:map,
  draggable:true,
  animation: google.maps.Animation.DROP,
  position: parliament
});
google.maps.event.addListener(marker, 'click', toggleBounce);

So let's say you have a ViewModel which holds a collection of interesting points, that will be visualized on the map. You have to iterate over this collection to show all of them on the map. One possible way around would be to use the subscribe method of KO. You could subscribe for example to the latitude of the point (assuming that the latitude would be an observable) and on any change perform the JS code. There is a better way.

Defining custom binding for Google Maps.

The way to go here is to define a custom binding, which will take care of the update of the point on the map, any time, that one of the observable properties (in basic scenario: latitude and longitude) would change.

ko.bindingHandlers.map = {
init: function (element, valueAccessor, allBindingsAccessor, viewModel) {
  var position = new google.maps.LatLng(allBindingsAccessor().latitude(), allBindingsAccessor().longitude());
  var marker = new google.maps.Marker({
    map: allBindingsAccessor().map,
    position: position,
    icon: 'Icons/star.png',
    title: name
  });
  viewModel._mapMarker = marker;
},
update: function (element, valueAccessor, allBindingsAccessor, viewModel) {
  var latlng = new google.maps.LatLng(allBindingsAccessor().latitude(), allBindingsAccessor().longitude());
  viewModel._mapMarker.setPosition(latlng);
 }
}

;<div data-bind="latitude: viewModel.Lat, longitude:viewModel.Lng, map:map" ></div>

So let's describe what is going on here. We have defined a map binding. This binding is used on a div element. Actually the type of the element is not important. There are also latitude and longitude bindings, which are not defined. That is because the map binding takes care of everything. The binding has two functions: init and update, first one called only once, the second one called every time the observable value changes.

The allBindingsAccessor parameter contains a collection of all sibling bindings passed to the element in the data-bind attribute. the valueAccessor, holds just the concrete binding (in this case the map value, because we are in definition of the map binding). So from the allBindingsAccessor we can easily obtain the values that we need:

allBindingsAccessor().latitude()
allBindingsAccessor().longitude()
allBindingsAccessor().map()

Notice that the map is passed to the binding in parameter (that is concretely the google.maps.Map object, not the DOM element. Once we have these values, there is nothing easier than to add the marker to the map.

And there is one important thing to do at the end – save the marker, somewhere so we can update it’s position later. Here again KO comes with rescue, because we can use the viewModel parameter passed to the binding and we can attach the marker to the ViewModel. Here I suppose that there is no existing variable with name _mapMarker in the viewModel and JS can happily add the variable to the ViewModel.

viewModel._mapMarker = marker; 

The update method has an easy job, because the marker has been stored, and we only need to update it's position.

viewModel._mapMarker.setPosition(latlng);

Almost full example

Just check it here on JsFiddle.

Possible improvements

One thing that I do not like about this, is the fact, that you have to pass the map as an argument to the binding and the div element has to be outside of the map. Coming from Silverlight/WPF you would like to do something like this:

<div id=”map_canvas”>
<div data-bind=”latitude: vm.Lat, longitude:vm.Lng”>Whatever I want to show on the map marker</div>
</div>

That is actually the beauty of declarative UI definition. You can save a lot of code only by composing the elements in the correct order. However this is not possible – at least I was not able to get it to work. I was close however:

init: function (element, valueAccessor, allBindingsAccessor, viewModel) {
var map = element.parentNode;
var googleMap = new google.maps.Map(.map.);
//add the pointer
var contentToAddToMarker = element;
}

Again thanks to KO, here the element variable represents the DOM element to which the binding is attached. If the div element is inside the map, than we can get the parent element (which is the div for the map) and we are able to create a new map on this element. The problem which I had is the once, the new map was created, the div elements nested inside the map disappeared. Even if that would work, some mechanism would have to be introduced, in order to create the map only the first time (in case there are more markers to show on the map) and store it somewhere (probably as global JS variable).

On the other hand, thanks to the element you can get all the div which should be for example given as the description to the marker.

Summary: KnouckoutJS is great. It lets me get rid of the bordelic JS code.

CodeProject

Using Xaml serialization to generate Design Time Data

Recently I have been working on a almost finished Silverlight project which was needed a big change in graphical interfaces, in other words I needed a designer to be able to change all the Pages and components in Blend.

The issue - well not real issues is that all the data was bounded to properties behind, and without the data, the desiner was not able to work change the UI.

Blend can help you in this case, thanks to it's ability to generate XAML data from an existing class. So you can just generate data from existing ViewModel. That is great, but the problem is that, the data generated by Blend is not always usable. When you have an account and Blend will generate the name of the account: "Rhoncus vulputate" and the string specifing currency "Ipsum hac phasellus", you are thinking, maybe I will have to do some manual changes. Well with one account it is ok, but if you have a list of accounts (like 10) and list of operations (another 10). Editing all the things manulally might not be the right approach.

Well it happens so that in our project, we are using AutoPoco for the data generation. If you do not know it, it is a great library for generating simple POCO's - it seems like the development has been stoped last year, but hopefully this project will live on.

Well basically this means that I have already a lot of meaningful data generated in the form of POCO's, later stored in the DB. I have also working application which connects to Web Services and generates the ViewModels on the client side, so I asked myself:
Why should'nt I just take these ViewModels, serialize them into XAML and givem them to Blend as Design Time Data Sources?

So my goal was to have the same data which I was using in execution time during design time. And as you can see at the two following pictures I got it to work.
Run-time

Design-time

So I set myself to try to do so:

First idea - use the XamlSerializer. The problem - this class is only presented in WPF - not in Silverlight. OK - I thought I will just use my ViewModels in WPF application.

First Problem: The ViewModels were not ready to be used in WPF. Concretely I had two issues:

• The INotifyDataErrorInfo interface does not exists (http://connect.microsoft.com/VisualStudio/feedback/details/568212/inotifydataerrorinfo-for-wpf)
• The Web Service proxies generated using Silverlight are not the sames as the proxies generated using WPF

Possible solulution to these issues:

• Try to make your ViewModel completely platform independent. Reuse a DTO's (Data Transfer Objects) on the server and client side (do not let VS to generate proxies for your DTO projects).
• Use #if SILVERLIGHT directive to specify which parts should be build for WPF and which for Silverlight.

Well this turned out to be too complicated. Maybe it is possible on a project which you start from the very beginning. But I already had several ViewModels and changing all of these would take too much time.

Second idea - I will stay in Silverlight and see if I can get the XamlSerializer working in Silverlight

The build-int XamlSerializer is available only for WPF but there is an open-source implementation done by David Poll, which is part of his Silverlight and Beyond Library.
The use of the serializer is described in the following blog post.

This saved my life, thought I had to do some changes.

• To determine which properties to serialize, the Serializer uses the description provided on this blog. So it will leave out all generic properties which are not read-only. In my case I had a lot of ObservableCollection properties which I wanted to serialize. Actually this was the reason to try this approach, because I did not want to edit these collections for Blend by hand. To solve this issue I had to modify the VisitProperty method of the Serializer in order to force him to serialize even the writable generic properties.
• Disable serialization of certain properties. Blend designer has problems with some special cases of XAML. Concretely I had problem with the serialized WCF proxies. When you generate a proxy, there are at least two classes generated. Service interface (ends with "Service") and Service implementation (ends with "Client"). Blend was unable to process the xaml when he was expecting property of type "Service" and I have given him "Client" it seems that it has problems regarding services and implementations. To overcome this issue I have introduced XamlSerializationVisibility attribute which can have two values (Visible and Hidden). This parameter allows me to specify to the XamlSerializer whether to seralize or not the property. Again a slight edit of the VisitProperty method was neede to check this attribute before serialization.
• And at last, the original WPF XamlSerializer does not serialize all basic types. In my case I had several Decimal values which I wanted to serialize and they were skipped out. This was solved quite easy. XamlSerializer contains a class BuiltInTypeConverter. This class contains a list of types which can be serialized by simple conversion to String (SupportedTypes. I have added to this collection the Decimal and DateTime and it just worked.

So thats it, grab the source code, try to serialize your ViewModels or whatever you need. Really a great thanks to David for the implementation of this class.