A new Javascript IPC library for KDB

Published

October 13, 2024

So … at “$Work” I’ve been doing quite a bit with our Dashboard product that is being used more and more to report on different aspects of the desk’s trading activities and KDB is a key part of delivering content. This of course has pushed us into the arms of Kx Systems’ c.js file (originally here).

This is another exercise in concission, but does come up short in a few areas, particularly serialisation. In no particular order, dates and times are all sent as the deprecated datetime types (-15 15h), symbols need to be encoded manually in the application code to have a leading backtick (e.g. "`VOD.L"), and all numbers are sent as float values (types -9 9h). In terms of deserialisation, it seems to be a GC nightmare in that it turns a table into an array of rows, each of which has a key/value property; there are comments in the file which mention this decision.

I’m particularly impressed by the way the author(s) have exploited the way the Javascript typed-arrays use an underlying ArrayBuffer and share this to decode int32_t and other types. It’s really rather cool. I don’t quite understand the decision not to use the BigInt64Array type; perhaps it was introduced later, perhpas the usability-hump of not being able to mix Number and BigInt types in an expression was deemed too “user hostile” to allow into the wild. The library doesn’t appear to support decompression, but then, if you own the websocket writer in KDB you wouldn’t encode your data with -18! anyway.

Writing KDB IPC decoders seems to have become something of a hobby, having written at least four of them for various employers in various languages, and more recently an open-sorce C++ library (here), with nice features like partial message decoding so you can get on with object-graph creation while the data is still being assembled into the network-receive buffer. Anyway, using the lessons learned I though I’d create one for Javascript that lets you be explicit about the types of things you want to encode: it has a KdbMonthAtom, for example, that contains a number that is encoded to an int32_t type and represents the number of months since January 2000. And so on, for KdbFunction, KdbProjections, KdbUnary and KdbTable; the latter does maintain primitive data in Javascript typed-arrays, does use BigInt64Array for int64_t values and only uses Array (with sub-objects) for KdbList. It’s obviously a deserialisation library too, the object-graph you get out on the Javsacript side is the same object-graph you write on the KDB side.

It’s obviously more faff to compose your message in the first place, since if you don’t just “stringify” everything (ugh), you’d end up with something like:

import * as Mg from 'ipc.js'

var func = new Mg.KdbSymbolAtom(".web.getTable")
var time = Mg.KdbTimeUtil.Timestamp.atomFromJsDate(new Date())
var request = new Mg.KdbList([func, time])
this.conn.sendRequest(request, this.onTableResponse)

It’s clearly a lot less faff on the KDB-side, since you won’t have to go through quite so much type-casting.

The library is hosted on Github here and the file you want is static/js/ipc.js.