My latest weekend project, called "Stella by Starlight" after a Charles Mingus recording, was to build an analytics-style dashboard for looking at random metrics and events generated by a faked-out backend. I started with these rules for myself:
- Write every 30-60 minutes. If I don't keep myself honest in journaling this, I'll never get around to writing about it.
- Use Scala for the backend service. More on this in a moment.
- Learn D3.js. This is a primary goal of this exercise.
- Maybe use Coffeescript. More on this too, below.
And now, my notes from periodic progress reports. I've annotated it with GitHub commits at each step; you can also peek at the whole repo.
Get started with Scalatra
Per my motivation to use Scala, I figured I'd start with Scalatra. It integrates with Akka, which seems like a great thing, sticks pretty close to the Sinatra-style of web app construction, and seems like its probably approachable by a Scala beginner.
Strike 1 is that you need to use this giter8 tool. Luckily, it's available via Homebrew, so it's not a blocker.
Strike 2 is all the work needed to setup a Coffeescript gizmo to work with Scalatra. I've been blocked on SBT-related stuff before, and the instructions don't match the style of SBT project that was generated by giter8.
So I have a decision to make: should I plow forward with Scala and drop CoffeeScript or make a strategic retreat to the comfortable land of Ruby where there's probably a thing that will automatically compile CoffeeScript every time I hit a page?
For this weekend, I'm going to tradeoff getting better with Scala instead of CoffeeScript. The latter's domain is browsers, a domain I have chosen not to optimize myself for.
First commit: I have no idea what I'm doing.
Boilerplates
With one decision down, I need to figure out what my first real milestone is. It seems like a spike/prototype project like this requires two setup steps before real work can begin:
- Put all the project boilerplate in place. Get the backend service running and responding to requests. Decide on any front-end boilerplate I'm going to use (Twitter bootstrap, et. al.) and put it where your server will hand it off to a browser.
- Get your feedback loop working. Make a trivial change to the backend app, make sure it appears in the front-end. Make a front-end change and make sure everything changes properly.
Once I've got these two nailed down, I'm ready to actually iterate on the idea in your head.
I've got the Scalatra part of this working, and just fetched Bootstrap. Now I just need to get its boilerplate working and I can start actually working on an analytics dashboard.
Templates and cargo cults
So Scala can manipulate XML as a language-level thing. This is both terrifying and, in the case of emitting HTML inline within a Scalatra action, useful. But the limit of this is that not-quite-valid XML, but perfectly reasonable HTML, will cause your Scala program to flat-out not compile. Ergo, I decided it was time to bite the bullet and move my HTML bits into an actual template (commit).
That turned out to be pretty easy. Read the Scalate (Scalatra view templates) docs, skim the actual Scalate docs and you're mostly good to go. The only catch is that I had pre-generated HAML-style templates laying around which were causing a weird error about title not being defined. Once I figured out I had cruft laying around and killed it dead, all was pretty good.
I cargo culted all the CSS from a Twitter Bootstrap example and ended up with something decent looking. Note to past-self: HTML and CSS are terrible, but things like Bootstrap will make it reasonably possible to put up a decent-looking app quickly without needing a designer or browser-bug expert.
The change loop for Scalatra is nice and quick when doing front-end work. The SBT feature that watches the filesystem for changes and automatically runs tasks on change is pretty handy and, IMO, a better place for that functionality than in something like Guard.
Let there be charts
Now I want to get Cubism in place. At first glance, I thought I was goinging against the grain here. Cubism has a slight tendency towards using Graphite or Cube as the metric source. However, the demo page for Cubism shows some charts using random data.
Peeking at the source showed me the way to creating a data source that isn't pulling from Graphite or Cube (commit). This saved me the effort of trying to reverse engineer the Graphite/Cube query APIs before I could make any progress at all.
This points to an important lesson of prototyping: when in doubt, steal from the example that looks like what I want to do. It's totally OK to cargo-cult things into your system at this point. Later on, I can do it with software engineering and craftsmanship. In the present, I want to make progress on exploring my idea.
Random numbers as a service
Now I want to emit some random numbers via JSON from the service. This ended up being a not-so-tough journey into actual Scala. Turns out the JSON support in Scalatra is pretty straight-forward. I had to take a side-trip through JodaTime, a library I'd heard about before but never worked with directly. Of course, that resulted in some temporary Maven confusion, but all was well in the end (commit).
I was pleased by how one can go about quickly emitting JSON from a Scalatra action. What you do is write a case class (somewhat analogous to a Ruby struct, but with more tricks up it s sleeve) for the structure you're going to convert to JSON. Then you return one or more instances of that class from your action and the library handles the rest. All of this mostly made sense when I read the examples and converted it to my own thing, so I guess the basics of Scala are starting to stick. Happy!
Better random numbers, an attempt
I wanted to generate more realistic data from the service. I figured I would port the random metric generator from the Cubism example's JavaScript to Scala. This would make it easier for me to grok the timeline windowing scheme that Cubism uses.
It ended up that porting this algorithm was a bit trickier than I thought. Oddly enough, you can paste the crux of the algorithm from JavaScript to Scala and it looks like valid Scala. However, doing so gave me compiler errors that took me a little while to work out. Basically, the algorithm expects to work with doubles, but the compiler infers integers if you specify any default value such as start = 0. Adding type annotations to declarations resolves all of this. With that worked out, making the Scala compiler happy was a little more obvious.
It turned out that the Cubism example, as I cargo culted it, passes timestamp strings to the service. It was getting late and the first few things I tried to parse timestamps in Scala didn't work out, so I decided to call it there (embarrassingly broken commit).
How'd I do?
On the bright side, I didn't get hung up on Maven dependencies, I roughed my way through the Scala type system, and I had a pleasant experience with Scalatra and Cubism. On the downside, I didn't get to streaming events to the browser from the server and I couldn't quite get random metrics flowing from the server into the browser.
These weekend hacks are like that. I learn about things I expected to learn about and I learn about entirely different things too. I didn't expect to find myself pressing ahead with Scala, but doing so was an entirely different kind of educational fun than if I hadn't.
The nice things about these weekend hacks is that they're just that; a hack over the weekend. It's not a big project that I am responsible for afterwards. But it's still enough progress that I can write about it here and share it on GitHub. That feels productive. Learning plus productivity feels really good!
