epepep

So, I’ve made my first Android application. That was a nice experience, especially after having hacked J2ME on Symbian for a short while. Only sad thing is that I haven’t got an Android phone, so I can only use my app in the emulator, which is boring as hell, since it’s a GPS based application. Well, I’m getting ahead of myself – so first some background about me, and then some more about that app.

Sometime in my early twenties, I once spent something like two whole days flying around the world in Microsoft Flight Simulator, in realtime. My old friends still like to remind me of this slightly autistic and highly nerdy feat once in a while. For my own part, it’s mostly a reminder of the oceans of free time I had while studying at university (I distinctly remember spending a considerable amount of the hours over the pacific studying for some math course I took at the time).

These days, I don’t even have flight simulator installed, but my interest in airliners and flying still linger in the back of my mind, and when the skies are clear, I often find myself drifting off, thinking about the contrails I see, what aircraft it might be, where they’re heading and where they’re coming from. In the last year, this little hobby has gotten a bit more interesting, thanks to a site called flightradar24.com, which makes it easy to check if it actually was a 747 that just passed by overhead, or if it was some other four engine model. (As a side note, flightradar24.com got a lot of media attention while the Icelandic ash cloud from Eyjafjallajökull grounded most air traffic over Europe for a couple of weeks, since everyone was very keen on seeing all the planes that weren’t flying. That was sort of weird. Anyway.)

The only thing that isn’t perfect about flightradar24.com for my little hobby, is that I’m not very often at a computer when I spot aircraft, and that their site is very Javascript-based, and hence not very useful in a phone browser. To solve this issue, some on and off hacking for a couple of months have resulted in a solution:

• A web service that uses data from Flightradar24.com to list the planes currently closest to a certain location, and presenting it as HTML, JSON or HTML
• A small application for Symbian/J2ME that uses a phone’s GPS to query the mentioned web service
• A small application for Android that does the same

The Symbian version is still a bit too much of a hack to actually show to anyone else, but the server parts and the Android application is in a state which I feel is sort of usable. So, if you’re interested, here’s the application file for version 1.0.0 of What plane?, which will show you the five aircraft closest to where you are, their direction and distance, and some more detailed information on model, airline, where they’re heading, and so on.

WhatPlane-1.0.0.apk

…and for those anxious of you who absolutely need screenshots, here’s a couple of those:

List of closest aircraft

Aircraft details

For those interested, I plan to make the source available at some point in the future (that I haven’t done it already is mostly because I don’t know where it’s best to host it, and I haven’t got that much time to work on this). Also, it must be noted that all the data shown in the application is from flightradar24.com – and it’s really their data. I haven’t checked with them if it’s ok to use it in this way, but since they’re a free service, aggregating information mostly from enthusiasts sharing the data from their ADS-B receivers, I hope this application is in the spirit of their vision.

In a pet project I’m spending my nights working on (hopefully more about that in a later post), I found myself in need of a JSON parser, or deserializer, for J2ME/CLDC. A bit to my surprise, I found that such a thing was not easy to find, even with the whole of the internets at my disposal.

To summarize, it appears that there has been a JSON lib for J2ME up on json.org at some point, but at least I can’t find it any longer. Also, some project on java.net is popular to link to, but come on, no download link? No pre-compiled JAR-file?

Anyway, after asking over at stackoverflow.com and getting surprisingly few answers, at least I found a link to some code that was easy enough to grab.

As some kind of attempt to give back to the community, I upload the compiled JAR from that source code here. So if you need to serialize, deserialize, marshal or unmarshal JSON from J2ME/CLDC, grab this JAR and go ahead:

• Compiled JAR: json-me.jar
• Source code: json-me.tar.gz

The code is most likely a copy of the one that was previously posted on json.org, and is distributed under the json.org license according to the copyright notice in the source (most importantly: “The Software shall be used for Good, not Evil.”)

As a very tiny modification, I have added the methods remove and removeAll to the class JSONArray, since I really needed them. I hope you don’t mind too much.

Efter misären med mitt inställda Göteborgsvarv har jag idag försökt kompensera åtminstone delvis genom att cykla Nationaldagsloppet, vilket jag nu tänkte fira genom att blogga för första gången på tusen år.

Har inte deltagit i något cykellopp tidigare i mitt liv, så det blev även lite studiebesök i cykelvärlden. Några iakttagelser:

• Till skillnad från löpning är cykling lite av en materialsport. Det ställer nya krav på tävlingsdeltagaren i form av förberedelser. Medan löpning enbart kräver något så när rena kläder, bör du innan ett cykellopp åtminstone funderat på din utrustnings kondition. Rekommenderad miniminivå är att pumpa däcken och se till att kedjan är smord. (Varför man inte gemensamt har ordnat med tryckluft och munstycken till ett evenemang med hundratals cyklister i behov av detsamma är för mig oklart, men sådana är kanske konventionerna inom cykelsporten.)
• Angående material är det även så att medeldeltagaren i ett cykellopp har skaffat duktigt med sådant. Extremt tighta kläder, slimmade hjälmar och en cykel som väger tio gram och med millimetersmala däck är snarare regel än undantag. Om du dyker upp på en smutsig, dåligt underhållen cykel iförd sladdriga shorts, t-shirt och en hjälm som inte riktigt matchar finns det risk att du känner dig lite utanför.
• Om man har ovan nämnda, svindyra utrustning skulle man kunna tro att det faktiskt går undan, men så är inte nödvändigtvis fallet. Vissa cyklar trots sin svindyra utrustning inte särskilt fort och får “kramp” i första uppförsbacken efter 12 kilometer. För mig känns det då som man missförstått det hela för en maskerad och klätt ut sig till cyklist, snarare.
• Under ett cykellopp dricker man inte bara (som i löpning), utan bjuds även på ätbara saker. Banan och i viss mån bulle kändes naturligt, men saltgurka stack helt klart ut när det erbjöds efter 35 kilometer; det gick dock utmärkt, och min kropp gjorde gällande att även en kebabrulle och kanske en prinsesstårta också hade varit på sin plats.
• Överviktiga herrar i besvärande åtsittande kläder är festliga att cykla om i uppförsbackar. Sedan kör de om i nästa nedförsbacke. Jag är mycket imponerad av att deras kolfibercyklar med däck smala som skridskor kan rulla snabbare nedför än min ompumpade loser-hybrid.
• De som kan cykla på riktigt cyklar sjukt fort och ropar glatt “opp-opp-opp-opp” om de tycker du ligger i vägen.

Sammanfattningsvis mycket nöjd med detta evenemang, kombinerad motion och sociologisk studie. Om du läst ända hit är du förstås galet nyfiken på hur det egentligen gick för mig. 60 kilometer på 2:22 är väl knappast snabbt för en tränad cyklist, men jag är nöjd med tanke på att det är det längsta jag cyklat i mitt liv och att min vardagsdistans är runt fem kilometer till eller från jobbet. Om du är besatt, se de riktiga detaljerna från min GPS. (Man får zooma in lite för att se bättre, och ja – kartan visar bara drygt 57 kilometer, men jag antar att arrangören mätt mer noggrant än min GPS tydligen gör.)

I’m a software developer. I like abstractions, I enjoy mentally putting things in little boxes, structuring my mental model of the world, or at least the model of the problem I’m currently struggling with. I don’t know if it’s a requirement, but if you’re a developer, it certainly helps to be border-lining to obsessive about knowing how stuff works, in excruciating detail.

Having said that, I find it fascinating how problems in programming constantly make you hit your head against the wall of reality – the wall located where pretty, simple and linear abstractions meet the real world. You know, the real world which refuses to be dumbed down into simple rules, and even when you think you’ve trapped it with you’re rules, always breaks free with new twists and turns that you hadn’t thought of. This is sort of the essence – problems that I’ve been working with more or less since I started programming, problems which also appear very benign from a casual viewpoint, that keep coming back, and simply refuse to be solved in a way such that you can put them behind you.

Time is the first and most obvious case. That something as fundamental and trivial (you learn to read the watch when you’re what? Five? Six?) can be so complicated and easy to get wrong is truly fascinating. And then I haven’t even mentioned dates yet. Just starting to think about dates will get any programmer into trouble. Before you know it, you’re knee deep in Gregorian calendars and leap years – and even then, you won’t get it right. In fact, I’d go so far as saying that any non-trivial date/time calculation in software will contain at least one bug, or one special case that you hadn’t really thought of.

Once again, then I haven’t even mentioned time zones yet. Time zones, combined with daylight savings time, is the real killer, if you by pure miracle got the first date calculations right. At the company were I worked before, we had a standing joke two times a year – was this going to be the first day lights savings switch in the company’s history where we didn’t encounter a bug related to it? From what I can recall, we never had a bug free switch (in five years!).

Maps, or perhaps rather geographic locations, is the second thing that springs to mind as a seemingly straightforward thing, that just never ever gets right.

From the day we learned how to use a two dimensional map, I actually think we’re doomed into living in the wrong paradigm. In grade school we learn that north is up on the map, and using a ruler to measure distances on it is the way to go. Sure, both work. Sometimes. But equally often, it just almost works, it’s almost true, kind of. “Almost true”, deeply ingrained in your mind, happens to be a perfect and never ending source of interesting and subtle software bugs.

Add to that the fact that the earth isn’t really a sphere, but almost, and while we have attempted to meaure its non-spheriness, we came up with several conflicting measurements, all giving slightly different results when you try to express where you are on this not-really-a-sphere thing. Sometimes, different measurements are mixed, and you will have to use completely non-trivial transformations to convert from one to the other, but those transformations only work for very special circumstances.

Even if you get it right, someone will probably want to calculate the distance from where you think you are to some other point, which isn’t at all trivial on a sphere. But it wasn’t a sphere, right? Yep, right – not a sphere, and that makes it just even worse. Luckily, I (and probably not you) don’t need the last kind of precision very often, but that doesn’t help, because we will still try to use a ruler (or its digital equivalent) to measure the distance on a map – which won’t work, even if the teacher in fourth grade said it would.

Character encoding is my last example. Yes, getting characters to show up on your screen, more or less. Again, something very mundane and something a non-developer would take for granted. And yet, this is something that has been a problem in just about any application I’ve written, or seen being written, or used, for I don’t know how long. I guess being a swede, using the funny å, ä, ö characters a lot (or “Ã¥, ä and ö”, as I’ve learnt to know them from years of UTF-8/ISO-8859 mangling), doesn’t help, since you’re so much more exposed to the problem.

Closing up, I want to attempt finding something that these three problems have in common, something that make seemingly simple things so very complicated. My first guess would be that its the perceived simplicity that is the core in all three – all are stuff that we in our daily lives take for granted, talk about while at the same time not paying much attention to the details: we look at our watches, make appointments and write them in our calendars, decide to meet at certain locations and talk about how far it is to places. We never ever think about the underlying details and complexities while doing this, it’s all very intuitive to us. On the other hand, I think it’s this very intuition that trips us when developing software; the fact that we think we know this very well gets in the way details (that most of us actually don’t know).

Also, all three share that they in some way involve problems regarding frame of reference: time zone and daylight savings time problems are hard since you very easily get confused about what is fixed and what is relative. The map problems are very much the same – what is absolute in one frame of reference (“up”, “north”, “distance”, etc.), does not necessarily translate to some absolute in another, and what you’re doing is switching frame of reference, if it’s going from a projected map to WGS84, or if it’s transforming between two geodetic references. Character encoding, and translation from one encoding to another, is also only meaningful if you keep track of your frame or reference (“current encoding”) through every step of the process – something which is very obviously much easier said than done.

At first, I also thought about adding the theory of relativity to the problems above. I decided against it, since it’s really not something you deal with every day, and it’s also far beyond my field of expertise. Interestingly enough though, it also talks about things we perceive as intuitive (time, length and weight being more or less constant) and turns it on its head, although more profoundly than the examples above. Also, the solution very much lies in keeping track of your frame of reference.

This is also the conclusion: a perceived intuitivity about a problem, combined with transitions between different frames of reference, makes for a problem that you will come back to again and again.

This entry will be in english, since it’s more likely to be helpful to others googling for solutions to the same problem.

A colleague of mine has used a D-Link DNS-323 as his RAID/backup solution at home. Apparently it’s been working great for ages, until he recently updated its firmware, which also caused all files and directories containing non-ASCII characters (mostly å, ä and ö for us swedes) to be completely inaccessible; the windows sharing (Samba/SMB) wouldn’t show the directories at all, and although they showed up in FTP, you couldn’t really access them. Downgrading the firmware did not help.

The fun thing about the DNS-323, and the reason the colleague asked me for help, is that it runs Linux internally. Although it looks like a USB disk with a web interface, it actually has a full-featured operating system underneath. Well, close to full featured.

Googling for solutions, I found another swede, Martin Bergek, who had at least similar problems (by chance, I also happen to know who Martin is). It seems that older firmwares used CP850 for filename encoding, while newer versions use UTF-8. Probably some upgrade didn’t go as planned, leaving the filenames in CP850, while interpreting them as UTF-8. Decoding swedish CP850 characters as UTF-8 results in invalid multibyte characters, causing the programs to refuse to handle files and directories containing them.

Now for the fun part. There seems to be quite a community developing hacks for the DNS-323. Using the so called fun_plug, you can very easily enable an SSH server, and get access to lots of useful commands. In my case, SSH access in combination with the rsync command turned out to be the key.

My solution was to get a backup disk large enough to copy all the material from the DNS-323 (actually, my colleague had already thought of this and provided me with a disk for this purpose). Once all the files where copied from the DNS-323, it could be wiped and the files copied back, but this time with the correct filename encoding.

As mentioned above, most of the problem results from applications interpreting the characters as invalid UTF-8 codes, refusing to work with them altogether. Even basic stuff, like doing recursive file lists with rsync, fails if a directory contains the invalid characters, probably since the runtime’s string library can’t work with the resulting strings. Shell commands like cd handle the directories, though, even if the characters are displayed as garbage.

Fortunately, rsync includes a command line parameter called --iconv, which lets you override which encoding is used when interpreting the filenames. This way, you can interpret the names correctly, and they can be converted into proper UTF-8. The trick is that you have to do this on the DNS-323 side, otherwise the conversion will be done on the backup unit’s side, still causing errors attempting to do recursive file lists. (In case you connect the USB disk directly to the DNS-323, you won’t have to think about this – in my case, the backup disk was connected to my Ubuntu desktop.)

So, to sum things up, this was the killer command line that made it possible to copy all the files out of the DNS-323:

rsync -azv --iconv=cp850,utf8 /mnt/HD_a2/ per@asta:/media/usbdrive/dns323

(Obviously, you’ll have to replace the paths to whatever directories you’re using on the DNS-323 as well as for the backup disk.)