Setting up u1sync in Ubuntu 11.10, Oneiric Ocelot

Maybe I’m just dumb or something, but I was sort of shocked of how hard it was to find any information at all about what had happened to the u1sync utility or why the package ubuntuone-client-tools had been removed. At last, I found that u1sync is now hosted on Launchpad – it doesn’t appear to be part of any package, there doesn’t appear to be any documentation, there isn’t even a downloadable file. You can get the latest code (no tags or branches or anything fancy here) using Bazaar, the VCS that isn’t popular anywhere. Sorry to be sarcastic, but it’s pretty much the caricature of an open source project. On the bright side: the code is there, someone has made this utility and is sort of maintaining it – I love you for doing this!

So, to get u1sync on your machine:

1. Install Bazaar:
   sudo apt-get install bzr
2. Download the latest code to your working directory:
   bzr branch lp:u1sync
3. Install the code:
   cd u1sync
sudo python setup.py install


…and we are more or less back to where we left off in my previous blog post. One minor issue: if you used the u1sync utility from the old package, you might get the error message:
ImportError: No module named u1sync.genericmerge
when running u1sync. In this case, you have an old file called .ubuntuone-sync/local-index in your synced folder – open the file and find the text ubuntuone.u1sync.genericmerge – remove the first part, so that only u1sync.genericmerge is left. (Yes, yikes! I spent a good part of a day swearing over this.)

Getting an authorization token for Ubuntu One using command line

A lot of the comments on my earlier blog post revolved around the fact that I reused the oauth token from my desktop machine when doing the backup from my server. That was an imperfect solution which resulted in file conflicts and other issues. On the other hand, not even Stuart Langridge, Technical architect for Ubuntu One, could post correct instructions on getting a new token from the command line

After digging in to this a bit more, writing some code of my own to do it, I finally found that of course someone else already did it. Roman Yepishev wrote a small script to create a new oauth token for Ubuntu One – it’s pretty self instructive if you use it. It dumps the new token to stdout, and you stick it into the u1sync commandline. Exactly what I wanted.

Hope this was helpful in getting Ubuntu One from the command line set up for you!

Ok, let me start by stating that I know, and you should also know, that Ubuntu One and the other cloud storage services like DropBox are not for serious backups. So if you’re reading this post to learn something about backing up your enterprisy data, you have come to the wrong place. But if you’re like me, and have a server in your closet as a hobby, and just want some sort of backup that’s also off site, you have come to the right place.

The problem at hand: I have a server in my closet at home, and I store some stuff on it that I would be very sorry if I lost. (I also store a lot more on it that I don’t care the least about.) So, I want to back that stuff up. It’s primarily a couple of MySQL databases and some version control repositories.

Ubuntu comes with a cloud storage service called Ubuntu One. Ubuntu One gives you a free account with two gigabytes of storage, and you can pay a small subscription fee to get significantly more storage if you need it. While there are lots of other similar solutions, Ubuntu One seems like a nice choice if you’re into Ubuntu, since its neatly integrated into the Gnome desktop.

The bad news is that for a headless (no screen) machine, like my server, Ubuntu One is currently a bit harder to set up. I really need the backups to work even if I don’t start a remote session and type my password, and so on. A backup that requires manual intervention is about as bad as no backup at all, in my opinion.

I was sort of frustrated when I found this out, and Twittered angrily about the lack of command line interface. A couple of days later, I got a very friendly reply from Stuart Langridge (@sil), who happens to be technical architect for Ubuntu One. Wow. Not only a response to my rant, but a short conversation ended up with enough clues for me to solve the problem at hand.

So, to save you from having to finding out the details yourself, here is how you could go about making Ubuntu One work as a backup for your headless server:

Install ubuntuone-client-tools

The program you need is called u1sync, and is distributed in a package called ubuntuone-client-tools. Install it:

sudo apt-get install ubuntuone-client-tools

Get an authentication token

The tricky part about getting Ubuntu One working standalone is really authentication. Normally, Ubuntu One authenticates using information stored in your Gnome keyring. On the server, you might not even have the keyring deamon installed, and even if it is, the backup must work even if you haven’t entered your keyring password, and so on. Fortunately, u1sync can authenticate without the keyring, but you will have to provide an OAuth token manually. Getting that token feels sort of like a hack, but this is how I do it:

1. Go to a desktop machine where you have Ubuntu One set up with the account you want to use
2. Go to System menu -> Preferences -> Passwords and Encryption Keys
3. Under the tab Passwords, you will find an entry called UbuntuOne token for https://ubuntuone.com – double click it.
4. In the dialog, open the password label and mark the Show password checkbox. It should now look something like this:

   

5. Copy the text from the password field into a text editor of your choice. It will look like this:
   oauth_token_secret=xxxx&oauth_token=yyyy, but with xxxx and yyyy replaced with much more gibberish. It’s those to parts of the strings you’re after.
6. In your backup script on your server (you have one of those already, right? Mine is based around AutoMySQLBackup and lots of tar commands), insert this line somewhere at the end:
   u1sync --oauth=yyyy:xxxx [your backup folder]
   Note the order of things: u1sync needs to oauth token (key) first, and the secret as a seconds argument; at least for me, they were stored in the other way around in the keyring string.

And that’s about it! The u1sync command will automatically synchronize the directory contents against your account every time the script is run. It will also output a lot of information about what it’s doing, so if you’re running the backup script from cron (you should), you might want to redirect the output to a log or /dev/null, or you will get lots of long mails.

Also note, that if you for some reason unauthorize the machine you got the OAuth token from, you will also unauthorize the backup server. Unauthorizing a machine really means revoking the OAuth token, and since the machines share the same token, both will now be unauthorized.

Finally, if you find out or already know a simpler way to get the OAuth token, I really interested in hearing about it. And yay for Ubuntu (One) and Stuart Langridge, thanks to them I now have an easy and free backup solution for at least the most critical stuff.

]]> http://per.liedman.net/2011/01/22/using-ubuntu-one-for-backup-on-a-headless-server/feed/ 11 http://per.liedman.net/2010/12/08/213/ http://per.liedman.net/2010/12/08/213/#comments Wed, 08 Dec 2010 11:51:37 +0000 Per Liedman http://per.liedman.net/?p=213 När Svenska Dagbladet publicerar en artikel om familjen Bramfors-Engelmarks försök att anpassa sitt liv efter klimathotet dröjer det inte länge innan det twittras i en lätt hånfull ton:

Är den här snubben verklighetens svar på uppdiktade bloggen Söderfamiljen? http://tinyurl.com/38d5vd2

skriver Eric Rosén (@detljuvalivet), och

Medelklassen tar till sig klimatpolitiken: “För ett par år sedan hade vi också en mindre Mallorca-kris” http://bit.ly/hPq3Ke i @svdse

skriver Jesper Nilsson (@redundans).

Jag har svårt att se vari problemet med artikeln egentligen ligger, bortom själva formen av utslätat familje-reportage med Sara Trus-inslag, som knappast familjen själv kan klandras för. Man kan naturligtvis slå sig för knäna över den senaste i den oändliga räckan av vad som minst sagt utslitet brukar kallas för i-landsproblem: “Mallorca-kris”, och en mängd andra ordval i artikeln, men bortom det beskrivs ett reellt problem: människor som har en övertygelse (klimathotet är högst reellt och överhängande) som är i konflikt med mycket i deras livssituation.

Jag är inte helt på det klara med var kärnan i Rosén och Nilssons kritik ligger, men det verkar ligga någonstans mellan den upplevda självgodheten: “det finns något väldigt dömande i att han är övertygad om att det redan är kört, men att han är en av få som kommit till insikt.” skriver Rosén i en senare tweet, och att familjen tar på sig ett personligt ansvar: “Om man hanterar klimatproblemet med konsumtionshets och ångest istället för politik skall man hånas.” skriver Nilsson. Det sistnämnda blir för mig lätt obegripligt i då jag varken vet var det står om konsumtionshets, mer än att de försöker handla ekologiskt, och uttryckligen nämner en politisk frustration:

Hon beskriver sig själv som en ganska realistisk person som mest känner ilska över att politikerna, som sitter på den verkliga makten att förändra, inte gör mer.

Utan att veta något om hur varken Eric Rosén eller Jesper Nilssons familjesituationer, förvånas jag snarare över att de inte ger uttryck för den minsta igenkänning. Jag vet inte om det beror på att de själva ignorerar all oro för klimatet, eller om de lever i klimatneutralt liv utan konflikter med andra intressen. Själv känner jag däremot igen mig mycket i Bramfors-Engelmarks bekymmer, vilket troligen är anledningen att jag överhuvudtaget orkar irritera mig över den raljanta tonen mot dem.

Ur en barnfamiljs perspektiv upplever jag dagens Sverige ur miljöperspektiv ungefär som att vara en gravt överviktig på julbord: ett överflöd av saker du verkligen inte borde ta av, men både smarriga och som du snarast förväntas ta så mycket du kan av. Att förklara att man inte vill äta, d.v.s. köpa en större bil, större hus, resa till Thailand och varför inte en teppanyaki-häll, blir vare sig man vill eller inte till slut något av ett socialt problem, så vida man inte har turen att enbart omges av andra personer med samma värderingar. För mig är det detta artikeln handlar om, inte att familjen upplever sig som goda, eller deras politiska visioner (eller avsknad av dessa).

Att inte kunna se bakom det dåliga ordvalet “Mallorca-kris”, d.v.s. problematiken i att förklara för barn och släkt att man inte vill bli bjuden på solsemester för att man av princip inte vill flyga, tyder bara på en tröttsam brist på insikt.

Utan att gå in på varför och vad man skall göra åt det, så menar jag att det i allra högsta grad är relevant att illustrera de här problemen, så som SVD gör. All politisk förändring börjar inte med att skrika sig hes om klass och blåsa i vuvuzela.

Uppdatering: Eric Rosén påtalar att han visst uttrycker förståelse för problematiken i en annan twitter-tråd: “självklart är en del av provokationen att det tangerar delar av ens eget liv. Ett uttryck för twittrarens egen klimatångest”, så helt svartvitt är det förstås inte (i samma tråd menar han dock också att Mikael Bramfors “låter som en pajas”).

]]> http://per.liedman.net/2010/12/08/213/feed/ 1 http://per.liedman.net/2009/03/04/reality-bites/ http://per.liedman.net/2009/03/04/reality-bites/#comments Wed, 04 Mar 2009 17:43:51 +0000 Per Liedman http://per.liedman.net/?p=135 I’m a software developer. I like abstractions, I enjoy mentally putting things in little boxes, structuring my mental model of the world, or at least the model of the problem I’m currently struggling with. I don’t know if it’s a requirement, but if you’re a developer, it certainly helps to be border-lining to obsessive about knowing how stuff works, in excruciating detail.

Having said that, I find it fascinating how problems in programming constantly make you hit your head against the wall of reality – the wall located where pretty, simple and linear abstractions meet the real world. You know, the real world which refuses to be dumbed down into simple rules, and even when you think you’ve trapped it with you’re rules, always breaks free with new twists and turns that you hadn’t thought of. This is sort of the essence – problems that I’ve been working with more or less since I started programming, problems which also appear very benign from a casual viewpoint, that keep coming back, and simply refuse to be solved in a way such that you can put them behind you.

Time is the first and most obvious case. That something as fundamental and trivial (you learn to read the watch when you’re what? Five? Six?) can be so complicated and easy to get wrong is truly fascinating. And then I haven’t even mentioned dates yet. Just starting to think about dates will get any programmer into trouble. Before you know it, you’re knee deep in Gregorian calendars and leap years – and even then, you won’t get it right. In fact, I’d go so far as saying that any non-trivial date/time calculation in software will contain at least one bug, or one special case that you hadn’t really thought of.

Once again, then I haven’t even mentioned time zones yet. Time zones, combined with daylight savings time, is the real killer, if you by pure miracle got the first date calculations right. At the company were I worked before, we had a standing joke two times a year – was this going to be the first day lights savings switch in the company’s history where we didn’t encounter a bug related to it? From what I can recall, we never had a bug free switch (in five years!).

Maps, or perhaps rather geographic locations, is the second thing that springs to mind as a seemingly straightforward thing, that just never ever gets right.

From the day we learned how to use a two dimensional map, I actually think we’re doomed into living in the wrong paradigm. In grade school we learn that north is up on the map, and using a ruler to measure distances on it is the way to go. Sure, both work. Sometimes. But equally often, it just almost works, it’s almost true, kind of. “Almost true”, deeply ingrained in your mind, happens to be a perfect and never ending source of interesting and subtle software bugs.

Add to that the fact that the earth isn’t really a sphere, but almost, and while we have attempted to meaure its non-spheriness, we came up with several conflicting measurements, all giving slightly different results when you try to express where you are on this not-really-a-sphere thing. Sometimes, different measurements are mixed, and you will have to use completely non-trivial transformations to convert from one to the other, but those transformations only work for very special circumstances.

Even if you get it right, someone will probably want to calculate the distance from where you think you are to some other point, which isn’t at all trivial on a sphere. But it wasn’t a sphere, right? Yep, right – not a sphere, and that makes it just even worse. Luckily, I (and probably not you) don’t need the last kind of precision very often, but that doesn’t help, because we will still try to use a ruler (or its digital equivalent) to measure the distance on a map – which won’t work, even if the teacher in fourth grade said it would.

Character encoding is my last example. Yes, getting characters to show up on your screen, more or less. Again, something very mundane and something a non-developer would take for granted. And yet, this is something that has been a problem in just about any application I’ve written, or seen being written, or used, for I don’t know how long. I guess being a swede, using the funny å, ä, ö characters a lot (or “Ã¥, ä and ö”, as I’ve learnt to know them from years of UTF-8/ISO-8859 mangling), doesn’t help, since you’re so much more exposed to the problem.

Closing up, I want to attempt finding something that these three problems have in common, something that make seemingly simple things so very complicated. My first guess would be that its the perceived simplicity that is the core in all three – all are stuff that we in our daily lives take for granted, talk about while at the same time not paying much attention to the details: we look at our watches, make appointments and write them in our calendars, decide to meet at certain locations and talk about how far it is to places. We never ever think about the underlying details and complexities while doing this, it’s all very intuitive to us. On the other hand, I think it’s this very intuition that trips us when developing software; the fact that we think we know this very well gets in the way details (that most of us actually don’t know).

Also, all three share that they in some way involve problems regarding frame of reference: time zone and daylight savings time problems are hard since you very easily get confused about what is fixed and what is relative. The map problems are very much the same – what is absolute in one frame of reference (“up”, “north”, “distance”, etc.), does not necessarily translate to some absolute in another, and what you’re doing is switching frame of reference, if it’s going from a projected map to WGS84, or if it’s transforming between two geodetic references. Character encoding, and translation from one encoding to another, is also only meaningful if you keep track of your frame or reference (“current encoding”) through every step of the process – something which is very obviously much easier said than done.

At first, I also thought about adding the theory of relativity to the problems above. I decided against it, since it’s really not something you deal with every day, and it’s also far beyond my field of expertise. Interestingly enough though, it also talks about things we perceive as intuitive (time, length and weight being more or less constant) and turns it on its head, although more profoundly than the examples above. Also, the solution very much lies in keeping track of your frame of reference.

This is also the conclusion: a perceived intuitivity about a problem, combined with transitions between different frames of reference, makes for a problem that you will come back to again and again.

]]> http://per.liedman.net/2009/03/04/reality-bites/feed/ 0