I suddenly switched to Mozilla Thunderbird for email.

I'm still using the local DBmail IMAP4/Mysql backend, with fetchmail to pull down email from the server, but switching from Claws-mail was surprisingly easy. I get to get away from the annoying UI quirks of Claws, and, curiously, I can't really remember what it was that prevented me from moving to Thunderbird in the first place.

Before 4am this morning, our cable went out. The TV also appears not to work. It's not in my name, so I'm SOL for the second time.

I am, however, connected using my Fido phone via USB modem. It was IMPRESSIVELY easy to connect using the Network Manager stack on Ubuntu Karmic Koala, unlike on previous releases.

I plugged in the phone. An item to set up a new mobile broadband connection magically appeared on the Network Manager widget's menu, taking me through the connection setup wizard. Fido was in the list, and all the default settings worked fine. Unlike before, I did not need to even install the blueman bluetooth connection manager, since I was usign the USB connection instead. All I had to do was turn off my wired LAN connection, and I was online.

Needless to say, it's not as fast as cable, but it's a damn sight better than dialup. It's more than adequate for the web and ssh sessions, if not for streaming media or Youtube.

Nice work.

The Multi-Pointer X extension to the X Windows server is included in the alpha version of the next release of Ubuntu, entitled "Lucid Lynx", coming in the spring. This will allow you to finally have multiple mouse cursors, each controlled by a different mouse.

I've been anticipating this for a while; too often using the mouse is like trying to work with three fingers and one hand tied behind your back.

Now how long until we get a usable open-source voice recognition library?

I think I just figured out how to mount multiple ecryptfs mounts as the same user.

Because ecryptfs is a kernel filesystem, and not a FUSE userspace fs (like encfs), it requires root privs to mount and unmount. To mount the user's private directory, a setuid script is configured to perform the mount, but it's not usable on any other filesystem for the same user.

I would imagine that putting an entry for the ecryptfs into /etc/fstab which gives users permission to mount it would work just like it does for any other filesystem.

Furthermore, there's an annoying additional step required before one actually runs the mount command for any ecryptfs filesystem: you have to add the mount's keys to the kernel keyring of the user performing the mount. I was stymied for quite a while by the fact that I was adding the keys to the keying of my ordinary user, but the mount was looking for them on the root user's keyring instead, since I had to use sudo to gain root privileges to perform the mount.

Hopefully I can make this work to simplify things.

I have been having an interesting experience with encrypted filesystems, sudo, su, and ssh public key authentication.

I like the transparent ecryptfs full home-directory encryption that was deployed with Ubuntu 9.10. However, I still feel kind of nervous having everything accessible all the time I'm logged in. As long as I'm logged in, my entire private home directory is unencrypted and accessible to all of my processes and the root user. If the PC is seized and shutdown, the encryption will protect it unless I'm forced to divulge the password. But what if, say, malware were to compromise my web browser? If I can read it, and a worm or virus achieves my user's privileges (or more), it's already defeated the encryption.

I would really like to have certain compartments of encrypted directories containing only certain data, that I can lock up after using without having to shut down my entire home directory. Call me paranoid.

Encfs, which I used previously, was a FUSE filesystem, meaning it was implemented in userspace. Thus, I could have lots of little closets and compartments that I could put anywhere I wanted. I even created AppDir scripts, so that I could just click on a folder in ROX filer and have the thing instantly prompt me for a password and mount it. Fairly convenient.

But the implementation of ecryptfs in Ubuntu, while faster and slicker, is not nearly as flexible. You either get an encrypted private directory, or an encrypted home directory, and that's it. You can't even have both, never mind more than two.

One thing I have noticed about ecryptfs mounts, however, is that they are accessible to all users, unlike FUSE filesystems (at lest by default). So, once mounted, it's easy enough to chown subdirectories of a mounted ecryptfs directory over to a different user entirely. So that's just what I did.

The first thing I needed was an ecryptfs file space for the mysql user, mounted, oddly enough, on /var/lib/mysql rather than in /home/. This worked brilliantly, since /var/lib/mysql/ is the default home directory of the mysql user anyway. I just opened up a shell for the mysql user (after allowing it in /etc/passwd), and ran the ecryptfs setup scripts. It was pleasantly surprising. Now my dbmail/mysql database containing all my email needs to be mounted with a decryption password as well as my home directory.

Next, I created another user, "crypt", to store porn other sensitive things that I need less often. Email pours in constantly, so leaving the MySQL encrypted filesystem up permanently (except after reboot) was warranted. This other stuff, not so much.

It was easy enough to create crypt's encrypted home directory, and then create and change ownership on a subdirectory for nato (my main account). But making it easy for my user to mount and dismount crypt's ecryptfs took a little doing. I spent a frustrating period of time trying to get this to work smoothly using su, sudo, and gksu.

Ecryptfs typically auto-mounts your encrypted home for any session you have logged in for. It even re-mounts it for any cron jobs you run in the background, provided you have logged in at least once, causing your keys to be saved inthe kernel's session keyring. When you logout, or your cron job completes, it auto-dismounts your private directory. What I did NOT want to do was have to open a terminal logged in as the crypt user and have to keep it open with nothing going on just to have my ecryptfs mounted. It turns out, fortunately, that if you run the ecryptfs-mount-private helper script AFTER you've already logged in and had your ecryptfs auto-mounted, the system makes a note of this, and doesn't auto-dismount your private directory when you log out - until you manually run the dismount helper script at least once. I think I read somewhere deep in a mailing list that ecryptfs now does some kid of instance-tracking to make sure it's got the nesting right to make this work.

Surprisingly, I ended up abandoning local terminals and su commands for the use of ssh - on the localhost, believe it or not. So I'd login to the localhost as crypt, be prompted for crypt's password. The system would auto-mount the ecryptfs and drop me to a shell as crypt. If I jsut logged out from there, the system would auto-dismount, but if I ran ecryptfs-mount-private, and THEN logged out, it would stay up. What's more, I could just run "ssh crypt@localhost ecryptfs-mount-private" , and get the same effect. I could even run this from a script, rather than from a terminal, becuase ssh knows how to prompt using the ssh-askpass GUI if it detects that it isn't in a TTY and has an X display available.

Then I decided to set up ssh public key authentication, and got a pleasant surprise. At this stage, the mounting was easy enough, but one drawback was that in order to dismount the ecryptfs, I'd have to log back in with crypt's password. That shouldn't be necessary to dismount an encrypted filesystem. At first, I thought enabling public key authentication would defeat the purpose of requiring a password to log in - except, it didn't. First off, you can always specify symmetrically encrypted passphrases for keys on the client side of ssh. But that wasn't necessary. When you want to use a public key to login to a remote host via ssh, the authorized keys are stored in the ~/.ssh/ directory for the user you want to login as. But if that user's home isn't mounted, then you have to fall back to using the password until you login, and get your encrypted home directory mounted. THEN, the ~/.ssh/authorized_keys file is available, meaning that all subsequent logins are handled by the public key. Clever. So the mount command requires the password - and the dismount does not. How nice.

it appears my fortnight of butting my head up against dbmail, ecryptfs, mysql, and sqlite is nearing its end.

My error in attempting to migrate from the Mysql to the SQLite database backend for dbmail became apparent as I began to import three months of mail archives dumped into mbox files through the dbmail IMAP4 server from the old mysql backend. It's somewhere on the order of 15,000-20,000 emails (I get a LOT of email). Once I got up and over ten thousand, the performance slowed down a lot. Even inserting new mail using fetchmail took a second or two per message. Some filtering and de-duping functions just plain stopped working, since they would always time out before completing.

So, it was back to Mysql. I now have most of the mail I've received since I deprecated the Mysql database dumped to mboxes and being transferred back via IMAP4 importation to the Mysql backend... and it's WAY faster, thank goodness.

To get here, though, I had to find a solution to my encryption problem, which is what drove me away from mysql in the first place. Well, no, not really. What drove me away was the fact that it's really not possible to encrypt SOME of a mysql database if it is required to be stored using the innodb storage engine. I had been trying to encrypt just the dbmail database, in its own folder. What was really called for was to just scrap the whole piecemeal and encrypt all of /var/lib/mysql wholesale.

At first, I was trying to do what I had been doing with the sqlite database, and symlink /var/lib/mysql to /home/nato/mysql . That would place the entire database space inside my encrypted home directory. Unfortunately, mysql would not, for reasons I did not investiagate throughly.

Instead, I just used the existing ecryptfs utilities to create a separate excryptfs encrypted file system just for the mysql database. That seemed to work great. All I had to do was tweak one file to make the mountpoint /var/lib/mysql .

So, now, I am celebrating, because my mail is now fast and usable once more. I can hardly believe it. It's been a depressing two weeks.

But, I win. Again. Through perseverance and patience. What a relief!

Now, on to that Google Wave preview I received from Jeff! It's a little slow, interface wise, but I'm digging it so far.

I spent all night last night and all day today importing several months worth of email from the mysql backend of dbmail into the sqlite backend of dbmail using Claws-mail to import mbox files over IMAP. I have been unable to use fetchmail to load new mail, and unable to browse existing mail. the dbmail-lmtp daemon that receives it from fatchmail sporadically complains that "the (sqlite) database is locked". shouldn't this be able to do two things at once? I didn't have this problem with MySQL.

To make matters worse, the IMAP performance of jsut browsing mail has slowed intolerably. Apparently, sqlite3 can't handle a GB database of email messages as well as Mysql can.

*wince*

I guess it's time to figure out how to migrate back to Mysql.

I also recall never seeing the corruption/header mangling problem on the Mysql backend either. I've even developed a theory about how it happens. It always only pops up when there are single quotes in the message header, and the symptom is that the header message block, as stored in the database is missing it's trailing characters. sometimes they're just missing the ending carriage returns; sometimes they're actually missing trailing characters on the final header line as well.

Could it be that dbmail is properly escaping single quotes in preparation for insertion into the database, by adding backslash characters as customary, but the forgetting to account for the extra length those escaping characters add to the header string? If the escaped string isn't the same length, but the length isn't updated, it might be chopping off the characters at the string's end by mistake.

When I finally get my email back to where I want (if ever), I may file a bug report with dbmail. but fuck this.

There is something terribly wrong with dbmail.

For some reason, it was mangling random email headers, such that the first line of the message body was being tacked onto the final line of the message header, minus one missing character on the end of the final header line.

I managed to isolate it down to the existence of at least two single quotes characters, which are not uncommon to find in the spam scan report tacked onto the header of each delivered message. dbmail does not parse it correctly when inserting it into the sqlite database - although it has never swallowed any messages whole, as far as I can tell.

I have worked around this by altering the exim4 mail server configuration to replace any single quotes int he spam report before inserting it.

What a horrible bug.

I'm settling into my new home on Ubuntu Karmic. It's taken a few days to migrate, clean, houskeep, prune, and integrate from it's home on the jaunty partition (and the encfs encrypted filesystem) onto the Karmic partition, which uses ecryptfs. It's all gone very smoothly, without a snag in sight.

It's startling when rox filer can fit every single file and folder in your home directory, including all the hidden files, into ONE screen. Neat.

I have also managed to get my dbmail email backend switched over from a mysql/innodb database to one using sqlite on my encrypted home. It was not difficult to discover the correct file permissions dbmail requires for the sqlite database file, and the dbmail IMAP daemon doesn't seem to mind that it's database is inaccessible until I login for the first time (the dbmail smtp mda runs entirely in userspace, so it doesn't mind either), and doesn't skip a beat when it becomes available on login. Not bad. I'm using fetchmail to transfer mail from my server to dbmail periodically.

Another great thing about ecryptfs is that, while yoru encrypted home is unmounted as soon as all your sessions are logged it, the kernel keeps your key on hand, just as long as you've logged in at least once, so any cron jobs that the user runs prompt the system to automatically re-mount the encrypted home partition before the cron job runs - and, of course, the partition is unmounted again when it completes. Pretty handy.

What this means is that all the authorized logins for my user will get access to the encrypted partition, so long as I have logged in with a password at least once since the last boot. As soon as the system shuts down, the keys are lost until I login again.

I still have to figure out what things to backup and what to delete, and reconstruct a remote backup scheme for everything. I have no fewer than *12* different places where email archives exist over the years; they need to be consolidated, de-duplicated, and stored in some uniform way.

I found a cache of old, old writings and music I was playing with from 2002 all the way back to 1998. That was fun.

After downloading the just-released Ubuntu 9.10 "Karmic Koala" live cd via bittorrent at a blistering 800-900Kb/second (seriously, it took 20 minutes to download, and 10 to burn to cd. wow.), I re-partitioned my hard disks and took it for a spin via a clean install.

If I recall correctly, I haven't done a clean install of an operating system since I switched from Debian to Ubuntu many years ago (and I'm not even sure about that - I may well have pulled off some weird hack to upgrade from Debian to Ubuntu without a clean install, but I doubt it). This is a tribute to the stability of Ubuntu, the underlying Debian OS, and the GNU/Linux bedrock at their foundation.

Nonetheless, there's been a LOT of cruft and hackery that have accrued to my filesystem over the years, and I felt the need to make a fresh start and explicitly import the parts of my data and settings I know I wanted to, rather than implicitly submit to whatever deeply ancient and subtle things which may lurk in my filesystem after years of being forgotten.

So, with an eye toward the future, and buoyed by my success with installing the Karmic beta release alongside the existing desktop backup image on my laptop, I decided against just upgrading my existing Jaunty Jackalope release on my desktop PC (an operation that has always gone quite smoothly), and decided instead to exercise the live cd installer's option to install Karmic alongside Jaunty, and dual-boot into another partition. The results have been excellent, and smooth. I'm quite happy with it.

Now, those who know me know I've been a GNU/Linux user for about ten years now. When Ubuntu came out after 2004, it prided itself on targeting average, non-technical desktop users. Since my introduction to GNU/Linux preceded the development of Ubuntu's suite of graphical tools, I knew how to do everything on the command line, instead. This turned out to be a liability with Ubuntu - not because of conflicts with the tools, but because friends of mine using Ubuntu on their own would often find and use the graphical tools to do things I only knew how to do with the obscure command line hacks. With this install, I was pleased to finally learn how to use those, and do things "the Ubuntu way". And it did everything pretty much as I wanted it too. I've put a lot of work into developing and maintaining my own solutions for things, but I am keen to relinquish that and use systems other developers maintain, if they work just as well or better.

First, I used Gparted, rather than fdisk, to edit my partitions, resizing the Jaunty partition down, and adding two smaller 16GB partitions, one for the karmic installation, and one reserved for future use. I plan to mount the Jaunty partition and use it's home directory for both installations, so I can boot back and forth if necessary with the same home directory data. Resizing filesystem partitions takes a while, which is normal, but it went flawlessly.

The second big change I'm going to make is to switch my encrypted filesystems from Encfs to the eCryptfs-based file encryption standard Ubuntu has been developing. When released in Jaunty, last spring, Ubuntu provided users with an encrypted directory, Private/ , which existed alongside Documents/, Desktop/, Music/, etc., where users could move data they wanted to protect, and link to it from its previous location instead. This was pretty much what I'd developed for myself with Encfs, explicitly choosing what I want to protect. The big difference was that the eCryptfs system used the user's login password, rather than a separate one, to wrap the keys that unlocked the encrypted directory, meaning that the Private directory would already be mounted as soon as the user entered their password to login. With my Encfs setup, I had to enter a separate passphrase in an additional step.

But in this release, Ubuntu has continued to improve this system. Now a user's entire home directory can now be encrypted, rather than just the Private/ sub-folder. These two improvements on my Encfs setup go a long way toward making encrypted protection of user data seamlessly transparent, and it's a wonderful advance for the privacy of the Ubuntu OS.

As to the details of installation, I had ZERO problems. Sound just worked (Pulseaudio and all, which frustrated me so much previously that I uninstalled it entirely), networking just worked (Network Manager now grants more user control), the installation of restricted Nvidia graphics drivers just worked (complete with Compiz-powered OpenGL desktop effects eye candy), installation of restricted extras like the Adobe Flash plugin, and mp3 codecs just worked, and a few annoyances I'd been tolerating for years have now disappeared.

Furthermore, it's even snappier in performance. Nautilus file management, Firefox 3.5, and the gedit text editor all loaded and responded noticeably faster than on Jaunty. My boot time was 20 seconds to login, and an additional 15 seconds to desktop - not mind blowing, but definitely better than Jaunty on the same hardware.

I've also noticed something on my laptop - since I installed the Karmic beta on it, I've left it on for longer periods, and it has not tended to overheat, hang, or spontaneously shut off as much as it did on Jaunty. Advertised Improvements to power management seem to be contributing to this.

I'm just tickled pink - this is a very solid release.

I have finally completed (mostly) a piece of software I've wanted for a long time.

I don't know how to nutshell it, exactly. It's a frontend for unix shell commands and terminal emulators that gives me flexibility in how it executes commands.

If I want to run a command in a terminal emulator:

# sketsh ls -al

a new terminal window pops up, executes the command (in this case, getting a file listing), and then drops me into a bash shell so I can enter more commands. in the meantime, it has added the command to my bash history so that it is immediately accessible for editing (in case I do something wrong). Simple and handy.

Now let's say that I'm writing a shell command or script that interacts with my graphical desktop environment, but isn't intended to use a terminal window at all. Like many programs, I use the console output to print debugging messages that aren't normally needed. If I execute a command by prepending sketsh with it's -C option,

# sketsh -C ls -al

it will skip executing the command in a terminal window entirely, and just run in the background. If something isn't working right, I can just delete that option from the command line in the script, and the command will execute in a window so I can see the output. When I've fixed any problems, I can then put the -C option back in, and the terminal goes back away. Neat and tidy.

At other times, I want a terminal window to pop up to execute the command, but I don't want it to hang around when it's done (unless, as before, I'm debugging it). For that, there is the -H option, which explicitly tells sketsh to close the window immediately after the command completes. To debug, simply take that option out of the line it's on, and a full terminal pops up and sticks around, with the command already in history for examination.

In this process, and over the years, I have learned a LOT about skillful BASH shell scripting - and how much more there is to it. This pleases me.

It's the perfect century to be a hacker.