The occasional scrivener

[This is an updated post!]

Twitter is a great resource, but once you follow a couple hundred people and don’t sit glued to it all day, it can become overwhelming. I’ve devised a way that works for me to keep track of my followers.

It’s home-brew, hackish, and not for everyone. But it’s the sort of stuff I like to faff around with.

Here’s an overview of the process.

• I run ttytter, an awesome console Twitter client. I run it under screen on a remote server, so I’m always “connected”.
• I’ve written an extension to ttytter that saves every tweet to a file, named after the date. The extension is based on the example “A Twitter logger” here.
• Every hour, I run a perl script that parses the twitter logs, and publishes them in one entry per hour as a Blosxom blog. This is then published statically on my site.
• I subscribe to the blog in Google Reader.

The publishing part of the process enables me to linkify URLs, twitter names, and hashtags, and lets me filter out crap like paper.li announcements and Foursquare checkins. I have links to each individual tweet so I can easily reply or retweet.

You obviously need some building blocks for this to work. Your own server capable of serving files for one. You should also be comfortable with scripting languages.

There’s another solution available if you don’t have access to the above. It’s a service called TweetedTimes. I’ve tried it, and while it does a decent job of keeping track of links posted by your followers it also posts links posted by their followers, which is not always what I want.

Let me know via the comments if you want more information!

I like using FreeBSD compared to Linux (or “lulnix” as we cool kids say nowadays) but distros like RedHat and Ubuntu have the edge when it comes to making it easy to keep your system up to date with the latest patches.

You’d think there was a simple one-shot way of doing this, but so far I’ve settled for the following method. It’s documented here as much for my own memory as anything else.

1. refresh your ports tree with cvsup: cvsup -g -L 2 /usr/local/etc/ports-supfile (you should edit the ports-supfile to reflect your geographical location).
2. run the portupgrade command: portupgrade -a -PP -Rr -i.

The options I use for portupgrade are:

• -a: all ports
• -PP: use only packages (use a single P if you want to compile a port where a package isn’t available)
• -Rr: both recursive and upwardly recursive
• -i, for interactive. I run this because once when I ran without it my perl got upgraded to a new point release under me and broke my CPAN modules. YMMV.

One of the sons in the house is in the last year of high school and as math is not his favorite subject he’s only now raised his need of a graphing calculator. I did a perfunctory research of the market and purchased a generic Texas Instruments model for around €70.

One thing that struck me is that today’s calculators aren’t that much different than the ones I remember when I was a first-year college student, lo so many years ago. We’re still looking at a plethora of multiply-labeled buttons, grayscale screens, and extremely limited memory. In an age where a basic calculator is practically disposable, memory is cheap and handheld UI innovation rampant thanks to cellphones and computers, it’s a bit weird to see a tech area looking so stable.

Part of the reason is of course that if you’re dealing with students, you don’t want them to be too connected or have too much computing at their disposal during exams. But that just raises another questions. Barring global catastrophe (in which case basic math skills would be nice to have, but knowledge of agriculture even more desirable) any pupil attending higher education will have a calculator within reach for the rest of their lives.

So, basic arithmetic has been automated, and in fact, even quite complicated algebra has been too, thanks to tools like Mathematica and Maple. In fact, as long as you have access to the web Wolfram Alpha will solve most equations faster than you can type them in.

Is it time to face the fact that doing arithmetic with pen and paper is like making pots on a pottery wheel? It’s a nice craft, probably will be quite popular for a small part of the population, but it’s essentially not needed in a modern society?

Please note I’m not saying mathematics is obsolete. It’s probably more important than ever. But I do think we need to find a better way of teaching it than just mechanically learning rules like long division and decomposition of factors. Leave the drudgery of math to the computers, and try to move on to teach the actual use of math in real life.

So I have an iPhone now, as have 60% of Stockholm’s inhabitants. It’s no longer a very interesting thing to say.

I moved to the iPhone from the Nokia E61, which I still think is a really good phone for its time. Here’s some random thoughts on the differences.

The iPhone excels at music playing and gaming. I’m not that big a fan of differnt social networking apps, but the web browser is miles better. Basic stuff like alarms work better. SMS rocks.

I miss having a physical keyboard though, but this is something I think I’ll get used to in time.

What the E61 was better at was multitasking, and IRC and Twitter apps. I used WirelessIRC and Gravity both by Jan-Ole Suhr, and they rocked. I use Rooms for IRC and Echofon for Twitter, and neither app has the ease of access and navigation that WirelessIRC and Gravity has.

However, it should be noted that these were the only apps I purchased for the E61. Purchasing software is so much better and easier on the iPhone it’s not even possible to compare.

Another thing I miss is the ability to “remote control” the phone from the computer. While Nokia PC Suite and its successor Ovi are a horrible conglomeration of ugly little apps, at least they made it easy to browse content and most especially read and write text messages from the desktop. I can’t really complain about iTunes syncing but I do wish there was something for Messages on the desktop.

When Google announced Wave the hype was immense, it was like everyone took Gmail and squared it and thought it would be that awesome! And basically that’s what the Wave developers thought too. Well I can’t remember what I did but I got an invite and hopped on, and right away realised that there was no-one there. The invite-only start (for performance reasons) limited the appeal and usefulness of Wave.

Because I think it was pretty useful, at least once they got the early kinks worked out. Me and some far-flung mates used it to plan some trips together, and Wave was pretty useful for that. It would have been better with a mobile interface but it was decent.

The thing is, Wave wasn’t the next coming of Gmail, it was the next coming of Microsoft Outlook. Outlook is merely the front-end to Exchange, which does all the boring and useful bits (routing email, planning calendars etc). And sure, it works, but not very well. At my current place of work we have Office Communicator Server (OCS) which is basically MS Live Messenger in a corporate suit. The nice thing about OCS is that it integrates into Exchange, so you can see whether a coworker is online, or in a meeting or whatever.

However, sometimes you want to start from the task, not the person. That’s where Wave began - you had a project, or a trip, or just a task involving some people. It could start with just you and then expand, like a lot of tasks do. However, instead of snowballing into a long mail thread, you actually had the option of keeping Waves clean and on topic.

Wave was flawed - it was before its time, it tried to solve problems that don’t often turn up in the wider world of the web, but which do turn up lots of times within organizations. Maybe Microsoft will learn from Wave and build better collaboration tools than Exchange.

[Warning, this post contains profanity.]

There’s a lot of blogther[1] around these here tubes about the closed nature of the Apple iPad. Exhibits: Doctorow and Danny O’Brien.

Basically the the thesis is that if this closed platform takes over, all creativity is stifled by unaccountable gatekeepers and humanity is doomed.

That’s bollocks. Here’s why.

Primo. The iPad is a toy for rich people. Period. Only a tiny minority will buy it, and only a minority of that minority will use it as their only way to access the internet. For fuck’s sake, you need a computer (PC or Mac) to sideload content to it[2]. There’s zero chance that this will become the only fucking way to access the internet, because Apple doesn’t sell to everyone on the planet. They sell to people who can afford Apple products. So far, they’ve made a lot of money doing this, while companies like Lenovo and HP are struggling. Why should they change?

Secundo. Closed platforms are the fucking greatest for some things, like gaming. I happen to think that the console systems are better for gaming than the PC platform. And if you disagree with me on this, you’re per definition in the minority. Seriously, check out the sales figures. Money talks. And if we’re gonna get serious about tackling spyware and malware and shit, we should mandate that 99% of internet users only get to use a closed, locked down, Trusted Computing-verified internet appliance. The general-purpose PC is like a fucking chainsaw in these people’s hands.

Tertio. If you happen to raise a geek or feel a pressing need to make him or her learn to code fucking Haskell, there are literally MILLIONS of used PC’s that can run your choice of Lulnix, and he/she can spend their precious time coding blog platforms when they should be out in the sun, getting D vitamins.

Quarto. Who died and made Apple god? If the iPad succeeds in taking over the world, they’ll end up like IBM or Microsoft. This is not the end of computing history.

In closing, shut the fuck up already, you apocalyptic freetards.

Update 2010-04-14: the tone of this post has been inspired by the Angry Mac Bastards podcast, who go forth and rip Doctorow and Winer to shreds regarding their iPad views in episode 55. Listen to them for even more sound and fury, signifying nothing much!

[1] a portmanteau of “blog” and “blather”, and yes, I realise if you have to explain it’s not very good. Go invent your own term.

[2] Conveniently. I guess you can buy stuff OTA too. Do I look like I do research?

Project Euler “is a series of challenging mathematical/computer programming problems that will require more than just mathematical insights to solve.” In other words, you need to get down and program to solve the problem[1].

I’ve recently reached Level 3 (solved over 100 problems) and thought I’d share some notes on what I’ve learned.

PE programming is “pure” programming, as in you’re working with algorithms and not systems. There’s rarely anything more complicated than file access needed. In fact, I’d say you need an aptitude for problem solving rather than programming, and the skills needed to solve PE problems are probably not really translatable to real work out in the real world, where you’re dealing with real people and real hardware and networks. To be sure, nowhere on PE do they say that this is programming in the wider sense, as opposed to Computer Science, but you’d be surprised how many CS grads think it does (I’m mocking you, Haskell nerds!).

The typical PE problem looks amenable to brute force, but it pays to research before jumping in. Wikipedia and Mathworld are great resources for a lot of the problems, often explaining the math behind them and sometimes having the algorithms you need.

Spoilers. I find it pretty hard to concentrate on a problem and not want to know the solution, especially when I’m into “grind” mode and just want to level up (this happens a lot with me and puzzle games too). There are plenty of spoilers around for the early problems (the ones that most people manage to solve) but as you move up they get sparser and sparser, and are usually in languages other than your chosen one. Often it’s enough for me to just get a hint. After you’ve solved a problem, you get access to a forum where others discuss their solutions. This can be an eye opener, but there’s no need to share your crappy solution if you don’t want to.

I use Perl. I thought of using PE to learn another language (Python), but finally I felt it was more fun to explore something I’m pretty proficient in than to learn something else. Perl has all the things you need for these problems (so far), especially as there are tons of libraries for specific applications. One of the more useful is Math::Pari, an interface to the Pari suite of numerical algorithms.

Perl is really nice for stuff like string manipulation and set operations. It’s also really easy to build ad-hoc data structures, although the syntax grows hairy really fast.

Python seems very popular among PE participants. Another popular language among the people who bother to post to answer threads is J), and of course Haskell has some vocal devotees.

I find it challenging and fun to work on these problems, and I encourage you to do the same!

[1] although there are some wizards who manage with pencil and paper, at least for the earlier problems.

Update 2011-09-03: I’ve just gotten my hands on an iMac running Lion, and Terminal.app now has built-in support for different colour schemes. This blog entry has a nice one.

The Mac has strangely few terminal emulators, unless you set up X and invoke the Deep Ones in other ways. So far, I’ve found iTerm to be the best one.

You cannot set iTerm to use black background and white text, but you can trick this with AppleScript. Put the following into your .bash_profile or similar:

setColor() {
  osascript -e "tell application \"iTerm\"
    set current_terminal to (current terminal)
    tell current_terminal
      set current_session to (current session)
      tell current_session
      set background color to \"black\"
      set cursor color to \"white\"
      set foreground color to \"white\"
    end tell
  end tell
end tell"
}
## now invoke the function, and clear the screen 
setColor
clear

When you start a new window, the background will be white but it will soon become white on black, as ghod intended.

Thanks Jedrek for helping me with this!

If you try to set the option “delete it” from the Outlook Rules and Alerts wizard, you’ll get a message that that particular rule is “client-side only”, which means that Outlook has to be running for the rule to work.

If you’re trying to set up rules for mobile use, for example, this is less than ideal.

One workaround is to choose “move to a folder”, and move to the folder “Deleted Items”. This achieves basically the same thing, with the added bonus of keeping the message for review if needed.

Sure, IRC has the /ignore command, but it just removes all mention of a nick from a channel. Sometimes this is what you want, but Mike recently shared a little irssi script that will simply censor anything an idiot says. That way the conversation isn’t as disjointed as it could be otherwise.

I’ve modified the script so that more than one nick can be censored. My version is here. Basic knowledge of Perl is required to use this. It’s also easy to evade by changing nicknames, I’ll try to fix this in later versions.

To install in irssi, follow these instructions.

I can’t help but feel that this is a previously rejected novel by Reynolds that’s now been exhumed by his publishers in search of a few extra quid.

It’s a half-baked space opera in the Iain M. Banks vein, ranging over the galaxy and across millions of years. It even has some Banksian foreshadowing, hinting at dark and mysterious secrets to come. But nothing comes of this, and there are plenty of other plot strands that are simply left dangling.

The scene setting is amateur, the plot is ludicrous, and the characters are two-dimensional.

Not recommended.

I recently had a spot of bother with my work lappy. The screen went all blurry after a short while of use. I didn’t mind this as it didn’t happen when the laptop was docked.

However I felt it needed to be fixed, so I called the Dell Gold support line in Sweden. Very professional service from them, a tech arrived the day after with a replacement motherboard and LCD screen. So far, so happy.

This morning I arrive to find an email from Dell asking me to participate in a survey about my experience. I’ve ignored these in the past, only to be spammed by reminders, and I thought that in this case I should take it as a I was very happy with the service I got.

So like a fool I click the link to some god-awful outsourced company (prognostics.com). A series of barely literate, badly sequenced survey questions pop up. Each takes about a minute to load. They are repetitive. They have no discernable order. They seem to reiterate the same damn point over and over again.

So I waste about 10 minutes of my time with this crap and feel obscurely that I’m more unhappy with the level of service I’ve received than before. This is the last time I’m answering these, I’ll be creating a filter to bin them from now on.

I’ve been using Ubuntu 7.10 on a Latitude D505 for a while now, and I’ve always been disappointed with the screen resolution. The max I could get was 1280x1024, while the native resolution is 1400x1050.

I recently had to reconfigure X on this machine (dpkg-reconfigure xserver-xorg) and chose the intel driver (recommended by the configure script). However, it turns out that this driver hangs the machine when you close the lid. The older driver i810 works in this regard, but the max resolution is too low.

The solution is to use the 915resolution application. This does some weird hacking of the video BIOS to allow Linux to use the full resolution.

Using it in Ubuntu is a breeze. Simply install the .deb package and reboot.

Instant nostalgia. Thanks to the Aleph One project and these instructions I was able to compile the Aleph One application on my Ubuntu lappy and play Marathon 2.

It’s a lot of fun, especially as I can point out “we had that in Marathon 13 years ago” while playing Halo 3 with Leo. This pisses him off no end.

The site isitfriday.net provided an essential service: answering the question “is it Friday today?”.

Apparently it wasn’t deemed essential enough, cause the service no longer exists.

David has stepped up to the breach with http://isitfriday.gnapp.org/. Here’s my contribution:

#!/usr/bin/perl -w
use strict;
use CGI qw(:standard);
my $wday = (gmtime( time ))[6];
my $msg = 'no';
if ( $wday == 5 ) {
    $msg = b( 'YES!' );
}
print header;
print start_html( 'Is it Friday?' );
print h1({-style=>"text-align: center"}, $msg );
print hr;

You can see it in action here.

N.B. this script is not tested for days other than Friday. Also, your timezone will have to be in the general vicinity of UTC for it to work for you.

I found out today that if you’ve installed a script in Nautilus, you won’t see it in the contextual menu until you’ve visited the script directory in Nautilus.

Install a script in ~/.gnome2/nautilus-scripts/ and making it executable.

Then visit that directory in Nautilus to “enable” it.

I’m not sure if you have to do this for every script you install, or only for the first one.

So there we were at work, wondering why part 2 of our home-brewed customer survey system[1] wasn’t sending the mails it should be sending.

Turns out that the perl script responsible read from one table in the database to find out the recipients of the emails. It then generated a form of authentication, updated the database to say that the mails had been sent, then called

system( "/usr/local/bin/email.pl" );

where email.pl was another script written by the same developer.

Note the lack of return value from that call to system.

So what happened was that in the distant mists of time (about 2 months ago) we switched source control systems. Apparently we lost our executable bits on some scripts, because email.pl was readonly and owned by Apache.

Had we checked the return value, we would at least have had a warning that something was wrong. But now the system apparently “worked” for a month or so, and we only realised that nothing was being sent when the responses dried up.

I’ve re-written the code to check if the call to email.pl was succesful, and only if it was would the database be updated.

Now we have to re-send all emails older than a month, which will probably put our server on a spam blacklist. Sigh.

Lesson: code defensively.

[1] Why we haven’t got a third-party solution for this is hard to understand, unless you worked there 2 years ago, when development time was considered free and maintainance was never factored in the cost equation.

Here’s a fun thing I learned for the first time today: you can have variables in formats for printf or sprintf.

Here’s a contrived example:

#!/usr/bin/perl -w
use strict;

sub variable_format {
    my ( $in, $len ) = @_;
    printf( "%0${len}d\n", $in );
}

variable_format( 42, 3 );  # prints 042
variable_format( 42, 9 );  # prints 000000042

Interestingly, I can’t find any mention of this in Perl’s documentation for sprintf.

Tip of the hat to Linus.

Here’s a stab at a script for posting to Twitter:

#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
use XML::Simple;
use URI::Escape;

my $username = 'your_twitter_username';
my $password = 'your_twitter_password';

my $message = shift or die "usage: $0 message\n";

if ( length $message >= 140 ) {
    $message = substr( $message, 0, 139 ); 
    warn "truncating message to: $message\n";
}
$message = uri_escape( $message );

my $ua = LWP::UserAgent->new();
# tell Twitter who we are
$ua->agent( "$0/0.1 " . $ua->agent );

my $req = HTTP::Request->new( POST =>
                  'http://twitter.com/statuses/update.xml');
$req->content_type( 'application/x-www-form-urlencoded' );
$req->authorization_basic( $username, $password );
$req->content( "status=$message" );

my $res = $ua->request( $req );

my $xml;
if ( $res->is_success ) {
    $xml = XMLin( $res->content );
} else {
    print "Posting failed " . $res->status_line . "\n";
}

my ( $created, $name, $screen_name, $id ) =
  ( $xml->{created_at},
    $xml->{user}->{name},
    $xml->{user}->{screen_name},
    $xml->{id} );

print << "EOF";
Your message

  $message

was succesfully posted at $created 
as user $name ($screen_name) with ID $id.
EOF

Thanks to Jim for the inspiration.

By the way, I’m gerikson at Twitter.

Update 2007-03-27: I removed the wget version as I don’t maintain it any more.

This Twitter API wiki is pretty helpful.

Nearly flubbed this. Time constraints suck.

First implementation, aka bog-standard:

#!/usr/bin/perl -w
use strict;
my $i = 1;
while ( $i < 101 ) {
    if ( $i % 3 == 0 and $i % 5 == 0 ) {
    print "FizzBuzz\n";
    } elsif ( $i % 3 == 0 ) { 
    print "Fizz\n";
    } elsif ( $i % 5 == 0 ) {
    print "Buzz\n";
    } else {
    print "$i\n";
    }
    $i++
}

Maybe I’ll play a bit with this later… the online comments have degenerated into golfing in people’s favourite languages.

Check here for a funny-cause-it’s-true “example” in Java, everyone’s favourite enterprise language… (first comment).

Quick start

• attach the disk to the computer
• Run the following commands as root or via sudo:

(The first command finds LVM volumes. The second one activates them.)

root@host:~# vgscan  
  Reading all physical volumes.  This may take a while...  
  Found volume group "foo" using metadata type lvm2  
root@host:~# vgchange -a y foo  
  2 logical volume(s) in volume group "foo" now active  
root@host:~# ls /dev/foo  
  root swap_1  
root@host:~# mount /dev/foo/root /mnt

Background

This may help others, I had a hell of a time getting it working, but that was just because I didn’t understand the concepts at all.

Background: a Linux laptop lost its video card recently. I was able to back up the most important files via FireWire to another machine before I couldn’t see the screen, but I felt I should have the entire hard drive handy in case I miss anything. So I removed the hard drive from the laptop and put it in a $40 USB enclosure.

An aside: this resulted in a very nice little wallet with 40G of storage, but with one big disadvantage. It gets its power entirely from the USB port, but unless you have powered port (generally one at the back of a tower PC) you need an external power brick. Most laptops cannot run the drive from their ports. Lugging a power brick around kinda negates the portability of the device. The point was moot in my case, as there was no power brick included in the package.

(In addition, el cheapo USB hubs can’t power this sucker either. I had a 3.5” enclosure with 3 USB ports that could, however.)

So, I plugged the drive to a laptop also running Linux (Ubuntu, same as the last one). The drive automounted and I could see the /boot partition. Obviously that’s formatted in some generic fashion that the automounter can recognise. A scan of /var/log/messages gave the device as /dev/sdb.

root@inspiron8600:~# fdisk /dev/sdb

The number of cylinders for this disk is set to 4864.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
  (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sdb: 40.0 GB, 40007761920 bytes
255 heads, 63 sectors/track, 4864 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          31      248976   83  Linux
/dev/sdb2              32        4864    38821072+   5  Extended
/dev/sdb5              32        4864    38821041   8e  Linux LVM

(note that I’m root here, no mucking around with sudo when there’s a lot of commands to issue!)

OK, so the data is on /dev/sdb5, in a LVM volume. What the heck is that? Linus has this to say:

You don’t want to use LVM, it’s a source of confusion and headache. It’s a toy variant of VxFS but doesn’t work nearly as well.

Words to live by, but Ubuntu has helpfully stashed my data into an LVM volume and I needed to get it back.

The short answer is to run vgscan:

root@inspiron8600:~# vgscan
 Reading all physical volumes.  This may take a while...
 Found volume group "Ubuntu" using metadata type lvm2

OK, this is where I went off the rails. You see, I thought that the volume group “Ubuntu” was the one on the host hard drive. So I spent a lot of time googling for solutions (someone somewhere must have had this experience before!), reading about using Windows (ugh!) and how not to do stuff. Then I read what I should have from the the beginning, the LVM HOWTO.

In the section Common tasks it tells you about Creating a volume group. This could then be mounted.

Aha! I tried

# vgcreate usb_volume /dev/sdb5

but got the message that it was already a part of a volume group, namely “Ubuntu”!

Mounting /dev/Ubuntu to /mnt was a moments work. I was able to copy the needed data easily.

(Update: /dev/Ubuntu showed up because I ran vgchange before, not knowing what that command did.)

Caveat: In the course of writing this, I tried to recreate the steps I used. However, I got this message back:

root@inspiron8600:~# vgscan
  Reading all physical volumes.  This may take a while...
  /dev/Ubuntu/root: read failed after 0 of 4096 at 38168100864:
    Input/output error
  /dev/Ubuntu/root: read failed after 0 of 4096 at 0: Input/output error
  /dev/Ubuntu/swap_1: read failed after 0 of 4096 at 1581187072:
    Input/output error
  /dev/Ubuntu/swap_1: read failed after 0 of 4096 at 0: Input/output error
  Found volume group "Ubuntu" using metadata type lvm2

I couldn’t mount the partitions either.

Looks like I was in the nick of time!

I have no clue whether the act of mounting the LVM volume via USB ruined it, or if this was just a bad disk. Be careful!

Thanks to Stefan and Linus for helping me with this!

I wanted a quicker way to transfer files between two laptops than via wifi. One’s is running Ubuntu Linux, the other Windows XP.

Both machines have a 4-pin FireWire port (labeled 1394, apparently someone owns the rights to the “FireWire” name and someone else doesn’t want to pay royalties). I borrowed a 4-pin to 4-pin cable from my brother-in-law and set out to try to connect the machines.

I knew that Windows has built-in IP-over-1394 support, the clincher was enabling it under Linux. Ubuntu has the required modules installed and the kernel I use can use them. There is a project page for Linux1394 if you need to get the modules and install them by hand.

I loaded the network module eth1394 by running

$ sudo modprobe eth1394

(Maybe this isn’t strictly necessary, but I didn’t find any reference to the module in the dmesg output.) Checking /var/log/messages, I saw that the interface got the name eth2.

Then I connected the computers with the cable (after enabling the interface in Windows). I heard a “cable connected” sound in Windows. Running

C:\> ipconfig /all

showed me that the interface was up. After a while I got a private address (169.254.203.16, netmask 255.255.0.0). I configured a static address on the Linux side:

$ sudo ifconfig eth2 up 169.254.203.17 netmask 255.255.0.0

I then started an FTP client in Windows, pointed it to the address defined in Linux, and started transferring files. I’m getting about 2.5 MB/s which isn’t too shabby.

Update: some quick perusal of the transfer logs gave me a average transfer rate of 1.45 MB/s.

Correctly formatted date strings are needed for RSS generation, among other things RSS. Diego shows how to do this in Java. This is my version in Perl

 #!/usr/bin/perl -w

 use strict;
 use POSIX qw(strftime);

 my $now = time();

 # We need to munge the timezone indicator to add a colon between the hour and minute part
 my $tz = strftime("%z", localtime($now));
 $tz =~ s/(\d{2})(\d{2})/$1:$2/;

 # ISO8601
 print strftime("%Y-%m-%dT%H:%M:%S", localtime($now)) . $tz . "\n";
 # RFC822 (actually RFC2822, as the year has 4 digits)
 print strftime("%a, %d %b %Y %H:%M:%S %z", localtime($now)) . "\n";

The output is

 2005-01-05T01:35:08+01:00
 Wed, 05 Jan 2005 01:35:08 +0100

There are some small issues I haven’t found clarification for at this late hour of writing. The first is whether the day should be zero-padded in the RFC822 case. As it is now it’s space-padded. Accordng to RFC822, the day is zero-padded.

The second is how to handle locale settings — RFC822 specifies that the weekdays and months be in the US locale. I’m pretty sure that you need extra magic in Perl for this not to work, but I’ll take a look at that tomorrow.

Yet another reason to standardize on ISO8601, I guess.

Update 2006-03-10: Eli Billauer was kind enough to mention this error to me, kudos to him.

Apparently you can’t format volumes larger than 32GB in FAT32 under WinXP. Anything larger will have to use NTFS. I’ve got a largish drive in a USB-connected enclosure that I’d like to use under Linux, so NTFS was out. As far as I know, the Linux support for it is still experimental.

Well, as the good book says, Google, and ye shall find. The first suggestions from MS was to boot into Win98/ME and use their FORMAT command to do this. This is akin to performing brain surgery with a butter knife dipped in pus.

But I also found Jens-Uwe Mager’s Windows port of the Linux command mkdosfs that worked like a charm.

So now I’m backing up the MP3’s on this machine. Can life get much better?

When posting my observations via email any Swedish characters are converted to quoted-printable ISO-8859-1 by Gmail. However, this blog is in UTF-8. This is how I translated the input from the mail message.

#!/usr/bin/perl -w
use strict;
use MIME::QuotedPrint qw( decode_qp );
use Encode qw( decode encode );

# split the mail message
my ( $headers, $body );
{
    local $/ = undef;
    ( $headers, $body ) = split( "\n\n", <STDIN>, 2 );
}

# decode the qouted-printable input
$body = decode_qp( $body );
# decode to Perl's internal format
$body = decode( 'iso-8859-1', $body );
# encode to UTF-8
$body = encode( 'utf-8', $body );

print $body, "\n";

The result is piped into a second script that formats the actual posting.

Pretty basic, eh? But until you know how, it can be a bit frustrating getting this to work.

I recently had to edit a bunch of mp3s. I found the following Perl modules helpful.

• mp3cut. Contains a script mp3cat that concatenates mp3 files. Very simple to use.
• MP3::Tag. For all your mp3 tagging needs. Contains a script mp3info2 that can read and update tags.

These modules work fine under Cygwin running in Windows.

I grabbed a bunch of audio books in mp3 format but was shocked, shocked I say! when the files weren’t bookmarkable (i.e. they didn’t remember where I stopped listening) and worse, the files were included in the shuffle. When you have 1,388 files, each a minute long, that kinda sucks.

I pecked around in iTunes a bit and saw that there was a setting in the Info->Options panel for a track that included the checkmarks

• Remember playback position
• Skip when shuffling

Perfectomento! BUT… you can only set these options one file at a time. Damn you Apple! Damn you to hell!!

(If you don’t use AppleScript. Which I don’t. Because I don’t have a Mac.)

I found 2 links on Google which offer some alternatives. The reason for this post is essentially that del.icio.us is down ATM and I need somewhere to post these:

• Shuffle, damn you! Shuffle!
• Turning MP3 audiobooks into iPod audiobooks

Update 2005-12-29: basically there are two ways of handling this.

• Convert to AAC format and rename

I tried this variant, but the resulting files were much larger than the files in mp3 format. I guess you can get around this by playing with the converting options, but I suspect they only apply to importing from CDs (see the second link above).

• Merge the mp3 files and set the appropriate settings manually.

I went with this variant. A quick search of CPAN resulted in a bunch of utilities I could use. I merged the tiny files into chapters, then added what ID3v2 tags I could. The rest were added in iTunes, along with the required playback and shuffle options.

I will try to find out more about ID3v2 tag handling later, so as to automate this even more.

Due to a little accident, my wife’s standard laptop is hors de combat — basically, I dropped our son on it (from a small height, I hasten to add) and the power connection broke. So it boots, for about 5 seconds, before discovering that the battery is flatter than Kansas.

I have in my possession an elderly laptop with no more than 256M of RAM, much too little to run a modern Microsoft OS. So I decided to check Ubuntu out. This is supposed to be “Linux for human beings”, so I figured this would be OK for non-geeks.

I downloaded an install CD via BitTorrent and used it to install it. The installation went really well. I had to fight my old-skool Debian and OpenBSD instincts and just let things go as designed. Ubuntu has made a great product — it installed seamlessly on the lappy and presented a clean, workable graphical interface that I think will be acceptable for Linux-o-phobes.

The only thing that didn’t work out of the box was the wi-fi pc-card I bought expressedly for its OpenBSD compatibility. However, I discovered that I had no less than four wi-fi cards lying around, and the oldest 3Com card worked fine.

I’ve been using this laptop all day (in between re-stacking our woodpile) and I really enjoy it. Small stuff, like Alt-Tab window-switching and Alt-F4 window close work exactly like in Windows. But having a real Debian system underneath just feels sooo much better than Windows crap leavened with Cygwin goodness.

Matthew: Generation VIC20.

Chris: So Dad, what’s a computer science class for at school?
Me: Well, you learn to program there for example.
Chris: Program?
Me: Yes, to write computer programs using a special language.
Chris: You mean like games?
Me: Well, yes I suppose.
Chris: Like Star Wars Knights of the Old Republic?
Me: Umm, no.

Classic.

I guess you could learn a lot by copying source code, even if it was just BASIC spaghetti code. With line numbers.

Cool hack in Unix: if you know the zonefile abbreviation for a timezone, you can use it to get the date there:

$ TZ=PST8PDT date
Tue Aug 30 05:07:39 PDT 2005

This also works:

$ TZ=America/Los_Angeles date
Tue Aug 30 05:08:17 PDT 2005

Timezone files are in /usr/share/zoneinfo on OpenBSD and Linux.

This is quick basic intro to GNU Screen.

If you haven’t installed screen, do so. It rocks.

Starting up

Start screen. You’ll get a splash screen. Hit space to make it go away.

Depending on your flavour of *nix, you’ll get between 1 and n screens. First one is numbered 0. Try to switch to screen 1: Ctrl-a 1. If you get a message, create screen 1 with Ctrl-a c (create). Now you can switch back and forth between 0 and 1 with Ctrl-a 0 and Ctrl-a 1.

Want a “real” Ctrl-a? (If you’re in Emacs, you do.) Type Ctrl-a a.

Detaching

Start an editor in one screen, load a file and start editing. Detach with Ctrl-a d (detach). Logout of the shell, then login again. Attach to the screen: screen -r (reattach). You should be back where you left. Continue editing.

Finally

Screen is perfect for flaky connections, you’ll be back where you left off even if you get a hangup.

You can use screen -r -d to cleanly reattach from another session.

If you want a bunch of screens at startup, edit your ~/.screenrc. Mine looks like this:

shell -$SHELL
screen -t SHELL0  0
screen -t ROOT    1
screen -t SHELL2  2
screen -t SHELL3  3
screen -t SHELL4  4
screen -t SHELL5  5

The first line starts the current $SHELL in login mode. The others set up six windows and gives them titles.

Type man screen for more info.

Happy screening!

Update: here’s a more full-featured tutorial. (Via NTK.)

Matt has written a nice tutorial on how to use Django in a simple CRUD situation: Django Generic Views: CRUD.

(CRUD, by the way, stands for Create, Read, Update, Delete — basic database manipulation. The title of this post is in no way a reflection on Django on my part.)

Matt and others are very excited over Django — it’s Ruby on Rails for Python. I’ve been meaning to learn Python for a long time, but lately I’ve felt I’m too old a dog for new tricks. More on this in a later post, maybe.

This list is here so that I can check out hardware in stores via the phone.

• ADMtek ADM8211 based CardBus/PCI adapters (atw) ^(G)
• Aironet Communications 4500/4800 ISA PnP, PCMCIA and PCI 802.11b adapters (an)
  • Aironet 4500/4800
  • Cisco 340/350
• Atheros AR521x based CardBus 802.11a/b/g adapters (ath), including: ^(G)
  • 3Com 3CRPAG175
  • Aztech WL830PC
  • D-Link DWL-A650
  • D-Link DWL-AB650
  • D-Link DWL-AG650
  • D-Link DWL-G650B
  • Elecom LD-WL54AG
  • Elecom LD-WL54
  • Fujitsu E5454
  • Fujitsu FMV-JW481
  • Fujitsu E5454
  • I/O Data WN-AB
  • I/O Data WN-AG
  • I/O Data WN-A54
  • Linksys WPC51AB
  • Linksys WPC55AG
  • NEC PA-WL/54AG
  • Netgear WAB501
  • Netgear WAG511
  • Netgear WG511T
  • Orinoco 8480
  • Orinoco 8470WD
  • Proxim Skyline 4030
  • Samsung SWL-5200N
  • SMC SMC2735W
  • Sony PCWA-C700
  • Sony PCWA-C300S
  • Sony PCWA-C500
• Atheros AR521x based PCI 802.11a/b/g adapters (ath), including: ^(G)
  • D-Link DWL-A520
  • D-Link DWL-AG520
  • D-Link DWL-G520
  • HP NC4000
  • Linksys WMP55AG
  • Netgear WAG311
  • Netgear WG311
  • Proxim Skyline 4032
  • Senao NL-5354MP
• Atmel AT76C50x based USB 802.11b adapters (atu), including: ^(G)
  • Acer Peripherals AWL300
  • Acer Peripherals AWL400
  • Aincomm AWU2000B
  • Bluetake BW002
  • D-Link DWL-120
  • Geowave GW-US11S
  • Linksys WUSB11
  • Linksys WUSB11-V28
  • Netgear MA101 rev B
  • OQO model 01 builtin wireless
  • Ovislink AirLive WL-1120USB
  • OvisLink AirLive WL-1130USB
  • SMC 2662W-AR
  • SMC 2662W-V4
• Intel PRO/Wireless 2100 802.11b adapters (ipw) ^(G)
• Intel PRO/Wireless 2200BG/2225BG/2915ABG 802.11a/b/g adapters (iwi) ^(G)
• Intersil PRISM-2-3 based 802.11b Compact Flash adapters (will be detected as PCMCIA adapters) (wi)
  • Buffalo AirStation
  • D-Link DCF-660W
  • ELSA XI800
  • Linksys WCF12
  • Netgear MA701
• Intersil PRISM 2-3, Lucent Hermes and Symbol Spectrum 24 based PCMCIA 802.11b adapters (wi), including:
  • 3Com AirConnect 3CRWE737A
  • ACTIONTEC HWC01170
  • Addtron AWP-100
  • Agere Orinoco
  • ARtem Onair
  • BUFFALO AirStation
  • Cabletron RoamAbout
  • Compaq Agency NC5004
  • Contec FLEXLAN/FX-DS110-PCC
  • Corega PCC-11
  • Corega PCCA-11
  • Corega PCCB-11
  • Corega CGWLPCIA11
  • Dlink DWL650 revisions A1-J3
  • ELSA XI300
  • ELSA XI325
  • ELSA XI325H
  • EMTAC A2424i
  • Ericsson Wireless LAN CARD C11
  • Gemtek WL-311
  • Hawking Technology WE110P
  • I-O DATA WN-B11/PCM
  • Intel PRO/Wireless 2011
  • Intersil Prism II
  • Linksys Instant Wireless WPC11
  • Linksys Instant Wireless WPC11 2.5
  • Linksys Instant Wireless WPC11 3.0
  • Lucent WaveLAN
  • NANOSPEED ROOT-RZ2000
  • NEC CMZ-RT-WP
  • Netgear MA401
  • Netgear MA401RA
  • Nokia C020 Wireless LAN
  • Nokia C110/C111 Wireless LAN
  • NTT-ME 11Mbps Wireless LAN
  • Planex GW-NS11H Wireless LAN
  • Proxim Harmony
  • Proxim RangeLAN-DS
  • Samsung MagicLAN SWL-2000N
  • SMC 2632 EZ Connect
  • Symbol Spectrum24
  • TDK LAK-CD011WL
  • US Robotics 2410
  • US Robotics 2445
• Intersil PRISM 2-3 and Symbol Spectrum24 based PCI 802.11b adapters (wi), including:
  • 3Com AirConnect 3CRWE777A PCI
  • Belkin F5D6000 PCI (a rebadged WL11000P)
  • Corega CGWLPCIA11 PCI
  • Eumitcom WL11000P PCI
  • Dlink DWL520 PCI revisions A and B
  • Global Sun Technology GL24110P PCI (untested)
  • Global Sun Technology GL24110P02 PCI
  • Intersil Mini-PCI
  • LinkSys WDT11 PCI (a rebadged GL24110P02)
  • NDC/Sohoware NCP130 PCI
  • Netgear MA301 PCI
  • Netgear MA311 PCI
  • US Robotics 2415 PCI (rebadged WL11000P)
  • Nortel E-mobility 211818-A
  • Symbol LA4123
• Intersil PRISM 2.5/3 based USB 802.11b adapters (wi), including:
  • Acer Warplink USB-400
  • Actiontec HWU01170
  • AirVast WM168b
  • Ambit WLAN
  • Apacer Wireless Steno MB112
  • ASUS WL-140
  • Compaq W100
  • Corega WLUSB-11
  • Corega WLUSB-11 Key
  • D-Link DWL-120 (rev F)
  • D-Link DWL-122
  • I-O DATA WN-B11/USB
  • Intel PRO/Wireless 2011B
  • Intersil Prism 2X
  • JVC MP-XP7250
  • Linksys WUSB11 v3.0
  • Linksys WUSB12
  • Melco WLI-USB-KB11
  • Melco WLI-USB-KS11G
  • Melco WLI-USB-S11
  • Microsoft MN510
  • Netgear MA111 (version 1 only)
  • Pheenet WL-503IA
  • Planex GW-US11H
  • Siemens SpeedStream SS1022
  • Sitecom WL-022
  • Syntax USB-400
  • US Robotics 1120
  • Z-Com XI-725/726
  • Z-Com XI-735
  • ZyXEL ZyAIR B-200
• Ralink RT2500 based CardBus 802.11b/g adapters (ral), including: ^(G)
  • MSI CB54G2
  • Surecom EP-9428-g
• Ralink RT2500 based PCI 802.11b/g adapters (ral), including: ^(G)
  • ASUS WL-130g
  • Minitar MN54GPC-R
• Raytheon Raylink and Aviator 2.4/PRO PCMCIA 802.11 FH adapters (ray)
• Realtek RTL8180L based CardBus 802.11b adapters (rtw), including: ^(G)
  • Corega CG-WLCB11V3
  • Netgear MA521

^(G): Drivers for hardware marked with (G) are only included in the GENERIC kernels, but are not included on the various distribution floppies (including the cd-rom boot image).

(Source: OpenBSD i386 hardware list.)

For a long time I have been having trouble sending mail to people using Microsoft’s mail clients Outlook and Outlook Express. I use Gnus, an all-singing, all-dancing news-mail-and-everything reader for use in Emacs.

The trouble was that if I included any 8-bit characters in the header of the message, Outlook would translate any 8-bit characters in the body of the message to an equal sign and two hex characters. This was intensely irritating, as I naturally assume that all free software is superior to commercial offerings, especially Microsofts.

The trouble is that the version of Gnus I’m using (v5.8.7) doesn’t encode the headers in quoted-printable, thus confusing Outlook no end.

The solutions is to place the following in your .emacs or .gnus file:

;; iso-8859-1 support for headers
(require 'gnus-msg)
(add-to-list 'gnus-group-posting-charset-alist
     '(message-this-is-mail 'iso-8859-1 (iso-8859-1)))

Thanks to Kai Großjohann for this info.

Keywords: Gnus, gnus newsreader, GNU/Emacs, Emacs, Xemacs, MIME, mime, quoted-printable, transfer-encoding, Microsoft Outlook, Outlook Express, mangled, 8-bit characters.

Note: this was originally posted on another server, I’m posting it here in an effort to clean up my online life.

Need to combine two or more PDF files into one under Unix? Use the following command:

  gs -q -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=merged.pdf \
  source1.pdf source2.pdf source3.pdf etc.pdf

Obviously, you’ll need Ghostscript for this to work.

(From macosxhints)

For some reason (probably because I feel an itch to hack) I was thinking about Plan 9 today. So it seemed an omen that /. had a call for questions for Rob Pike, co-creator of Unix and Plan 9.

I read some of the links in the article, and this pessimistic view left me thinking. Pike’s point is that (academic) software research no longer matters. We’re in a sterile wasteland of Windows, Linux, and the Web. No new ideas are being explored.

Well, that’s fine as far as it goes, but a meme that’s brewing is the coming dominance of mobile devices and content — quite different from desktop or server computing.

These points from the article show some possible fields for research:

Only one GUI has ever been seriously tried, and its best ideas date from the 1970s. (In some ways, it’s been getting worse; today the screen is covered with confusing little pictures.) Surely there are other possibilities. (Linux’s interface isn’t even as good as Windows!)

Ties in nicely with this post.

There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts.

Again relevant in the mobile space.

The future is distributed computation, but the language community has done very little to address that possibility.

Who knows? Mobiles are getting more and more powerful. GPS, encryption need processing.

The Web has dominated how systems present and use information: the model is forced interaction; the user must go get it. Let’s go back to having the data come to the user instead.

Also very relevant from a user perspective. Data must come to you, when and where you want it, with a minimum of fuss.

Will academia pick up the thrown gauntlet? Lets hope someone does.

Matthew asks how one goes about to teach kids to code. Viking is too small yet, but it’s an interesting question. I know Hanna is quite proficient in HTML, mostly by copying and pasting, but Leo has shown no interest whatsoever in coding.

Part of the problem is the polished and complex nature of todays computers. In our day, you could slavishly copy pages of code and get something that worked. Even if it was just copying, you got down and dirty with the code. Some of it stuck. A curious kid (which I was not) could explore further, learning more and more. Whether learning Basic and VIC-20 assembler was a good thing is another question…

But now? Who can feel that they can produce something like Doom 3 by themselves?

Having said that, I believe a programming environment should have a graphic component. A former co-worker’s son loves (loved? it’s been a while) a DOS-based program for scripting dungeon adventures. A language of that kind could introduce the building blocks of programming — loops, conditionals, events — in a fun way that gives instant feedback and makes debugging fun.

An OO component could make it easy to “clone” your succesful monster, trap, whatever, and re-use the code. Introducing test cases is perhaps overkill at this stage…

I haven’t seen Lego’s Mindstorm stuff, but if anyone can make IDEs for kids, it should be them.

Update: Bill Ward writes in a comment:

For me it was BASIC on the Commodore too. But today’s kids have options as well. I think Javascript may be a good choice. My wife is taking a Flash class at the local college, and teaching me what she is learning. That could be a good choice too, except for the fact that it’s rather expensive.

I remember someone prophesying that Windows Scripting would be the next “laymans programming language”, but I haven’t seen MS promoting it that way. Having an easy to learn powerful scripting language built into the OS would introduce lots of people to programming, not just kids.

I learned Perl in my first real consulting gig at Agero. A large business directory company in Sweden wanted to synchronise their print catalogue with the Web. Additionally, they wanted an interface for customers to create their own ads on the Web. This was the sexy part of the project. I wasn’t involved there.

The synchronisation didn’t work yet, so every Monday my colleague had to take a 650 MB XML-file and feed it to a Java program that inserted the contents into a big old Oracle database running on a Sun Starfire. She was much more billable than I was, so as I incautiously admitted to Un*x knowledge I was asked if I could take over this job.

The XML was full of errors, unescaped ampersands, invalid characters… The Java program choked if it couldn’t parse the file, so you had to manually search for the error and fix it, then try again. A successful run took about 9 hours.

I started by chopping up the file into the component entries and checking for bad stuff. This is trivial, just set $/ to whatever end element suits your fancy, but it took some reading of the Perl Cookbook before I had it nailed.

Then I started looking at how to automate this stuff. I eventually wrote a sophisticated run-control program that could be started with at, and that sent email when something went wrong.

Just when I had cut down the effective load time from three days to about 11 hours, the whole project got axed. I later learned that this was the third attempt to integrate the print version with an online database.

The contractor more or less blamed the whole debacle on us, even though it could be fairly laid at bad project management and unrealistic promises from the client to its customers. Oh well.

In the middle of my next project, I was cding up from a directory over a slow ssh link and accidentally rmd all my perl code. When I called the admins of the machine they helpfully informed me that as the machine wasn’t in production it didn’t have backups.

So now I know more Perl than I really want to. But I’m still learning more every day.

After a tip from Rui, I’ve started to sort my work mail (mail addressed to me personally, and the support box) into quarterly archives.

Long experience has told me never to throw away mail, and the quarter seems to be a good time period in which to ask yourself “when did I get that mail?”

Learn about the fifteen spaces defined in Unicode at this page.