best effort technologies

It occurred to me today, that this principle when applied to software design is quite effective. Then I thought about all the softwares we use today and tried to find instances of this principle applied in practice. And I came up with this post.

The 'best effort' approach comes into picture when you are trying to solve a problem that is very difficult to solve. The solution for the problem is not defined, to the extent that it might seem infeasible.

Here are some examples we see in today's digital life that obey this principle. Although they are specific instances I think they are representative of many others like them.

Tivo
I had read about how Tivo started and evolved. Initially they had to figure out how to collect the show times of variety of shows on different channels, so that their users can program their boxes to record their favorite programs. Their initial solution was as rudimentary as employing manual labor to go through the show times published in newspapers and formatting and feeding them to their boxes over network. They surely didn't have any solid scheme to collect the meta data of the shows. Being a startup, they couldn't expect all channels providing such information for a technology that was so unusual and unproven. But Tivo applied a best effort strategy. And we all know how well it paid.

Amarok
I haven't seen this in other players, it might be there, I am just not aware. But only today I found out how Amarok's context tab fetches the lyrics of a song I am playing on the fly using the best-effort strategy. I don't know if they scan ID3 tags or just file name or what else, but their strategy just worked for a song I randomly received from a friend. I have been using iTunes for more than a year, I know they provide lyrics in some cases (I got continuum's lyrics in a pdf when i bought the album). But iTunes controls the music they deliver. The problem to solve here is: "Given an audio song, show its lyrics in real time". In an ideal world you could do speech processing. But in a more practical world, meta-data is the only solution. For iTunes it's fairly easy to acquire the meta-data, because they control the data itself. Softwares like Amarok, won't have the luxury of well defined meta data. So they do the best they can. They might be subscribing to some well defined repositories of lyrics and just doing best effort match against ID3 tags or filenames. If they find exact match, they show it; if not, they give suggestions. It works like a charm though.

Thunderbird
Have you used MS Outlook in your office and tried to search through the mail by typing in that box on upper right. If yes then by now your mind must have fixed this action to a non-responsive interface hung for at least 5 seconds. I experienced this in my last job and have heard same comments from colleagues. Fortunately I switched to Thunderbird in my current job. Although Thunderbird isn't a perfect mail client, just see how slick it makes searching through your email. The reason it gives results so fast, is because by default it searches only through the subject line and To, Cc list of the emails in the folder. This is their best effort for speed. And it yields desired results almost always, because it is based on a right assumption - a subject line is the summary of the email. Surely this search might not be sufficient ~5% of times when you don't know what you are looking for and remeber only vague keywords. But that is covered by giving you an alternative to include entire mail body for search. (Why MS misses this is topic of another discussion)

Google search
I am no expert in search technologies. But just by observing the output of the software I can tell, that Google search (or any search for that matter) wasn't full proof in the beginning. In the beginning of search era, one could get unexpected search results than what they originally intended to find. But Google did the best they could do to try to understand what user was looking for. They didn't waste their time implementing a classic solution for Natural Language Processing, they just tried their best to figure out the user's intent. No need to mention how successful the strategy has been.

Google desktop
I remember Microsoft planning a filesystem (for longhorn/vista) designed on top of a database, so that it will make searching through your desktop files as efficient as SQL query processing. Everyone knows what happened to it. Compare this to Google's best effort approach to solve this problem. They didn't need to redesign the whole file system. They instead made a daemon which runs non-intrusively in background indexing your files. Initial indexing will take time, so if you install Google desktop and immediately fire a search it won't give you proper results. But that's only for first few hours. Once indexing is done Google desktop will give you what you're looking for. It must be doing some incremental indexing for the new files/emails that keep coming in. But that works so well. I remember using it as an alternative to MS outlook's search box at my last job. It would show up fairly recent mails in the search results.

.... I think we can find more of such examples easily.

I don't think this is a radically new concept/term I am talking about. I think this approach - "best effort strategy" - to software design is just one of the shades of extreme programming. It is just one of the lessons that can be learnt from the software startups in recent past. But it works so well, doesn't it?

formula of a horror movie

Imagine you are working all day at your office. You interact with so many people. Many people come to your office or desk uninvited. They just show up standing besides you, you at first fail to notice them, then suddenly you become aware of their presence. You travel on a subway or in a bus or shop in a department store, but when a totally stranger face or body shows up in your eyesight, you just take it granted - let alone stun or scare.

Now you come back from your office or bus or mall to your two bedroom apartment where you live alone. You go into the kitchen, start cooking... after few minutes you realize that a total stranger is standing 4 feet from you looking at what you are cooking.... you will be just short of a heart attack. However benign and harmless that strange face is, you will skip few heart beats before coming to senses. It is a regular stranger whose presence you wouldn't even acknowledge, had it been a grocery store and not your kitchen.

Then why is it so scary?

The answer is simple. We feel anxiety if something unexpected happens. And non-expectancy of someone other than you being in your apartment is extremely high "in your mind". Naturally your anxiety in response is also very high, almost a shock or horror.

This is what I think about the concept.

Imagine of a finitely practical situation in which one person's non-expectancy of an event is entirely commonplace occurrence for other person. And the situation should be practical in the sense that the audience should identify with both persons' points of view.

Imagine such a situation and you can have an intelligent horror movie.

I guess this is a general equation for horror movies, everyone just finds a different solution that satisfies all sides of it.

next desktop jump

Now Fedora 7 is out for couple of months, Fiesty Fawn was out even before that. So it was obvious that I would find something wrong with my Fedora 6 installations and convince myself to get bored of it.

I am the Redhat guy, and never could make myself like package management of other distros. But Ubuntu making so much noise and after reading Mark Shuttleworth's blog for a while, I couldn't avoid getting excited about Feisty Fawn. I gave it a first shot within 2-3 days of its release. But I found myself afraid of diving into the LVM configuration that Kubuntu-alternative installation CD offers. I didn't want to blow my existing LVs. I did some experiments in a virtual machine. Finally I braved myself into swallowing the pill. Seconds later I felt like choking when the system went into freeze at a screen after I chose to configure LVM. Thanks to the shell, I realized it wasn't frozen but was blocking. When I tailed the system logs, I found it was waiting for each volume to get discover and was waiting 30 seconds on each of them. With 7 volumes I could manage to steal a nap. Once it was done, it listed all LVs and rest of the things went smoothly. After entire installation the system booted just fine, without any video driver problems etc. Usual setup tasks (firefox, thunderbird,....) went fine. Liked the GUI on top of apt-get system. But wondered why it was giving some red colored "Bad install" errors during installation of some packages. It's dependencies might be failing, but couldn't find any straightforward error saying so. My biggest motivation for desktop switch was Beryl. But after spending an hour or so and following all instructions on forums, I still couldn't get beryl-manager and emerald-theme-manager to install. The repo that provided them didn't provide many dependencies that were needed. Big bummer! Never could find any repos that could satisfy those dependencies (and there were lot). I left it aside for a while and started installing gaim. Realized pidgin is the new name. The same story repeats. I couldn't install pidgin after spending some 10 minutes...... Thankfully I had my FC6 installation on one of the volumes, so I switched back.......


....enter Fedora 7. Installed it first day (or second?!). Everything went fine. But it wouldn't boot alright - giving me tons of Red "[Failed]" results for various services. I always have these problems office machines where NIS, NFS all doesn't work unless you do everything right. I was lucky with FC6 last time. But F7 got screwed. so back to FC6 for few days....

Since I can't reboot my office machine quite often (with so many things open all the time), it took till this week to me try again. My FC6 was very sluggish. The reason I found out was: my home directory was on the network drive. And our file server is overloaded nowadays. So I had decided to create local account and home directories next time. On Monday, I reinstalled F7 and didn't configure NIS at the installation time. Everything worked like a charm. No... Beryl won't work out of the box. But I knew what to do. It seems the stock NVidia driver that comes with F7 (nv) isn't sufficient (at least for the graphics card I have). But the additional steps I had to do were straightforward:
Goto Livna and install their RPM which automatically adds its repo address to your yum configuration.
Install kmod-nvidia using yum install. [Note: Do NOT install this RPM using rpm commandline, USE yum. More likely this driver is compiled with a newer linux kernel that the stock F7. So "yum install" will automatically do this upgrade, you might miss that out with manual rpm installation. I read some posts which messed up because of this]
Reboot (Now you should see nVidia logo before X starts)
Run Beryl-manager it should work.


........... So Fedora 7 gets my vote again this time.

I think Ubuntu is a nice system. Might be very user friendly, once it's installed. But it is still not good for power users (or at least not enough to lure RedHat power users to its camp). I wish just like kubuntu, xubuntu, they come up with "Pubuntu - Ubuntu for power users" - which has support for LVM management at installation time, etc.

P.S. Pidgin was a great find. It has awesome GUI compared to gaim.

The "Screen"

Thanks to a friend who introduced me to the screen utility about a year ago. It is a very fine utility for unix programmers/users who spend most of the time using CLIs. I would say it is a desktop virtualization solution applied to command line terminal. It is to the command prompt, what VNC is to the graphical desktop. To learn more follow the above link or google. It is installed on almost all linux distributions by default, so you can just start using it right away.

I was however slow in adopting it. Mainly because switching between the screens wasn't so quick. The screen command line utility gives an option of creating tabs in one screen and switching between them, but that meant mastering several key sequences. So I wrote my own wrapper script to do the job. Following is the code. Save it in a file and name it suitably (use a short name that's handy - I call it 'sc') and just execute it.

The code is available here

Here is a sample run:

jayesh@bluebox [~] sc --help
Usage: sc
Generates list of screens if available and gives a prompt
Valid commands to enter at prompt -->
new-<NEW_SCREEN> - creates a new screen with name NEW_SCREEN
                   and connects to it
[number]         - number is the index of the screen you want
                   to connect to
q                - quit

jayesh@bluebox [~] sc
Create new screen. Use the command 'new-'
>>new-root
[Now you can login as root on this user to do all administrative tasks]

..... 
[create bunch of other screens for others tasks you do -
 compiler,editor, etc.]

jayesh@bluebox [~] sc
0] compiler
1] root
2] editor
>> 1

[Now you connect to screen 1, where you are logged in as user. 
You don't have to enter root password again-n-again]

Suggestions are always welcome!

Nice review article on virtualization technologies

This one is pretty thorough analysis of status quo on Virtualization.

I liked this quote best...

"Between Microsoft, Novell, Red Hat, XenSource, VMware and the scarcity of good benchmark data, it's going to be tricky for users to figure out which operating systems will deliver the best virtualization performance on whose version of which hypervisor."