JavaScript is the next C

I've been coding in JavaScript for almost a year now. I find it a decent language to write code in. The code written in JavaScript doesn't look as pretty as python, but it doesn't look as bloated as Java either. But that's not why I chose the title of this post. It's everything else that's happening around JavaScript that makes me think that it's going to play as significant a role in computing as C has played.

Before C, there was assembly. (I know there were many other languages before programming evolved from assembly to C, but I am talking about only the mainstream candidates.) The abstraction that C provided was perfect enough that programmers didn't have to learn about the Assembly details. That made C the perfect candidate to build a solid layer on top of Assembly. Programmers could write accounting software, graphic programs, games without bothering about what CPU architecture and memory bus size they were running on. It all worked and C became an integral part of computing stack. Today we don't use C to write new programs, but all operating systems, their device drivers, native libraries are written in C. C has become so ubiquitous that it's now invisible.

We have spent a long time searching for a candidate language to build the next layer, on top of the one built by C. C made our code hardware-agnostic. But over time we developed a variety of operating system platforms that created their own stacks of libraries. With the advent of Internet we needed to transfer code over the wire and be sure that it can be run on all the machines irrespective of which platform they were running. We needed a write-once-run-everywhere solution. Java emerged as the solution to specifically fill that need. For a period of time, it seemed it would indeed be the one. But for several reasons it failed. Today when we are deciding a platform independent solution to write a GUI program, what number does Swing score in our preferred list? Java did a great job of freeing the programmers from worries of memory management. But the main reason for its failure is probably its awkward ownership by a single commercial entity Sun (and now Oracle). The lawsuit Google is facing over Android is enough to corroborate this. If someone can sue you for using their language, then how can such language be adopted by entire industry.

The latest candidate to build the next abstraction layer in the computing stack is JavaScript. It's hardly a new language. It's hardly a perfect language. But there are two technologies that will make JavaScript the next C - HTML5 and Node.js.

HTML5 (and whatever goes under that umbrella name) is a new web framework whose primary programming language is JavaScript. It is bound to succeed, in fact if you consider it as just a new fancy name for HTML then it has already succeeded. It's not developed by a single company, but many big guns are simultaneously promoting it  - Google, Mozilla, Apple, Microsoft (by accepting most of the standard for IE9), are writing virtual machines that keep improving the speed of JavaScript. The new age behemoths - Twitter and Facebook - have strengthened HTML5 merely by adopting it.

On the other hand, Node.js has become a great success. It has a teeming community around it. The success of a platform depends on the libraries its provides to do various tasks. Just look at this wiki page that lists different Node.js modules to accomplish various tasks. I spent last week writing a factory server for 3DTin using Node.js. Everything I needed - from binary encoding library to canvas rendering libraries - I got it from this page. And there are more than one option for each job. Most of the libraries are nascent and will mature over coming few months. But they are a strong sign of a solid platform taking shape.

Yet another sign of JavaScript's growing might is, its choice as a target language for compilers of other languages. There are projects compiling Ruby, Python, Java right into JavaScript. The shortcomings of JavaScript as a language are being fixed by many frameworks successfully, think of jQuery. Libraries like Underscore.js or projects like CoffeeScript are making programming in JavaScript more fun.

Between HTML5 and Node.js you can now be sure that if you write your next application in JavaScript it will run on any server, desktop or mobile. JavaScript will build that next layer in computing stack that we are waiting for and that's why it will be the next C - ubiquitous and then invisible.

Talking Twitter client

So I learnt this afternoon about this Linux utility called festival. It's a text to speech conversion program. Running it is as simple as

echo "Hello world" | festival --tts

Moreover, installing it on Fedora is as easy as

sudo yum install -y festival

After that, a bit of a bash and a bit of a python and I had a twit-to-speech utility running.

The code is simply this much:

#!/bin/bash

TWITTERURL="http://twitter.com/statuses/friends_timeline.json"
JSON="/tmp/twittline.json"
SPEECH="/tmp/twt.message"
PYCODE="/tmp/twt2speech.py"

read -p "Username: " TUSER && \
read -sp "Password: " TPASS && \
curl -s -u $TUSER:$TPASS $TWITTERURL > $JSON

cat > $PYCODE << "EOF"
import json
import sys
import re
urlp="(https?|ftp|file)://[-a-zA-Z0-9+&@#/%?=~_|!:,.;]*[-a-zA-Z0-9+&@#/%=~_|]"
twits = json.load(open(sys.argv[1]))
for twit in twits:
    text = twit['user']['name']+' says: '+twit['text']
    text = re.sub(urlp, '', text) 
    print text
EOF

python $PYCODE $JSON > $SPEECH

while read line
do
notify-send -t 15000 "$line"
echo $line | festival --tts
sleep 1
done < $SPEECH

echo "THE END" | festival --tts

code syntax highlighting by GVIM

The above script will ask your twitter credentials, fetch latest 20 twits in friends' timeline, save into a JSON file. A short python script parses the JSON, extracts twit text and user's name from it and outputs in a sanitized format (it removes URLs, because there is no use hearing them).

The sanitized output is  saved in another text file, which is piped one line at a time to festival. In case the speech is not clear, it also shows the text in a pop-up using notify-send.

A full script with some error checking can be found here.

Makeshift XML beautifier in python

Lately I had to deal with some XML dumps. It's a pain to analyse XML if it's not properly indented. Browsers do the best job of rendering XML. Those collapsable XML elements are very handy. But they work only if the XML file is downloaded with right MIME type. That's why XML dumped to a local file and opened using a browser doesn't get the same treatment.

I am sure there are other XML beautifiers, but I couldn't find one that will work for me. (I am sure in comments someone will post better solutions). Finally following simple python script did the trick. I found it here and corrected a little to take care of </> tags. It worked perfectly on many XML dumps I worked with.

#!/usr/bin/python
import sys
import re

data = open(sys.argv[1],'r').read()

fields = re.split('(<.*?>)',data)
level = 0
for f in fields:
   if f.strip() == '': continue
   if f[0]=='<' and f[1] != '/':
       print ' '*(level*4) + f
       level = level + 1
       if f[-2:] == '/>':
           level = level - 1
   elif f[:2]=='</':
       level = level - 1
       print ' '*(level*4) + f
   else:
       print ' '*(level*4) + f

It's all about keeping track of depth.

Inkface+pygame v0.2.0

As per the plan for v0.2, I am ready with a working implementation of inkface. I also made few more design decisions after my last design post. Here are the highlights:

1. All code is in python
2. For X11 surfaces, pygame is used.

I found that pygame (python bindings for SDL) is a mature library for drawing on X11 surfaces. It also has some optimizations which help in 2D animations. There is also OpenGL support, but I do not plan to use it inside inkface library. It is also well supported on Maemo.

I have written my favorite twitter client app with the new inkface-pygame library. The following video demo shows, how I could quickly code 2D scrolling animation with the help of pygame.



The v0.2.0 is delivered as a tarball. I have also updated the inkface wiki with some getting-started tips.

Debian repository on Google App Engine

So here is the post about the eureka moment I promised on Twitter.

Last week before releasing Inkface v0.1.2 and twitter-inkface client, I was working on submitting the packages in maemo-extras-devel repo. I was following a long list of instructions. Halfway through the list, I noticed that the example package is assumed to have had autotools kind of build framework. One of my packages uses SCons. I had a doubt (yet unconfirmed) that it won't work. So I scratched my head and that brought to the surface one of the ideas I have had at the back of my mind for a long time. Using one of these "cloud" services to host a debian repo.

And what cloud service can be better than the one that is absolutely free (for such a low traffic purpose, at least) - Google App Engine. So I went through the basics of Debian repo and put together an app for Google App engine. Under couple of hours, I had my very own Debian repo. (Yeah, that's when I leapt out of my bathtub and ran to twit 'Eureka Eureka!') That's what you hit when you point your apt-get to repo.altcanvas.com.

I have submitted the small script as a recipe in Google App Engine cookbook. You can also find it in altcanvas source tree.

Inkface v0.1.2, Twitter-inkface client

Finally it is ready. Actually it has been ready for more than couple of weeks now, but I wanted to make sure the bugs in my demo app were not due to problems in core library.

The demo app is a client I wrote for the Twitter service. The intention was to demonstrate how an intuitive GUI can be designed using an Image editor. With the GUI designed using Inkscape and an off-the-shelf python library to talk to Twitter service (Thanks to DeWitt for python-twitter), I only had to write simple glue code to get the whole app working. Check out the video.



Good news is I have setup a debian repo to host packages for this app.

In the Maemo Application Manager, add following Catalog:

Name: Altcanvas
Web Address: http://repo.altcanvas.com
Distribution: testing
Components: main

Refresh the application list. You should see 'twitter-inkface' app in the list. All the dependencies will be automatically pulled from the repo.

Since the repository implementation is currently experimental (No it's not a standard web server! I will save it for a separate post), you may face a problem while installing or I may have to take it offline for modifications. In that case, you can download following four packages and install them with dpkg ("dpkg -i *.deb")

libaltsvg,inkface-python, inklib, twitter-inkface.

For more info on Inkface project, visit the project page.

Debugging python reference counts

If you are wondering about the progress of inkface, it's coming soon. I was planning to release version 0.1.2 couple weeks ago with a cool app demonstrating the improvements in the library (A twitter client), but after I saw the abysmal memory performance of the app on my n810, I decided to wait. Past two weeks I spent understanding the memory leaks in the library. In this post I would like to share my experiences while debugging reference count leaks in my python bindings.

The primary documentation on the subject from python.org is the first stepping stone. Considering how daunting the task of memory debugging is - some inspiration always helps. Read this post from Guido which explains how to approach the task.

In most of the cases we start debugging the memory leak after we have seen the program crawling or ever-increasing heap profile graph from valgrind massif. Whichever is the case, the theory of reference counts as explained in above docs, doesn't make sense in this scenario. In such case, the following steps may help you get started.

• Put debug statements in the dealloc functions of your Python Type objects. This will tell you when are they getting called, if at all. They will be called when Py_DECREF call on your python object decrements its refcount to zero.
• Identify the python objects that you suspect to be leaking (i.e. the objects whose ref counts are not reaching zero when you expect them to) and write a simple routine that dumps their refcounts. You can write this routine inside your C module or in python - depending upon how your code is organized. In C you can find PyObject's refcount by its member ob_refcnt. In python you can do the same by passing the object to sys.getrefcount() function. Call this refcount-dump routine from various places in your code and monitor how the reference count of these objects varies.
• By calling the refcount-dump routine at strategic points you will soon narrow down to the area that is increasing the ref counts unexpectedly (or more correctly forgetting to decrease it when the job is done). Now look in the Python-C API docs for the behavior of the API calls you are making in this particular code segment. Understand the meanings of borrowed reference and new reference. Soon, you will find the places where you are supposed to call Py_DECREF, but you haven't.
  

While debugging my code I had following observations.

• The ref counts on my objects were astronomical. After a while I figured that they were increasing with time. This observation clearly showed that the bug was in a loop structure. Soon I found the problematic code and put Py_DECREFs.
• One common place of forgetting to release the ownership of an object is while iterating over a sequence using PyIter. Note that, PyIter_Next returns a new reference and you have to release it (roughly at the end of the iteration)
• Sometimes Heisenberg's law kicks in ("you can't observe something without influencing it" as mentioned by Guido in above post). In my case, the python objects I was tracking were saved as values in a dictionary. So the only way to refer to them was

  sys.getrefcount(dict.values()[index])

  This leads to a creation of list of values, thus incrementing the reference count of the object I am monitoring. There is no easy way that I know of to work around this, but just taking into consideration these additional refcounts helps.
• In my code, I have a pure python class which encapsulates the objects created by C library. So the deallocation of C objects was dependent on the deallocation of pure python object. So I had to track its refcounts as well. One surprising thing I noticed was this object had very high refcount immediately after its instantiation. e.g.
  

  o = someclass()
  print sys.getrefcount(o)  # 11
  

  This was very unexpected. The reason behind it lied inside its constructor of course. This class has several methods. And inside the constructor I register these method objects as callback handlers. This understandably increases the refcount of method objects; however as it turns out it also increases the refcount of the parent object. So in my case, I got rid of these references by unregistering the callback handlers when I wanted to release the object.
  

After the whole exercise, I concluded that debugging memory leaks by means of debugging reference count leaks is a more useful way, compared to going through valgrind logs of a misbehaving C/C++ program.

These are my findings based on less than one day's work. If you find any mistakes in my understanding, please do point out.

And yes... an update on Inkface is coming soon.

Inkface v0.1

After spending about a month on bug fixes and writing python bindings, Inkface v0.1 is ready. Check out the video that demonstrates two applications written in Inkface framework. The video also shows how the GUI is designed in Inkscape and how to get the app running with few lines of python code.



If you want to try these applications yourself:

• Install the .deb packages (libaltsvg, inkface-python)
• Download python scripts and GUI SVG images: Keyboard (.py .svg) IRC (.py .svg)
  

# Install Inkface libraries
dpkg -i libaltsvg_0.1.0_armel.deb inkface-python_0.1.0_armel.deb

# Keyboard demo
python inkface-keyboard.py keyboard-entry.svg

# IRC demo
python inkface-irc.py irc.svg

Check out the project wiki for future plans of the project. It has some notes to get you started. A look into simple python scripts mentioned above will also help.

Here is my last month's post that discusses the idea behind the project.

Flickr uploads from n810

Finally I am done with new version of publishr for Nokia n810. Check out the demo video.



You can install from here. Check out the project page.