Debian repository on Google App Engine

So here is the post about the eureka moment I promised on Twitter.

Last week before releasing Inkface v0.1.2 and twitter-inkface client, I was working on submitting the packages in maemo-extras-devel repo. I was following a long list of instructions. Halfway through the list, I noticed that the example package is assumed to have had autotools kind of build framework. One of my packages uses SCons. I had a doubt (yet unconfirmed) that it won't work. So I scratched my head and that brought to the surface one of the ideas I have had at the back of my mind for a long time. Using one of these "cloud" services to host a debian repo.

And what cloud service can be better than the one that is absolutely free (for such a low traffic purpose, at least) - Google App Engine. So I went through the basics of Debian repo and put together an app for Google App engine. Under couple of hours, I had my very own Debian repo. (Yeah, that's when I leapt out of my bathtub and ran to twit 'Eureka Eureka!') That's what you hit when you point your apt-get to repo.altcanvas.com.

I have submitted the small script as a recipe in Google App Engine cookbook. You can also find it in altcanvas source tree.

Inkface + Clutter

Clutter is the easiest way to start coding GUIs with accelerated graphics. I liked it when I first saw the demo in March last year. So all the time while working on inkface, I was waiting for some free time to use inkface with clutter. After getting Inkface v0.1.2 out, I tried my pyclutter... pretty soon figured out a simple way to combine the two.

Check out the following 12 minute video. I put together a simple program in few hours and the video documents all the steps I needed to do. It demonstrates a slider widgets which controls the rotation of a steering wheel.



The program is 169 lines of python code in a single file (including comments and some boilerplate code). It loads the GUI from two svg files (slider, steering). As you can see the slider is unlike any GTK or Qt widget and it was put together in less than 10 minutes in Inkscape (the video is real-time).

[You can skip through the video from 1:00 to 9:40, if you don't want to see all the details of drawing in Inkscape]

Now animated "clutter" GUIs can be designed with smooth curvy shapes and smart color palettes, which are difficult to program, but easy to compose in an image editor. Libraries like clutter are making it possible to design out of rectangular widgets. A library like inkface makes it possible to design those new widgets in an image editor.

Inkface v0.1.2, Twitter-inkface client

Finally it is ready. Actually it has been ready for more than couple of weeks now, but I wanted to make sure the bugs in my demo app were not due to problems in core library.

The demo app is a client I wrote for the Twitter service. The intention was to demonstrate how an intuitive GUI can be designed using an Image editor. With the GUI designed using Inkscape and an off-the-shelf python library to talk to Twitter service (Thanks to DeWitt for python-twitter), I only had to write simple glue code to get the whole app working. Check out the video.



Good news is I have setup a debian repo to host packages for this app.

In the Maemo Application Manager, add following Catalog:

Name: Altcanvas
Web Address: http://repo.altcanvas.com
Distribution: testing
Components: main

Refresh the application list. You should see 'twitter-inkface' app in the list. All the dependencies will be automatically pulled from the repo.

Since the repository implementation is currently experimental (No it's not a standard web server! I will save it for a separate post), you may face a problem while installing or I may have to take it offline for modifications. In that case, you can download following four packages and install them with dpkg ("dpkg -i *.deb")

libaltsvg,inkface-python, inklib, twitter-inkface.

For more info on Inkface project, visit the project page.

Inkface update

So as promised, here is an update on Inkface. I am ready with v0.1.2, almost.

A main change in this update is, the definition of a new abstraction called 'Face'. This was introduced to make programming the app more intuitive. A 'Face' object is initialized by giving it a name of SVG file that holds UI elements. One can create several face objects that hold elements from different SVG files, and as per the program logic can add-remove these 'face' objects from the canvas. The elements in the SVG files can be addressed as attributes of the face objects in which they are loaded. This makes the app source code very intuitive. The management of the element objects is done behind the scene without the app programmer having to worry about it.

To demonstrate these features, I have written a simple yet useful app using above concepts. It's a twitter client, written using python-twitter library as backend and inkface-python as front end. This app depicts friends' and public's twits in a more playful manner as clouds and banners, rather than rectangular gtk widgets. It enables text-entry by a face object called 'Keyboard' which implements the logic of text entry (from touch screen interface as well as actual keyboard of n810). I want to polish the app further before making a demo video, so I am not going to rush it today.

The inkface infrastructure is now divided into three libraries: the SVG parser and renderer library - libaltsvg (derivative of librsvg); the python bindings for inkface - inkface-python; a helper python library which defines above mentioned abstractions - inklib. There are deb packages for each of these components. A good news is, I have got permission to upload these packages to maemo extra repo; so users now won't have to worry about dependencies.

Soon I will have to seriously concentrate on the performance issues. As you will observe while running the twitter client, the app takes nearly 50% of n810's total 128M memory. So it surely is not healthy. The python ref count debugging made sure that the memory was no longer leaked in buckets; but a vector graphics based app is known to need more resources. I plan to look deep into librsvg and cairo data structures to trim down the memory usage.

Meanwhile, this update also has some code for OpenGL backend, however it's disabled by default. It is targeted for desktop and OpenGL ES based handhelds.

Tommorrow I am leaving for a vacation, so after I return next week, I will do some final testing and release the deb packages.

That's all for now.

Merry Christmas to all!!!

Debugging python reference counts

If you are wondering about the progress of inkface, it's coming soon. I was planning to release version 0.1.2 couple weeks ago with a cool app demonstrating the improvements in the library (A twitter client), but after I saw the abysmal memory performance of the app on my n810, I decided to wait. Past two weeks I spent understanding the memory leaks in the library. In this post I would like to share my experiences while debugging reference count leaks in my python bindings.

The primary documentation on the subject from python.org is the first stepping stone. Considering how daunting the task of memory debugging is - some inspiration always helps. Read this post from Guido which explains how to approach the task.

In most of the cases we start debugging the memory leak after we have seen the program crawling or ever-increasing heap profile graph from valgrind massif. Whichever is the case, the theory of reference counts as explained in above docs, doesn't make sense in this scenario. In such case, the following steps may help you get started.

• Put debug statements in the dealloc functions of your Python Type objects. This will tell you when are they getting called, if at all. They will be called when Py_DECREF call on your python object decrements its refcount to zero.
• Identify the python objects that you suspect to be leaking (i.e. the objects whose ref counts are not reaching zero when you expect them to) and write a simple routine that dumps their refcounts. You can write this routine inside your C module or in python - depending upon how your code is organized. In C you can find PyObject's refcount by its member ob_refcnt. In python you can do the same by passing the object to sys.getrefcount() function. Call this refcount-dump routine from various places in your code and monitor how the reference count of these objects varies.
• By calling the refcount-dump routine at strategic points you will soon narrow down to the area that is increasing the ref counts unexpectedly (or more correctly forgetting to decrease it when the job is done). Now look in the Python-C API docs for the behavior of the API calls you are making in this particular code segment. Understand the meanings of borrowed reference and new reference. Soon, you will find the places where you are supposed to call Py_DECREF, but you haven't.
  

While debugging my code I had following observations.

• The ref counts on my objects were astronomical. After a while I figured that they were increasing with time. This observation clearly showed that the bug was in a loop structure. Soon I found the problematic code and put Py_DECREFs.
• One common place of forgetting to release the ownership of an object is while iterating over a sequence using PyIter. Note that, PyIter_Next returns a new reference and you have to release it (roughly at the end of the iteration)
• Sometimes Heisenberg's law kicks in ("you can't observe something without influencing it" as mentioned by Guido in above post). In my case, the python objects I was tracking were saved as values in a dictionary. So the only way to refer to them was

  sys.getrefcount(dict.values()[index])

  This leads to a creation of list of values, thus incrementing the reference count of the object I am monitoring. There is no easy way that I know of to work around this, but just taking into consideration these additional refcounts helps.
• In my code, I have a pure python class which encapsulates the objects created by C library. So the deallocation of C objects was dependent on the deallocation of pure python object. So I had to track its refcounts as well. One surprising thing I noticed was this object had very high refcount immediately after its instantiation. e.g.
  

  o = someclass()
  print sys.getrefcount(o)  # 11
  

  This was very unexpected. The reason behind it lied inside its constructor of course. This class has several methods. And inside the constructor I register these method objects as callback handlers. This understandably increases the refcount of method objects; however as it turns out it also increases the refcount of the parent object. So in my case, I got rid of these references by unregistering the callback handlers when I wanted to release the object.
  

After the whole exercise, I concluded that debugging memory leaks by means of debugging reference count leaks is a more useful way, compared to going through valgrind logs of a misbehaving C/C++ program.

These are my findings based on less than one day's work. If you find any mistakes in my understanding, please do point out.

And yes... an update on Inkface is coming soon.