Category Archives: Uncategorized

python & qt

I wanted to learn GUI programming in QT for a long time but never really found enough motivation and a good toy problem to start from. Finally the motivation presented itself as a more or less unconstrained task in one of the courses at school, called computer graphics. I decided to implement the IFS fractal generator that I once saw in The Computational Beauty of Nature . It’s really strange how couple of simple equations can generate such natural looking objects and they would look even more realistic just by adding some random.

Here are couple of screenshots from the application:

fern_colordepth

ifs31

QT turned out to be a really elegant library and it’s documentation in the form of qt-assistant was probably the best library documentation that I have ever seen. It is easily navigable and provided lots of good examples that helped a lot. For PyQt specific stuff I skimmed through Rapid GUI Programming with Python and Qt (seems to be a good book) and read some of the sources of the Eric IDE. Altogether it took about 5 working days to write this and most of the time was spent learning QT, so probably now it would take a lot less time, which is impressive considering it has several dialogs and runs rendering in a separate thread etc.

Best of all – things written this way are really portable [1] and will run just fine under linux, *bsd, Mac OS X, windows and several other operating systems. Anyway, here’s the source and you need to have PyQT installed in order to run it (python-qt4 package under debian/ubuntu linux).

1 – as opposed to MS style portable where it usually means you can get it running on other windows versions if you are lucky enough.

swedbank ja KMyMoney

Kunagi sai kirjutatud pisikesest scriptist millega ma Hansapanga konto väljavõtteid KMyMoney’sse importisin. Vahepeal on aeg veidi edasi läinud, Hansapangast on saanud Swedbank ja ka nende konto väljavõtte CSV formaat veidi muutunud.

Sai ühesõnaga eelpoolmainitud scripti veidi uuendatud. Lisatud sai võimalus kasutada -c võtit mis võimaldab kontrollida kas mapingu listis (payee_map) on kõigi maksjate kohta defineeritud regexpid. Ühtlasi saab scripti confida nüüd ka QIF formaadi profiili input filtriks.

Script ise siin.

Mõned pildid QIF Swedbanki jaoks QIF profiili confimisest:

kmm1

kmm2

kmm3

kmm4

kmm5

kmm6

kmm7

kmm8

snow

Finally we have a decent winter with lots of snow like it used to be in my childhood. Actually it’s even better since according to meteorologists we just set a new snow record for the last 100 years with 62cm of snow, the last record was 59cm from the year 1968. And the snow still keeps falling…

my with bsd daemon

laitse seltsimaja

our homestreet

skiing

Python namespace hack

If you import a module into your Python program an isolated namespace is built for it which doesn’t contain objects from the caller’s (importer’s) namespace. While this is a lot better than PHP’s everything in single namespace approach, it sometimes gets in my way too.

For example I have an application where I want an imported module to see some things from the importer’s namespace.
The application is basically a web app server written entirely in Python and it imports modules that implement specific applications.
A custom namespace is built for the modules before import and it contains some dynamic objects that are
managed by the core server – for example a public API object that all the modules can register their own API functions into, database connection pools etc.
All is good as long as the module lives in a single file but if it consists of several files it has to do imports in extravagant ways (like the server itself imported the modules) to pass these global server objects along correctly.

Here’s a simplified example:

server.py

api = {'test': 1}
 
def build_app_ns():
    return {'api': api}
 
execfile('app.py', build_app_ns())

app.py

import app_views
print "APP:",api

app_views.py

print "APP VIEWS:",api
$ python server.py
APP VIEWS:
Traceback (most recent call last):
  File "server.py", line 6, in 
    execfile('app.py', build_app_ns())
  File "app.py", line 1, in 
    import app_views
  File "app_views.py", line 1, in 
    print "APP VIEWS:",api
NameError: name 'api' is not defined

Luckily it seems to be Python’s philosophy that ugly things are not impossible but just hard to do and discouraged so I implemented a namespace helper function that I can use to import any variables from the namespaces present in the call stack, to my own namespace.

import inspect
 
def import_parent_vars(varnames):
    """Import variables with names specified in the varnames to the local namespace.
 
    We start the search from the outermost stack frame and move inwards.
    @arg varnames: a list of strings or a string containing the variable names that should be imported to the   
    caller's namespace.
    """
    if isinstance(varnames, basestring):
        varnames = [varnames]
    stack = inspect.stack()
    try:
        caller_frame = stack[1][0]
        for f in reversed(stack[2:]):
            fobj = f[0]
            for v in varnames:
                if v in fobj.f_locals:
                    caller_frame.f_locals[v] = fobj.f_locals[v]
                    varnames.remove(v)
    finally:
        del stack

Now if we use this helper function from the app_views.py to get the variable api into its namespace everything works as desired:

app_views.py

from ns_helpers import import_parent_vars
import_parent_vars('api')
 
print "APP VIEWS:",api
$ python server.py
APP VIEWS: {'test': 1}
APP: {'test': 1}

The need for this namespace hack can probably be removed altogether if we just refactored all the shared stuff into a separate module and imported that from all the application files but it would require a lot more testing and the restructuring of our existing code.

e-voting and Linux

For the last 4 elections (from about 2005 or so) we (Estonians) have been able to vote electronically from the privacy of our own homes using our national ID card.  It really is rather convenient and helps to increase voter turnout. Since the current (local elections) are not finished yet it’s not clear how many of the votes were given electronically but at the moment it’s more than 60000 votes. This is a rather large percentage considering that the whole population of this country is about 1.5 million and voter turnout on the last local elections was a bit over 500 000.

Anyway what I really wanted to talk about is the e-voting support for Linux, that really managed to surprise me in a positive way. The Linux binary of the voting app has been available since the beginning, but a couple of months ago when we had elections for the European Parliament I discovered that this binary was only available for 32 bit Linux and I couldn’t get it running on 64 bit Linux even in the 32 bit compatibility mode. So I sent a complaint letter to the Election Board and used a Windows machine for voting. I really didn’t expect that they will get it fixed in a couple of months, even though it probably only needed recompiling for amd64 target. But they did and the new 64 bit Linux election app actually worked fine and was half the size of the old one even though the old one was dynamically linked and the new one is statically linked.

I guess it’s now time to ask for FreeBSD support and insist that they open source it 🙂

icebiking, again

Another nice day for icebiking – this time the roads were all literally covered with thick layer of ice and I actually even passed couple of ice skaters. I Also saw another icebiker who by the looks of it didn’t use studded tires so his trajectory was rather erratic.

icebiking

apache deaths with signal 4

Yesterday I did an major upgrade to an rather busy website (should be in the top 10 most used in my country). Basically I moved the code from the old 32 bit single-core webserver running FreeBSD 4.8, Apache 1.3 and PHP 4.X to a new 64 bit, 4 core server running FreeBSD 7.0, Apache 2.2, PHP 5.2. As you might imagine changing so many variables at once caused several surprises. Most interesting one was that Apache died on some pages with signal 4 (SIGILL) which stands for illegal instruction. It’s not that uncommon to see apache deaths with signals 11 (SIGSEGV) and 6 (SIGBUS) which are usually caused by bugs in some PHP modules but signal 4 was something new to me. Quick Google search turned up some threads were people reported getting rid of it by changing module orders in PHPs extensions.ini file and a bug report about PHP function preg_replace() sometimes causing it.

I tried both theories and neither one was the case that I had. Then I spent some time looking at the coredump with GDB but in the end I resorted to good old binary search strategy. So to make a long story short the cause turned out to be a recursive function in the website code that under some circumstances never stopped recursing until it overflowed it’s stack.

using kmymoney with hansapank

I’m using KMyMoney (KMM) to keep track of my finances and in general I think it’s a very nice program but I find inserting the transactions (ledgers) by hand a very tedious thing to do. I’m a client of the Hansapank and their web banking interface allows me to download transaction history in a CSV file. Since I practically always pay with debit card this transaction history covers almost all my transactions so the only question remaining was how to import it into KMM.

KMM seems to be able to import QIF and OFX formats, so in order to get my CSV file imported I wrote a python script that converts Hansapanks CSV to QIF (with KMM transaction ID extension).

Here it is, maybe it’s useful for someone else too. You can run it like so:

python hansacsv_to_kmymoney.py my_transaction_history.csv > out.qif

and import the resulting file into KMM. There’s an payee name based mapping mechanism too that allows you to map payee name from the CSV file to different name for the QIF file and attach category to it. There are several examples of such mappings near the beginning of the file.