Author Archives: hadara

SVG fun

I have lately been interested in vector graphics in general and SVG in particular. My hope is that it would potentially allow us to do web design in a far saner way than using about 6 images for each box that requires rounded corners and some gradients. The result would also be nicely scalable to different resolutions and probably take far less bandwidth to serve since the aforementioned box would probably just be represented by a couple of hundred bytes of XML in the SVG file and this can be packed to take even less space. As an added bonus you can also get reasonable diffs from the revision control for your SVGs.

Anyway here’s a simple SVG analog clock widget, a kind of Hello world if you will.
The idea was to write a clock widget that can be easily redesigned by non-programmers with a WYSIWYG SVG editor like Inkscape.

Here are a couple of examples that should be visible in all the reasonable browsers (ie. not IE) that have been released since 2002 or so:

Basically you can take either of these files and clock.js from the same directory and edit the SVG file in Inkscape and it should still work.
There are couple of things you have to keep in mind while editing though:

  • There should be clock hands with the ID’s second_hand,minute_hand and hour_hand
  • There should be a circle with the ID clockface which is used to determine the center of rotation.
  • The IDs of objects can be changed from the Object properties dialog
  • You should probably use the align tool to place the clock hands vertically to the center of the clockface element, otherwise the clock will act funny
  • The clockface doesn’t have to be visible, the code only needs it to know the center of rotation. So you can make it invisible if you want to create non-circular clockfaces

This kind of simple things can actually be done without any ECMAScript whatsoever using only SVG SMIL animation. I haven’t really investigated SMIL yet though.

house sensor network

Lately I have spent some time with a friend on building a temperature sensor network for my house. We used CAT5 cables for the wiring and DS18B20+ digital temperature sensors (~2.4€ a piece).

The network runs in two branches and is powered directly from the serial port so the sensors run on the phantom power.
The serial port connector is based on a schematic from this article (in Estonian). It’s actually connected to a serial-to-USB converter which is plugged into the powered USB hub which in turn is plugged into the Beagleboard. I use the digitemp utility to read the sensors.

Since we used CAT5 cables there are still 6 wires left that can be used if we need to transmit external power for higher power actuators and/or sensors in the future.

The connection points to the one-wire network are done with RJ45 surface wall mount boxes.
Here’s a picture of one splitter box for sensor connections and one sensor embedded directly into the box:

sensor_connection_small

Here’s what the sensor cables look like:
sensor_small

And finally here are a couple of graphs from last night:

house_temp_short

house_temp_long

Live data is available @ pachube.

power management & measuring

I recently bought myself a couple of Plugwise smartplugs that can be used to control and measure power usage over the wireless ZigBee protocol. I wrote a bit about these and the alternatives in the last post.

The Home Start kit consists of a USB stick that is reasonably enough called Stick and two smartplugs: Circle and a Circle+ (a circle that has ZigBee router capabilities). The Stick takes care of the ZigBee communication and provides the operating system with a simple serial port interface.
The Circles are really nicely designed and reasonably priced at somewhere around 40€ per Circle (125€ for the kit), which is a lot cheaper and smaller than the alternatives mentioned in the previous post.

plugwise home start kit

Plugwise Home Start kit

Plugwise Circle

Plugwise Circle

The only problem with these things from my perspective was that they use a proprietary communication protocol and you are basically supposed to use their Windows based software for managing them. I haven’t used Windows in a decade or so and I’m interested in integrating it into my own system instead of using some special GUI app so I had to find some other method for communicating with these things. Luckily most of the protocol has been reverse engineered by Maarten Damen so I just wrote my own Python library called python-plugwise that implements this spec. It’s available under the MIT licence here.

The library also contains a small command line utility that can be used to test the basic functionality of the plug (measuring power, switching on/off). The Circles also support uploading an on/off switching schedule and reading the power usage log so you wouldn’t have to keep a computer constantly running just to do these things. I haven’t yet implemented these functions since I control these and other things from the Beagleboard. It is a very small and low powered ARM based fanless computer that uses around 3W of power so it doesn’t waste much resources to keep it always on.

I haven’t really tested the working range of these things but currently the Stick is about 7m away from both Circles and there’s a brick wall and a door between them and the communication seems to be stable so far.

Anyway here’s the first quick and dirty RRD graph of power usage from a couple of appliances:
power usage

thoughts on energy usage

I have lately spent considerable amount of time on reading various home automation related stuff from the energy efficiency front. With a goal of finding some solution that would allow me to measure and control electricity usage down to specific device. The idea being that in general people have found ways to reduce their energy consumption by ~20% if they see where it’s actually going. One interesting case study is here which shows that almost 1/4 of the energy in the measured office building is used up by the printers and coffee&vending machines account for almost half of the total energy usage.

In my case I believe I already have a pretty good idea what to optimize and I just need something cheap and open enough to do it.

I concentrated ZigBee smartplug solutions since it’s just really convenient to connect it anywhere without having to deal with the wiring and I already have some ZigBee stuff around for the greenhouse controller.

I came up with the following options:

  • Digi’s XBee Smart Plug. It’s currently priced at 84$ and seems rather developer friendly since simple Python examples are available. From my perspective the showstopper is that they only have US 110V version of this device available and I happen to live in europe where we use 220V.
  • Plogg. Currently priced at 97£ for their older version which as far as I can gather has proprietary interface and 135£ for the ZigBee Smart Energy profile compliant version, which hopefully means the interface is standardized. In addition I would probably need ZigBee USB dongle with Ember chip (34£) since as far as I understand from couple of google-translated form posts it wouldn’t probably work with my current XBee nodes. Add shipping to that and you get ~170£ which is a large amount of money for measuring and controlling just a single socket. Another thing that bothers me about this solution is that they don’t seem to be as open as Digi’s solution is – their FAQs talk about using their own commercial SDK for communication which is Windows only and costs you 125£ which is far cry from couple of lines of Python that it should be. On the bright side several people seem to have tapped the SDK communications and have simple scripts available that seem to work.
  • ePlug by 4-Noks. Seems nice but I couldn’t find any information on the pricing. Usually this means either you don’t want to know or we aren’t actually in production yet.
  • Plugwise. They provide starter kit with 2 smartplugs and USB dongle for 124€.You can buy extra plugs for around 30€ per socket. The kit also contains some software, but I have no use for that since it’s closed source and Windows only. Besides they have had a privacy fiasco with older version of that software which turned out to be sending your data to their servers without asking you first (it’s said that the current version of the windows software does contain option to turn it off). Luckily the protocol they use seems to be reverse engineered. Currently the main problem with this solution seem to be the shipping costs, they want 101.39€ for shipping that rather small package to me which is a bit outlandish considering the kit itself costs about as much and shipping it with normal air mail should cost max. 15€. Some resellers are shipping for less but their kit price itself is 20-30€ higher so there’s not much difference in the end result. Update: Plugwise was kind enough to send it with TNT which was a lot cheaper @ 17€

Then there’s the more fundamental question if controlling your devices should be done over such wireless protocols at all since it makes attacking your network that much simpler. The low power aspect doesn’t really help much against that since similarly to Bluetooth long-range attacks it can easily be worked around with well directed powerful antenna. I haven’t really looked well enough at ZigBee security aspects and neither am I security expert, but judging by how far more evolved WiFi security has been taken down several times I wouldn’t be hopeful that ZigBee security stands for long without major holes. Some attack tools are already available and the referenced presentation describes several awfully simple attacks (even simple replay attack is effective against several products!).

I don’t care much if an attacker can read my sensor network and determine how much power I use or what is the temperature in each room but being able to control central heating, sauna, water and gas valves etc. remotely is quite another thing. Another rather interesting question is what do you do when the protocol that is embedded in chips in wall sockets, lamps, switches and whatnot becomes trivially exploitable? Replace the whole thing? Reflash the firmware over the air? Some of the security stuff can be implemented in hardware for efficiency so firmware replacement wouldn’t help even if you could do that…

So in general I think I will use ZigBee for things that I don’t care about and for which randomly turning on and off doesn’t have any bad consequences. For critical things that can burn out or burn down the house if kept in too long I will use well shielded cable or no computer control at all.

beginnings of greenhouse automation

Last year I bought a new larger greenhouse (lwh: 6x4x2.8m). Currently it looks like this:
greenhouse in winter

Which means that I don’t have to deal with actually growing things at the moment and now is a good time to make preparations for the next season.

My ideal is to automate it as much as possible, so first I will try to automate the obvious things like watering and ventilation. In the future things like heating and growlights might be added too.
I have spent several weekends on that project with my friend Kalle who helps me on the electrical engineering front and finally we have something that more or less works.

The current solution consists of controller board & sensor board in the greenhouse. Controller is run by Arduino and controls 220V power outlet & gathers information from various sensors. Bidirectional datalink with the house is done over ZigBee radio. In the house I have a Beagleboard which logs data, draws nice graphs, serves these over the internet and can control Arduino.

The greenhouse controller looks like this:
controller_smaller
It consists of:

Some of the Arduinos IO pins are connected to the CAT5 cable that runs to small external sensor board:
sensor_board_smaller
The sensor board has following sensors connected to it:

The sensors and connections are protected from the environment mainly by having them covered by a thick layer of hot glue.

First version of the sensor board used LM35 analog temperature sensors which were nice and easy to interface with but thanks to their analog nature were rather sensitive cable length and small voltage flux on the Arduino that was caused by the relay switching and other sensors. Besides LM35 outputs negative voltage for temperatures below 0 deg. C which you can’t measure in a direct way from Arduino.

I haven’t had time yet to document the schematics & connections but for similar well documented project you can take a look here and here.

And here’s how it’s currently “installed”:
ad_hoc_installation_smaller
The power socket/switch that you see right next to the controller box is meant for the water pump and switch is used to override Arduinos decisions if need be.

Software wise things are quite simple at the moment – controller periodically gathers data from all the sensors and prints them in a line similar to:

H121 T-28.8 T-10.0 L192

Where letter indicates sensor type and number is determined by the position. Since the arduinos serial output is really connected to the ZigBee module I can just read it from the other side of the radio link as though the devices were directly connected with serial cable.

So the code running on Arduino is just this:

#include <OneWire.h>
#include <DallasTemperature.h>
 
#define ONE_WIRE_BUS 2
 
OneWire oneWire(ONE_WIRE_BUS);
 
DallasTemperature sensors(&oneWire);
 
#define MEASURE_CYCLE_TIME 5000
 
int humidity_pin = 1;// input pin for the humidity sensor
int light_pin = 5; // input pin for the light sensor
int relay_pin = 12;   // relay switch
float val = 0; // variable for storing sensor value
 
void setup() {
  pinMode(relay_pin, OUTPUT);  
  digitalWrite(relay_pin, LOW);
  Serial.begin(9600);
  sensors.begin();
} 
 
float get_temp(int pin) {
  float val;
 
  return sensors.getTempCByIndex(pin);
} 
 
void show_temp(float v) {
  Serial.print("T");
  Serial.print(v);
  Serial.print(" ");  
} 
 
void loop() {
  sensors.requestTemperatures();
  val = get_temp(0);
  show_temp(val);
  val = get_temp(1);
  show_temp(val);
 
  val = analogRead(humidity_pin);
  Serial.print("H");
  Serial.print(val);
 
  val = analogRead(light_pin);
  Serial.print(" L");
  Serial.print(val);
  Serial.print("\n");
 
  delay(MEASURE_CYCLE_TIME);
}

And on the Beagleboard it’s even simpler:

import serial
import os
import time
import traceback
 
SAMPLE_WRITE_TIME = 5*60
DATA_DEVICE = "/dev/ttyUSB0"
 
"""
to create the RRD file:
rrdtool create temperature.rrd --step 300 \
  DS:temp1:GAUGE:600:-273:5000 \
  DS:temp2:GAUGE:600:-273:5000 \
  DS:light:GAUGE:600:0:1000 \
  DS:humidity:GAUGE:600:-273:5000 \
  RRA:AVERAGE:0.5:1:1200 \
  RRA:MIN:0.5:12:2400 \
  RRA:MAX:0.5:12:2400 \
  RRA:AVERAGE:0.5:12:2400
"""
 
last_ts = None
 
ser = serial.Serial(DATA_DEVICE, 9600)
 
def calculate_relative_humidity(raw_humidity_val, temperature_c):
    """converts raw HIH4030 output to relative humidity %"""
    # see http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1267245927/3 for discussion
    if __debug__:
        print raw_humidity_val,temperature_c
 
    ZERO_PERC_VOLTAGE = 0.8
 
    max_voltage = (3.27-(0.006706*temperature_c))
    relative_humidity = ((((raw_humidity_val/1023)*5)-ZERO_PERC_VOLTAGE)/max_voltage)*100
 
    if relative_humidity > 100.0:
        # if it's  saturated/frozen it outputs raw values around 920 which would lead to
        # RH% in the range on 120 which is theoretically possible but probably just an
        # measurement error / problem with the equation
        return 100.0
 
    return relative_humidity
 
def write_sample():
    global last_ts
 
    l = ser.readline()
    # input is expected to be similar to "T-23 T-10 H100 L20\n"
    if __debug__:
        print "raw:",l,
 
    temp1, temp2, humidity, light = [float(x[1:]) for x in l.strip().split()]
    humidity = calculate_relative_humidity(humidity, temp2)
    if __debug__:
        print temp1,temp2,humidity,light
 
    if last_ts is None or (last_ts + SAMPLE_WRITE_TIME) < time.time():
        cmd = "rrdtool update temperature.rrd N:%f:%f:%f:%f" % (temp1, temp2, light, humidity)
        os.system(cmd)
        last_ts = time.time()
        if __debug__:
            print "cmd:",cmd
 
while 1:
    try:
        write_sample()
    except KeyboardInterrupt:
        raise
    except:
        print "write failed"
        traceback.print_exc()

Here’s some sensor data from today:
temp_211220101
light_21122010
humidity_211220101

live data available through Pachube feed

And here’s a picture of the Beagleboard at the garage that gathers data & serves the graphs
beagleboard_installation

Actually getting the Beagleboard working in a suitable manner turned out to be the most complicated part of the whole project. It took 3-4 days in total over a month or so while I reinstalled different distributions, replaced broken SD card, tried more than 10 different versions of the kernel and u-boot and finally removed a capacitor from the Beagleboard to get it stable. Hopefully I will write a separate post about that in the near future 🙂

top opensource hardware from 2010

At the end of every year Makezine publishes a list of most interesting opensource hardware projects. And here is the list for 2010. Descriptions are a bit terse this time around but there were several new discoveries even for me even though I follow progress in open source HW all year round.
For me the most interesting projects from that list are Freakduino-Chibi, An Arduino-based Board For Wireless Sensor Networking and the Bus Pirate.

Hopefully I can use Freakduino to replace Arduino + XBee combo that I currently use to communicate with greenhouse and the Bus Pirate looks just interesting and might be helpful sometimes. Pandaboard is missing from that list for some reason, probably because it was released so close to the end of 2010 and is already backordered well into 2011.

sharpening the saw

A friend of mine called the other day and asked how to basically loop over a tree. The actual description of the problem was that there’s an SQL table with N fields that has a different enum defined for each field.
His question was how to print out all the possible records for such a table.

While it’s a simple task there are many ways to do it with different upsides and downsides. So I decided to code up several implementations to show him the possibilities.

So the data structure is something along the lines of:

table = (
  ('a', ('a1','a2')),
  ('b', ('b1', 'b2', 'b3')),
  ('c', ('c1', 'c2', 'c3')),
  ('d', ('d1', 'd2')),
)

The first solution that comes to mind is obviously the simple recursive algorithm:

def construct_tree(t, depth=0):
    if depth == len(t)-1:
        return [[x] for x in t[depth][1]]
 
    retl = []
    futures = construct_tree(t, depth+1)
    for attr_val in t[depth][1]:
        for f in futures:
            retl.append([attr_val] + f)
 
    return retl
 
print ','.join(f[0] for f in table)
for l in construct_tree(table):
    print ','.join(l)

This works and is easy to understand but consumes lots of memory since it will build full tree as list of lists before returning.

Another solution is to avoid recursion completely and use a separate list to keep track of where you are in the tree.

def construct_path(idx_map, t):
    """return path through the tree as a iterable"""
    return (t[fieldno][1][field_val_idx] for fieldno, field_val_idx in enumerate(idx_map))
 
def reset_subtree(from_idx, idx_map, t):
    for idx in range(from_idx+1, len(t)):
        idx_map[idx] = 0
 
def construct_tree(t):
    idx_map = [0]*len(t)
    cur_field = len(t)-1
 
    while 1:
        if cur_field == len(t)-1:
            # we have reached the leaf node, print the whole path
            yield construct_path(idx_map, t)
 
        if idx_map[cur_field] < len(t[cur_field][1])-1:
            # we still have some work at this depth
            idx_map[cur_field] += 1
            # always jump to the end after changing index
            cur_field = len(t)-1
        else:
            # can't increment this field anymore, try previous if any
            cur_field -= 1
            if cur_field >= 0:
                reset_subtree(cur_field, idx_map, t)
            else:
                # there is no previous field
                break
 
print ','.join(f[0] for f in table)
for l in construct_tree(table):
    print ','.join(l)

Performance wise this is a lot better but certainly harder to write and understand.

So how to get the performance of the iterative solution while keeping the simplicity of the recursive one? The solution is to use Python’s generators instead of lists:

import collections
 
def construct_tree(t, buf=None):
    if buf is None:
        buf = collections.deque()
 
    for x in t[len(buf)][1]:
        buf.append(x)
        if len(buf) == len(t):
            # leaf node, stop recursion
            yield buf
        else:
            for x in construct_tree(t, buf):
                yield x
        buf.pop()
 
print ','.join(f[0] for f in table)
for e in construct_tree(table):
    print ','.join(e)

Since I have lately been re-learning Prolog I also wanted to write a solution in that language since this seems to be an ideal task for it:

attribute(a, [a1, a2]).
attribute(b, [b1, b2, b3]).
attribute(c, [c1, c2, c3]).
attribute(d, [d1, d2]).
 
fgen(Field, X):-
    attribute(Field, L1),
    member(X, L1).
 
table:-
    fgen(a, X), fgen(b, Y), fgen(c, Z), fgen(d, I),
    write(X),write(','),write(Y),write(','),write(Z),write(','),write(I),nl,fail.

And here’s how to run it:

hadara@hadara-laptop:~/code$ prolog -s table.pl -t table -q | head -3
a1,b1,c1,d1
a1,b1,c1,d2
a1,b1,c2,d1

Update
While learning Erlang I noticed that their list comprehension syntax allowed you to use multiple generator expressions in it. You can think of it as basically using nested loops so you can write my specific example in Erlang like this:

[{X1, X2, X3, X4} || X1 <- [a1,a2], X2 <- [b1,b2,b3], X3 <- [c1,c2,c3], X4 <- [d1,d2]].

Sure enough Pythons list comprehension syntax allows the same:

[(x1,x2,x3,x4) for x1 in ('a1','a2') for x2 in ('b1','b2','b3') for x3 in ('c1','c2','c3') for x4 in ('d1','d2')]

or in a much more effective way using the generator expressions:

((x1,x2,x3,x4) for x1 in ('a1','a2') for x2 in ('b1','b2','b3') for x3 in ('c1','c2','c3') for x4 in ('d1','d2'))

This of course would be useless in real life since you obviously do not want to hard code expression for specific size of list. Luckily you can just use recursion inside the comprehension expression to generalize it.
In Erlang it would look like this:

build_tree([Head]) -> [[X] || X <- Head];
build_tree([Head|Tail]) ->
    [[X1] ++ X2 || X1 <- Head, X2 <- build_tree(Tail)].

and in Python (using generators instead of list comprehensions):

def construct_tree(x):
    if len(x) == 1:
        return ([i] for i in x[0])
 
    return ([i] + j for i in x[0] for j in construct_tree(x[1:]))

some no-NOOP’s

Sometimes every programmer finds something in the code that shouldn’t do anything, but in reality is essential to the correct functioning of the code because of some side effect. Someone really should make a site that gathers these since they are usually really interesting and you can learn much from the reasoning behind them.

One of the first of encounters with these subtle no-NOOPs was back when I was at university and had to hack something together for a deadline that was just a few hours away. The program worked fine under the FreeBSD that I was using for development but crashed under the Solaris where it was supposed to be run by the supervisor.
The bug seemed to be a Heisenbug since it went away when I added printf()’s to see what’s happening. I inferred that it must be something that corrupts the stack and I added some debug variables along with printf’s. These must have provided enough buffer space on the stack to avoid overwriting critical stuff.

Since I didn’t have any time left before the deadline to actually debug it I just added declaration of a small char array and set some of it’s element so the compiler wouldn’t complain that it was unused. This stabilized the program even though in a extremely ugly fashion. Of course I also mentioned this problem in the report and fixed the real bug later.

The next one is probably known to every embedded programmer. If you want to wait for something for a very small amount of time and do not have any operating system that would provide you with sleep() or other similar facilities the logical thing is to do a busy wait – basically a simple loop similar to:

for(int i=0; i<1000; i++);

The only problem was that it didn't work. It took me almost an hour before I was desperate enough to check the compilers assembler output and saw that the loop was just optimized away since compiler inferred it wasn't necessary.

The next one comes from a embedded platform browser that I use at work. All the pages that used the device specific JS APIs had the following line as the first JS line on every page:

var v = document.width;

The v variable was never used anywhere, but the code wouldn't work without that line in place since some of the platform specific objects were missing.
So executing that line probably initialized JS engine into some good state.

And the last one is from the wonderful world of JS where one of the things you have to do to work around some of the same domain origin policy limitations is this:

document.location = document.location

I have read the explanation several times and I still can't fully understand why would it change anything.

pandaboard

Beagleboard now seems to have a sister project called Pandaboard which is similar in it’s form to the Beagleboard designs but uses more powerful dualcore 1Ghz OMAP 4430 CPU and has 1GB of RAM.
Size is ~11x10cm and like beagleboard it’s fanless and actually doesn’t seem to have any heatsinks too. Since it should be capable of decoding h264 videos at 1080p resolution it will probably be widely successful with the HTPC crowd.
It even has an ethernet port, 802.11b/g/n wifi & bluetooth directly on board so there’s no need to use USB networking adapters that we had to use with the Beagleboard.

Power consumption of the beagleboard was around 3W when playing videos so I speculate the pandaboard uses at most 6W.

I will probably buy it for home automation & robotics experimentation. Also hacking it will probably become a good Meego devel platform.