Using Python to detect activity in Sensory Software's Grid 2

posted Oct 2, 2017, 9:50 AM by Matt Oppenheim   [ updated Oct 4, 2017, 5:49 AM ]

Following on from the eyeBlink post, with the help of Fil at Beaumont, I modified the algorithm I'm using to detect when the Grid 2 or Grid 3 software is being used. The image below shows Sensory Software's Grid 2 software being used to construct a sentence. The new text appears in the white area at the top of the window. Fil suggested that I change the Python script to just monitor this area at the top of the window. The script now looks for a change in the amount of black text in this area. After the usual software wrangling I think I got it working. The Python script looks at the top 20% of the window and counts the number of black pixels in this area. Every half second it recounts the number of black pixels. If there is a change in the number of black pixels above a threshold, then a trigger is sent to indicate that the Grid software is being actively used. I'm using a threshold of 20 pixels, so there needs to be an increase or decrease of 20 or more black pixels for a change to be detected. This allows you to move your mouse cursor around in the text area at the top of the Grid window without triggering that there has been a change. The activity detection script needs more testing, but preliminary results seem to show it works. Prior to this, I was monitoring the entire Grid window and looking for a change in the whole window above a threshold. This led to false triggers when cells were selected, but not activated. When a cell is selected, the colour of the cell changes, even when it is not activated to produce text. This change in colour was being detected.

Each time we test the script, we find new ways to break it, leading to some 'try except' exception handling clauses. The script is designed to run on Windows as Grid 2 and Grid 3 only work on this operating system. I use the win32gui library to interact with Windows and the python imaging library, PIL (known as pillow), to do the image processing.

Sensory Software's Grid 2 Chatterbox grid being used to construct a sentence.

The latest code and installation details on how to get this running using the BBC microbit to give a flash when the Grid software is being actively used can be found on my github site at:

If you have any questions, please ask.


posted Jul 14, 2017, 9:51 AM by Matt Oppenheim   [ updated Jul 14, 2017, 9:56 AM ]

It can be difficult to tell when a student who uses an eye tracker to operate their communications software is actively using the software. So the temptation is to go and look over their shoulder. The Head Technologist at Beaumont College asked if it is possible to have a light flash to indicate when the communications software is being used. This makes for a more natural two way conversation. You talk to the student and you see the light flashing, so you know that a reply is being composed.

After some false starts, I think I have some code that will detect when the software is being used. My script makes an image of the screen each second and looks for a difference. I set a minimum threshold for the difference between the images so that a blinking cursor will not continuously indicate a change.

I use a BBC micro:bit to do the flashing as they are cheap, cheerful and reliable. I reckon to have two of these qualities.

Please find a picture of Craig, one of the technologists at Beaumont, testing the prototype of the eyeBlink system using a Tobii eyetracker and Sensory Software's Grid 2 software. You will have to take my word for it that the microbit does indeed flash when the Grid 2 is being used.

Once I get the software tested with some of the students at Beaumont and the code tidied up, I'll post it on Github so that anybody else can use the idea.

BBC Micro:bit based hand shake detector

posted Jul 13, 2017, 5:14 AM by Matt Oppenheim   [ updated Jul 13, 2017, 5:14 AM ]

I tested out the latest version of the HandShake gesture detection hardware at Beaumont last week. The latest hardware is based on the BBC Micro:bit. Finally it looks like I have a solution for enabling disabled people to communicate through gesture using cheap, off the shelf, technology. Here's a video that we made:

HandShake testing at beaumont, July 2017

There is still a lot of work to be done, but I have some hope now of a working solution that can easily be replicated and used elsewhere.

More details on the HandShake project are here.

Booting a new Lenovo Thinkpad from USB stick to use Clonezilla

posted Jun 28, 2017, 5:14 AM by Matt Oppenheim

All I wanted to do was boot from my trusty Clonezilla USB stick to make a system back up of my shiny new Thinkpad X260. Long story short, you need to disable the 'Secure Boot' option in the UEFI (what used to be called BIOS) to boot from a bootable Clonezilla USB stick.

I bought the X260 a couple of month's ago. This is last year's model, so I got it at a discount. Usually I'd buy a 2/3 year old Thinkpad and replace the drive and keyboard, but found I could buy a new laptop, albeit last year's model for about half of what it would've cost a year ago. A quick cleansing of the OS by installing Linux Mint. The usual kerfuffle to configure the system and remember how to partition the drive, then try to remember how to mount said partitions. Time to make an image of the OS partition. I've learned this is a good idea the hard way. When I tried to boot from my Clonezilla USB sticks, none of them would work! Somehow I had made a bootable USB stick that would install Linux Mint. I spent a good hour before checking on the 'Secure Boot' option in the UEFI screen, which by default is Enabled. Flicking this to Disabled solved this issue. 

Now I have an image of my OS on an external drive for when I manage to destroy the installation. Not if. When. Still, that's how we learn, by breaking and fixing. Probably a good thing that I don't work in medicine.

Microbit wrist holder

posted Jun 16, 2017, 5:06 AM by Matt Oppenheim   [ updated Jun 19, 2017, 9:03 AM ]

This post shows you how to convert a £3 iPod 7 arm band into a wrist or arm holder for a BBC Microbit board and battery. The photo shows the creation with a Microbit and battery pack being modeled by the local electronic's wear supermodel.

I am hoping to use the Microbit as assistive technology as a communications aid. The Microbit on the wrist detects a gesture and then communicates this to a second Microbit which is connected to communications device. This second Microbit says 'hey, there's been a gesture!' to the communications device, so acts as a switch. This enables the wearer to control software through hand motion. More details of this work are here

I need a wrist holder for the Microbit as I am not allowed to just superglue it on to other people according to the University Ethics Committee. I couldn't find anything suitable for sale, so broke out my sewing kit and made one from an armband aimed at one of the many iPod incarnations. With the Microbit on it, I am reminded of the Power Glove from 1989. Who wouldn't want one of those? The tagline for the Power Glove was 'it's so bad', which kind of fits.

A friend pointed out that the holder could be used to strap the Microbit on the leg or any other suitable appendage. For instance, you could log accelerometer and temperature data during sports. Use a second Microbit connected to a laptop to wirelessly log the data from the Microbit attached to your body.

I started with a 'King of Flash New iPod Nano 7th Generation Premium Water Resistant Armband Case' from eBay for the princely sum of £3.

Remove the plastic iPod holder bit with a few minutes of cutting with a suitably sharp instrument.

You need to sew back one of the plastic belt reinforcements on to the first of the two belt slots.

You can cut off the neoprene with the second slot. If you are using the modified arm band to attach to the wrist, then you can also trim down the length of the belt as it is sized to be long enough to go around a biceps. Even my weedy biceps are much larger than my wrist. Sew on some elastic loops to hold the Microbit board and the battery holder. I bought a 1m piece of 3mm black elastic from eBay to do this. 

To get the belt to stick to itself and close the holder, I used some sticky back velcro of the 'hook' gender. I stuck two pieces back to back to form a velcro 'gender bender'. This sticks to the belt part of the velcro on the holder, next to where the board and battery will sit. 

The assembled wrist holder with the Microbit and battery pack is shown below. I use a AAA battery holder with a switch on it that I also bought from eBay for about £2 instead of the 'always on' battery pack that came with the Microbit. I buy a lot of things from eBay.

If you are worried about the battery pack flying off after a violent wrist motion, you can double secure it with some sticky back velcro between the back of the battery pack and the neoprene of the wrist mount.

There you have it. How to make your own Microbit and battery pack wrist/arm/leg holder. 'It's so bad!'

Installing linux mint 18.1 onto a Lenovo 260 with an encrypted home drive

posted Apr 18, 2017, 8:21 AM by Matt Oppenheim   [ updated Apr 19, 2017, 6:32 AM ]

The simplest way I found to install Linux Mint 18.1 on to my Lenovo 260 with an encrypted home drive and a separate installation partition is to install the system using the simplest options, then afterwards encrypt your home drive and shrink down the installation partition using gparted. The rest of this post is how I failed to do this several times. Which is undoubtedly due to my lack of linux wisdom.

I tried and failed to install Linux Mint 18.1 on to custom partitions for my root, home and swap. The system would not boot after I completed the installation. I could not install grub to the /mnt partition to fix this. I tried some stackoverflow solutions with no joy.

So I did a simple install, clicking on the option to encrypt the home folder. Then I used gparted on the installation USB stick to shrink down the partition. However, my swap space was also encrypted, which I understand increases security. Every time I booted I was presented with message asking for a non-existent password to mount the encrypted swap space. No real issue, I just hit enter and carried on to the regular login screen. Then I tried updating the system. For each update I had to press enter to mount the encrypted swap space. A little tedious. So I went on stack overflow, found a 'fix' and rendered the system unbootable. This was getting a little tedious.

So I again installed Linux Mint 18.1 from my USB stick. This time I chose the vanilla, easiest options, no encryption. I used the instructions here to encrypt my home drive. I used the installation stick to run gparted and shrink down the partition. So now I have Linux Mint installed on a partition and an encrypted home drive.

Simple. Hind sight always is.

how to configure the accelerometer range on the microbit using micropython

posted Apr 13, 2017, 10:01 AM by Matt Oppenheim   [ updated Apr 13, 2017, 11:00 AM ]

This article details how to set the range of sensitivity on the accelerometer on the microbit board using micropython and the i2c interface. I am using v1.7.9 of micropython for the microbit, the mu editor and linux mint v17.

After listening to Joe Finney talk about his role in developing the microbit board I realised I could use it for some of my hand gesture assistive technology work. The accelerometer on the microbit board is an MMA8653FC, data sheet here. There are programming notes for this chip here. The default range for this chip is +/-2g. This can be reconfigured to be +/-4g or +/-8g. For some of the students I work with on gesture recognition I need the higher ranges. So I entered the world of microbit i2c programming. I chose the micropython platform as python is always the 'second best choice' for any programming application. Actually, I'm a fan of using C for embedded hardware, but in this case using micropython looked to be fastest way of getting a solution. I used the simple mu editor. Long story short, it's all about syntax. Thanks go to fizban for his example microbit code to interface a microbit with an lcd display using i2c. After reading this code I fixed the mistake(s) I'd been making. The documentation for the i2c microbit micropython is here.

Here's my working code:

''' microbit i2c communications with onboard accelerometer '''
from microbit import *

ACC_2G = [0x0e, 0x00]
ACC_4G = [0x0e, 0x01]
ACC_8G = [0x0e, 0x02]
CTRL_REG1_STANDBY = [0x2a, 0x00]
CTRL_REG_1_ACTIVE = [0x2a, 0x01]
PL_THS_REG = [0x14]  # returns b'\x84'
PL_BF_ZCOMP = [0x13]  # returns b'\44' = 'D'
WHO_AM_I = [0x0d]  # returns 0x5a=b'Z'
XYZ_DATA_CFG = [0x0e]

def command(c):
    ''' send command to accelerometer '''
    i2c.write(ACCELEROMETER, bytearray(c))

def i2c_read_acc(register):
    ''' read accelerometer register '''
    i2c.write(ACCELEROMETER, bytearray(register), repeat=True)
    read_byte =, 1)
    print('read: {}'.format(read_byte))

def main_text():
    ''' send accelerometer data as a string '''
    print('starting main')
    counter = 0
    while True:
        x = accelerometer.get_x()
        y = accelerometer.get_y()
        z = accelerometer.get_z()
        counter = counter + 1
        print('{} {} {} {}'.format(counter, x, y, z))

print("sending i2c commands...")
print('reading PL_BF_ZCOMP :')
print('reading WHO_AM_I')
# check the initial accelerometer range
print('reading XYZ_DATA_CFG:')
# change the accelerometer range
print('commands sent')
# check the accelerometer range
print('reading XYZ_DATA_CFG:')
# main_text()


reading PL_BF_ZCOMP :

read: b'D'


reading WHO_AM_I

read: b'Z'


reading XYZ_DATA_CFG:

read: b'\x00'


commands sent

reading XYZ_DATA_CFG:

read: b'\x01'


The onboard accelerometer has an i2c address of 0x1d. There is a good article on how to scan for and verify this address here. I set the variable ACCELEROMETER to be this value in line 4 so that I could refer to it throughout the code without having to remember the hex value. Too many hex values flying around - I'd be bound to make a mistake if I didn't give them names.

To send a command over i2c, as shown in line 18 of the example code, you need to address the target then send the commands as a bytearray. In this case the target is the accelerometer. Typically we send two bytes to the accelerometer. The first specifies the register we want to change, the second the value we want to write to this register. For example, to set the accelerometer's range of sensitivity, we need to set the value of the register called XYZ_DATA_CFG to the value that corresponds with the range we are after. The address of this register is 0x0e. To set the +/4G range, we want to set this register to be 0x01. Now the variable I set in line 6 should make sense. Look in the data sheet linked above for more details. Before we can change this register we have to set CTRL_REG1 to be inactive by writing 0x00 to it. After changing the XYZ_DATA_CFG register we have to set CTRL_REG1 to be active again by writing  0x01 to it. This is detailed in the accelerometer application notes which I linked at the start of this article. 

If you uncomment the last line, then the raw accelerometer values will stream out. The last column are the values for the z-axis of the accelerometer. Lay the board flat on the table. With the default +/-2g range you will see the z-axis values being around +1024 or -1024 depending on if the board is face up or down. This corresponds to +/-1g on the +/-2g range. Now that the board is set to +/-4g, the values for +/-1 g will be +/-512. The maximum and minimum value for the accelerometer stays as +/-2048, but it is now spread over +/-4g. Similarly, if you go crazy and set the range to be +/-8g, then you will see +/-256 for the z-axis value from the accelerometer for the board laying flat. As you would expect, you have to wave the board harder to get it to max out when you set the sensitivity to the higher ranges compared with the default +/-2g range.

So what about the PL_BF_ZCOMP and WHO_AM_I registers that I read from in lines 43 and 45? These are two read only directories. Reading the values stored in these is a sanity check that the chip is turned on and I have working code. I read the XYZ_DATA_CFG before and after setting it to verify that the sensitivity range has been set. Read up on these registers in the data sheet.

Look at line 23. The repeat=True flag has to be set. This clears the 'message end' flag in the write command. The default for this flag is False, which means that the i2c write command has a 'message end' flag at the end of it, which terminates the operation. As we want to read from the chip in line 24, we need to not set the 'message end' flag. Otherwise you will just read 0xff. Can you guess why? The data line is held high for i2c, so if there is nothing coming out of the chip you are trying to read from, you just read a bunch of '1s'. Line 24 means 'read 1 byte from the device with address ACCELEROMETER'. 

Where I initially came unstuck was by sending data as individual bytes, using e.g. b'\x0e' followed by b'\x02' to try and change the XYZ_DATA_CFG register. This looks to be valid for the Adafruit implementation of micropython, but I couldn't get it work.

parsing and unpacking python3 serial data containing double backslashes

posted Apr 11, 2017, 9:23 AM by Matt Oppenheim   [ updated Nov 15, 2017, 9:50 AM ]

edit 11th October 2017: The 'eval' statements in the code shown below can be replaced with the safer 'literal_eval' from the ast class in the standard library. From the python docs: 'Safely evaluate an expression node or a string containing a Python literal or container display. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None.'

This can be used for safely evaluating strings containing Python values from untrusted sources without the need to parse the values oneself. It is not capable of evaluating arbitrarily complex expressions, for example involving operators or indexing.

I lost a day of my life figuring out how to parse serial data sent as bytes from the BBC Microbit using micropython. The problem is that the data byte string appears with double backslash characters instead of single backslashes when read in over a serial interface.
Actual data:
What I wanted as data:
So how to convert from one misformed byte string to the clean one that python 3 would use?
I really went around in circles on this one. In the end I used a kludge. But it works. My life can now move on.
I convert the double slash byte to a string. Then I use the replace method to replace '\\' with '\'. Then I use the literal_eval function to recast it as a byte. I am open to suggestions for a cleaner way of doing this!
Here's some example code I used in a jupyter notebook session. test2 is the misformed byte string received over the serial interface and test3 is the cleaned byte that I can now unpack and extract the data from.
from struct import *
from ast import literal_eval
PACKER = ('2s5f2s')
test3 = str(test2)
test3 = test3.replace('\\\\', '\\')
test3 = literal_eval(test3)

(b'ST', 7516.0, 50.0, -224.0, 144.0, -976.0, b'EN')

The data was produced from reading the accelerometer on a BBC Microbit board then using struct.pack(PACKER,scan). I am programming the boards using micropython.
The data is packed using the packer format:
PACKER = ('2s5f2s')
The transmitted scan is constructed using:
values = (START, counter, DELTA, x, y, z, END)
scan = struct.pack(packer, *values)
Where values contains a START and END string ('ST' and 'EN' respectively), a constant called DELTA which represents the time in between samples and the x, y and z readings from the accelerometer. So PACKER means '2 characters followed by 5 floats followed by 2 characters'.
I was being obstinate in sending bytes over the serial interface instead of a string. Why use bytes and not just send a text string? Using the pack and unpack enforces a structure to the data packets and reduces the amount of data needed to be transmitted compared with a string. Consider a number '2048' sent using the packer function. This is coded as an 'f' meaning a float. This is 2 bytes long. Sending '2048' as a string would require 4 bytes, one for each of '2', '0', '4' and '8'. 
If I encode the string 'ST 7516.0 50.0 -224.0 144.0 -976.0 EN' using packer '2s5f2s', the message is 26 bytes. If I send it as a string, it will be 37 bytes. Please see the example code and its output below.

from struct import *
PACKER = ('2s5f2s')
test = 'ST 7516.0 50.0 -224.0 144.0 -976.0 EN'
test2 = (b'ST',7516.0,50.0,-224.0,144.0,-976.0,b'EN')
print('string length: {}'.format(len(test)))
packed_data = pack(PACKER,*test2)
print('packed length: {}'.format(len(packed_data)))
print('unpacked data: {}'.format(unpack(PACKER,packed_data)))
string length: 37
packed length: 26
unpacked data: (b'ST', 7516.0, 50.0, -224.0, 144.0, -976.0, b'EN')
The second reason for using pack and unpack for data packed transmission over sending a stream is that this enforces error checking. If the data is corrupted while reading from the sensor, then an error will be raised during the pack process at the transmitter end. If the data packet is corrupted during transmission, an error will be raised during the unpack process at the receiving end. This can be caught using a try-except clause.

syntax highlighting with pygmentize

posted Apr 4, 2017, 7:28 AM by Matt Oppenheim

Syntax highlighting means making the code examples you put into your blogs look pretty. How to do this? There are some online highlighters where you post your code, then copy and paste the result back into your blog. For instance, I had been doing this. Now I am using pygmentize, which is part of the pygments library for python.

Here is the content of a simple example test file called

import math
for i in range(5):
    print('hello {} {:.2f}'.format(i,math.sin(i)))

The command:

pygmentize -l python -f html -O full,style=default,linenos=1 -o ./pyg_test.html ./

Produces a file pyg_test.html which contains the prettified text. This can be opened in a browser, then copied and pasted into your blog:

import math
for i in range(5):
    print('hello {} {:.2f}'.format(i,math.sin(i)))

As you tweak your code and rerun the pygmentize command, all you have to do is refresh the browser window to get the latest prettified text. There's a bunch of styles that can be applied. Pygmentize works to highlight many languages. Have a look at the docs for more details.

CD player for the visually impaired

posted Jan 15, 2017, 7:49 AM by Matt Oppenheim   [ updated Jun 15, 2017, 9:44 AM ]

This blog details a method to allow somebody who is visually impaired to easily listen to their CD collection again. My Mother lost her eyesight through macular degeneration. She has a decent collection of classical music that she built up over a few years. But she can't see well enough to easily use a CD player anymore. On top of that, her mobility is restricted. I bought her a Roberts Concerto 2 CD player and radio designed for the visually impaired. Please find details and a review here. This is the best that I could find, but it is still fiddly and difficult for somebody without sight to load the CD. It is quite a bulky device, which makes putting it next to an elderly person awkward as it takes up most of chair side table, or the user has to get up and go to where it is placed. Which is a barrier to it ever being used when just getting out of a chair is no longer straight forwards.  I looked at a few potential solutions. I found a portable CD player on eBay and tried that. But again, it takes up a little too much table top space and it is fiddly to load. You don't realise how poorly controls are laid out on most devices until you try to explain to somebody without vision how to use it. I found some lovely projects where custom built players are built using tags. Audio books are loaded onto a memory card and played using something like a Raspberry Pi single board computer. An NFC coil is used to read a tag placed inside the case of an audiobook or CD and the audio is played from the memory card. Here is an example on Hackaday. I started going down this route. Then had another think. This will add another device to my mother's chair side table. I will have to:
1. Build it
2. Run off all her CDs to memory card
3. Show her how to use it
4. Maintain it
My mother uses a Sovereign USB stick player to listen to her talking books and newspapers. This is a well designed player aimed at the visually impaired. It has decent sound quality. The build cost of a custom device would exceed the cost of a Sovereign and for me to think I would match the sound quality is a tad arrogant. One of the design features of this player is that it will remember the place you were last at on the stick. You can even swap sticks and it remembers the last play position on the previous five sticks you played. Mum already has this next to her and knows how to use it. As a side note, there is now a smaller version of this player available called the sonic which I bought to listen to my podcasts with and loan to Mum when she visits.
So I ran off her CDs to MP3 and put each one on to a cheap USB stick. I bought some small key rings and used these to connect the memory sticks to postcards on which I printed the title of the CD. So far I have run off 10 of these. These 10 sticks and labels fit in an little box on her table next to the player. If the idea works, I will run off some more.
I've written this up so that other people in my position have a potential solution to enable others with disability to enjoy their music collections.
Having to pay a couple of dollars each for a cheap USB stick from eBay for each CD may seem a tad pricey, but compared with the time and cost of building a custom device, I think it is money well invested.
The proof of the pudding is in the eating. Please find a photo of Mum at my house, having dozed off while listening to Aled Jones with the bear she gave me about 40 years ago. The bear's arms are a little saggy now, but we all get a little infirm and need some help as we age.


1-10 of 15