Testing Searduino’s Debian specifics with VMM and Jenkins

Posted on by sandklef

Problem and solution

Searduino provides headers and libraries to program the Arduino boards. It would be stupid to rewrite the whole Arduino software so we’re of course using the Arduino source code. Currently the default way of building Searduino is to download (automatically done by Searduino) the Arduino source code from their download site and unpack that to a specific folder. This solution doesn’t work with Debian so we discussed a bit back and forth how to solve this.

The solution we reached was to add a configure option to make it possible to build Searduino from the Arduino source code as shipped with Debian’s Arduino package. Use it like this:

./configure --enable-debian-sources

Providing source code dist to Debian

Every night we’re building Searduino (yeah, autobuilds are back again) and provide source code dist (tar.gz) and binary versions for 32 and 64 bit GNU/Linux systems. These scripts are internal and look so ugly at the moment so there’s no way you will se them ;) . To test the Debian specific option (–enable-debian-source) we’ve set up a dedicated build server (Debian, unstable, 64 bit) that downloads the source code dist file (.tar.gz) and builds and tests Searduino from that.

Testing the new option in the source code dist provided to Debian

We use Jenkins to do the tests of the Debian specific stuff. We’ve been trying out the two VirtualBox plugins but we never got it to work properly. So instead we started using VMM and now it works nice.

Searduino and GNU Xnee have a Jenkins site to automate builds:

http://dhcp2-pc213045.itu.chalmers.se:8080/

To build Searduino from a dist file using VMM in Jenkins we ceated a “a free-style software project” and added some “Execute shell” to do the building. The calls we do to build are:

/opt/vmm/bin/ats-client --start-client-headless Debian-unstable-64
/opt/vmm/bin/ats-client --wait-for-ssh Debian-unstable-64 300
/opt/vmm/bin/ats-client --client-exec Debian-unstable-64 \
       "rm -fr searduino-dist"
/opt/vmm/bin/ats-client --client-exec Debian-unstable-64 \
      "mkdir searduino-dist"
/opt/vmm/bin/ats-client --client-exec Debian-unstable-64 \
      "cd searduino-dist && wget http://129.16.213.45/searduino-build/dist/searduino-git-$(date '+%Y%m%d').tar.gz"
/opt/vmm/bin/ats-client --client-exec Debian-unstable-64 \
      "cd searduino-dist && tar zxvf searduino-git-$(date '+%Y%m%d').tar.gz"
/opt/vmm/bin/ats-client --client-exec Debian-unstable-64 \
      "cd searduino-dist && ln -s searduino-git-$(date '+%Y%m%d') latest"
/opt/vmm/bin/ats-client --client-exec Debian-unstable-64 \
      "cd searduino-dist/latest && make -f Makefile.git"
/opt/vmm/bin/ats-client --client-exec Debian-unstable-64 "export CFLAGS=\"-I/usr/lib/jvm/java-6-openjdk-amd64/include/\"; export CXXFLAGS=\"-I/usr/lib/jvm/java-6-openjdk-amd64/include/\"; cd searduino-dist/latest && ./configure --enable-debian-sources --prefix=/tmp/tmp-searduino-dist-debian"
/opt/vmm/bin/ats-client --client-exec Debian-unstable-64 \
      "cd searduino-dist/latest && make"
/opt/vmm/bin/ats-client --client-exec Debian-unstable-64 \
      "cd searduino-dist/latest && make check"
/opt/vmm/bin/ats-client --client-exec Debian-unstable-64 \
      "cd searduino-dist/latest && make install"
/opt/vmm/bin/ats-client --stop-client Debian-unstable-64

Results from test:

The trend seems to be ok. So we’re now finally able to say that Debain can ship Searduino.

Log file from a build we made today:

http://dhcp2-pc213045.itu.chalmers.se:8080/job/Searduino-debian-unstable-dist/12/consoleFull

Comments

The sun seems to be up. So we’re now finally able to say that Debain can ship Searduino. Also we’ve seen that it is possible to use VMM with Jenkins.

About VMM

VMM is a piece of software to make it easy to start, stop and use virtual machines/clients (currently supporting Qemu and VirtualBox).

Searduino autobuils (again) using VMM

Posted on by sandklef

Searduino autobuils (again) using VMM.

(Added title to blog post: March 8, 2013)

Why autobuild in the first place?

Your software has been building fine, manually, for a while so one might be tempted to think that autobuilds are superflous. Why check if your software builds and why test it on all the platforms you target automatically when you so easily can do it manually?

In Searduino’s case most of the code is written in C (and C++) using POSIX threads so it should be possible to say that if it build son platofrm A it should build equally well on platofrm B, shouldn’t it? The Java and Python extension could potentially be tested on one platform alone. I don’t think so – but I wouldn’t argue to strong for this case.

I would say you always gain from building your software on various platforms. And a side effect could be nightly dist files and even binary releases.

Searduino is a bit tricky – we don’t claim it is tricky in a unique way nor one of the trickier projects. Anyhow … Searduino provides libraries cross compiled for the various Arduino boards so we’re getting some more interesting cases where things can go wrong. Several times we’ve ran into problems with various combinations of the versions of the cross compiler (avr-gcc), libc (avr-libc), Arduino software and Searduino itself. Add to that the operating systems we target (GNU/Linux (Debian and Ubuntu – Fedora developers are more than welcome!!!), Mac and Windows (using cygwin)). We need to make sure it is easy to build the software – on as many platforms as possible and in an automated fashion. The old build server crashed recently, so we’ve been setting up a new. We’ve had Ubuntu (amd64), built natively, for some days and we needed to add Ubuntu (32 bit). We will continue with other OS:s and GNU/Linux distributions soon.

We don’t have a 32 bit computer lying around and even if we did we don’t want it to be powered on all day. So we decided, as most developers do I assume, to run the builds and test on a virtualised environment. Enter VMM.

How to test the Arduino libraries, built for the Arduino boards? Should we test on actual boards – how do we check if the code is working? Is it enough to build and upload with no errors? This something we will deal with another day. But this blog post is more about VMM so we’re leaving that for now.

Autobuilding using VMM

With VMM it’s easy to start, stop and execute commands on virtualised machines (currently VirtualBox and qemu). On the github site there’s a manual guiding you how to set VMM up.

Here’s what we did to get VMM to autobuild Searduino.

  • Install VMM
  • Install Ubuntu (32bit) in Virtualbox and call the client/guest Searduino-32.  
  • Added a user, autobuilder, to the client/guest Searduino-32. This user should run the actual builds.
  • Add your (local) ssh keys to the autobuilder (.ssh/autohorized_keys) user and to the root account (this is needed to close the host as graceful as possible).
  • Create a basic configuration for VMM called .ats/ats.conf

    # ATS Settings
    LOG_FILE_DIR=/tmp/ats
    # ATSVM Settings
    VM_STARTUP_TIMEOUT=300
    VM_STOP_TIMEOUT=20
    SSH=ssh
    SSH_TEST_OPTIONS=” -o connectTimeout=3″

  • Create a client configuration file on the host (not on the guest/client) called  .ats/clients/Searduino-32.conf
VM_NAME="Searduino-32"
VM_TYPE="VirtualBox"
VM_IP_ADDRESS=localhost
VM_USER=autobuilder
VM_SUPERUSER=root
SSH_PORT=auto
SSH_SHUTDOWN_COMMAND="shutdown -h now"

… and now we should be ready to write our script to do the actual test.

Stripped down version of Searduino’s autobuild script:

# start the client

ats-client –start-client-headless Searduino-32

# make sure the directory is empty

ats-client –client-exec Searduino-32 “rm -fr searduino”

# clone the Searduino git repo

ats-client –client-exec Searduino-32 “git clone git://git.sv.gnu.org/searduino.git”

# Build Searduino – the command line is nasty. Blame Searduino for this

ats-client –client-exec Searduino-32 “export BUILD_DOC=false ; export CXXFLAGS=\”-I/usr/lib/jvm/java-6-openjdk/include/\” ; export CFLAGS=\”-I/usr/lib/jvm/java-6-openjdk/include/\” ; cd searduino && bin/build-and-test.sh”

# Build a binary dist

ats-client –client-exec Searduino-32 “export BUILD_DOC=false ; export CXXFLAGS=\”-I/usr/lib/jvm/java-6-openjdk/include/\” ; export CFLAGS=\”-I/usr/lib/jvm/java-6-openjdk/include/\” ; cd searduino && bin/build-bin-dist.sh”

# Close down the client

ats-client –stop-client Searduino-32

Schedule the builds the way you want it. We simply put the script in /etc/cron.daily/

…. you may, which Searduino does, have to transfer some files from the client. This will be explained later – and we’re currently developing better support for this in VMM. A more complete example (a step by step guide) is currently being written. Meanwhile, check out this script https://github.com/tis-innovation-park/vmm/blob/master/examples/searduino.tmpl.

Results from the execution of the Searduino script:

Builds logs: http://129.16.213.45/searduino-build/build-logs/20130301/i686/
Binary dist: http://129.16.213.45/searduino-build/bin-dist/

Concluding remarks

The configuration needed for using VMM is limeted to some 15-20 lines. Writing a script like the above is only some 10 lines worth of work. So using VMM to do autobuilds is easy.

We’re currently working hard on getting VMM to be useful to as many people as possible.

  • It would be great if you use it
  • It would be great if you want to join us.   

/Henrik and Shaun

BTW, ats will be renamed to vmm shortly

Searduino: @FOSDEM, new GUI simulator and some thoughts on Java

Posted on by sandklef

First a small introduction to Searduino. Searduino consists mainly of two things. First, a couple of libs (mainly the Arduino source code compiled and archived) and some Makefiles that together make it possible and easy to write pure C/C++ code for Arduino boards. Secondly, Searduino comes with a simulated Arduino board environment. This can be used to unit test your Arduino code (think automated tests). There’s also a couple of simulators, using the simulated environment, with which you interactively typically can validate your software. The simulation is implemented in a library which loads the Arduino code (on GNU/Linux and Unices using dlload) and runs it in a thread (pthread).  This library can quite easily be wrapped by simulators and extensions for other languages That’s as much of an introduction I’ll give here.

After several attempts to get the Python/Gtk simulator called Pardon, using the Python extension to Searduino (called Pearduino), to work well , I gave up trying. The threading support in Gtk seems to make it impossible to get both speed (and we do need speed) as well as an app that doesn’t crash due to too quick/unsynchronised GUI updates from the underlying thread. Don’t get me wrong – I like working with Gtk and I am a Gnome user since … hmm around 2001 (I think). The problem we’ve seen relates to the rather complex problem of combing C binaries loaded by Python and run that in a separate thread and have the results (via callbacks) update the Gtk GUI. So what to do next?

How about a GUI in Qt? I have tried to get into Qt many times but always lacked the time to give it the time it needs.

We didn’t have to think for a long time, since part of my new job at Gothenburg University will be supervising students in Java. Wow – or perhaps uhh – it’s been a while since I did some Java hacking. A long while. The students should also learn Swing so that will be supervised too. To be able to supervise in Java and Swing I needed, it turned out I really needed it, to refresh my almost forgotten Java and almost non existing Swing skills. What better way to do this than implementing a GUI frontend to Searduino in Java/Swing? Yup, that became the task during the Christmas holiday and also the answer to what gui framework to use for the simulator.

I am more of an imperative/procedural (think C) hacker than an OO hacker (think C++, Java). I often use some concepts from OO when hacking C though – hey I don’t go about saying or thinking that mixing C++ and C is a good idea. Either you write OO (C++) or you write procedural (C). For me writing Java code for a Free Software project is a new thing.

Anyhow, over the last days I have, to my surprise, found that I’ve started to like hacking GUIs in Java. Ok, I fall in to the traps of writing C-ish Java code (send the Spanish inquisition after me for doing that!!!) and I figure most pattern fascists probably will cry so much their bodies will dry out when looking at my Java code. Nevertheless I am happy with the speed of which I wrote the GUI. And apart from some rare and odd behavior when stopping (pthread_cancel) the pthread running the Arduino code it works really well. Updates in the GUI are really really quick.

To be able to communicate with and control the C code I had to write a small JNI layer. Just as we did for Python with the Python extension. Comparing writing a Java extension to writing an extension for Python I would say they are equally easy to write.

I think the fresh GUI is the “final” piece for Searduino to make it really useful for many people – so this blog post feels really nice to write. But hey, so many words and yet nothing has been said. I will try to say at least something useful.

  • I am slightly surprised to say that I enjoyed writing a GUI in Java.
  • Searduino has now become useful for more people
  • It’s easier to join the project now – so join us!!!! We need you!

Next step then. Next steps for Searduino that is? For me personally, I would like, for the fun of it and inspired by Andy Wingo‘s post on Scheme to finally be giving Scheme a go – it may not be the strategically most optimal choice but I’ve been wanting to learn Scheme for a while so why not. Leaving myself and my ego alone for a while and instead start thinking about what the next steps for Searduino should be…. I guess that’s up to you to decide.

… and finally. I will be doing a lightning talk about Searduino at FOSDEM (thanks all FOSDEM volunteers for your friendly and informative emails!). So if you’re interested in seeing Searduino simulate some leds blinking and also actually seeing some real leds blinking – and all of this done in C/C++ – pop by the Searduino lightning talk on Saturday at FOSDEM.

GNU Xnee 3.15 (‘Shankar’) released

Posted on by sandklef 
We are pleased to announce the availability of GNU Xnee 3.15

GNU Xnee is a suite of programs that can record, replay and distribute
user actions under the X11 environment. Think of it as a robot that can
imitate the job you just did. GNU Xnee can be used to:
    Automate tests
    Demonstrate programs
    Distribute actions
    Record and replay 'macro'
    Retype the content of a file

Getting the Software

   ftp://ftp.gnu.org/gnu/xnee/xnee-3.15.tar.gz
   ftp://ftp.gnu.org/gnu/xnee/xnee-3.15.tar.gz.sig

or one of the mirror sites as found in:

   http://www.gnu.org/prep/ftp.html

Checksums

  md5sum:
    32c8ac9f354741f03d7736383599984e  xnee-3.15.tar.gz

  cksum:
    3996773279 1776676 xnee-3.15.tar.gz

New in this release

  * Fixed bugs:

    Savannah:
      bug #36662: 2 extra lines, not really needed, printed to recorded log files
      bug #37895: pnee/data/pnee.server.in file missing in 3.14 tarball?

    address@hidden
      xnee patch for Red Hat Enterprise Linux

Why 'Shankar'?
 In December 2012 Ravi Shankar passed away. This is a way to pay tribute.