Cryptocurrency quotes from the terminal

I made a small tool to retrieve prices of cryptocurrencies from the command line. It uses by default Kraken‘s API, first retrieving the available asset pairs (e.g. bitcoin to US dollars), caching them locally, then allowing for quotes to be retrieved with these assets.

For example, you can get the price of Bitcoin in British pounds with:

cq quote BTC GBP

This outputs something like this:

BTC price on Kraken as of 16/10/17 11:14:10:
 Ask: £3938.80
 Bid: £3713.50
 Last: £3501.00
 Today low: £3501.00 (last 24h: £3501.00)
 Today high: £3969.30 (last 24h: £3502.00)

Simple!

Additional assets (such as Etherium) are supported; additional exchanges can be made available using what I hope is a straightforward set of code additions.

Visit the project’s GitHub page for installation instructions.

Publishing my thesis in book form

This is a story and guide to publishing my thesis in book form. If you’re looking for the good stuff, feel free to skip to the instructions.

Last year I completed my PhD studying interferometry for the detection of gravitational waves. My thesis is now available in electronic form that anyone that wishes to read it can obtain, and the source code is on GitHub. A paper copy – following the standard fabric-bound, A4 format specified by the university court – is available courtesy of the university library: at the time of writing, my thesis is stored in the research annexe on 29 Saracen Street (interestingly, that’s not on campus, but near the Bowmore whisky bottling plant in the north of Glasgow).

The title page of my thesis. Pretty, eh?

One of the ways in which I procrastinated while writing it was to try to produce diagrams, charts and figures which were entirely made from vector graphics such that they could be scaled for display or print at any resolution without losing fidelity. This was unfortunately due to rather of an obsession I have with data integrity: why throw away information when you don’t have to? It might come in handy later. Information is lost in photographs when they are scaled up – for example in preparation for print. This is in general true of all raster graphics like the JPEGs, PNGs and GIFs used all over the web and in designs for print. Vector graphics typically look nicer when displayed in a PDF viewer at high screen resolution or printed with a high quality printer. There are downsides, and there is definitely an appropriate time and place for raster graphics, but not for most of the plots and diagrams in my thesis.

The upshot of all of this time spent making nice, scalable graphics was that it was possible to render the thesis in (theoretically) infinite fidelity for printing. My printed thesis graphics looked really nice in the A4 tome I submitted to the library, but the book design itself looked dated and boring. That’s what gave me the idea to investigate professional print services.

University of Glasgow approved book binding service from Downie Allison Downie in Partick, Glasgow. While the workmanship is good, the university-enforced standard design is boring. No thanks!

Publishing my book

My thesis is almost entirely made up of text, plots of data and diagrams. Almost all of the diagrams are vector graphics. The scant few raster graphics were of circuit board layouts (which could, in hindsight, also have been produced in vector format). I had no photographs, both because my work was mainly focused around simulations and designs for new, not-yet-built experiments, and because of my aforementioned dislike for raster graphics.

As I had written the thesis in LaTeX – a scripting language for documents – I realised it would be straightforward to resize my A4-formatted thesis into a nicer A5 size more suited for reading, and in the process lay it out the way I really wanted to instead of conforming to the boring old University of Glasgow approved format. If nothing else, I’d save on paper by decreasing the margins and text spacing to result in a book more like the ones you buy today. Don’t you hate it when you have to turn the page so frequently?

So, what follows below is a guide to how I published my thesis as a nice, custom book.

Step 0: find an online book printer

The obvious preliminary step. After a bit of searching I found InkyLittleFingers, who had good reviews, made everything at their own factory (in Gloucester) and offered what seemed like a pretty comprehensive binding service that could be arranged completely online. Importantly, they are based in the UK, which cuts the cost of postage, allows for efficient communication regarding specifications, and provides a VAT exemption due the rules in the UK regarding book tax.

One of the appealing parts of InkyLittleFingers’ offer was the level of passive support they provide in the form of guides on their website. They even have a YouTube channel with some video tutorials.

There is a comprehensive guide to designing the artwork for the book. “Artwork”, as I learned from their helpful glossary (under “Useful terms” on this page), is the term in the book binding industry for the content you want printed. That means both the case (i.e. the front and back covers and the spine) and the book block (the inside pages) – the whole thing.

Form on Inky’s website for creating hardback book orders.

The website is a little rough around the edges, and sometimes you might find yourself hitting a 404 or some other sort of error, but for the most part it works as intended. To order a book you first need to upload your artwork and ask for an automatic quote. As I wanted a hardback, I went to this quote form and filled in the details. You can guess the page count just to get a quote, but you need to know the exact page count before making a proper order. The page thickness you choose is combined with the number of pages you specify to calculate the spine thickness, which you then have to use to appropriately dimension your cover design. After some research of their FAQs, I chose the following options:

  • Quantity: 4. Enough for my two supervisors, my parents and myself. It seemed that ordering a fifth made the price higher than five times the cost per book when ordering four. Maybe they prefer to make batches of 4?
  • Size: 210x148mm A5.
  • Orientation: portrait (bound long edge).
  • Case (cover) type: custom full print colour. You can also choose a coloured cloth, but because I wanted to make a design for my cover I chose a printed form.
  • Corner protectors: no. These would look a bit silly on a full colour cover.
  • Ribbons: no. These are used for keeping your place in the book. They weren’t yet available when I ordered, but I may have taken them if had they been.
  • Headbands: no. These are little coloured ribbons that go above and below the paper block inside the cover. These would look weird alongside a glossy, printed cover.
  • Cover laminate: gloss. As noted in the guide, gloss cases are not the same as standard glossy photo paper. That stuff has a mirror-like sheen that looks pretty ugly on anything but colourful photographs. “Gloss” is the de-facto cover laminate for most new books, contrasting with “matt” which is the duller variety. This one is personal preference.
  • Hot foil cover: no. This is the gold or silver writing you put on standard theses. This only suits cloth covers.
  • Dust jacket: no. I decided it’s a bit pointless for a full colour cover to have a dust jacket: these are normally found covering cloth cases as a poor man’s full colour cover. It would only have been a few pounds more to order them, though, and you can always remove them if you don’t like them, so if you fancy it then go ahead.
  • Inner pages printed: mixture of colour & HiQ B&W. My thesis is mostly black text, but some pages have colourful graphics. If you don’t want a monochrome book, choose this, and you won’t regret the high quality black and white. When you select it, it asks for a count of the black and white and colour pages, so you’ll need to do that (just count the colour ones and subtract from the total to get black and white).
  • Paper type: 100gsm matt uncoated white. You can get thicker, thinner, shinier paper, but I went for standard and I’m very happy with it. If you’re publishing a chic novel you might want to go for cream.
  • Proofing type: PDF (online soft proof). This was the only option for me; it’s possible other options exist for higher order quantities. If you want a “real” proof to check everything looks good in print, you need to order a single copy with the same form. See the note on the PDF proof later.
  • Print, delivery, postcode service: tailor these to your requirements. I went for the (cheapest) defaults as I was not in a hurry.

My order ended up costing around £79 all-in, or just under £20 per copy. Considering that binding the single, “official” copy of my thesis for the library to the official university format cost £26 at one of the recommended companies, even having provided them with the printed pages to use, I consider that a pretty marvellous deal. That’s especially true given the creative control I was allowed.

Once you submit your order, the site works out how thick your book will be. Thicker books cost more to make and send, but more importantly you need the exact dimensions to use to design your case artwork in the event that you chose to make one yourself. The thickness of the spine is obviously determined by the number of pages and the weight of the paper, but also the type of cover you specify.

You are given an automatically generated note with your quote. In my case it said:

NOTES:
(1) Please make sure you read the online help explaining the differences between standard and high quality black and white printing.
(2) Spine width is approximately 21.8mm.
(3) Case artwork dimensions, height: 234mm, width: 345.8mm.
(4) Dust jacket (if ordered) artwork dimensions, overall height: 222mm, overall width: 491.8mm, front/rear cover width (each): 154mm, flap width: 77mm.
(5) If there is anything that you do not understand in this specification, please contact the helpdesk.
(6) Please make sure that you read the artwork preparation guide before placing your order.

So, my spine will be 21.8 mm. Given that I selected A5 paper, the case artwork needs to be equivalent to two A5 sheets next to each other, plus the spine, plus some extra bleed. For me, it was 345.8 × 234 mm (width × height). That budget breaks down as:

  • Two A5 pages: a single A5 page is 148 × 210 mm (width × height), so I need 2 × 148 mm in width, i.e. 296 × 210 mm
  • The spine: 21.8 mm, as defined above, added to the overall width, i.e. 317.8 × 210 mm
  • Extra 28 mm (width) and 24 mm (height) bleed, i.e. 345.8 × 234 mm. Note that you get grooves at the spine, where the glue that attaches to the inside paper ends, allowing for flexibility at the spine. This makes the actual A5 covers slightly wider. This adds to the extra width you need to account for.

The equations for the dimensions are, essentially:

  • Total height = book block height + 24 mm, so in my case the height is A5’s 210 mm + 24 mm = 234 mm.
  • Total width = 2 × book block width + spine width + 28 mm, so in my case the width is 2 × A5’s 148 mm + my spine’s width of 21.8 mm + 28 mm = 345.8 mm.

    The “hang over” of the cover over the book block. This is an extra 4 mm of material that helps to protect the pages of the book.

The extra 24 mm (height) and 28 mm (width) are required to allow the design to fit the cover. The budget for the extra width (28 mm) breaks down as:

  • 8 mm used to wrap the cover artwork around each edge of the book (16 mm total)
  • 6 mm used to allow the cover to “hand over” the paper, and to allow for extra material for the two “grooves” at the book’s spine (12 mm total)

The budget for the extra height (24 mm) breaks down as:

  • 8 mm used to wrap the printed design around the edges, as above (16 mm total)
  • 4 mm above and below the book block height to allow the cover to “hang over” the paper

Step 1: design the cover

If you choose to provide custom cover artwork like I did, you need to create a PDF of the correct size and upload it alongside your book block. Fortunately, Inky provide a video tutorial showing how to achieve this with the free, open source design software Scribus. Fantastic! It’s easy to set the dimensions in Scribus using guides, which help to define the area you need to provide artwork for, and show what will and will not be visible on the front, back and spine of the book. The video walks you through calculating the dimensions of each part of the cover (as I also listed above), so you’re finished with a grid of guides showing each area.

Guides for my A5 book in Scribus. In addition to the standard margins, I added guides to show the horizontal and vertical centre lines to assist with placing text and graphics.

Of course, the trickier bit is actually coming up with a cool cover design! For that, you’re on your own, except a note that you should design your cover with the back on the left, so that when folded over the book the text still reads the right way up.

Here’s what I came up with:

My thesis cover design.

I’m no design pro; I have the creativity of Status Quo when it comes to art. I included the usual stuff: the title and my name on the front cover and spine. I also added a generic picture related to my work*, prettified for the front cover with the removal of the labels, tweaking of axis limits and the use of a nice complementary colour wheel (it took a very long time to render, but that’s a story for another time). Tempted as I was, I decided against adding the thesis abstract to the back – this is not a novel. I managed to find a nice vector graphic of the university’s logo, so I added that to the front cover opposite my name. The background is a nice bold blue – my favourite colour. I’m quite happy with it!

*For anyone that is wondering, it shows a surface map of the power of the error signal from the second RF control sideband in the signal recycling cavity of ET-LF as the Schnupp asymmetry and exact cavity length are tuned (see Chapter 7).

Step 2: compile the pages in A5 format

I spent over half a year writing my thesis, formatting it for A4 paper as I went along. LaTeX, however, takes most of the control over page layout away from the user and uses algorithms to decide where to place text, figures and everything else. This generally works quite well, but occasionally has issues, especially when your text is figure-heavy. I found that changing the paper size from A4 to A5 led to some unintended blank pages at the beginning or end of chapters and sections, and small figures hogging their own page. Some of these issues were because of blank pages I had added when it was formatted for A4, which were no longer appropriate for A5, so let’s not blame LaTeX too much. Some of them, though, necessitated moving the figure definition in the source code to force LaTeX to alter its positioning behaviour, which is a bit more time consuming.

I had to go through the whole document and fix these as they occurred. To prevent wasting time in a recursive doom fixing one part and breaking another, I did this sequentially from the start. Whatever you do, don’t just fix layout issues at random as you find them – you’ll no doubt break the layout later in the document.

Naturally, going to a smaller paper size while maintaining the margin and font sizes leads to extra pages. Going from A4 to A5 in my case, the page count went from 244 to 271 – not as much a jump as I expected. As I did not alter the text size, the relatively small increase in page count is probably due to the page-spanning figures taking up less physical real estate where otherwise text could go. Put another way, the ratio of the graphic dimensions to the text dimensions was reduced.

Step 3: upload the sources and proof the full book

The PDF proof of my book generated by InkyLittleFingers. The cover is naturally larger than the rest of the book, as it has to wrap around the two cover pages and spine. The proof contains bleed marks, where the paper will be cropped after printing (to allow graphics to “bleed” over the sides of the paper, so that when it is cut the graphic is flush with the edge).

After creating the order (and paying!) you need to upload your work. The book block (the pages of the book) is uploaded as a separate document to the case artwork. In my case, the book block was a PDF compiled with LaTeX, and the cover was as PDF exported from Scribus. I thoroughly examined these documents before uploading, but the main check to make is of the automatic proof that InkyLittleFingers generate for you after uploading. It contains the case artwork and book block rendered as you will see them once printed: with the colours set, the pages appropriately sized, the fonts chosen and the images appropriately cropped. Check this proof thoroughly – this is exactly what you’ll see, minus the marked bleeds, when your printed books arrive.

In my case, the automatic proof contained a few small artefacts on some diagrams. I phoned Inky to verify where this was coming from, as I was not sure if this was just an effect of my PDF viewer. It turns out that they were real, and somehow produced by their systems: the book block I uploaded did not contain these artefacts, but only the proof generated by them. In hindsight, though, I doubt it is a bug with their own software, but rather some bug with the way that the Adobe PDF format handles embedded graphics. The worst artefact I found was in a graphic that was itself a PDF embedded within the PDF book block. This was originally generated with Inkscape using a library of SVG parts themselves made with Adobe Illustrator. Somewhere along this myriad format conversion process something happened that led to this artefact in the final version.

One of the minor artefacts in a vector graphic within the book block. The beam splitter – designated “M6” – should not have pixellated edges. Indeed, the source file I generated the graphic from does not have this artefact – it’s only when I got the proof from InkyLittleFingers that it appeared.

I never found out the culprit for the artefacts, but I did manage to get rid of almost all of them by playing with the SVG sources in each case. Flattening images from many layers to one, and merging overlapping segments were two strategies that seemed to work.

The nice thing about the Inky interface is that you can continually upload new book blocks and case artworks and generate new proofs until you are happy. I did this at least three times, gradually getting rid of all of the artefacts, until I was happy. In the end I left the worst artefact in, as I had managed to make it better than to begin with and it was very, very minor and hardly noticeable. My last hope was that it would not appear in the final print despite the proof.

Once you sign off on the final proof, Inky get to work making the book. Now the waiting begins!

 

Results

After about a week, I got the books in the post. Accompanying the delivery was a note saying that I should wait at least 12 hours before opening the pages fully; they are apparently immediately dispatched after production so there’s a chance the glue might not have fully set if the delivery was unusually quick. I was so eager to look at my creation that I did open one copy – the one I intended to keep myself – and it seems in hindsight that it didn’t do any damage.

The black and white pages were printed to very high standard.

The quality is superb. The whole book seems tough and durable. The book as a whole feels sturdy. The glue is neatly applied to the paper block, which is firmly glued to the inside of the case, over the edges of the cover which are tucked properly in place. It easily sits free-standing on its narrow side, able to sit on a shelf and support its own weight. The binding of paper and cover has been achieved flawlessly, and the book block sits exactly in the middle of the cover. The paper quality is great. The print fidelity on the black and white and colour pages is equally excellent. I am extremely pleased and impressed with the results, especially given the price I was charged.

The artefact mentioned above was, while present in the final print, hardly noticeable (see photo below). It was more than offset by the superb print quality for the diagrams and images (and indeed text!).

Highly recommended!

Online Labbooks for Scientific Research

Over at my workplace, the Institute of Gravitational Research at the University of Glasgow, we undertake lots of experimental research (primary physics) and produce lots of data. For the most part, this data is saved on individuals’ computers (centrally backed up but only accessible to that user), a shared hard drive on the (Windows) network, or written in paper labbooks (though these do have serial numbers for archival and accountability).

In my group, Stefan Hild’s European Research Council funded Speed-Meter experiment, we were slightly ahead of the curve. My interferometry colleagues and I for many years used a simple SVN-based labbook that involved placing a text file called “Notes.txt” and any images associated with the post into a directory with the current date. This was then parsed by a simple web interface to provide an online labbook that could be viewed by anyone on the network with the URL. It worked quite well, and was certainly better than a paper-based labbook in most regards, but it lacked a few key features:

  • Ability to link to previous labbook posts directly (via URIs/URLs): we instead had to quote the labbook title so that the user, if they were interested, could find the referenced labbook post.
  • Users: we used a single login for the SVN to add posts, all which which were marked as having been uploaded by “jifadmin”. It was only possible to tell who made a post by context.
  • Categories: the ability to group posts together in logical sets.
  • Versioning: otherwise known as “keeping track of edits”. We kind of hacked this to work with the use of the SVN, but the ability to roll-back or view a diff of two files involved the use of a separate system (command line or GUI).
  • Image/media upload: display of in-line images in posts, where they would logically go. Instead, our images always appeared at the bottom of the post and we would have to reference them by file name in the appropriate part of the post.
  • Centralised user credentials: this was part of a wider issue in the group which was gradually getting better. We would require a login for the SVN to add content to the labbook, but also a different login for other services like email, WiFi and the shared calendars.

We put up with this system for a number of years, but eventually when we started a major project we ditched it and turned to some software called ‘OSLogbook’.

Open Source Logbook

Imaginatively named ‘OSLogbook‘, this software is used in the Virgo and LIGO collaborations, which are scientific organisations that we are members of and frequently collaborate with. This brought a number of new features to the table:

  • LDAP login: centralised login system using centralised credentials, via the LDAP protocol.
  • Categories and tasks: each category can have subtasks, and each post can be assigned a single task within a category.
  • Authors: each user has their own login and can post their own content that is marked as their own.
  • HTML posts: a basic WYSIWYG editor allowed us to add headings, tables and coloured text to our entries.

We used this for a couple of years and found it useful, and a vast improvement over the SVN-based labbook. However, it still annoyed us in a few places and had some quirks which were not intuitive. For instance, posts could only be assigned to one category. In our daily work we often conduct lab work or experiments that involve multiple aspects of the same experiment, such as the organisation of power supplies to various corners of the lab or temperature and humidity changes affecting sensitive measurements.

The labbook also allowed us to assign multiple authors to each post, but in the form of a text field. That sounds useful at first, but it’s actually a pretty bad way of implementing this feature. If a person makes a labbook entry and puts their name before another collaborator, and the next day the other collaborator puts their name before the first collaborator, that author list is then a different set of characters despite containing the same names! Searching for posts by a certain author was therefore difficult because you don’t know in which order their name appear. Furthermore, it was left to the whim of the author of each post to correctly specify the names of their collaborators, meaning that occasionally a nickname or mis-spelling found its way into this field.

Perhaps the weirdest feature was that comments made on each labbook entry actually showed up as their own labbook entry. The list of labbook posts therefore contained in chronological order the comments made on each post as well as their respective posts – meaning the homepage was a cluttered mess of crossing conversations.

Not good.

The sum total of a number of small niggling issues with OSLogbook led me to investigate an alternative. We wanted our alternative to have the following features, in no particular order of importance:

  • Centralised login system: using our already-established LDAP system to authenticate users.
  • Permission system: we wanted our summer students and undergraduates to contribute to our labbook, but not be able to mischievously edit or delete other users’ posts.
  • Comments: labbook entries in an experiment with dozens of members more often than not require organised discussion. A comments system allows each user to contribute their thoughts in a coherent manner, without making a new labbook entry as a follow-up.
  • Media upload: beyond images, but also for zip files, analysis and plotting scripts, PDF documents, etc.
  • Rich text editing: we wanted to be able to add tables of data, links, coloured text, lists, special characters, centrally-aligned text… every little detail. Don’t constrain the user in making a post as clear as possible.
  • Multiple authors: the ability to assign a post to multiple authors, and allow any of those authors to further modify the content, and to have all collaborative posts an author is part of show up in a search of that author’s posts.
  • Reference pages: the ability to create static pages with reference information that does not fit in a labbook post. For instance, a list of procedures for ordering equipment or links to useful intranet/internet pages.
  • Ability to add new features: ideally, rather than a black-box monolithic software like OSLogbook, we wanted to be able to modify the software to fit our needs. There are a few programmers in the group, so with a nice API it would be able to tailor the labbook the way we want it.
  • Looks nice: people don’t read ugly websites unless they really have to. Let’s not make it difficult!

The obvious choice: WordPress!

Almost all of these features can be found on full-blown news sites, blogs and other content mills. It turned out that the popular blog software WordPress fit the bill… with the help of some plugins. Its vibrant community and extensive theme and plugin support made it a no-brainer. However, it lacked a few of the desired features that would not necessarily be useful on full-blown sites, but useful to a scientific organisation, like multiple authors and LDAP support. Before taking the plunge and moving my colleagues over to the system, I had to research whether there were plugins that could help us meet our list of requirements. In the end, the huge WordPress community delivered. There were plugins or native WordPress settings for all of our desired features!

Implementing Desirable Features in WordPress

This section is a guide to the plugins required to implement cool features, listed below, in our labbook software. The same plugins should work for anyone wishing to build a scientific labbook.

Multiple authors

As described above, we wanted to be able to assign multiple authors to individual posts, and allow these posts to show up in searches of any of the individual authors of that post. This can be achieved with the Co-Authors Plus plugin – and it does exactly what we want! After a single author finishes writing the post, they can start typing the names of their collaborators. As they type, other registered users with names matching the text spring up and can be assigned authorship. If a user is named as an author on a post, they can then edit it. This lets users create and edit any post they are named as an author on, but not necessarily edit other posts should you not wish them to.

Mathematical markup in posts

As a scientific group, we frequently share mathematical equations with one another. The plugin MathJax-LaTeX lets us type LaTeX equations in our posts and displays them in-line. It also utilises MathJax rendering, which displays in modern browsers nicely and allows the corresponding LaTeX code to be copied-and-pasted.

Email alerts for new posts and comments

We wanted to have the option of receiving an email alert when a new post or comment was made. Somewhat surprisingly, this is not a standard feature of WordPress. A Google search will probably point you towards JetPack, but this plugin is too bloated for what we require – it’s focused on commercial sites and making money, not a simple task like notifying registered users via email.

Email Alerts

Some users wanted alerts via email when new posts and comments were made on the site. This was accomplished by using the already built-in feature of WordPress, RSS (“Rich Site Summary” or “Really Simple Syndication”), which allows an external feed-reading service to download a copy of the recent posts on the site and send a digest to a user’s email address. Perfect, right? Not quite. Our labbook was to be private – scientifically sensitive measurements were under discussion – so external feed readers would not be able to access the RSS feeds by default. This hurdle was overcome by the plugin Private Feed Keys, which allows users to create a special URL which allows external readers to download RSS feeds. This doesn’t require a login, but each special URL is unique to each user and can be given to a trusted feed reading service (most of us use Blogtrottr).

 

With our plugins sorted, we still had to do something about the old labbook content. Since we had used our old labbook for two years, we had plenty of content already on it that we would not want to lose. We couldn’t just delete it, so I had to investigate a way to transfer the old content across.

Importing posts from old labbook

The number of posts was not so high (roughly 250, including comments) that it was beyond the scope of hacking together an import script. WordPress also has a nice API to allow a PHP script to directly create posts in the WordPress database – so I was able to create a script to extract the old posts from the OSLogbook database and inject them in to the new labbook. Comments were a little trickier, as were image and file attachments – but I managed to do it in the end with a mix of crafty PHP scripts and some elbow grease!

Styling the new labbook

The last job was to make the labbook look great. As I said earlier, people don’t like to use nasty looking sites with bad user interfaces. Luckily, WordPress is of course designed for ease-of-use, so we already had a great default theme. To allow the site to operate more like its intended labbook and less like a blog, we adopted the theme Simple Life and modified it slightly for our particular purposes. This led us to create a child theme (see it on GitHub) and a custom plugin or two to provide specific functionality that we wanted:

  • A recent edits widget to show any posts that were recently modified, to keep track of changes across all content (GitHub page)
  • A widget containing a list of comments and links to recent SVN commits (GitHub page)

Our awesome labbook

This is our labbook:

Our Speedmeter labbook software in use.

Our Speedmeter labbook software in use.

Pretty cool, huh?

Since we made the labbook, it has garnered the attention of other colleagues in our group who have since adopted similar blogs. The IT manager simply uses WordPress’s network administration tool to create a new site in seconds and clicks a few buttons to enable the plugins we use.

I hope this guide will be of use to the wider academic community and the science industry. You don’t have to suffer with poor labbooks!

Real time updating slideshow with webcam photos

I recently hosted a Halloween party and I thought it would be a cool idea to have my Raspberry Pi set up to take photos every few seconds and display them in some sort of slideshow. After a few hours of tinkering with my Raspberry Pi and Camera Module (technically the Pi Noir module), I ended up breaking the camera by sticking it too violently to a mount with Blu-tac. Oops. Not to worry though – I fished out a decent USB webcam and hooked it up to my laptop instead. I didn’t get to use my Raspberry Pi in a nerdy way, but this still did the job.

On the laptop I ran standard webcam software (Cheese) which saved the photos into a specified directory (/home/sean/Pictures/Webcam/). I then ran a cron job every five minutes to synchronise these photos with my web server. In the crontab, I specified a line:

# m h  dom mon dow   command
*/5 * * * * /home/sean/scripts/sync-photos.sh

And in /home/sean/scripts/sync-photos.sh I made an executable script with the following contents:

#!/bin/bash

cd /home/sean/Pictures/Webcam/
rsync *.jpg user@example.com:/path/to/web/directory/halloween/

I had already setup my laptop user account with an SSH key for the server, meaning the rsync command would not ask for a password. This can be done quickly on Ubuntu by doing something like this (but you should Google for a proper guide if you’re unsure):

~$ cd ~/.ssh
~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa): <filename here>
Enter passphrase (empty for no passphrase): <password here - this can be blank>
Enter same passphrase again: <password again, or blank>
The key fingerprint is:
a1:a3:32:ff:24:dc:ff:21:10:44:0e:ed:95:c2:ac:e3 user@hostname
The key's randomart image is:
+--[ RSA 2048]----+
<random art>
+-----------------+
~$ ssh-add -i <filename here>

So now, anyone taking a photo with the webcam would have their photo uploaded to my server. So far, so good. I still needed some way of displaying the photos. It turns out that there don’t seem to be very many decent, free and open source slideshow scripts which work in a browser and update their slideshow images periodically. Since the server would potentially receive a bunch of new photos every five minutes, I couldn’t be hitting refresh all night.

I ended up using a nice, simple slideshow script called slides.js. It uses jQuery, a really awesome Javascript library which makes Javascript a bearable programming language. I then crafted a bit of AJAX to grab the directory index page. By default, Apache web server will list all files in a directory if there doesn’t exist a file called index.html, index.php or similar. Index pages are predictable structure – they use simple HTML lists to display a link to each file – so it’s easy to grab out the relevant data with jQuery.

Finally, I had to find a way to update the slideshow periodically. This was not done with Javascript, but rather a rudimentary piece of HTML in the page’s header which tells the user’s browser to refresh every X seconds. I set this to fifteen minutes.

Here is the full slideshow script, which sits in the same web-accessible directory as the images on the server. I also added a piece of code to shuffle the images found in the directory, just to mix things up a bit!

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="refresh" content="900">
    <script src="jquery.js"></script>
    <script src="jquery.slides.min.js"></script>
    <style>
      #slides {
      /* Prevents slides from flashing */
	display:none;
      /* Sets width of photos to fit the screen (in my case, 1750px) */
	width: 1750px;
      }
    </style>
    <script>
      function shuffleArray(array) {
	for (var i = array.length - 1; i > 0; i--) {
	  var j = Math.floor(Math.random() * (i + 1));
	  var temp = array[i];
	  array[i] = array[j];
	  array[j] = temp;
	}
	
	return array;
      }

      function getImages() {
	var images = [];

	$.ajax({
	  url: "http://example.com/path/to/images/",
	  success: function(data){
	    $(data).find("td > a").each(function(){
		var name = $(this).attr("href");

		if (name.slice(-4) == '.jpg') {
	            images.push(name);
		}
	    });

	    // randomise
	    images = shuffleArray(images);

	    var imgstr = "";

	    for (var i = 0; i < images.length; i++) {
	      imgstr += "<img src=\"" + images[i] + "\">";
	    }

	    $('#slides').html(imgstr);

	    $(function(){
	      $("#slides").slidesjs({
		//width: 200,
		//height: 200,
		play: {
		  active: true,
		  effect: "slide",
		  interval: 5000,
		  auto: true,
		  swap: true,
		  pauseOnHover: false,
		  restartDelay: 2500
		}
	      });
	    });

	  }
	});
      }
    </script>
  </head>
  <body>
    <div id="slides"></div>
    <script>
      getImages();
    </script>
  </body>
</html>

The above script cannot be saved as index.html, otherwise it won’t be able to get an index page list of the files in the directory! There are ways around this, e.g. with PHP or similar, but as this was just a hack I renamed the file to slideshow.html.

With all of this set up properly, I opened a browser on my TV, pointed it to this script and entered into full screen mode. It displayed a nice slideshow of photos people took of themselves on my laptop, and updated throughout the night with the latest photos. Awesome!

A Handy Tool for Cleaning Up Disorganised Backups

During my recent trip to Germany, I took a whole bunch of photos with my smartphone. While I was out there, I experimented with a different operating system on my phone. Since my regular contacts from home were communicating with me using different channels (email, WhatsApp, etc.), I could afford to screw up my phone and not go completely off the grid.

It’s always a good idea to backup the contents you can’t afford or don’t wish to lose, even if you’re pretty confident you won’t screw up. So, before my experiments I lazily took a copy of my photos from my phone and dumped it on my laptop’s hard drive. Naturally, after each experiment I did not go back and remove my backup.

Now that I’m back, I’ve been slowly migrating the files associated with the last six months of work from my laptop onto my desktop. Today was the turn of the photos, and I found myself with three separate directories full of largely the same photographs, but not the same names. Removing each duplicate manually would be a pain, so I did some searching and found a nifty program called fdupes. Exactly what I needed! It was available via apt-get, which was even better. Here’s how to do it:

  1. Install fdupes. If you use Ubuntu (or Debian), it’s easy:

    ~$ sudo apt-get install fdupes

    If you use something else, have a look in your package manager or compile the source code (available from fdupes’s GitHub page).

  2. Move the folder(s) you want to remove duplicates across into another folder, e.g.:

    ~$ mv CameraBackup ~/tmppics
    ~$ mv CameraBackup2 ~/tmppics

  3. Run fdupes like this (though with the caveat that you should be careful here, since you’re deleting files – you should backup these files while you remove duplicates!):

    ~$ fdupes -rdN tmppics/

This will look for duplicates in each of the directories within tmppics recursively, and remove all instances of duplicates except the first one found. The -r flag tells fdupes to scan folders recursively, rather than just looking at the files in the top level of the directory specified. The -d flag tells fdupes to delete the files it finds. The -N flag tells fdupes not to prompt you before each deletion. The combination of -d and -N behaves in such a way that the first file found in each matched set is preserved, which is the behaviour I want.

Sonos-Like Synchronised Streaming, Part 2

In my last post I outlined my intention to make a Sonos-like audio streaming system. Today I managed to get a single Raspberry Pi playing music from a Spotify through a server on my home network. I’ll explain how I did it, and what I’m going to do next.

As I hinted at in my previous post, I don’t particularly fancy playing about with a Perl behemoth like Logitech Media Server. It also just seems a bit of a dead end to me – the server’s code is open source, but it’s hosted by Logitech, and it was open-sourced after they mothballed their Squeezebox range. Who can say how long Logitech will keep hosting and maintaining the code? Surely they will lose interest, without a product range to back up their investment of time and money. Additionally, the software seems so featureful that the community aren’t forking it and actively adding new features or refactoring the code. This state of play led me to try to find another solution, and it seems I have found it in Mopidy.

Mopidy

Mopidy is a Music Player Daemon (MPD) service, hence the name. It supports a subset of the standard features of MPD, but most of the important ones. Staying close to MPD means that Mopidy can be controlled with one of the many MPD clients available. For instance, during my testing I was using ncmpcpp, a terminal client.

The feature I love about Mopidy is that it is written in Python. I can open it up and work out what’s going on, with enough effort, and I can add extensions if I am so inclined. It also supports PulseAudio, which brings me on to the next part of the setup…

PulseAudio

PulseAudio is a piece of software that acts as an interface between audio sources (programs, games, microphones, etc.) and audio ‘sinks’. A ‘sink’ is a place where audio goes, be it a sound card, an audio editor or a network stream. PulseAudio supports streaming via RTP, which brings me on to the next part…

Real-time Transport Protocol

Real-time Transport Protocol, or RTP, is a method of transport on networks which tries to make sure that data is sent in a timely fashion. It is intended for use with audio, and often forms part of VoIP packages a la Skype. We can tolerate packet loss with audio, in favour of keeping everything timely.

Setup

Since I currently only have two Raspberry Pis, and one is spending its life forwarding electricity and temperature data to another server, I can only set up one Pi as a receiver. This is a good opportunity to test the concept of the Mopidy-PulseAudio-RTP audio stack.

In terms of hardware…

  • I need a server to host Mopidy and stream its audio via PulseAudio-RTP. I have a Philips Living Room PC Core 2 Duo server that’s about 5 years old that I use for this kind of thing, running, as of yesterday, Ubuntu 14.04 LTS Server Edition.
  • I have a Raspberry Pi, of course, with a 16 GB memory card with Raspbian. The 16 GB is only there because I don’t have any other SD cards larger than 2 GB (standard install Raspbian needs about 3 GB). Mopidy and PulseAudio won’t need anywhere near 16 GB of storage, and I’ll probably eventually strip out all the extra stuff in Raspbian I don’t use and copy the whole card to a 2 GB one. I am using headphones on the Pi to listen to music. The output jack won’t drive anything more powerful, so you’ll need an amplifier if you want to play music through speakers. It’s also possible to output audio via the HDMI port, so that might work for others.
  • A laptop to control what’s playing. Not strictly necessary, if the server is also a useable desktop computer. You can also use one of the many MPD clients available for Android and iOS to control Mopidy, if you want.
  • A router running DHCP or similar, with every device connected to it. Again, strictly, the DHCP server can be the same as the music server, but I use my home router as the DHCP server so I’m using that. The Raspberry Pi and server are connected via wire to a switch which in turn is connected to the router. The room I’ve got this stuff set up in has only one ethernet wired connection, and I split it into two with the switch. The laptop is connected wirelessly.

In terms of software, this is what I’ve used:

  • Server: Ubuntu 14.04, PulseAudio, Mopidy
  • Raspberry Pi: Raspbian, PulseAudio
  • Laptop: Ubuntu 12.04, ncmpcpp terminal client

Instructions

Here’s how to do it:

PulseAudio setup on both the server and the Pi

server~$ sudo apt-get install pulseaudio
pi~$ sudo apt-get install pulseaudio

That should work for both Ubuntu and Raspbian.

Next, set up an RTP sink on the server. Change directory and edit /etc/pulse/default.pa:

server~$ cd /etc/pulse/
server~$ sudo nano default.pa

Fill it with the following content:

#! /usr/bin/pulseaudio -nF

load-module module-native-protocol-unix auth-anonymous=1
load-module module-suspend-on-idle timeout=1

load-module module-null-sink sink_name=rtp format=s16be channels=2 rate=16000
load-module module-rtp-send source=rtp.monitor

This will set the audio sample rate to 16 kHz. This is low, but I found that setting it to 44.1 kHz or 48 kHz would congest the network to the extent it is unusable. I need to tweak this later and get it to work, perhaps using a separate network altogether.

Next, edit daemon.conf:

server~$ sudo nano daemon.conf

Leave everything in there, but uncomment and edit the line exit-idle-time to be:

exit-idle-time = -1

And uncomment and edit the line default-sample-format to:

default-sample-format = s16le

On the Raspberry Pi, configure the /etc/pulse/default.pa file:

pi~$ cd /etc/pulse
pi~$ sudo nano default.pa

Uncomment the line that says #load-module module-rtp-recv .

Next, edit daemon.conf on the Pi:

pi~$ sudo nano daemon.conf

Add at the bottom:

default-sample-rate = 16000
resample-method = src-sinc-fastest
default-fragments = 10
default-fragment-size-msec = 10

Mopidy and Spotify on the Server

Following the installation instructions from Mopidy, on the server, add the repository key:

server~$ wget -q -O - http://apt.mopidy.com/mopidy.gpg | sudo apt-key add -

Add the repository sources as a file:

server~$ cd /etc/apt/sources.list.d/
server~$ sudo nano mopidy.list

Add to the file:

# Mopidy APT archive
deb http://apt.mopidy.com/ stable main contrib non-free
deb-src http://apt.mopidy.com/ stable main contrib non-free

Update the repositories and download the packages:

server~$ sudo apt-get update
server~$ sudo apt-get install mopidy mopidy-spotify

Run Mopidy once to make it generate the configuration file:

server~$ mopidy

Press Ctrl+C to kill the server. Edit the file ~/.config/mopidy/mopidy.conf:

server~$ sudo nano ~/.config/mopidy/mopidy.conf

Edit the configuration options under [audio] and [mpd] to look like this (you can leave the commented values already there):

[audio]
output = pulsesink device=rtp
[mpd]
enabled = true
hostname = ::
port = 9999

Set the port in [mpd] to whatever you want. I found the default Mopidy port was already used, even on the default Ubuntu 14.04 install, so I set it to 9999.

For Spotify, edit the values under [spotify]:

[spotify]
enabled = true
username = [username]
password = [password]

Obviously setting [username] and [password] as appropriate. I found that this pair was all I needed to get Mopidy to access my playlists successfully. It’s also possible to specify bitrates and timeouts, but I left these commented.

(Optional) ncmpcpp on Controller

Follow the instructions on the ncmpcpp website. I didn’t want to add the unstable Debian repository so I compiled it from source, but there are alternatives.

Getting it to Work

Everything is configured, so it should then be possible to start everything up and play music on the Pi. On the server, start PulseAudio as a daemon and Mopidy in userspace:

server~$ pulseaudio -D
server~$ mopidy &

Pay attention to /var/log/syslog if you have any trouble.

On the Pi, start PulseAudio:

pi~$ pulseaudio -D

Again, any trouble and check the syslog.

Now, play something on the server using the laptop or other controller:

laptop~$ ncmpcpp -h 192.168.1.10 -p 9999

The IP address 192.168.1.10 corresponded to my server. The port is set to the one I specified in the configuration.

In ncmpcpp, I can look at my Spotify playlists and play them, and I hear the tunes through my Raspberry Pi’s headphones. The music quality isn’t great, because I set the sample rate quite low. I will need to investigate network issues to see how to improve audio quality while keeping network congestion minimal. PulseAudio’s RTP uses multicast, and this in general is a heavy protocol. For instance, reports all over the web say it can floor a WiFi connection with ease. Sonos uses a separate wireless network, and presumably this is because quality and throughput can’t be guaranteed on a regular network.

So, I have the ability to play music across the network. Next up, I will get another Raspberry Pi and set it up to listen to the same stream, to test how multicast works with two clients. Hopefully the timing provided by RTP will mean the music steams will be in sync.

Credit to ianmacs’ posts on this Raspberry Pi forum thread, and this blog post from rozzin.

Sonos-Like Synchronised Streaming, Part 1

I got a Sonos kit a little while back through an employee offer at my old workplace. For anyone that doesn’t know, this is an easy-to-use solution to playing audio from local computers, internet radio, Spotify, Napster and so on. You buy a Sonos speaker and base station, and connect the latter to your internet connection. It finds network attached media servers and can interface with your Spotify account (or other online music service). Then, you either buy the ridiculously expensive Sonos controller or download the free app to control what the speaker plays.

I’ve used a controller at a friend’s house and it is nice, but the app is just as featureful. It combines searches across the different sources quite nicely, though this is only a recent feature. You can create queues, random shuffles and so on, just like any good media player.

The really nice thing about the system is the fact that it works seamlessly. You can add new speakers to the mix by plugging it in and holding down two buttons. It then appears in your app, and you can control what is played on it. You can play different songs in different rooms, or synchronise one audio stream across all of the players. Really cool.

I can completely understand the problem the Sonos system tries to solve. It has the Apple-esque “it just works” feeling about it. I would love to purchase a second Sonos to make use of the multi-room synchronisation features. However, the Sonos system has an Apple-esque pricetag, and, as a massive nerd and tinkerer, I find myself wondering how much effort it would take to make a homebrew version.

The obvious way to do this is with a Raspberry Pi. This credit card sized computer is great for hacking together web-connected controllers. It has its own native Linux distribution and there are loads of tutorials on the web concerning most features of the device. What’s more, they are cheap. Whereas a Sonos system is many hundreds of pounds, a Raspberry Pi starts at £25. Once you add a few extras for playing music, I don’t think this will come close to breaking the bank.

There are numerous reasons why a homebrew setup might be difficult:

  • The Raspberry Pi’s analogue audio output is not so brilliant in terms of quality, and therefore of little use for a Sonos-rivalling music system. Instead I expect a solution would either involve the Pi’s HDMI output or a USB sound card.
  • Synchronisation across distributed units is potentially difficult, and might involve something like Real-time Transport Protocol.
  • Interfacing with music services like Spotify and Napster together with files on a home server or connected laptop is a whole project in itself.
  • Latency can cause all sorts of issues, especially for synchronised audio. The Raspberry Pi has limited CPU power, and connecting it to a WiFi network or congested wired network will no doubt impact performance.
  • Controlling the various distributed units will require some topology thoughts. Should there be a web interface to a single server (not necessarily a Raspberry Pi but a computer on the same network), which can control the connected Pis? Or a mesh network where each Pi is both a client and server, passing around packets to each other? How should the server know about each connected unit? Should it send a message to the whole network to find them, or should the user need to register the IP address of each device in an admin panel or configuration file?

Sonos has addressed some of these issues by insisting that it has its own dedicated network. The base station that is a requirement of the Sonos system creates a wireless network of its own which the networked speakers can listen to. Other network traffic therefore does not impact the system. It also, handily for the shareholders, means that only Sonos speakers work with other Sonos speakers. Once you’ve shelled out £250 for the first kit, if you want to add any more you need to shell out more for further Sonos speakers.

The open-source community excels at providing alternatives to systems exactly like Sonos. The gap in the market that Sonos fills is to take these ideas and create a product. Open-source implementations often involve many different pieces of independent software, and lots of configuration to get them to talk to each other. This is what I would like to try to impact. There are various homebrew setups of Sonos-like systems described in blogs, but there doesn’t appear to be a system that covers the whole Sonos stack from app to speaker to Spotify.

Technologies exist for the individual layers on the Sonos stack:

  • Synchronisation of audio streams across different devices: Real-time Transport Protocol
  • Playing of audio from network: various Linux technologies such as PulseAudio or ALSA
  • Streaming audio from a combination of local and remote sources: Mopidy
  • Control via web interface or Android/iOS app: Moped or similar

It can be done. It will take time to do though. Others appear to have managed to get the synchronised audio part working properly, but I don’t know if anyone has managed to get Spotify to work too. Various solutions also involve Logitech Media Server, which is (somewhat weirdly) open-source and written in Perl (eugh). That might be an option, but I doubt I’d want to make any code modifications in Perl. Happily, Mopidy is written in Python, a glorous language, so I would be happy to dive in and start tinkering.

I’m going to give this a go at some point. I’ll keep you posted.

Update: view the second post in this series.

Hattrick Player Tracker

One of my current projects is the building and running of a player tracker for Hattrick. Hattrick is an online football management game where players can start teams, enter leagues, buy and sell players and eventually try to win some trophies. There are a few things about it that I think are nice:

  • The games take place in real time, so league games are weekly. No need to be online all the time to be successful.
  • A decent community of fellow football fans, and special forums for fans of particular clubs or nations to come together.
  • A national team aspect, allowing you to take part in the ups and downs of your club’s country’s national team.

It’s this last aspect that in particular is exciting. Every 6 months, the players in a nation can vote for an individual to become the national team manager. This person then has to call up players from clubs around the world in order to try to win national team games. This process is often tricky, because the manager does not see the skills a particular player has until they are called up. The manager can’t call up more than a handful of players at a time, and every time they call someone up the squad’s morale takes a hit. This means that it is very much advantageous to have a network of scouts working for the national team, keeping track of player skills wherever they can get their hands on them.

The scouting aspect is where I come in to play. I thought it would be fun to get involved in my national team in Hattrick (Scotland), but I’m by no means the most tactically astute person in the country. My talents are better served in making an automatic tracking system, which is what I’ve done. Around this time last year, I started working on Hattrick Scotland, a website where users can register to have their players’ skills tracked. After registering, a special access token is retrieved from Hattrick to allow the website to access Hattrick on their behalf. Then the user may leave, and not necessarily have to come back ever again. With the token, I can use scripts to systematically and periodically access the required data that the national team scouts require.

With a database of player skills available, scouts can then access the site to view this data privately. Scouts are assigned by the national manager and so the site can identify them and let them access the database accordingly. It was a lot of work to put this together, but I found it fun learning the reasonably new aspects of PHP 5. The last time I used PHP on a large scale was when I was in high school, hacking together forum software for my friends and I. I learned it the way that most kids do, by online examples. It turns out that PHP is very much a victim of its own ease of use, and a sizeable chunk of its users are equally oblivious to basic programming paradigms. For a long time, the examples available regarding aspects of PHP were hacky and didn’t follow good programming practice. This can be important, because programs are increasingly being reused rather than rewritten as they become more complex, and powerful new features to modern languages often require programs to be organised in a certain fashion. The consequences of this meant that coming into the task of programming a player tracker, my experience with PHP was biased towards the ‘get it done quickly’ attitude.

The initial site I made was a reasonably badly written pile of code. After a little while I started to realise that I had made some fundamental mistakes in the design of the tracker, and went back to the drawing board. The design I eventually ended up using turns out to be quite similar to Django and other web frameworks. This was a happy accident, and it meant that I could add new features quickly towards the end of summer last year, after weekends and evenings spent here and there on the project. Now the tracker is more or less complete in its initial purpose, and additions I’ve made since then have been along the lines of ease of use and look and feel rather than functionality.

Running a large site like this has some interesting challenges. Although there are only around 80,000 Scottish players to track in Hattrick, which is a small number to relational databases that power most large websites, there are plenty of other things that also need tracked. Each player’s skills are of course the primary information to track, but it’s also useful to track this across time – is a player improving with time or are they being neglected? This is important information that the national team manager must be able to find out. Also, details of the matches the players have played in are important, so the manager and scouts can work out if the player has been getting the correct game time they require to improve. All of this leads to a database with many millions of rows of data, each needing to be organised and stored. It’s certainly been a good learning exercise.

Since some of the code I’ve written I am quite proud of, I think I will eventually release this project as an open source program, letting other Hattrick countries adopt their own trackers. I have not yet done this because I want Scotland to gain an advantage from having a shiny, new and featureful tracker, at least for a while. My work has appeared to have inspired other countries to start developing similar trackers, so when the advantage diminishes I will open up the code. I am sure there are plenty of improvements that could be made by other members of the community.