Music and Artificial Intelligence

There was a great BBC click podcast this week, covering some of the advances made in technology and music.  There was some interesting stuff around Bjork’s new music video and art exhibit, but also a section on IBM Watson research.  They did some interesting work on computer/musician collaboration with Alex Da Kid.

They also discussed learning algorithms, used to identify vocal stem samples in songs, and attribute them to the original song.  Many vocal lines are sampled hundreds, or thousands of times, and it’s not actually clear how derivatives of original work have evolved.  Some interesting stuff in there – worth a listen.

Registrator, a Docker Service Registry Bridge

I’ve been diving into golang recently, partly by accident, given that we decided to use an open source docker service registration bridge called registrator.  Registrator fills in a niche with docker containers – how to get them into an existing service registry such as (the excellent) consul.

We plan to use this to keep our existing netflix eureka service registry updated with new container based services as they are brought online, and allow us to integrate containers into our existing VM based microservices environment.

There was a great PR to provide eureka support to registrator which someone had already submitted.  However, we had additional requirements too:

  • Add the AWS metadata about the host the container is running on
  • A way to register metadata that they are actually containers, and our own stuff like routes for the service.
  • Support for registering ELBv2 (application load balancer) endpoints in eureka automatically. We plan to use these in most cases rather than directing traffic straight to containers from nginx.

Given that the original eureka PR hadn’t been accepted yet to gliderlabs/registrator, we forked the project, and will carry on making changes as necessary.  You can have a look here at the first PR for the main changes.  Hopefully gliderlabs will be interested in some of our additions to be merged back in the future.

Next, I started trying to write some golang unit tests – but that’s stuff for another post.

 

Rise of The Robots

I recently finished listening to the audiobook version of Rise of The Robots, by Martin Ford.  It was a fascinating and insightful look at how robots and computers are used today to replace human work, and how the expansion of artificial intelligence and the reduction in costs for new machines are accelerating their creation.

It got deep inside my psyche – I’ve been thinking about the implications for a couple of weeks afterwards, and it started a big conversation on Christmas Day with my family.

Collective bargaining, a degree and an engineering skill set just don’t cut it any more, if you’re competing with a robot that can produce more widgets, more quickly, with higher quality and lower cost.  Some of the traditional views about how you protect jobs in a capitalist economy just don’t apply to this threat.  Nedd Ludd would be spinning in his grave.

One of the central tenets of the book, is the reduction in jobs that will result from the rise of AI, and the increased automation of jobs that humans used to have to do.

Interestingly, this week, FoxConn, a major electronics manufacturer in China revealed it had automated away 60,000 jobs from it’s factories.  Though it’s questionable whether you’d really want to work in such a place unless you had no choice.

Many, from technology futurists, to economists, agree that automation is going have a big impact on the jobs market in the future.

One of the main problems mentioned in the book is that most of the jobs we do today, are not necessary for survival.  And a lot of the old middle-class positions are either already non-existent in the 21st century, or primed for automation.  There are potential software solutions to replace jobs, whether you are a bookkeeper, lawyer or a fast food chef.

Fast food is a good example of an area that has thus-far resisted automation, because of low wages, and highly efficient staff training and procedures.  Both make employee turnover a less serious issue for the big fast food giants.  However, at a certain crossover point in wage growth and automation cost, a point is reached where companies will find it cheaper to invest in automation – that is, machines to flip the burgers, dunk the fries, and package them up.  At that point, lots of people’s jobs will become threatened very quickly, with the power that two or three large food franchises have around the world.  Our local McDonalds have already automated most of their order points.

So what about solutions to these problems?  How can we keep producing careers for people, in an age where so many jobs could be automated given sufficient money and motivation?

I think one important mention by Ford is that we don’t really need to work as much as we do anymore in our developed economies.  We have chosen to take the time we’ve saved from automation of agriculture, and manufacturing, and instead directed much the same time to office work, for 40 hours a week.

Why do we all need to spend this arbitrary number of hours doing these, often unimportant, things?  For many people, they don’t even produce a great deal from the time they spend at work.  Everyone wants to feel like they produce something important – but I doubt any most office output is really important as the loaf of bread we produce by machine, or the wheat grown in the field, and harvested by a machine.

Will the idea of full-time employment for everyone even be desirable, let alone possible in the late 21st century?

Some ideas to change the way we think about work, are the universal basic income, higher wages for less hours of work, or just plain artificial job creation mechanisms.  I’m not sure which of these, if any, will be used, but I am sure that something will have to change, lest we get a future where the divisions of our society grow much larger.

Unfortunately, I think in the meantime, before we realise the real threat, there will be big fights over a return to trade protectionism, amongst nationalism and fear of immigration that we are seeing across the developed world, and the force down of wages in many sectors – something we have seen in Britain already in recent years.

To summarise – a great, thought provoking book, with lots of well argued points of interest.  I’d recommend the audible version if you don’t have a lot of time to read.  It’s quite a lengthy work.

Links:  Audible  |  Amazon

Coworking and Mental Health, plus Nottingham Co-Working Spaces

There’s a great article here on the benefits of co-working, and how it can transform what can be a lonely and difficult struggle for some people working at home, into an energising and supporting environment.

I’ve found it’s becoming more and more important to me too, now in my 10th year of working at home for the majority of the time.

 

The co-working spaces I’ve used in Nottingham are:

Antenna

This is my regular haunt, which I typically use once or twice a week.  The £30 a year membership fee is really a bargain, though you don’t get the free tea/coffee that some more office like environments provide.

Cafes suit me for working in, but the noise and bad wifi doesn’t.  Antenna solves both of those problems.  It’s really a cafe bar with fast wifi (100Mb) and a nice working atmosphere.  I feel at home here, and like to go to the Tech Nottingham events that are on the first Monday of the month too.  The food at Antenna is excellent.  I often treat myself to a breakfast sandwich, which are delicious.

They also have virtual tenancy options for the use of an additional member’s lounge, with desks available from £49+VAT per month.  There are well equipped meeting rooms available for hire.

Cobden Place

I had a co-working rental here for a few months, but I didn’t use it as much as I thought I would.  It’s got a nice community, and is a real co-working office, but it’s a bit more bare bones than Antenna, and I had a few network issues a couple of times when I was in that put me off.  The location is excellent, in the Hockley are, in a little alleyway opposite Wired Cafe.

They also provide studios, a meeting room, and resident desks for various costs.  The basic (once per week) package starts at £35.

Minor Oak

This is a new place in the refurbished Sneinton Market units which opened in September.  At first glance it looks promising – they have a quiet area, and phone booths for quiet meetings, which might be ideal for me.  It’s also a short hop from Antenna.  Since it does daily/half-day pricing, there’s a low barrier of entry to trying it out too.

 

Glastonbury Ticket Site Update

Three years ago I wrote a detailed post with observations and advice on the Glastonbury Festival ticket sale, and how their website appears to work. Again, I was involved in the big refresh this year, trying to buy tickets for friends and family.

I wasn’t successful in purchasing tickets this year, although my sister was able to buy two sets of tickets (I.e go through the checkout process twice). I also spent a lot of time dealing with a grey page, with connection errors and the like, like many people.

Immediately after the sale, after the tickets sold out, the pages refreshed fine. This suggested to me that it’s not just about traffic this year – they are likely using IP rate limiting. This explains several things:

  1. Before the sale, people were advised not to try to buy tickets from commercial premises (typically these share one IP address).
  2. See tickets repeated the advice to only use one device during the sale.
  3. Emily Eavis again repeated the same advice during the sale.

This change means that my previous advice about using multiple tabs with a refresh plugin to your browser with an auto refresh feature may actually be counterproductive now.

In fact, although I was unable to get the registration page with this Uber-refresh approach, my stepdad got the registration page on his tablet over a Mifi 4G connection, and my sister got the registration page twice (and successfully booked tickets) only using two browser windows, one in incognito mode.

This doesn’t tell us anything for sure really, but I strongly suspect the traffic volume approach will no longer help you get through and book tickets.

I’ll be researching this a bit more and coming up with some updated advice for those who missed out to use in the resale. 

There are some other techniques that can now be used instead, including using a VPN to improve chances of success in the future.

Install a Babun (Cygwin) Shell and Ansible for Windows

Update: You might find this thread useful if you have issues getting ansible to work in babun/cygwin  – thanks mcfo for the comments.

Update2: Another option is to run ansible inside a docker container using Docker for Windows.  This way you are actually using Linux to run ansible in, so are likely to have less problems.  Startup time, (after the first time you run it), should be very quick, so has none of the disadvantages of running inside a heavy VM.

Babun is a nice distribution of cygwin with lots of pre-installed packages, and also a built in package manager called pact. It has an auto update tool, and includes most of the ansible requirements already, such as python and gcc. Another advantage is that it won’t affect your existing Cygwin install, should you already have one.  As such, it’s a pretty good way to get started with ansible on a windows workstation.  Note that this is not officially supported, but it is often needed, and works pretty well for the most part, with a few tweaks.

This guide covers installing Babun, and the appropriate packages to get ansible working on Windows.

Getting and Installing Babun

  1. Download Babun from the website at: http://babun.github.io/
  2. Unzip the file and run the install.bat which comes with it. This will install Babun under your user profile at c:\Users\youruser\.babun
  3. A zsh shell will be launched – you can switch to bash with the below:
babun shell /bin/bash
bash

VIM

There is an issue with the right click paste when inside the vim program.

Update: It should work if you add set mouse-=a to your ~/.vimrc file – thanks mcfo for that!

Bootstrap Ansible

Run the bootstrap script as follows:

curl https://raw.githubusercontent.com/chrisgilbert/scripts/master/bootstrap_ansible_windows.sh | bash

This will set up the dependencies for ansible on windows.

Accessing files

Babun adds a mount point /c for the C: drive. If you wish to add another drive, then you can add an entry to the /etc/fstab in a similar fashion, and run “mount -a”.

E.g. to access files at c:\src\ansible, you should use

cd /c/src/ansible

Setting Up SSH Keypair

  • Create an SSL public/private key pair. This will allow you to connect to servers remotely over SSH without usernames and passwords. This is required for easy operation of ansible, and also to use any corelogic git repositories. Ansible manages servers using public/private keypairs. It can also fall back to usernames and passwords, but this quickly becomes very laborious when working with multiple servers, especially when using roles. As such, you should set up a keypair as below, and protect it with a strong passphrase.

In order to do this in a secure way:

ssh-keygen
[press enter to choose rsa]
[enter your choosen passphrase twice]
chgrp -R Users ~/.ssh
chmod -R 700 ~/.ssh/
chmod 600 ~/.ssh/id_rsa

This secure passphrase is yours alone, and should be a good strength password. You can also add it to other programs which use SSH keypairs in many cases (e.g. putty, filezilla, etc).

  • Next, you can add the ssh-agent for convenience. To avoid entering the passphase every time you connect, which would be no easier than a username/password, you can instead just enter it once per session. This is achieved using the ssh-agent package. The ansible bootstrap script you ran above should have added the appropriate lines to your bash profile. Just source it again with the following command:
. ~/.bash_profile

You will be prompted for your password now, and each time you open a new shell.

Test Your Ansible

To test things are working:

# Add a server to test to your default inventory file (ansible-training will already be here if you used the bootstrap script)
nano /etc/ansible/hosts

ssh-copy-id root@ansible-training

# You will be asked to input the root password once here (ansible), after which you should be able to connect without a password.

ansible ansible-training -m ping -u root

Ansible-training is just an example – add your own server as required.

If that works, you should get a “pong” back from the server in green text to say the connection was successful. You may have spotted the “-u root”, which means connect as the root user. You can of course connect as other users, but ansible playbooks provide a way to switch users using sudo also, so don’t worry about that too much at this moment.

Off to Hudl

Hudl_PrimaryAfter almost 10 years at Servelec-Corelogic I’ll be starting in a new position at Hudl on 1st June.

I’m excited and nervous about the change and looking forward to meeting my new colleagues and learning about how they do things over there!  Some of the things to look forward to are their open-minded approach to technology, working culture and cutting edge continuous deployment processes.  I’m sure I’ll learn a lot, and hopefully I can add some of my experiences to the team.

It looks like I’ll be off to Lincoln, Nebraska in June to do some training and meet everyone.

It’s going to be really tough leaving Corelogic.  But it’s about time to see what the outside world is like – and I hope it will be a fun and positive move for me.