Install a Babun (Cygwin) Shell and Ansible for Windows

Babun is a nice distribution of cygwin with lots of pre-installed packages, and also a built in package manager called pact. It has an auto update tool, and includes most of the ansible requirements already, such as python and gcc. Another advantage is that it won’t affect your existing Cygwin install, should you already have one.  As such, it’s a pretty good way to get started with ansible on a windows workstation.  Note that this is not officially supported, but it is often needed, and works pretty well for the most part, with a few tweaks.

This guide covers installing Babun, and the appropriate packages to get ansible working on Windows.

Getting and Installing Babun

  1. Download Babun from the website at: http://babun.github.io/
  2. Unzip the file and run the install.bat which comes with it. This will install Babun under your user profile at c:\Users\youruser\.babun
  3. A zsh shell will be launched – you can switch to bash with the below:
babun shell /bin/bash
bash

There is an issue with the right click paste when inside the vim program. This cannot be fixed directly unfortunately – you can however access the system clipboard by using the vim command “*p to paste. As such, I suggest using the “nano” editor when following this guide, as pasting works correctly in that.

Bootstrap Ansible

Run the bootstrap script as follows:

curl https://raw.githubusercontent.com/chrisgilbert/scripts/master/bootstrap_ansible_windows.sh | bash

This will set up the dependencies for ansible on windows.

Accessing files

Babun adds a mount point /c for the C: drive. If you wish to add another drive, then you can add an entry to the /etc/fstab in a similar fashion, and run “mount -a”.

E.g. to access files at c:\src\ansible, you should use

cd /c/src/ansible

Setting Up SSH Keypair

  • Create an SSL public/private key pair. This will allow you to connect to servers remotely over SSH without usernames and passwords. This is required for easy operation of ansible, and also to use any corelogic git repositories. Ansible manages servers using public/private keypairs. It can also fall back to usernames and passwords, but this quickly becomes very laborious when working with multiple servers, especially when using roles. As such, you should set up a keypair as below, and protect it with a strong passphrase.

In order to do this in a secure way:

ssh-keygen
[press enter to choose rsa]
[enter your choosen passphrase twice]
chgrp -R Users ~/.ssh
chmod -R 700 ~/.ssh/
chmod 600 ~/.ssh/id_rsa

This secure passphrase is yours alone, and should be a good strength password. You can also add it to other programs which use SSH keypairs in many cases (e.g. putty, filezilla, etc).

  • Next, you can add the ssh-agent for convenience. To avoid entering the passphase every time you connect, which would be no easier than a username/password, you can instead just enter it once per session. This is achieved using the ssh-agent package. The ansible bootstrap script you ran above should have added the appropriate lines to your bash profile. Just source it again with the following command:
. ~/.bash_profile

You will be prompted for your password now, and each time you open a new shell.

Test Your Ansible

To test things are working:

# Add a server to test to your default inventory file (ansible-training will already be here if you used the bootstrap script)
nano /etc/ansible/hosts

ssh-copy-id root@ansible-training

# You will be asked to input the root password once here (ansible), after which you should be able to connect without a password.

ansible ansible-training -m ping -u root

Ansible-training is just an example – add your own server as required.

If that works, you should get a “pong” back from the server in green text to say the connection was successful. You may have spotted the “-u root”, which means connect as the root user. You can of course connect as other users, but ansible playbooks provide a way to switch users using sudo also, so don’t worry about that too much at this moment.

Off to Hudl

Hudl_PrimaryAfter almost 10 years at Servelec-Corelogic I’ll be starting in a new position at Hudl on 1st June.

I’m excited and nervous about the change and looking forward to meeting my new colleagues and learning about how they do things over there!  Some of the things to look forward to are their open-minded approach to technology, working culture and cutting edge continuous deployment processes.  I’m sure I’ll learn a lot, and hopefully I can add some of my experiences to the team.

It looks like I’ll be off to Lincoln, Nebraska in June to do some training and meet everyone.

It’s going to be really tough leaving Corelogic.  But it’s about time to see what the outside world is like – and I hope it will be a fun and positive move for me.

Some Thoughts on Almost 9 Years of Remote Working

Since I started working for Corelogic in 2005, I only spent the first year working in the office every day.  I lived in London for that year, within walking distance of the office, in a nice, albeit small 1 bed flat.  That flat probably cost around the same price as our 4 bed semi in Nottinghamshire (and that was years ago).  But I digress.

Anyway, I didn’t enjoy living in London too much.  It’s a nice place to visit, but it will always be there to visit.  That doesn’t mean I had to live there.  I found it too noisy, expensive and crowded for me.  It’s still noisy and crowded, but even more expensive than it used to be.  There’s a lot of good things about it, but I am not a particularly outgoing person, and I felt there was a lot of chaos there that you have to put up with day to day.

Since I moved back to Notts, I have worked remotely, in various different arrangements, and have been doing that for the last (almost) 9 years.

  1. For the first year or two I tried two days a week in the office, and an overnight in a hotel.  This was self funded, and became expensive, though I learnt lots of tricks for cheap train fares and the best (and worst) places to stay.  Very advanced bookings at a Travelodge was the best way to make it affordable, in case you wondered.  However, staying in identical Travelodge rooms every week is not my favourite pastime.
  2. After a while I asked to change my arrangements, and work from home permanently which the company agreed to, and helped with travel expenses after that.  I visited less, sometimes once a week, sometimes less up until the present day.

I have learnt a lot of things about remote working in this time, and I thought I’d put some of these in a blog post, to hopefully help others who are considering it, or struggling with it.

The Negatives

Lets put these first, since everyone can think of some positives about home working.

It’s (Still) A Bit Weird

People still get envious or curious about this sort of arrangement, even in 2015.  From the tradesman confused that I will be in during the day, to neighbours thinking I lived off benefits, it is apparently still an odd working arrangement, even in today’s world.  Over the years, I’ve met a few other people who do this, and work with a few too, but it’s still pretty unusual.  I get the impression it’s much more common in the states, where many job adverts mention remote working.  In the UK, there’s still very much a culture of “presenteeism” among many company’s recruitment regimes.  I think the overwhelming fear in our zero-hours culture is that the employee will not do any work unless their manager is standing over them with a big stick at all times.

It’s Hard to Communicate

You must work hard to communicate.  I’m not a big phone user, but use email (hated but required), IM (try Slack, or HipChat), and Skype an awful lot (or Google Hangouts).  You must communicate as much as you can, and as often as you can to make up for the fact that no-one can pop over to your desk and chat with you.  Sometimes when involved in something complicated, I forget to communicate enough, so this is something you need to remind yourself to do regularly.  I also begin to find communication more work at times, especially when I am trying to get something done.  But it reassures people, which is half of the reason why you need to do it obsessively if you can.

People Can’t See How Busy You Are

Inevitably, because you work at home, you will get some people assuming you don’t do anything all day, and others assuming you have plenty of free time to help them.  People who see your output don’t worry about it, whilst people who don’t, often assume there isn’t any.  I am not sure this is unique to remote working – people across offices are very skilled at not knowing what other people do, or what their job actually is.

If you are lucky, people will be understanding and not look at your lack of presence in the office as the same as a lack of work output.  If you are unlucky, they will be the sort of manager who sees all employees as trying to get away with whatever they can, whenever they can, and will assume you are trying to skive off at any opportunity.  Mostly, you can prove yourself by working hard to the second group of people, although I’m sure some managers will never get over the general idea that they can’t watch you.

It’s Lonely

Most days, I like not being interrupted, having thinking space, and a really efficient way of getting down to work.  However, I miss the social aspect.  I can’t go for a drink after work very often.  There’s no impromptu chats at my desk, or round the water cooler.  My phone either doesn’t ring enough or rings too much.  I am barraged by emails without being able to see someone’s expression or hear the tone of their voice.

It intrudes onto personal time

Many nights I work late, and don’t even notice the time.  Mostly these days, the boundaries are enforced by my fiancée, and our young daughter who has needs more important than my own.

But it’s so easy to work at any time of day or night, that you just do, especially if you care about your job (which is pretty much an essential requirement of a home worker).  The counter side is that you must take quiet moments during the day as breaks, get out of the house as much as you can, and generally try to avoid cabin fever, or you will go insane.

The Chains

You may feel chained to the desk and to your house.  You will go out less, and therefore can become much more hermit like during the working week if you are not careful.  You will find it harder to get enough exercise, and going to the office will seem much more of a chore than it used to.  The difficulty of inertia will make simple tasks seem much more difficult, and since you sat at home the whole day, you don’t get the pleasure of returning home after work.

The Positives

The Freedom

The freedom of avoiding a commute, the expenses and stress of parking or public transport every morning are incredible.  The ability to roll out of bed and be at work is legendary.  The lack of wasted commuting time can potentially mean a much better work-life balance.  You can have a nap at lunchtime to catch up on sleep lost through your child’s demands.  You can pick up your kids from school, or be able to run errands near to home that you’d not otherwise be able to do.  Things like doctor’s appointments are easier to arrange and attend.

The Lack of Distractions

I would have left my current job a long time ago if I didn’t work at home.  The main reason was the constant distractions, questions and conversations in the office made it very hard to concentrate.  Now, when I am in the office, I expect and welcome this, and don’t try to do things which require too much focus for too long.  The impact of context switches has been well documented, and sadly this isn’t something Corelogic is particularly focussed on avoiding (except for perhaps in the development team, where it’s quite well recognised).

Working at home is completely different.  If you need to, you can sign out of IM, and close your email program and just work on something for an hour or two at a time.  This allows me to do many things I wouldn’t have had the brain power for otherwise, and be many times more productive on some days.

It Saves Desk Space and Office Costs

Companies spend an awful lot housing their workers, with the business rates for the premises on top of the lease costs.  It’s cheaper to have remote workers, especially for a startup business.  There are a lot of arguments for offices, but there are plenty of ways this can be done these days.  For micro-business, appropriate options may be places like Regis shared offices (giving the best of both worlds, a bit of buzz in a shared environment, without constant interruptions), for bigger companies, hot desking, and more low key breakout areas instead of static desks are sometimes a good option.

It’s Better for The Environment (and reduces congestion)

Most of the UK’s railway rolling stock is only used for two hours per day.  Those two hours, millions of people produce many tons of carbon moving their meat suit from one location to another.  How could it not be more environmental to work at home?  I suppose a counter argument is that I commute further when I do it.  I don’t think this has even been close over 9 years though.  I would say my carbon footprint is generally lower.

It Teaches You Self-Sufficiency and the Skills to Self-Start

Since you don’t have anyone looking over your shoulder, you need to look over your own shoulder instead.  This means things like guilt can keep you motivated, but also that you gain a good work ethic, and learn more about what keeps you motivated and productive.  No one else is responsible for this other than yourself now.  That’s a good feeling.  A routine helps, even if the routine means you wake up late, and work late.  If that’s what works for you, then great.

You Can Control Your Environment

You can control how and where you work, which is a major plus.  Not only can you work from home, but if you so choose, a beach in Hawaii.  You probably wouldn’t want to though, there would be too much screen glare.

My chosen working environment is a shed-office.  I am excited to see that this practice is taking off a lot, and becoming more popular each year.  The reason I choose this as an option was because a screaming baby isn’t conducive to concentrated work time.  I then got the best of both worlds – I wasn’t far away if my partner had a panic for any reason, but I was far enough away that I could concentrate on work without being too distracted.  I love it.  Here’s my office:

My Office
My Office

That’s It

I am still learning the best way to work – it’s an ever changing process.  What I do know is I get to spend a lot more time in the “zone”, the productive place of high concentration that today’s knowledge workers need to do their best work.  Otherwise, distractions and interruptions can just turn us into reactionary slaves, constantly emoting, whilst never really thinking about the best way to do things.  I love home working.  Maybe in my next job I may not do it, but I feel it’s made a big difference to my life and given me a lot of freedom I wouldn’t have otherwise had.

If you are reading this, and have any questions, or tips, please let me know.  It would be interesting to hear from other home workers, or people considering it.

Sysdig – A general purpose system capture and analysis tool

I’ve just been looking at a nice new tool called sysdig, which seems to be really useful for analysing and troubleshooting on production systems.  There’s a great blog post by Gianluca Borello, detailing how he set up a number of honey-pot servers with poor passwords, and then captured system activity with sysdig, showing exactly how his server was compromised, and what the hacker did at each stage.  The level of detail he was able to garner is astounding, and I can see how powerful this tool could be in the future, for any sort of troubleshooting where it’s not clear exactly what has happened/is happening on a system.

http://draios.com/fishing-for-hackers/

There’s also a nice post on using sysdig to find data from logfiles, without knowing the name of any of the log files before hand, and being able to correlate the output across multiple files, belonging to different applications.  Really cool stuff!

Active Directory to OpenLDAP Sync with LSC

I have recently had to sync accounts and groups from Activc Directory to OpenLDAP, for a requirement for a directory server in the DMZ.  A DMZ (De-millitarised zone) is an area of the network open to the internet.  It’s supposed to be separate from the rest of your LAN, so you can have services running on the internet without fear that people can break into your LAN from these.

There are other options for doing this, including a read-only domain controller (RODC), a AD LDS (Lighweight Directory Server) and so on, but they all require connectivity back from the DMZ to the LAN, which is precisely what we are trying to avoid.

If you start from the premise that no traffic at all be allowed to flow into the LAN from the DMZ, then how do you authenticate your user’s accounts?  The only real answer is a directory server in the DMZ, and to save our own users having to have multiple logins, clearly some sort of account sync would be required.

We looked at a tool called LSC (LDAP Syncronisation Connector) which is designed for syncing various directory sources to and from each other.  It’s a very capable product, and now I’ve gone through the learning process, I will have to remember if for similar functions in the future (it can’t read/write from databases, CSV files and so on too).

In order to get it set up, there are some gotchas, not least password sync, which requires another method.  But I will leave discussion of that until later.  First of all, I needed to get our users and groups into OpenLDAP from Active Directory.

To set this up required a config file, a modified version of which is below:

<?xml version=”1.0″ ?>

<lsc xmlns=”http://lsc-project.org/XSD/lsc-core-2.0.xsd&#8221; revision=”0″>

<connections>

<ldapConnection>

<name>dst-ldap</name>

<url>ldaps://myopenldapserver:636/DC=mydomain,DC=co,DC=uk</url>

<username>cn=Manager,DC=mydomain,DC=co,DC=uk</username>

<password>mypassword</password>

<authentication>SIMPLE</authentication>

<referral>IGNORE</referral>

<derefAliases>NEVER</derefAliases>

<version>VERSION_3</version>

<pageSize>-1</pageSize>

<factory>com.sun.jndi.ldap.LdapCtxFactory</factory>

<tlsActivated>false</tlsActivated>

<saslMutualAuthentication>false</saslMutualAuthentication>

</ldapConnection>

<ldapConnection>

<name>src-ad</name>

<url>ldap://myADserver:389/OU=mydomain,DC=mydomain,DC=local</url>

<username>chrisgilbert@mydomain.local</username>

<password>mypassword</password>

<authentication>SIMPLE</authentication>

<pageSize>1000</pageSize>

</ldapConnection>

</connections>

<audits/>

<tasks>

<task>

<name>MySyncTask</name>

<bean>org.lsc.beans.SimpleBean</bean>

<ldapSourceService>

<name>MySyncTask-src</name>

<connection reference=”src-ad” />

<baseDn>OU=mydomain,DC=mydomain,DC=local</baseDn>

<pivotAttributes>

<string>userPrincipalName</string>

</pivotAttributes>

<fetchedAttributes>

<string>cn</string>

<string>sn</string>

<string>givenName</string>

<string>mail</string>

<string>sAMAccountName</string>

<string>userPrincipalName</string>

</fetchedAttributes>

<getAllFilter>(&amp;(mail=*)(objectClass=user))</getAllFilter>

<getOneFilter>(&amp;(userPrincipalName={userPrincipalName})(objectClass=user))</getOneFilter>

<cleanFilter>(&amp;(userPrincipalName={userPrincipalName})(objectClass=user))</cleanFilter>

<interval>6</interval>

</ldapSourceService>

<ldapDestinationService>

<name>MySyncTask-dst</name>

<connection reference=”dst-ldap”/>

<baseDn>ou=mydomain,DC=mydomain,DC=co,DC=uk</baseDn>

<pivotAttributes>

<string>mail</string>

</pivotAttributes>

<fetchedAttributes>

<string>cn</string>

<string>sn</string>

<string>objectClass</string>

<string>givenName</string>

<string>mail</string>

<string>uid</string>

</fetchedAttributes>

<getAllFilter>(objectClass=inetOrgPerson)</getAllFilter>

<getOneFilter>(&amp;(objectClass=inetOrgPerson)(mail={mail}))</getOneFilter>

</ldapDestinationService>

<propertiesBasedSyncOptions>

<mainIdentifier>”mail=” + srcBean.getDatasetFirstValueById(“mail”)

+ “,ou=mydomain,DC=mydomain,DC=co,DC=uk”</mainIdentifier>

<defaultDelimiter>;</defaultDelimiter>

<defaultPolicy>FORCE</defaultPolicy>

<dataset>

<name>objectClass</name>

<policy>FORCE</policy>

<forceValues>

<string>”inetOrgPerson”</string>

<string>”organizationalPerson”</string>

<string>”person”</string>

<string>”top”</string>

</forceValues>

<delimiter>,</delimiter>

</dataset>

<dataset>

<name>uid</name>

<policy>FORCE</policy>

<forceValues>

<string>

srcBean.getDatasetFirstValueById(“sAMAccountName”)</string>

</forceValues>

</dataset>

<dataset>

<name>default</name>

<policy>FORCE</policy>

</dataset>

</propertiesBasedSyncOptions>

</task>

<task>

<name>GroupSyncTask</name>

<bean>org.lsc.beans.SimpleBean</bean>

<ldapSourceService>

<name>GroupSyncTask-src</name>

<connection reference=”src-ad” />

<baseDn>OU=mydomain,DC=mydomain,DC=local</baseDn>

<pivotAttributes>

<string>cn</string>

</pivotAttributes>

<fetchedAttributes>

<string>cn</string>

<string>member</string>

</fetchedAttributes>

<getAllFilter>(&amp;(objectClass=group)(member=*)(|(cn=-sec-Jira*)(cn=-sec-Confluence*)))</getAllFilter>

<getOneFilter>(&amp;(objectClass=group)(cn={cn}))</getOneFilter>

<cleanFilter>(&amp;(objectClass=group)(cn={cn}))</cleanFilter>

<interval>100</interval>

</ldapSourceService>

<ldapDestinationService>

<name>GroupSyncTask-dst</name>

<connection reference=”dst-ldap”/>

<baseDn>ou=groups,DC=mydomain,DC=co,DC=uk</baseDn>

<pivotAttributes>

<string>cn</string>

</pivotAttributes>

<fetchedAttributes>

<string>cn</string>

<string>member</string>

<string>objectClass</string>

</fetchedAttributes>

<getAllFilter>(objectClass=groupOfNames)</getAllFilter>

<getOneFilter>(&amp;(objectClass=groupOfNames)(cn={cn}))</getOneFilter>

</ldapDestinationService>

<propertiesBasedSyncOptions>

<mainIdentifier>”cn=” + srcBean.getDatasetValuesById(“cn”)+

“,ou=groups,DC=mydomain,DC=co,DC=uk”</mainIdentifier>

<defaultDelimiter>;</defaultDelimiter>

<defaultPolicy>FORCE</defaultPolicy>

<dataset>

<name>objectClass</name>

<policy>FORCE</policy>

<forceValues>

<string>”groupOfNames”</string>

<string>”top”</string>

</forceValues>

<delimiter>$</delimiter>

</dataset>

<dataset>

<name>default</name>

<policy>FORCE</policy>

</dataset>

</propertiesBasedSyncOptions>

</task>

</tasks>

</lsc>

I had a few specific requirements – I wanted to sync only some of the groups, as we have hundreds, so I filtered out the security groups related to Confluence and Jira (our customer facing systems).  I also wanted to change the distinguished names (DNs) of the objects as I moved them, from mydomain.local to mydomain.co.uk.  In LDAP speak that is from dc=mydomain,dc=local to dc=mydomain,dc=co,dc=uk.

I won’t go into explaining the config too much, you need to know a bit about LDAP to understand it.  I am “lucky” enough to have delved into LDAP search criteria before, so it wasn’t too challenging.  However, for everyone else, a good bit of googling and trial and error will hopefully produce something useful.

This works for me – use it and modify it if you like.  I wasn’t able to find a good example for the 2.0.x series of LSC (with the new XML config files) so hopefully this will help some people.  I put it together from some other posts I found on the mailing lists, along with suggestions on how they should be fixed, to come up with something that worked.

The second phase of this is making password sync also work.  LSC cannot do this by itself, but does come with a perl script which can be hooked in via a third party program called hkpassword, and send the password changes to OpenLDAP as they occur.  I got this working fine, with a few modifications.  I’ll post this at a later date when I’ve had a chance to write up and document the process.

HTTP 2.0 Is Coming

It’s been almost 15 years since the last standard of HTTP was ratified.

HTTP is the protocol which transports web pages across the internet.  It’s a brilliantly thought out and quite simple protocol.  It’s beginning to show it’s age though, especially with the rise of many requests across multiple web servers. AJAX and newer technologies like WebSockets have worked around it’s limitations.  It’s inefficient at connections, is designed to be stateless, so doesn’t maintain data in it’s headers between requests, and it’s not great at doing lots of requests in parallel.  So, some clever bods at Google and elsewhere have come up with a new standard, which is due to join browsers next year.

There’s some technologies like Google’s SPDY which have helped in the meantime, but this protocol change is really what the web needs for efficiency improvement, especially on mobile devices.

When this makes it to browsers, (and web servers) we’ll see some real speed ups on many web sites.