The Hosting of exubero.com has Changed

This is just an administrative announcement: I have moved exubero.com to be hosted by wordpress.com. It was previously hosted on a EC2 server and database. At the time I moved exubero.com to AWS, I felt it was an interesting project to learn the AWS tools and infrastructure. In that sense, it was a good way to learn. However, years later I’m running months behind on keeping everything updated with security patches and paying too much for AWS services. Given that 90% of my website was already running under WordPress, I decided to bite the bullet and let WordPress.com manage my site for me, allowing me to kill my EC2 server and save some money.

The transfer wasn’t quite painless: some of the URLs have changed. All the content is still available, but if you have any bookmarks, they may need to be updated, and the RSS feed location has changed too, so you may need to update your subscription.

Software Development is an Engineering Discipline

There is a view that software development is a craft; that “Software Craftsmanship” is a better metaphor for software development than is engineering or science. I’m going to argue against this view: software development is an engineering discipline, and relies strongly on scientific principles. Firstly, what is an engineering? Here’s one definition of the engineering method:

The Engineering Method is the use of heuristics to cause the best change in a poorly understood or uncertain situation within the available resources. – Billy Vaughn Koen

What heuristics? In general, they look something like this:

Engineering method

(from the article “Comparing the Engineering Design Process and the Scientific Method“)

The first thing to notice is the feedback loops here. The steps brainstorm ⇒ develop ⇒ test corresponds to the TDD cycle in software development:

TDD cycle(from “Growing Object-Oriented Software, Guided by Tests“)

The difference between the engineering design process with physical products and TDD in software is mainly the speed of the feedback loop. A skilled developer can often go through the red-green-refactor loop in less than 5 minutes, as there is no physical materials to manipulate. When you then add in the steps specify requirements ⇒ … ⇒ meets requirements around the TDD cycle, you get the ATDD cycle: ATDD Cycle

Depending upon the size of feature that the the acceptance test describes, this could take anywhere from an hour to a week, though high-performing teams learn that quicker feedback is critical to quality.

Zooming out, there is a larger feedback loop surrounding the ATDD loop: the lead time between new software capabilities being prioritised and the capability being used by an end user. The DevOps movement is built on the focus on the software and organisational structures required to get software into the hands of end users as quickly as possible.

The size of the feedback loop is probably the most critical part of successful engineering. Consider, for example, the first human powered flight across the English Channel. The Kremer Prize was established in 1959 to give some motivation to show that human powered flight was both possible, as able to go long distances.

The challenge was picked up by many peoples and teams, but there was very little progress until the problem was examined by Paul MacCready, who noted that the engineers were trying to solve the wrong problem.

MacCready’s insight was that everyone working on solving human-powered flight would spend upwards of a year building an airplane on conjecture and theory without the grounding of empirical tests. Triumphantly, they’d complete their plane and wheel it out for a test flight. Minutes later, a years worth of work would smash into the ground.

The problem was the process itself… He came up with a new problem that he set out to solve: how can you build a plane that could be rebuilt in hours not months. And he did. He built a plane with Mylar, aluminium tubing, and wire.

After two and a half years, MacCready team successfully flew the Gossamer Albatross II across the English Channel in just under 3 hours.

Gossamer Albatross

(Gossamer Albatross II in flight, from Wikimedia Commons, public domain)

Compare this feedback loop to a more traditional engineering discipline based upon something less malleable than software: traffic engineering. This is a branch of civil engineering that uses engineering techniques to achieve the safe and efficient movement of people and goods on roadways. In this case the engineering medium is road signs, lane markings, asphalt, road size and shape. In some cases, the traffic engineer will look to move roads through entire buildings. The behaviour of traffic can be very complex, with surprising effects from interventions (think of adding a lane to a busy road, causing induced demand, causing worse traffic delays). In all these examples, the feedback loop for understanding if the intervention was successful can range anywhere from days to years. In order to reduce the risk of untested changes, traffic engineers must use models of the road, and run the simulation in software.

In summary: software development that makes use of feedback loops to help understand and direct the problem solution is most definitely engineering.

One question is left to ask: where does Software Craftsmanship fit in the context of software engineering? My take is that Craftsmanship is the focus on the skills that are known to be effective, a body of knowledge on the techniques that are effective. Engineering without skills won’t get you very far. Skills without engineering means that you can build the wrong thing very well. Both are required.

(This post was prompted by a discussion between Dave Farley and myself when we were both at the book launch of Infrastructure as Code by Kief Morris. A subsequent discussion with Andy Parker helped refine the topic)

Elephant Carpaccio

Carpaccio
Carpaccio – thinly sliced meat

I co-facilitated Alistair Cockburn’s elephant carpaccio exercise yesterday. This is an exercise to teach developers and business analysts:

  • How to slice large applications into 1-day to 1-week requests, from the business perspective
  • How to slice application requests into 15-30 minute work slices, from both the business and programming perspective

Previously, there had been some discussion at work about problems people were having with large stories. Coincidentally, @jtf mentioned the facilitation guide for the exercise around the same time. Looking through the guide, it covered a lot of the issues we had been talking about, and seemed like a better scope for true learning, as opposed to just pointing people to the story splitting resource. Given that the facilitation guide was so detailed, it didn’t seem too much of a stretch to have a go at running the course.

Main problem: after deciding to actually do it, I had about 2 days to prepare, and a huge amount of loose ends to tie up before I escaped work on paternity leave. Luckily, I had a very experienced business analyst to help me run the exercise: Lisa Ellis. Lisa was able to smoothly cover my lapses.

What worked:Team

  • the exercise is a very good way of teaching the reasoning and benefits of small yet complete story slices. The approach is focused on the coaching aspects of asking participants go think through the issue for themselves
  • I think it was very fun for everyone. 40 minutes to go through 10 – 20 stories? Go!
  • The facilitation guide was incredibly detailed, which made it very easy to help participants reach their learning goals.

What didn’t work:

  • some participants were not ready with a development environment – note to self: ensure everyone is prepared before turning up.
  • I wasn’t fully prepared, and didn’t have enough answers at my fingertips, and wasn’t always quite sure about what the next steps of the exercise were. I had to refer to the notes quite a lot. Luckily Lisa was there to help.
  • We ran out of time in the end, and had to cut the coding time short by 1 iteration. Interestingly enough, this actually helped highlight the benefits of small slices and prioritising by value

This was my first time running this exercise. Even with a scant 2 days notice, we had 20 people sign up. I think there’s a lot of demand yet in the company. We’ll definitely be running this exercise again (with a bit more finesse)


Carpaccio image by Manoel Petry, and used under the terms of the Attribution 2.0 Generic licence.

Develop a Team Vision

This post describes the first day of my Bringing Distributed Teams Together series (read that first to see the larger context of this post).

TL;DR

Using combinations of clean language, active listening, conflict resolution models, and directed questions from the facilitators, all the team members were able to gain empathy and understanding for each other and learn some very useful communication techniques. We all created individual “Rough Guide to Me”, and to agreed on a shared vision of how we work.

Planning

I asked Paul Field to advise and facilitate the first day of the distributed teams onsite. My brief to Paul was to use the day to help all the team members to work more effectively together, and be able to move towards forming a high performing team.

Before planning started, Paul asked for some more concrete and measurable outcomes. With some probing from Paul, I came up with the following:

  • more positive interaction from all participants during planning and technical meetings (I was concerned that communications were too one-sided)
  • more informal and ad-hoc communication outside of formal meetings (as would be expected when people are comfortable with other and able to emphasise with them)
  • more distributed decision making (I’m betting that better knowledge of everyone’s interests, drivers and concerns would help team members to propose solutions that would work with everyone)

Paul strongly recommended that the team work towards developing a shared vision of how we all work together, which struck a chord with me, and we added the following as a desired outcome:

  • A shared team vision for how we work

After a number of meetings, Paul and I had settled on a day structured around learning a set of soft skills that would help the team work effectively, and using those skills to learn more about each other and to help develop our team vision. Based upon the plan for the day, we also brought in a second facilitator: Marian Way, an expert in clean language workshops.

The success of the day depends upon both internal team interactions as well as how the teams interact with the wider organisation. As such, I invited other people closely associated with the teams, so that we ended up with:

  • 2 development leads (that includes myself)
  • 7 developers
  • 1 enterprise architect / subject matter expert
  • 2 business analysts
  • 2 product owners
  • 1 production services representative (that it, IT operations)

There were 15 people in total, plus the 2 facilitators.

Workshop

The first half of the day was devoted to learning some aspects of clean language that would help us later in the day. I was fascinated to see how Paul and Marian introduced the subject: by example.

We started with the “clean set up”. This involves asking everyone 3 questions:

  • For today to go just ad you’d like, it will be like what?
  • And for today to go like that, you’d like to be like what?
  • What support or resources do you need?

These questions are designed to help prepare all the participants to understand their goals, and to put them in the frame of mind to enable them to reach their goals. Once you learn the technique, it can be done quite quickly as preparation for any meeting.

Next, Paul instructed us all to “see an elephant”. A nonsensical request! Paul and Marian used a combination of active listening and clean questions to elicit statements from us. We learned just how each participant had very different thoughts about that particular question had. It emphasised just how different everyone was in interpreting a question (we discussed the ladder of inference later in the day). Through repeated demonstrations of the technique, and a challenge to all of us to gain insight into the techniques they were using, and to allow us to try it ourselves.

After this, we started work on one of the main products from the workshop: every individual created a “Rough Guide to Me”. We didn’t complete it in one go, but filled in various sections at points during the rest of the day. The sections were:

  • When I’m working at my best, I’m…
  • My ideal team is like…
  • My patterns…
  • How I become #!#!#!
  • What’s important to me…

These rough guides can be shared with everyone, allowing a really transparent view into each persons preferences.

The afternoon centred about getting a shared vision of our ideal team. We started with our own vision of an ideal team from the “Rough Guide to Me”. We were each asked to finish of the sentence “My ideal team is like…”. The very structure of the sentence invites us all to create a metaphor for our individual vision. We were also asked to draw a picture to represent our vision (this was a cruel and clever trick, as we later found out). The metaphor I came up with was this:

My ideal team is like the staff in a busy kitchen in a high end restaurant

I’ve been watching Masterchef lately, so I liked the kitchen metaphor. I thought that there were a number of parallels between a restaurant kitchen and software development:

  • Good chefs are skilled craftsmen (like good software developers)
  • Different chefs have different skills, all the kitchen workers must cooperate to produce the end product
  • Value is delivered to the end user in small increments (of delicious and beautiful plates of food)
  • Feedback cycles are short and obvious (you’ll hear quickly if the diner didn’t like their food)

The next part was very interesting: we formed pairs, and were asked to come up with a joint team vision. This was an act of negotiation. We were reminded to use the clean questions as taught to us in the morning. Most pairs were able to synthesise their visions into something that both could subscribe to, but not everyone. One pair in particular didn’t negotiate effectively. Rather, one member didn’t like the suggestions from the other, and refused to abandon any aspect of their individual vision. This was something Paul and Marion were counting on! (by getting us to draw a picture, they made us become more attached to the vision that might otherwise have been).

Conflict! Paul and Marion were able to use this as an exercise to teach us about the Karpman drama triangle:

drama-triangle

With an awareness of our roles in the drama / conflict, we are better able to understand a way out. Using a conflict resolution model, the members were able to reach a mutually acceptable consensus.

With 8 pairs of people now having a joined up vision, we then repeated the exercise of forming a joint vision. Each pair joined up with their neighbours to negotiate on a vision that all 4 people could accept. Then we did it again: the 4 groups of 4 joined formed 2 groups of 8, and then once again we finally combined the vision for each group of 8 into a fully synthesised cross-functional team vision.

Our final vision is both nonsensical and awesome: it involves a rocketship bank with bee chefs. The value comes from how the team arrived at a shared metaphor, the understanding of the negotiation process to get us to the metaphor, and the values that go into defending the aspects of the metaphor that are important to each of us.

Post Workshop

Everyone said that they enjoyed the day. It was fun working together on the team vision. A number of people said that they learned some skills that would radically change how they communicated with others. Others looked very thoughtful. It’s too early to say if the desired outcomes will all be achieved. I’ll share any insights become evident to me later.

Bringing Distributed Teams Together

It was a very busy week for me – representatives from all the teams that I work with were in London for three days of group workshops. I was running around trying to make sure everything happened when it should, which was quite exhausting. In the end, pretty much all the participants gave a big thumbs up to the experience, which was a big relief.

To give some background, I work in an investment bank with teams in London, Kiev and Krakow. The team in Krakow is also very new to the bank, while the London and Kiev teams are quite seasoned. We have a number of applications we maintain together, and so effective teamwork is an important concern to us. The distributed nature of the teams makes communication and empathy much more difficult.

I had arranged the onsite event with three goals in mind:

  • help all the team members to gain empathy and understanding for each other, and to allow us to gain a shared vision of how we work
  • allow the teams to work on a shared technical vision and approach
  • allow the teams members to practice some some technical skills together

Luckily for me, I was able to bring in an experienced consultant to help with the planning: Paul Field from Clarity of Purpose. Paul was hugely influential in helping me define the outcomes from the onsite, as well as acting as facilitator for the first day.

With the three goals defined, and three days together, it was a short step to divide the onsite into three parts:

To avoid making this post too big, and to keep each day focused, I’ve put the details of each day into a separate post. Follow the links above for the story of each day.

Agile Applications with Java

I’ve been re-reading a couple of resources on modern application design:

Those links contain a huge amount of distilled knowledge and advice on how we should build applications today. If you’re working on applications that don’t follow the advice in those articles, take note!

As an exercise in preparing for a Hackathon at work last month, I wanted something to allow me to quickly get up and running with a Twelve-Factor app without too much fuss. With this goal in mind, I ended up with something that I’m quite happy with:

https://github.com/tumbarumba/dropwizard-marionette.

It’s interesting to review how I ended up with that starting point. I went through a number of decisions:

Problem domain?

Before the Hackathon there were a number of ideas proposed for projects that could demonstrate skill and business value in 24 hours. I already had narrowed in on one: automation of some administration that affected everyone in the company. It seemed like a focused problem with a clear business benefit.

Architecture? 

I thought that an application server to coordinate the administrative tasks, with a web front-end to allow user interaction would be a reasonable way to solve the problem (and give us something to demonstrate at the end of the Hackathon).

Server language?

A Hackathon is not a great place to learn a new language, you need to be able to hit the ground running, and so a familiar language is required. For me, this would me a choice of:

  • Ruby
  • Java (preferably Java 8)
  • Scala

However, the most of my team mates would not be that familiar with anything but Java, so that was an easy choice in this situation.

Server library/framework?

At the time of the Hackathon, I was aware of these frameworks:

The reactive optimisations in the Play Framework were tempting, but I thought that Dropwizard gives a much better coverage of the features of the Twelve-Factor app, plus I was much more familiar with the libraries used by Dropwizard.

Front-end language?

Javascript or Coffeescript? My last experiment with Coffeescript was a couple of years ago. It was OK, but I didn’t think it was a massive win. Given I would need to be working with other people who are probably less familiar with Coffeescript, Javascript was the only real choice.

Front-end framework?

There are lots out there, but I have had experience with Marionette (effectively an improved Backbone).

All the above questions led pretty directly to dropwizard-marionette.

Result: we won! Twice! My team won both the regional UK Hackathon, and the global Hackathon in a head-to-head with all the other regional winners. I’d like to say that the technical preparation was the real winner, but I suspect that the main decider was based more upon the clear business benefit of our solution. I’m happy either way: my team now has some pretty big trophies we can use to taunt our co-workers. Win-win!

Devops 101

I recently saw a reference to Devops 101 on Twitter. It looked interesting, so I bookmarked it, and forgot about it for a few weeks.

Devops 101 - Part 4

Last night I finally got around to trying the exercises – it was a bit of an eye-opener. There’s a huge amount of functionality on AWS that I’m not really up-to-date with (and I say this as a person who has been running an AWS EC2 instance for many years now).

The “101” part of the name implies that it’s a starting place for graduate developers. I would clarify to say that it’s very useful for anyone who isn’t working on infrastructure-as-code day to day. If you’re developer who is not yet embedded in a team working devopsly, give it a go.

I had never had a chance to place with the AWS command line, and leveraging that with cloudformation and ansible was an real pleasure to use.

I previously worked at a company where we had built our own stack builder tool. I never got to grips with it at the time, but I now want to go over it again.