The Loneliness of the Self-Taught Programmer

I have a friend who is very well-read in all kinds of fields. He has a PhD in Organic Chemistry from Cambridge University but he is also widely read in Russian and French literature. He bundled through the Penguin Classics well before we left sixth form. He likes all kinds of pop music and he also likes opera. What's more he achieved all this culture and erudition from a relatively humble background - most of his reading was done in the ice-cold back bedroom of a council house in West Yorkshire where he grew up.

Now aren't we all lead to believe that education and culture are a good thing? Shouldn't having read all these books have made my friend more confident and well-received when he did finally get into polite society? Well, it should have done, but there was slight problem. Pronunciation. He'd read Proust, he loved him, but he didn't know that his name was pronounce to rhyme with "roost" and not "roust". He loved Puccini but he didn't know that the word "aria" isn't supposed to rhyme with "Mariah".

And so when he did get into polite society, people laughed at him. People who had never read Proust still felt it was all right to laugh at his pronunciation. He learned the correct pronunciations really quickly of course, but he never forgot the sharp humiliation of those condescending smiles.

I was at lunch yesterday with 3 coders, one of whom I'd never met before. We talked about Agile stuff for about 10 minutes and he didn't say anything. Finally he said that working on a project that used Agile methods - particularly in his case, test driven development - had made him completely miserable. After he'd left the project he'd lost all faith in his abilities to code. It took him several months to regain any confidence.

Learning to program is a back-bedroom, late-night activity for most people. Most people who do it, are autodidacts - they teach themselves. There are university courses that claim to teach it, but most of them miss the mark. Either the course is too prosaic and equates teaching programming to teaching the syntax of a certain language or the course is too theoretical and imagines programming to somehow happen as a natural bi-product of understanding certain branches of mathematics.

Even if you're lucky enough to find a course that does teach you programming instead of computer science or discreet mathematics, in order to actually write anything that works there's a ton of arcane detail that you need to learn and there aren't many ways to do that other than dragging through tutorials, manuals, FAQ's and forums and other people's code by yourself. By the time you've done that it's not likely that your knowledge is going to be rounded, flexible and generally applicable. It's more likely that your knowledge is going to be, at least in some areas, brittle, specific and sketchy. So what are the chances that you're going to want someone to look at your code?

And maybe this is what is behind what seem like stubborn denials, refusals and condemnations of pair programming. When you suggest to someone that you write code with them and their response is "I taught myself to code, why should I be forced to do all the work for some rookie who can't be bothered to stay up nights himself," or even "I don't have to drink a bucket of vomit to know it's a bad idea." You might guess from the emotive reaction that there's fear involved. Perhaps the fear of the autodidact.

Making creative and business sit together with less conflict

One of the big questions raised time and time again by those involved in supporting and developing creative businesses is why it is that creative people are so good (and prolific) at starting businesses but not so good at sustaining and growing them?

Many more businesses are started by creative practitioners than those from a business background. Creative businesses are responsible for more new job creation than any other area of economic activity in the UK. London is a world powerhouse of creative business and yet despite this the failure rate of creative businesses is very high and of those that make it past the 3 year mark, many never grow beyond a dozen or so employees.

While there are a multitude of reasons given for this, such as the unwillingness of those that run such businesses to break through the 'lifestyle' barrier needed to grow or sustain a business to the difficulty in accessing investment, there is an important factor that is common to most, if not all, such businesses. This is the conflict between creative process and business process. It is not an untruth to reflect that these two areas of discipline are simply very different and require different attitudes, skills and knowledge but to end the consideration here is also neglectful.

One way to consider the root of the creative and business conflict is to look at the way that the processes that traditionally underpin creative and business activity are shaped. Business planning and execution is understood as linear. To attract investment or secure borrowing in order to build a business so that it can be sold or can realise the long term exploitation of IP is understood to require 3 year projections that provide a month by month picture and use language that suggests risk reduction achieved through careful long term planning. Here change is to be managed rather than embraced.

Creative people are at their strongest and happiest when thinking and working cyclically, embracing risk and dealing with constant change. This is true of those engaged in the creative application of science and art. Such people make hypothesis, explore and test such hypothesis, review the results of this activity and then adjust their hypothesis accordingly. It also true that business planning should be constantly reviewed and updated in light of progress made and lessons learned. Therefore cyclical activity is also common to the ongoing delivery of such plans even if it does not make the initial research and preparation of such plans any more palatable to the creative person. It does however give us a very important pointer to finding new ways of addressing this challenge.

Clearly we need to continue developing new processes and practices for engaging business heads with creative practitioners in ways that allow them to develop long term sustainable relationships. One such process I will refer to as Agile Business Planning. By using Agile process as the basis for business planning and development delivery we allow the creative practitioner to use processes that are familiar as they are cyclical, embrace change and risk continually and yet deliver continual and visible outcome. Such process is also SMART (specific, measurable, achievable, realistic and time managed) and can dovetail with the long term visioning and projection orientated nature of established business planning practice. It is simply delivered week by week, month by month, using a set of tools that are owned and understood equally well by the business head and the creative head and therefore reduce conflict allowing the creative business to grow and become sustained.

There has to be a better way of getting software developed

Why is it that many people have had painful experiences of procuring software? You would think this should be a wonderful opportunity to engage in creativity and innovation, to learn something new and see some of your ideas become realised in ways you could not quite imagine. But the reality turns out to be completely frustrating. You don't get what you want or what you were promised, you get it late, over budget and it never works properly. Then you find out that what you just paid said web tech company many thousands of pounds for is available for free on the internet anyway!

The fact is that paying for software development at a time when technology is moving so fast will always be a risky business. Ironically, despite the painful stories and battle scars of those that have had bad experiences of software development procurement, much of the problem can be put down to the processes that are used. If you are paying for development then the perception is that you want to know what it is you are getting before you sign a contract. Therefore you work with your supplier to develop and agree a list of functional requirements. These are costed and their development is plotted on a some kind of time-line with milestones, deliverables and so on. Then you press the go button and in theory everything works out as planned. Or, as is more likely the case you end up cutting requirements like crazy to make deadlines, getting half-baked functions that kind of work but not really as you had hopped and the rest is history.

So if this Agile malarkey is so great then what could it do to this process?

Agile provides the software purchaser with a suite of methods that can help them get what they want, when they want it and on budget. However, for this to work it requires a potentially seismic step outside of the comfort zone of a tight requirement list and schedule into a space where creative thinking and constant change are embraced jointly by the supplier and the supplied in order that everyone gets what they want. The supplied gets great software and the supplier gets to make some money and you are still on talking terms at the end of the project.

This works by the supplied selecting who they work with on the basis of who they want to work with because they understand the problems faced in the environment the software is required for, have an attitude and approach that gives you confidence that they can deliver really good code and can talk to you in a language you understand. 'Ahhhh', I hear you say, 'but we do that and still things end up pear shaped'. Yes, but despite having these feelings towards said developers, you also probably made them sign up to a fixed set of requirements and specifications and a tight and rigid development schedule so you were asking them to work with a big fixed design up front and a deliverables time frame that required first this, then that, then we pay, then you do this and then and then and then. And then it didn't work out like that.

If you were working in an Agile way you would be only taking one step at a time and according to a continually changing requirements list. You would be able to add, change and re-prioritise your requirements as you go so that you were always addressing the real needs of the project rather than a set of needs that made sense before you had any working code to play with. You would also be in a position to learn more about what your needs are as prioritised working functions begin to emerge rather than having to rely on very early sketchy ideas of what you imagine you will need before you really know.

Then their are stories. Agile uses narrative and dialogue between client and supplier, stakeholder and user to ensure that people really get useful stuff. Stories are much more effective than pretty pictures and technical descriptions. We can all understand them and that means that their use greatly improves communication around the project.

I could go on here. But I wont. I have been involved in enough projects that haven't gone smoothly to know that Agile makes a real difference. Given that much software development is paid for by the public purse surely it is time that those responsible for this expenditure start to look for more effective ways of working. Anyone who has been close to software projects in the public and private sector will recognise some of what I have written from their own experience so surely it is time that we innovate with process instead of using broken old fashioned approaches to software procurement that often leave everyone feeling dissatisfied?