Tooling-up for agile state-transition

This post started out in life as an answer to a question about ‘backlog tooling’ on the LinkedIn ‘Lean & Agile’ group. Someone had given the culturally acceptable answer that the best solution is simple cards or post-it notes on a board or wall. I normally just let that pass because I don’t have a better solution to offer but this time, THIS happened:

I’m about to be intentionally provocative. We know that we are engaged in transforming a multidimensional network of business functions from one poorly understood and transient state to another, currently ill-defined, future state that we hope will emerge from the mist as we travel in it’s general direction. In organisations of any size, this change process is likely to run in parallel with other change programmes, some of them probably deliberately kept secret by people whose pay grade exceeds their ability to make rational judgements about the basis of who “needs to know”.

Amongst this chaos, the chosen tool of ‘the Agile Community’ is a single, 2-dimensional view of a ‘list of lists’, sometimes known as ‘a tree’ or ‘a star’, all of which are topologically equivalent representations of items’ states in the backlog of each Product development. Our best software tools are little more than a model of cards on a board.

Why do we expect the complex, dynamic agile change process to map any more adequately onto a tree of cards than it does onto a hierarchical management structure? Have we learned nothing from our mistakes of modelling within the limitations of the filing cabinet and the typewriter? Perhaps agilists don’t value tools because our tools aren’t fit for purpose.

If all you have is lists representing vague descriptions of changes between two mental models you hope your whole team all share, perhaps the limited nature of the backlog tool isn’t your biggest problem. The backlog items reference changes to an implicit model of roles and the objects in the business domain of your product. My advice is to make it explicit.

Advertisement

Agile as something you do

I have spent the last 2 evenings in Birmingham listening to talks by @diaryofscrum at @ScrumUK and @stevejpitchford at @bcsbrum about management and ‘Agile’ software development, which brought some of my own concerns into sharper focus, particularly about the Scrum framework. In many discussions with practitioners over the last couple of years, I’ve heard the following phrases:

“Agile is an adjective not a verb”
“Agile isn’t something you do, it’s something you are”
“Agile is a philosophy not a method”
“Agile isn’t a process”

Someone who ISN’T agile has to start somewhere. They typically need to DO something, to write software. Would we claim,”Scientific” is an adjective not a method? We wouldn’t, because it is both. The scientific method is a function which delivers what we call “scientific knowledge” as its value. If it didn’t, it would be pointless.

Managers are generally trying to get things done. Each team must agree its own Agile Operating Model (thanks to BCS’s ‘Agile Foundations’ book for that useful phrase.) What came out of the last couple of evenings was pragmatic application of philosophy. Many organisations take Scrum as a starting point, without realizing that “framework” is to be taken very literally. Scrum doesn’t paint the complete picture. It is (part of) a process to organize work. It says almost nothing about how to do that work. It is an alternative to writing a project plan “up-front”, when you know least.

An Agile Operating Model is a process which delivers a value, so it is a function. My scientific hypothesis is that it delivers valuable business function change, sometimes in the form of software. It is itself a business function. Agility has business functions as first class citizens. It doesn’t meet general expectations of a process because it can recursively self-modify. That doesn’t mean it isn’t one. As the kids say, “get you an agile function that can do both”.

Practising your Process

My very sincere thanks to Simon Powers for posting the ‘onion diagram’ in his ‘What is Agile?’ post on LinkedIn and for answering my question. The post is also available on his own blog http://www.adventureswithagile.com/2016/08/10/what-is-agile/
It shows ‘tools & processes’ separate from ‘practices’. I’ve been thinking for a long time about whether there is any real difference between process (what) and procedure (how) or if they are simply different levels of detail. I think I’ve just been convinced that the equation I’ve been searching for is:

• process + practises = procedure

Simon actually listed in his answer to me, ‘roles, interactions and artifacts’ as the difference between the set of Agile practices and the set of Agile processes, so I’ve corrupted his definition for my own purposes but I haven’t broken his diagram so I hope he’ll forgive me. (Or maybe I don’t understand whether the layers of an onion diagram are inclusive or exclusive.)

I think making the process one of the practises would make the function recursive and this is supposed to be one of my Lisp rest-days. If my process diagram shows roles or artifacts then I’m sure I’ve moved into the realm of specifying practice. Interactions may be input-output that is part of the definition of the process, so it is probably necessary to split them down more, into message type & format.

Software Development Science

I’ve tried to write this post before https://andywootton.wordpress.com/2015/07/07/software-life-cycle-part-2-from-craftsmanship-to-computational-science/
but I just answered a question about ‘software engineering’ on LinkedIn and I’m pleased with the brevity:

Agile software development is the application of the scientific method to understanding the requirements for a software product. Each iteration is an experiment to confirm a hypothesis of what the customer should really have wanted at the beginning, if they’d known what they know now.

To Be Derived

I thought I’d said something about this but I can’t find it, so sorry if you’ve heard this before:

Professor Brian Cox turned to Dr. Hannah Fry and said, “You have 30 seconds to define ‘calculus'”.
She replied, “It’s the study of change.”

What we call “The IT Industry” deals almost exclusively with change. As a business analyst, I’ve worked in business process re-engineering which is the process of changing the processes which cause those changes. I’ve worked in agile software development teams that try to change software while the requirements for the target they’re travelling towards are changing, so the changes have to change.

I’ve met calculus in maths and physics but never in computer science. Why on earth aren’t computer scientists up to their necks in calculus?

Is this Important or Urgent?

This post refers to a technique often used in Agile software development, including within the Scrum framework. It is not an introductory text so not recommended for non-agilists.

‘user-stories’ are classified as: Must (do), Should (do), Could (do) and Won’t (do), known as: MoSCoW. User-stories are then usually prioritised by an integer representing value, which represents a calculation of return on investment, or benefit:cost.

Paul Oldfield, Chief Referee at LinkedIn ‘Agile & Lean Software Development’ group said:

I find a bit of a problem with MoSCoW – distinguishing between “Must have eventually” and “Must have in release 1”. Get beyond release 1 and a high value “should have” can be prioritized in front of a low value “must have”.

And then, a lot of the “must have in release 1” turn out not to be, if we look closely.

“If you want all these in release 1 you get nothing for 6 months.
Or you can get these in 2 weeks, those in 4 weeks… would you like that?”

I gave (a slightly worse version of) this reply:

I think MoSCoW is about ‘importance’ not ‘urgency’.
Urgency comes into the prioritisation choices when the benefit of the story is time sensitive.

Delivering benefit early starts summing value for longer, so total value delivered in the life-time of the product or service will be higher but now we’re talking about delivering a different absolute ‘spot value’. Putting it another way, value can be a function of time.
e.g. “If this isn’t ready in 2 weeks then we’ll be fined by the regulator” or
“We need this before the Summer Sale starts. If you miss that, it’s useless until Christmas.”

I didn’t know this before today. I’m sharing the idea in case it helps someone else or they can improve it and give me a copy. It’s how things worked before science had to make a profit.

Lean or Agile? Pick any two.

I still see many people writing about adopting Lean and/or Agile software development. I can remember how difficult it was for my team to work out what ‘Agile’ was and I think it has got harder since, as growing popularity has drawn charlatans into the area. I see two main types of useful articles.

1. What (theory) : “It’s a philosophy” articles which usually point first towards the different values of agile and lean practitioners. But you can’t “do” a philosophy, so we get:
2. What (practice) : Methodology – the study of methods that embody the philosophies. Many will say that Lean & Agile are not processes but I disagree; I think they are ‘software development process’ change processes.

I’d like to try something different: WHY?

The old ways of planning engineering projects, used for building a tower block, didn’t work for software. We don’t know enough, with sufficient certainty at the beginning of development to design top-down and are rarely sufficiently constrained by physics to be forced to build bottom-up.

Unusually for computing, the words ‘lean’ and ‘agile’ have useful meanings.

Lean is about ‘travelling light’, by avoiding waste in your software development process. It uses observation and incremental changes of your current process, while you incrementally deliver business value via working software.

Agile is about doing only valuable work, being nimble and able to change direction, in response to changed requirements or better understanding. It recognises that there are very few completely stable business processes, so software developers need to identify changes that will have impact on the software under development and apply effort in a new direction.

I recommend that you consider both approaches, as they are complementary. Neither removes the need for appropriate engineering practices. We’re only throwing out hard-engineering stuff we packed but didn’t prove useful on a software journey. We throw out what we don’t need, to prevent the weight of unecessary baggage holding us back.

Agility at Human Scale

[ I wrote the post below on LinkedIn, as a reaction to something I read which annoyed me. One person whose opinion I respect liked it, so I thought I’d share it here. I don’t understand where old LinkedIn posts go and you never know when you might need a positive thought. ]

I’m always surprised how many people believe software development is about turning what they think someone wants into code, when the really hard part is understanding what they will turn out to have really needed, at the end.

OK, coding is hard too; so don’t waste effort writing the wrong thing. The answer is not a better specification, a more demanding project manager or more people. The answer is to embrace your human limitations and think small. Be lean and agile. Work as a team. Go back for the wounded. Be a human.

Remoting-in to the Virtual Box

As no-one spoke up yesterday, I’ll assume everyone has accepted the notion that all software runs in a virtual universe, free from the laws of physics. That’s why we don’t need to run software developments like an engineering project. They are not subject to all of the constraints that make engineering hard. We can ‘build the roof first’ and worry about ‘how strong the foundations need to be’ later, when we understand more about the model we have built of our incomplete idea. Like this blog post, Agile products are almost free-floating in a world of our invention, until they need to communicate with people. We are stuck here, interacting with this parallel universe, using our big, heavy ape arms and clumsy interfaces. We drag behind them like tired children.

Did you see the original ‘Tron’ film? Do you remember how the programmers’ personalities were represented by the programs? That was a true story. Programs can be gentle, kind, beautiful but shallow, or bullying ego-maniacs, just like their creators. They can appear to have a certain character while actually being something else entirely. Software reflects aspects of the personalities of it’s creators, as expressed within their self-imposed cultural boundaries.

I think most people reading this will accept that evolution theory is most likely true and that genes carry the necessary code to make new life. I want to propose my own hypothesis. I don’t know if I’ve re-invented an old idea so please tell me if you’ve heard of it before. I think that every form of life has its own culture and that DNA and culture have evolved together in a symbiotic relationships, like a third interlocked spiral. The main difference is the speed at which the invisible cultural strand can change. We may still have the emotional responses of cave dwellers due to our DNA but we can change our political and religious opinions in a day. Every system that survives, protects itself, so we have evolved early-adopters, fashion-victims, people who want to fit in and reactionaries, to quality-check dangerous ideas. As a species we resists change, because change has proved to be really dangerous. At the same time, we constantly strive to try something new because that has been proved to give evolutionary advantage, if you don’t die trying. The variation in the attitude of humans is one of our evolutionary advantages. The two are kept in balance by death of the over adventurous and economic failure of the over-cautious.

“Where’s he going with this?”, you may ask. Well: just as Richard Dawkins put forward the idea that we are carriers of our selfish genes, I’m in turn proposing the idea that selfish us and our selfish genes are carriers of our cultures and that if we can project human culture into software, we can free it, and ourselves from the rules of physics and the constraints of limited resources and thereby, finally, from the drive to be selfish.

Applying the Science Process to Process Change (see Recursion)

Now that we’ve established that very few people who work in “IT” have anything to do with technology, except as a tool, and that “computer science” isn’t about computers, we’re back at the original reason I began to write a book. I was once an ‘Information Systems Engineer’ and wasn’t very sure what any of those words meant, particularly “information”.

Software development teams often see their role as solving their customer’s problems. Software package providers say they are “solution providers”. What does the ‘unspeakable profession’ actually do? We got some clues from Hal Abelson in the video I linked to in my last post, that new areas of intellectual endeavour often confuse the ‘essence’ of their subject with their tools; so do we really engineer software?

Hal said that writing software is the process (or function) of formalising our intuitions about process (function.) Our software is a speculative formal abstraction of our intuitive understanding of a process we may not entirely understand. No wonder software projects so often fail. Like the rest of science, software is built on ideas that haven’t been proved wrong yet.

Software developers are presented with, or attempt to discover, experts’ (declarative) knowledge of the business process in the ‘domain’ we are about to change. This is ‘the abstract requirements’. Some of this may have to be implied from imperative knowledge embedded in existing software. It may be presented as imperative solutions. It may be incomplete.

We then follow our own process (function) for: ‘the way we do software’, in order to design a new process, some of which is also likely to be have to be embedded in software. Applying functions to change functions? That’s what functional programming does, isn’t it?

I believe that the ‘stuff’ of ‘computer science’, is state-change of systems of process and data. Who remembers ‘Data Processing’? Functional programming points out that the processes themselves are data and dynamic state-change is unpredictable and therefore dangerous.

Engineering would apply ‘project thinking’ to this unstable, poorly defined change. I may once have tried to be an information systems engineer but I saw that it didn’t work and took a 20 year break from software development.

Agile frameworks recognise such changes as risky experiments and carefully apply the scientific method, incrementally with feedback loops, to check assumptions.

We may need to return another time to see what functional programming has to say about 2 projects making concurrent changes from the same initial state. Until then, good luck with those.