Category Archives: Software Development

A Functional Mindset

When I started learning Clojure, I thought I knew what functional programming was but I’ve learned that the functional paradigm is now more than I expected.

Everyone agrees that it’s a computational model based on evaluation of mathematical functions, which return values. This is generally contrasted with imperative programming languages such as FORTRAN, C, JavaScript or Python, which are also procedural and some of which are object-oriented but may make functional coding possible, in a hybrid style. I wouldn’t recommend learning functional concepts in a language that gives you short-cuts to stray back  to more familiar territory.

Clojure is a member of the Lisp family, first specified in 1958. The unusual feature of Lisps is their homoiconicity – code and data are the same thing. Learning Clojure has informed my thinking about business process change.

Some modern, functional languages such as Clojure use immutable data whenever possible, to eliminate side-effects. This allows better use of multi-core processors but requires a complete change in thinking, as well as programming style. ‘Variables’ are replaced by fixed ‘values’, so loops have to be replaced by recursive functions. New data can be created but it doesn’t replace old data. Yesterday’s “today’s date” isn’t automatically wiped when we decide today has happened.

Objects with their methods and local data were designed for simulating the current state of real-world objects by changing (mutating) object data state. The object model, like relational databases, has no inbuilt representation of time. Functional programming splits these objects back into separate functions and data structures and because values can’t change, they may be transformed by flowing through networks of functions, some recursive, to keep doing something until a condition is satisfied. Eventually, code must have a side effect, to tell us the answer.

Rather than computation being a conditional to-do list with data being moved between boxes, it becomes a flow of data through a network of ‘computing machines’; and the data and machines can be transformed into each other.

I hear that map, reduce & filter data transformation functions will change my world again.

Advertisements

Becoming Functional

I’ve been playing with the idea of doing some functional programming for a while now. I’ve been trying to learn and paddling around in the shallows but this week I dived right in the emacs/CIDER pool. I was aware of some dangers lurking beneath the surface: recursion, immutable data structures and the functional holy trinity of map, reduce & filter, so I came up with some ideas to face my fears. I’ve also realised my maths has got rusty so: Some of That Too.

  1. I’ve ‘done recursion’ before but I thought I’d read that my chosen weapon Clojure didn’t do tail-end recursion. This isn’t true. What it can’t do is automatic optimisation of tail-end  recursion, to stop it blowing the stack after a few thousand iterations but Clojure has a ‘recur’ expression to manually signal tail recursion and fix that. I knocked off the programme in a couple of hours and went to bed happy. My code was happily printing the first n numbers of the Fibonacci sequence but a day later I still couldn’t get it the return the numbers as a sequence.
  2. I was finding out about immutable data the hard way. You can’t build up an immutable vector, 1 element at a time. You get to keep the empty vector you created first. It’s a big mind-set change to not have variables that can vary. In my next post, I’ll try to say what I’ve learned. On this occasion it was lazy sequences.
  3. I mentioned the Algorave in my last post. I only found out about that because of an idea I had for improving my theoretical understanding of music. I realised that I could write, for example, a function that would return the 1st, 3rd and 5th notes in a major scale, using a map function.While working the theory out, I found out that Lisps are already popular in the live-coding world.
  4. At Algorave, I was inspired by the live-coded graphics to try automatically generating some graphics too, to work out the maths of mapping triangular grids onto Cartesian co-ordinates. I need that for another idea.

Three basic working programmes in about a week. They aren’t ‘finished’ but is software ever? They have delivered value via increased Clue.

Lispbian Pi. A Lambda Delta.

I’m conflicted. Part of me says that ‘us old timers’ shouldn’t assume ‘the way things were when we were kids’ were better but we know the Raspberry Pi was an attempt to recapture the spirit of the BBC Micro Model B and that seems to have gone quite well. I got a Pi 2 and I’ve worked out that it is more powerful than the first computer I worked on, a DEC VAX-11/780 which supported about 16 terminals, most used for teaching college level computing. Having that machine to myself would have been an unimaginable amount of processing power for one developer. Banks ran their financial modelling software on boxes like that. So why does the Pi feel so slow? We wasted our gains on GUI fluff.

When I started computing you learned just enough of the command language to get going. So, that’s bash on a pi. Then an editor. For reasons that should become obvious, let us choose emacs. When I first used the VAX/VMS operating system, it didn’t have command line editing. If you made an error, you typed it all again. Getting the facility to press up-arrow, edit the command and re-execute it was a big advance. We should keep it. Bash has that, using a sub-set of the emacs keys, so that’s a way into emacs.

The next big improvements I remember were X windowing and symbolic debugging. We got debugging first but it became far more powerful with multiple terminal windows. The GUI was OK, I guess but DEC didn’t give us many free toys so the main advantage to a developer was having lots of terminal windows. emacs can do that, without the overhead of X.

When I decided to re-learn coding a while back, I got my shortlist of languages down to Python, Java and JavaScript but picked Python because I was already learning a new language and the Object paradigm, so I didn’t want to have to learn web at the same time. I heard about the modern Lisp dialect Clojure and changed horses mid-stream. I’m convinced by the argument that functions and immutability can save the universe from the parallel dimension.

Last night I deep-dived into emacs and found myself in an editor session with 4 windows. Why do I need more than that to learn about computation and data transformation? This guy seems to have come to a similar conclusion http://hackaday.com/2015/09/23/old-lisp-languaged-used-for-new-raspberry-pi-os/ I’ve also wondered whether a purely functional OS might make Sun’s ‘the network is the computer’ dream a little easier. emacs is written in Lisp.

I think a dedicated Lisp machine may be a step too far back. How would you browse in the world-wide hypertext library when you got stuck? But a Linux with bash, emacs, the Java Virtual Machine and libraries, Clojure via Leiningen and Cider to plug everything together might make a fine Lispbian Pi! Is all this chrome and leather trim completely necessary in the engine compartment?

It is unfortunate that the Raspbian upgrade left Leiningen broken.

The Unvexing

This post comes to you courtesy of Graham Lee at: http://www.sicpers.info/2016/11/the-vexing-problems-in-programming/ who reminded me about Value. We agilists (people who believe in agile software development) talk about value a lot but haven’t  agreed on a consistent definition of the word. As an illustration of two of the options:

“The value of this car is £9,500.” Is that to a seller or a buyer? Unless there is a difference, no trade will take place. Or is it a market rate? The long acquisition dance between Yahoo! and Google makes an interesting case study. This is: a value. A ‘fact’, right or wrong. It is the kind of thing developers in imperative programming languages put into variables and mutate but pure functional programmers squirrel away for ever, to embarrass themselves at their past Wrongness.

In contrast, “This car cost £9,500 pounds. I think that was quite good value but the model that costs £1,000 more is even better value.”, is a relative benefit:cost calculation, yet the manufacturer may market a ‘value’ model, which is just cheap. This is: a value judgement. An explicit or implicit comparison relative to something else, followed by a decision. It is a computation.

When Graham says, the most vexing problem of software product  developers is their inability to “compare the expected value of their work to the expected cost of the work.”, I think he means  ‘business benefit of their work’, and cost normally equates closely to development time. All the customer really wants to know is: “Is this the best investment I could make now?”

Graham goes on to say that we are very bad at estimating how long something will take. This is true but we are much better at estimating small jobs accurately than large ones.  Uncertainty increases exponentially with length of sequence of actions, to slightly corrupt Shannon’s Information Theory. This is why the Fibonacci sequence is often used as a sizing tool.

Agility accepts this reality. It addresses the list of things the customer currently wants, in highest benefit:cost ratio order (guessed by a business domain expert, based on guesses by an agile develooment team.) It doesn’t yet know whether the ‘whole job’ is worth doing. It decides only whether to risk the next small, cheap step and keeps doing that, as long as the ‘value’ is Good Enough. While value is high and risk is low enough, keep going.

A journey of any length starts with the first step, so why worry about whether or not it is going to be 1000 Miles? The hard part is to make it a journey, rather than simply wandering about, lost.

Then there are: our values. Our personal decisions about what matters most to us. Do we go home to read our children a bedtime story or work late and win that promotion so they have greater financial security in the future? What do we really care about and what are we willing to pay for it? This too is relative. Politics is the art of persuading you to modify your personal values. Currently, cheating is allowed.

Moderately Grouped

One of the rules I try to live my life by is: “Small pieces, loosely joined”

Then this happened.

http://phys.org/news/2015-06-social-networks-group-boundaries-ideas.html

I don’t know who I am any more. I already feared de-selection from the cult of Unix and now this.

Then I realised that although I favour hi-fi separates, I don’t  design my own amplifiers and hand-wire the components. I don’t compile Linux from source every time. I’m not a fanatic.

Tooling-up for agile state-transition

This post started out in life as an answer to a question about ‘backlog tooling’ on the LinkedIn ‘Lean & Agile’ group. Someone had given the culturally acceptable answer that the best solution is simple cards or post-it notes on a board or wall. I normally just let that pass because I don’t have a better solution to offer but this time, THIS happened:

I’m about to be intentionally provocative. We know that we are engaged in transforming a multidimensional network of business functions from one poorly understood and transient state to another, currently ill-defined, future state that we hope will emerge from the mist as we travel in it’s general direction. In organisations of any size, this change process is likely to run in parallel with other change programmes, some of them probably deliberately kept secret by people whose pay grade exceeds their ability to make rational judgements about the basis of who “needs to know”.

Amongst this chaos, the chosen tool of ‘the Agile Community’ is a single, 2-dimensional view of a ‘list of lists’, sometimes known as ‘a tree’ or ‘a star’, all of which are topologically equivalent representations of items’ states in the backlog of each Product development. Our best software tools are little more than a model of cards on a board.

Why do we expect the complex, dynamic agile change process to map any more adequately onto a tree of cards than it does onto a hierarchical management structure? Have we learned nothing from our mistakes of modelling within the limitations of the filing cabinet and the typewriter? Perhaps agilists don’t value tools because our tools aren’t fit for purpose.

If all you have is lists representing vague descriptions of changes between two mental models you hope your whole team all share, perhaps the limited nature of the backlog tool isn’t your biggest problem. The backlog items reference changes to an implicit model of roles and the objects in the business domain of your product. My advice is to make it explicit.

Popular Scrum tweaks

Contentious, I know, but the Scrum framework for agile software development may not be completely perfect yet. That’s fine because because we can improve on it, like science.
A few problems that often come to light are:

  • Very few people agree what Agile is. I’m one of them.
  • Few people agree exactly what Scrum is (despite it being defined by a very short document, The Scrum Guide.) “Like chess, Scrum is very simple.”
  • There is a lot of common agile practice that is used by Scrummers and taught on courses but isn’t part of Scrum.
  • Many people think they are ‘doing’ agile and/or Scrum but may not be. Who knows? The rules are: there are no rules.

In my last post I mentioned pragmatic changes to Scrum. Below are some that seem common and I’m not sure are always wrong. I think they come from the fact that Scrum makes assumptions about the mere humans who fill the Scrum roles that are, to be polite, idealistic:

  • The use of Business Analysts to supplement the Product Owner’s knowledge and skills and the developers ability to listen and ask the right questions. Developers are not all good at dealing with people or at business analysis. The people who are, are not all good at development. Part-people who add up to a whole role may be the best a team can realistically achieve.
  • The use of stakeholders who know areas of business better than the Product Owner (PO.) Product Owners need to be super-human: trusted by whoever is paying, knowledgeable, decisive, able to write good stories and constantly available. The PO is there to make dangerous business decisions, so the Team don’t have to. If they make bad decisions, it isn’t the Team’s fault. They are allowed to have help. By this logic, any BA who helps the BA should be outside the Development Team because business process is not “IT”. I have not yet seen any organisation with a business process department. I think we will soon. Process design is often considered a management responsibility but very few managers have appropriate experience.
  • The use of technical/engineering/architecture specialists to supplement the skills of the Development Team. One small team is assumed to consist of generalising specialists with knowledge of everything that will be necessary to complete the project, though you don’t know what that is yet, so ‘be lucky’
  • Network communicators/organisational specialists. Scrum assumes autonomous teams but at scale, organisational efficiency considerations start to apply pressure to centralise scarce and expensive skills. Co-ordination of networked teams becomes necessary. Traditionally, managers are likely to have filled similar roles but in future there may be more collaboration than typical managers have experienced while fighting for influence and resources inside a hierarchical organisation.