All posts by andywootton

About andywootton

A modelling metaphor mixologist

Truth is beauty; but is that all?

I was reading about Clojure’s views on truth and falsehood this morning. Some of them are interesting:
(true? true) ; -> true
(false? false) ; -> true – a classic double negative

Clojure also has the value ‘nil’.
(true? nil) ; -> false
(false? nil) ; -> false

Then I went on LinkedIn and someone asked if there is ever an absolute truth. It’s a question I’ve been thinking about, so I wrote this:

I think that “at non-quantum scales”, it seems likely there can be only one set of events which actually happened but every human is wandering around in their own model of reality, based on their perception of incomplete data. Sometimes, a slow-motion replay helps fill in some of the gaps but it will be the same every time we look and it may not reveal ‘the whole truth’ we seek. We all have a simplified view of what reality is, based on our personal knowledge and beliefs and we can’t go back in time for more data, so our view of truth is an approximation. Heisenberg’s Uncertainty Principle says that it can only ever be an approximation.

In summary: I believe there is one truth but that doesn’t mean we will ever know what it is. Alternative perceptions may be just as valid as our own; possibly more so.

[ The original version of this post said nil meant “I don’t know.” I immediately discovered that was wrong. In Clojure, only ‘false’ returns ‘true’ to ‘false?’ but nil is ‘falsey’, so each of the following forms returns “F”:
(if false “T” “F”)
(if nil “T” “F”)
You can see how that could confuse a stupid person ]

Process + Data + Structure = Engineered Software

I tried to address a question about data structure on Quora. This post is a stand-alone version of the answer I gave.

‘In the beginning’ there was ‘Data Processing’. That is: ‘data’ and ‘process’, expressed in the form of a program. Programs implement algorithms.

In 1976, the practice of ‘Structured Programming’ was trending and a book was written: Algorithms + Data Structures = Programs ( Wikipedia entry)

Processes and their structure + Data and it’s structure = Programs.

If you ignore interactions with the real world, that’s all there is. If you take any working program and ignore the processes and their structure and the raw data, then whatever is left is data structure.

We structure data because the alternative is data sauce, traditionally only ever served with spaghetti code.

Reality has Levels

It’s been a while since I blogged. I’ve been busy.

A major theme emerging from ‘writing my book’ is that we humans are very bad at confusing our models of reality with the reality we are modelling.

I started planning with the ‘Freemind mind-mapping tool for hierarchical brains’ before finding my own creative process had a network architecture and discovering ‘concept mapping’ which uses graphs to represent concepts and propositions. I saw that graphs were what I needed and decided to experiment with building my own software tools from bits I had lying around.

I didn’t have a current programming language, so I set out to learn Clojure. Being a Lisp, Clojure uses tree-structures internally to represent lists and extends the idea to abstractions such as collections but the only native data structures available to me appeared to be 1-dimensional.  I confidently expected to be able to find ways to extend this to 3 or more dimensional graphs but despite much reading and learning lots of other things, I’d failed to find what I was looking for. I had in mind the kind of structures you can build with pointers, in languages like ‘C’. There are graph libraries but I was too new to Clojure to believe my first serious program needed to be dependent on language extensions, when I haven’t securely grasped the basics.

This morning, I think I ‘got it’. I am trying to build a computational model of my graphical view of a mathematical idea which models a cognitive model of reality. There was always scope for confusion. Graphs aren’t really a picture, they are a set of 1-dimensional connections and potentially another set of potential views of those connections, constructed for a particular purpose. I was trying to build a data structure of a view of a graph, not the graph itself and that was a really bad idea. At least I didn’t get software confused with ‘actual’ reality, so there’s still hope for me.

Yesterday, I used Clojure/Leiningen’s in-built Test-Driven Development tool for the first time. It looks great. The functional model makes TDD simple.

When your Netbook falls off its Sky-hook

I have an Eee PC 1000 ‘netbook’ that has been with me for a while. It’s not very fast but then I type quite slowly. It’s become a personal challenge to see how long I can keep using it. It’s always run Linux but the latest version of Ubuntu won’t fit into its (ample, in my opinion) 1GB of memory. I can’t upgrade it in place either because it has 2 SSDs, currently configured as / on the 8GB and /user on the 32GB. A couple of days ago my desktop environment went AWOL.

Trying to deal with the space imitations, I’d tried booting it from a Lubuntu Live memory stick. It seemed quicker with LXDE but Lubuntu also has leaner apps than the ones I’m used to, so I decided not to install it permanently. I may find another way to keep my current apps but replace Unity by LXDE. Afterwards, I think I rebooted it to check it was OK but I may have shut down from the login screen. The following morning I logged in and got an empty, frozen desktop display. I couldn’t even open a terminal window but I found I could log in to the Guest account. Odd. I opened a console window from my normal account and rolled up my sleeves. I had a /user (a different one, I later discovered.)

To cut a long story short, the answer was:

$ sudo mount -a /dev/sdb1 /home

My home directory wasn’t there but because it wasn’t it had fallen back to the original /user folder on the system disk. The Guest account logs in on /tmp on the system disk, so didn’t have a problem. Now, I just need to work out why whatever was auto-magically mounting it for me and why it decided to stop.

[ Update: The permanent fix was to find out the ID of my device with
$ sudo blkid

then add the following line to /etc/fstab

UUID=4b18fe5c-2d2a-4d12-938b-a38046a3cf84 /home ext4  errors=remount-ro 0  0

I still haven’t found the hole in the sky where the hook came detached. ]

Living a Virtual Life

There is a Taoist story about it being impossible to know at the time whether an event is lucky or unlucky. At my age, you start to reflect how things have gone, from a safe distance.

I planned to go to Birmingham University to study mathematics with a side-order of computer science. My ‘A’ Level results were, to put it mildly, ‘below expectation’ so I scraped into Aston through the Clearing process, to study mathematics, computer science and physics. The teaching language was Algol 68 and the visionary assumption throughout the course was that within a few years all computers would be virtual memory systems. We would never have to worry about physical restrictions on memory allocation. We had a linear address space to play with, that could be as big as the available disk space and there would be a garbage collector to tidy up after me. A few years later, PCs were to make those assumptions invalid.

I actually graduating into a recession caused by a war to the death between Margaret Thatcher and the unions. Many large companies cancelled their graduate recruitment programmes. I was unemployed until just before Christmas, when I took the first job I was offered, as a programmer in a College of Higher Education in Cambridge. I’d never heard of the computer they used. It was one of the first batch of half a dozen DEC VAXes delivered to the UK: a 32-bit super-mini running the new Virtual Memory System OS, VAX/VMS. I specialised in VMS/OpenVMS for the next 25 years, gradually becoming a system manager and specialist in high-availability clusters and development environments. I had side-stepped Bill Gates’ “No-one needs more than 640K” pronouncement and all the mess that went with it.

I lost direct touch with software development until a few years ago when I joined an agile team as analyst and decided I wanted to get back into writing code. Initially I picked Python, until I saw a demonstration of Clojure. I knew I had to have it. Clojure designer Rich Hickey says that we can treat disk space as effectively infinite. That has a huge impact on our ability to design software as temporal flow rather than last known state. Servers have become virtual too. Software is doing everything it can to escape the physical realm entirely. I’m holding on for a free ride, hoping to stay lucky, a link to a virtual copy of ‘The Wizard Book’ on my Cloud-drive. Nothing is Real. I’m not even sure about Time.

Things I used to be Wrong about – Part 1

I get very annoyed about politicians being held to account for admitting they were wrong, rather than forcefully challenged when they were wrong in the first place. Unless they lied, if someone was wrong and admits it, they should be congratulated. They have grown as a human being.

I am about to do something very similar. I’m going to start confessing some wrong things I used to think, that the world has come to agree with me about. I feel I should congratulate you all.

You can’t design a Database without knowing how it will be used

I was taught at university that you could create a single abstract data model of an organisation’s data. “The word database has no plural”, I was told. I tried to create a model of all street furniture (signs and lighting) in Staffordshire, in my second job. I couldn’t do it. I concluded that it was impossible to know what was entities and what was attributes. I now know this is because models are always created for a purpose. If you aren’t yet aware of that purpose, you can’t design for it. My suspicion was confirmed in a talk at Wolverhampton University by Michael ‘JSD’ Jackson. The revelation seemed a big shock to the large team from the Inland Revenue. I guess they had made unconscious assumptions about likely processes.

Relations don’t understand time

(They would probably say the same about me.) A transaction acting across multiple tables is assumed to be instantaneous. This worried me. A complex calculation requiring reads could not be guaranteed to be consistent unless all accessed tables are locked against writes, throughout the transaction. Jackson also confirmed that the Relational Model has no concept of time. A dirty fix is data warehousing which achieves consistency without locking by the trade-off of guaranteeing the data is old.

The Object Model doesn’t generalise

I’d stopped developing software by the time I heard about the Object Oriented Programming paradigm. I could see a lot of sense in OOP for simulating real-world objects. Software could be designed to be more modular when the data structures representing the state of a real-world object and the code which handled state-change were kept in a black box with a sign on that said “Beware of the leopard”. I couldn’t grasp how people filled the space between the objects with imaginary software objects that followed the same restrictions, or why they needed to.

A new wave of Functional Programming has introduced immutable data structures. I have recently learned through Clojure author Rich Hickey’s videos that reflecting state-change by mutating the value of variables is now a sin punishable by a career in Java programming. Functional Programmers have apparently always agreed with me that not all data structures belong in an object

There are others I’m still waiting for everyone to catch up on:

The Writable Web is a bad idea

The Web wasn’t designed for this isn’t very good at it. Throwing complexity bombs at an over-simplified model rarely helps.

Rich Hickey’s Datomic doesn’t appear to have fixed my entity:attribute issue

Maybe that one is impossible.

Agility vs Momentum

[ This post is aimed at readers with at least basic understanding of agile product development. It doesn’t explain some of the concepts discussed.]

We often talk of software development as movement across a difficult terrain, to a destination. Organisational change projects are seen as a lightening attack on an organisation, though in reality, have historically proved much slower than the speed of light. Large projects often force through regime change for ‘a leader’. Conventionally, this leader has been unlikely to travel with the team. Someone needs to “hold the fort”. There may be casualties due to friendly firings.

Project Managers make ‘plans’ of a proposed ‘change journey’ from one system state to another, between points in ‘change space’, via the straightest line possible, whilst ignoring the passage of time which makes change possible. Time is seen as distance and its corollary, cost. The language of projects is “setting-off”, “pushing on past obstacles” and “blockers” such as “difficult customers”, along a fixed route, “applying pressure” to “overcome resistance”. A project team is an army on the march, smashing their way through to a target, hoping it hasn’t been moved. Someone must pay for the “boots on the ground” and their travel costs. This mind-set leads to managers who perceives a need to “build momentum” to avoid “getting bogged down”.

Now let us think about the physics:

  •  momentum = mass x velocity, conventionally abbreviated to p = mv.
    At this point it may also be worth pointing out Newton’s Second Law of Motion:
  • force = mass x acceleration, or F = ma
    (Interpretted by Project Managers as “if it gets stuck, whack it hard with something heavy.”)

What about “agile software developments”? There is a broad range of opinion on precisely what those words mean but there is much greater consensus on what agility isn’t.

People outside the field are frequently bemused by the words chosen as Agile jargon, particularly in the Scrum framework:
A Scrum is not held only when a product development is stuck in the mud.
A Scrum Master doesn’t tell people what to do.
Sprints are conducted at a sustainable pace.
Agility is not the same as speed. Arguably, in agile environments, speed isn’t the same thing as velocity either.

Many teams measure velocity, a crude metric of progress, only useful to enable estimation of how much work should be scheduled for the next iteration, often guessed in ‘story-points’, representing relative ‘size’ but in agile environments, everything is optional and subject to change, including the length of the journey.

If agility isn’t speed, what is it? It is lots of things but the one that concerns us here is the ability to change direction quickly, when necessary. Agile teams set off in a direction, possibly with a destination in mind but aware that it might change. If the journey throws up unexpected new knowledge, the customer may wish to use the travelling time to reach a destination now considered more valuable. The route is not one straight line but a sequence of lines. It could end anywhere in change-space, including where it started (either through failing fast or the value of the journey being exploration rather than transportation.) Velocity is therefore current progress along a potentially windy road of variable length, not average speed through change-space to a destination. An agile development is really an experiment to test a series of hypotheses about an organisational value proposition, not a journey. Agile’s greatest cost savings come from ‘wrong work not done’.

Agility is lightweight, particularly on up-front planning. Agile teams are small and aim to carry everything they need to get the job done. This enables them to set off sooner, at a sensible pace and, if they are going to fail, to fail fast, at low cost. Agility delivers value as soon as possible and it front-loads value. If we measured velocity in terms of value instead of distance, agile projects would be seen to decelerate until they stop. If you are light, immovable objects can be avoided rather than smashed through. Agile teams neither need nor want momentum, in case they decide to turn fast.