Push Driven Business Process Modeling Considered Harmful

Several years ago, I had this intuition that 'locking can't be abstracted away' and that 'pessimistic locking was impractical'. They were a somewhat controversial back then. My arguments then were grounded on the fact that transactions could be in conflict with what the business requires. It certainly fell on deaf ears for those developers who wanted to be isolated from the concerns of a business.

Today however, there is considerable consensus has been building that you can't always require transactions in a purely technical sense. In fact, there is formal conjecture (see Brewer's Conjecture) that it may simply be an impossible task to have availability, consistency and partitioning all at the same time. In short, if you want massive scalability then be prepared to sacrifice transactions. In a physical world that where instantaneousness doesn't really exists (see: The night's sky), then the notion of relativism should be more of the norm.

The question however these days that my mind has been stewing over is the notion of the explicit Business Process Modeling. Codifying knowledge is at the heart of software development. Defining and streamlining process is a the heart of a successful enterprise. I have no argument against the need of streamlining the handling of physical goods. You can't break the laws of physics, the movement of mass requires energy and anyone who has filled up his gas tank lately knows that energy costs money. My question about explicit Business Process Modeling (BPM) and whether it makes sense confined to the context of knowledge based industries. That is, where value is created based on virtual goods.

The general prescription of a workforce automation or re-engineering activity is to first uncover and make explicit the underlying business process. It is through this discovery phase that one can identify points of improvement. BPM streamlines the codification of the business process by allowing practitioners to build from process model a machine executable rendition of them. It is an extremely appealing idea, especially for management, a modeled process is now changeable on a whim (so they claim) and there is better visibility and therefore control of what's going on.

Process is extremely important, however process for process sake can be extremely harmful. Software Development is one of the most knowledge intensive industries known to man. We have painfully discovered the failures of rigid waterfall methodologies and have begun transitioning to more agile, lean and iterative methodologies. This evolution from rigid specification to a more iterative approach gives me a hint that codifying process models can only take you only so far as big upfront design. On the flip side, BPM tools should also place a premium on supporting the iterative development of process models.

BPM tools however are best able to codify repetitive processes. These however are are low hanging fruit. The 'not-happy-path' or 'exceptional-path' process model is what's difficult to capture and in many knowledge based industries, this path is where most of the value comes from.

The key to understanding business processes is that they are highly asynchronous and concurrent. Sequential dependencies exist not because they are defined by a control-flow process diagrams but rather because the knowledge that's being pushed around have intrinsic dependencies. The fatal conceptual flaw of control-flow is that it places artificial constraints on the movement of information. Furthermore, control-flow driven process is dehumanizing because that's simply not how humans do work. Mark Miller explains this best when he writes:

Remarkably, human beings engage in concurrent distributed computing every day even though people are generally single threaded.
How do we simple creatures pull off the feat of distributed computing without multiple threads? Why do we not see groups of people, deadlocked in frozen tableau around the coffee pot, one person holding the sugar waiting for milk, the other holding milk waiting for sugar?
The answer is, we use a sophisticated computational mechanism known as a nonblocking promise.
(read his piece for the details)
human beings never get trapped in the thread-based deadlock situations described endlessly in books on concurrent programming. People don't deadlock because they live in a concurrent world managed with a promise-based architecture.

Requests are 'pushed' as signals into the to-do lists, the responses are promises for future action and the execution of these actions are pulled on demand from the to-do list. This precisely mirrors Kanban that is practiced by Lean and JIT manufacturing. Matter of fact, any decent Issue Tracker would suffice and a process would just be a template for creating a bunch of 'signaling' tasks. In short, to-do lists are all that's essential to support human workflow. Richard Pawson of NakedObject's fame writes:

I understand that all of this is a somewhat controversial view of business, but it is based on real experience. It reflects a very strong personal viewpoint that, to the extent that an information system involves human users, then the system should be designed to be the servant of the user, not the master. Most workflow systems attempt to be the master of the user.

Pull driven processes make sense for humans. Unfortunately their essence are cumbersome to capture using control-based push based diagramming notations. A formal BPM specification can best be used as an active monitor that notifies users if events slip through the cracks. Beyond that, it should best act more as a facilitator rather than the manager.

In fact, to support 'exceptional-path' processes, we can derive inspiration from the ideas and tools we employ in lean development. I've previously written about how 'Web 2.0 enables Lean production'. What becomes strikingly obvious is that software development is rarely ever driven by a BPM tool. It is however supported by tools like version control systems, issue trackers, wikis, automated build systems and continuous integration dashboards. That is tools that are more focused in managing knowledge.

A BPM tool apparently doesn't seem to add enough value to be of any use in the process of developing software. There seems to be little value in refactoring out control flow from knowledge (or documents). In other words, if there are any data dependencies then that would provide constraints to the flow. Work that is exerted to refine exactly what that control flow should look like is an exercise in splitting hairs.

I am forever dumbfounded when I stumble upon development organizations that lack of even a rudimentary VCS and Issue Tracker. It is the height of hypocrisy to sell to customers the use of information technology to enhance processes and at the same time not know how to use information technology to enhance one's own processes. So if a BPM isn't used to drive one's own process, then why in the world is it even sold to drive someone else's? Perhaps someone else's process isn't as complicated as one's own? Perhaps someone else's business is less human centric that software development? Perhaps a more advanced collaboration tool like IBM's Jazz is the cure to our process ills?

To conclude, as the saying goes, "practice what you preach". Good software process is based on sound underpinnings and should also be supported by automation. If you are able to migrate this wisdom over to support your customer's processes then you would have added more value than just codifying their business processes. So if you asked me for a solution to re-engineer a business process, I would recommend at a bare minimum a document management system that supports versioning and an issue tracking system. Couple this with a good dose of design patterns ( Issue Tracking Patterns, SCM Patterns )