Of People, Processes and Programs
By Barry Briggs © 2004
It has become almost axiomatic that machine-driven application integration and human workflow and collaboration stand diametrically opposed. The one describes creating and connecting complex but deterministic business processes, such as the classic order management scenario, in which one computer program receives an electronic purchase order and perhaps updates the general ledger before passing it to another computer which checks the customer’s credit; another then selects and implements a preferred shipping method, and so on.
Such easily modeled processes lend themselves to highly automated solutions and can be elegantly described and implemented in software. They are sufficiently well understood that their fundamental structures can be abstracted: the Business Process Execution Language (BPEL), for example, contains primitives for task sequencing, parallelism, and synchronization, among others.
These processes find great application within the real world: the global financial network SWIFT manages millions of financial transactions daily; across SWIFT some $7 trillion in funds flow -- daily.
On the other end of the process spectrum lies the far more indeterminate, impulsive, occasionally chaotic space of human workflow, in which the executors of tasks are human beings, not computer programs. Approving a document, such as a purchase order or performance review or FDA drug application, is the classic use case, and generally there are so many exceptions to the “rules” – when to escalate or collaborate, when to hold off, what to do when a tasked individual is out sick for a week – that such scenarios seem to defy comprehensive modeling. Any attempt to do so seems doomed by the curse of reductionism: whether purposeful or inadvertent, we humans always find ways around the system.
But such is the nature of being human: we cannot model ourselves, for we and our computers would become recursive!
Or so we might think.
Systems-Up and Humans-Down Integration
Yet the two worlds often meet. Machine-to-machine integration nearly always fails unless the modeled process has some sort of “trap door” allowing humans to intervene, or to be notified in an exceptional case. Left to itself, a computer program sees no difference particularly between a $10 withdrawal and a $ 1 million one: numbers are numbers. But a human being might suspect an error, or an attempt to defraud.
In such integration applications humans must be involved at critical moments. I call this paradigm systems-up integration: multiple computer programs and systems functioning more or less autonomously but occasionally – as a result of business rules – bubbling up to ask for guidance from humans.
Here is a very simplified example of systems-up integration, in which a human only gets involved in rare cases:
Example of “Systems-Up” Integration
Humans-down integration works the other way. Consider the common Performance Review process, an annual ritual at most large companies. Typically the process begins with the employee composing a self-appraisal – her or her own judgment of the last year’s work according to previously agreed upon goals. Then the manager adds an assessment; next-level managers analyze the performance of the organization; salary guidelines are implemented; and so on. While the process is largely driven by humans according to their own timeframes, access to corporate resources – HR systems, payroll, taxation, and so on – is mandatory.
Procurement is another such example, equally common, requiring the participation and cooperation of various systems but ultimately controlled by humans:
Example of “People-Down” Integration
In our world we are surrounded by process: from mortgage applications to drivers’ license renewals to income taxes, how many of our interactions with other people and with computers are tasks within a larger process?
A Late Example
So far we’ve looked at fairly simplistic business processes; it’s time for something a bit more complex – and even melodramatic!
Let us for a few moments consider that most profound human event: death.
Administratively, the act
of expiring is one of the most complex and involved interactions an individual
can have with civil authority. Procuring a death certificate involves
unambiguously identifying the silent body, ascertaining the cause of death,
gathering various demographic information for
statistical purposes. If there is indication of a crime, an inquest or autopsy,
generating their own idiosyncratic volumes of reports,
must be conducted. If a serious illness or potential health threat is
indicated, the Centers for Disease Control in
The death certificate
itself is passed to a number of authorities including public health, the Social
Security Administration, the
In fact, the administrative context of the death, that is, its supporting documentation, may involve the collation of any number of individual, heterogeneous documents, some paper, some electronic, including the individual’s birth certificate (if one exists), medical forms, criminal forms, coroners’ reports, and many others. The sheaf of paperwork that the process carries may vary enormously from one case to the next: and all this, as you can see, is very different from the purchase order scenario in which the document context is precisely fixed and well known.
Death Registration, Amendment and Dissemination Process
(courtesy Bureau of Health Information, Vital Records Section,
Moreover, given the enormous variety of human funerary customs, some of these functions must be accomplished quickly – within 24 hours in some cases. Time, it would seem, is of the essence -- for everyone, perhaps, except the unwitting initiator of these processes.
Can such complexity be modeled and automated?
A New Model for Applications
Yes: but we have to introduce a new way of thinking about applications and integration.
First, we must note that the most basic element in all the scenarios we’ve described is a task: not a document, not a form, and not a program. Paradoxically enough, a task is primarily defined by how it ends, that is, what is required to declare the task complete and move on to the next task. String a group of well-defined tasks together:
And you get a business process, that is, a set of ordered activities that cooperate in order to perform some larger function, in this case, selling things to customers and collecting their money.
Now this basic abstraction works for both the humans-down and systems-up scenarios, although what you do with tasks is different. Systems-up applications tend, as we mentioned, to be simpler and more deterministic; tasks are marked as complete by the fact of a particular return code from some function or subroutine.
In a humans-down scenario, tasks can take very long times, can be delegated, can be reassigned, can be postponed, modified, or even ignored; and as often as not a task is declared complete when someone says it’s done. And how we say it’s done may be anything from pushing a button on an electronic form to scribbling “OK” on a paper purchase order.
Indeed, humans often decide to reopen a task (“Oops, I forgot to append this document,” or “I need to change something”) which may involve those individuals forcing the workflow back to a previous state. We are complicated creatures, but messy.
And, unlike computers, we humans have the notion of “good enough.” If three of the five reviewers of my article like it – well, that’s good enough, let’s publish it! (Tomorrow it may be two out of five.) Computers tend to be more binary, more deterministic, and, let’s admit it, emotionless.
Moreover, experience informs us that human beings are somewhat resistant to being wrapped in an envelope of highly structured, highly deterministic systems-up-like process: we like the help that computers can give us, but because computer software often demands a high degree of consistency between moving parts, and because we require large amounts of flexibility, the cost, in dollars or in aggravation, is often too high. We want automation, but we want to swallow it in small chunks.
Tasks, however, appear to be a universal human paradigm, and can be defined and modeled. Whether those models describe the particular format of a document or the constraints on a field in a database on the one hand, or simply that a user may write “OK” to signify the task’s completion -- tasks and processes containing them can be declared, and thus modeled, tracked, and to varying degrees, automated.
It’s tempting to say that all business processes converge upon the collaborative creation of a document, or a set of documents. And in many cases this is true.
Consider an application to the Federal Drug Administration for approval of a new drug. Literally hundreds of professionals contribute information of all kinds – clinical trial results, statistics and the methodology behind the statistics, the “drug master file” describing the manufacturing process – and so on. The end result of the process is, concisely, the finished document, which may run to hundreds of thousands of pages.
Fascinatingly, one can see such a document like a modern spreadsheet: as content backed by formulas and external data, in widely varying formats; as the result of processes, long-running collaborations with various methodologies for filling it in. All are interdependent upon one another; a change to any one section of the document may trigger very significant consequences.
However, other processes are less centered. As we mentioned, the death certificate scenario involves possibly dozens of different organizations with many different documents, of which few and possibly none are standardized. In this case the application is not about documents; documents play an important supporting role but the larger function is finally about assuring that society takes the appropriate measures to recognize this change in its status.
Indeed, for small business the most important tool for communicating with partners is not the computer (horrors!) but the fax machine. Managing faxes, correlating orders with approvals, proposals with acceptances, negotiating contracts and home purchases – how many hours do educated people spend standing around finicky fax machines trying to navigate through busy signals and paper jams?
So what does all this mean? How can we make sense of all this and take action upon it?
Process Requires Progress
Our first and most important conclusion is that the nature of applications is evolving. The act of writing a document should not be seen as the application-in-itself, but rather as a (potentially small) part of a much larger function. Registering a birth or death involves the creation of many documents, the interaction of many individuals and organizations.
So perhaps we would like to see on our desktops not individual applications and documents, but the overall functions or workflows we would like to start, or participate in. I would like to compose a new expense report, and track it through its various stages (and not launch a new spreadsheet document and do Save As expense report!); I will approve or disapprove the various items members of my team would like to procure, or suggest alternatives; I want to know if the various performance reviews I’ve approved fall within salary guidelines for my division, and how many are being reviewed by my own manager.
The software application is about the business function: call it a business process, a workflow, or a collaboration, but the essence, of people working together and using computers to accomplish a business task is the grand challenge of our time.
In the early nineties a famous MIT study suggested that the advent of the personal computer actually lowered individual productivity; with the passage of time, it’s easy to see why now. For the unconnected PC seduced users into spending long hours interacting with their own applications, and not with each other in a coordinated way to advance their business goals.
Our second conclusion is that while the current fad surrounding services-oriented-architectures (SOA) is indeed significant, from the standpoint of making applications easier to build, what those applications are -- how these services can be aggregated into tasks is a far more important and valuable exercise. Again: business processes are the applications of tomorrow.
Thirdly, as we’ve mentioned, workflows and business processes introduce a notion of temporality and order to programming which did not exist before. A task must complete before the next task or tasks can begin; a task may not start, conversely, until its predecessor task(s) complete (again, if this starts to sound like how a spreadsheet recalculates, I think the analogy holds). Business processes are constructed according to new models and paradigms of programming.
Fourthly, how this is exposed to us quirky and utterly nondeterministic humans is of utmost importance. The software industry is positively littered with failed companies whose products have tried to treat us like cogs in machines. The application developer or solutions provider must be granted both the power to do enforce a systems-up process as strictly as possible and conversely to make a humans-down application as loose and flexible as possible – depending on the need.
We are on the verge of an entirely new and exciting class of software programs. Based upon a new paradigm of programming and providing a virtual revolution in value and utility, this new generation of software will transform our computers from glorified typewriters and browsers into indispensable tools for driving business goals, for all kinds of organizations and for all levels within them.