Of People, Processes
and Programs
By Barry Briggs © 2004
Introduction
It has become almost
axiomatic that machine-driven application integration and human workflow and
collaboration stand diametrically opposed. The one describes creating and
connecting complex but deterministic business processes, such as the classic order management scenario, in which one
computer program receives an electronic purchase order and perhaps updates the
general ledger before passing it to another computer which checks the customer’s
credit; another then selects and implements a preferred shipping method, and so
on.
Such easily modeled
processes lend themselves to highly automated solutions and can be elegantly
described and implemented in software. They are sufficiently well understood
that their fundamental structures can be abstracted: the Business Process
Execution Language (BPEL), for example, contains primitives for task
sequencing, parallelism, and synchronization, among others.
These processes find great
application within the real world: the global financial network SWIFT manages millions
of financial transactions daily; across SWIFT some $7 trillion in funds flow --
daily.
On the other end of the process
spectrum lies the far more indeterminate, impulsive, occasionally chaotic space
of human workflow, in which the executors of tasks are human beings, not
computer programs. Approving a document, such as a purchase order or performance review or FDA
drug application, is the classic use case, and generally there are so many
exceptions to the “rules” – when to escalate or collaborate, when to hold off,
what to do when a tasked individual is out sick for a week – that such
scenarios seem to defy comprehensive modeling. Any attempt to do so seems doomed by the curse of reductionism: whether
purposeful or inadvertent, we humans always find ways around the system.
But such is the nature of
being human: we cannot model ourselves, for we and our computers would become
recursive!
Or so we might think.
Systems-Up and Humans-Down Integration
Yet the two worlds often
meet. Machine-to-machine integration nearly always fails unless the modeled
process has some sort of “trap door” allowing humans to intervene, or to be
notified in an exceptional case. Left to itself, a computer program sees no
difference particularly between a $10 withdrawal and a $ 1 million one: numbers
are numbers. But a human being might suspect an error, or an attempt to
defraud.
In such integration
applications humans must be involved
at critical moments. I call this paradigm systems-up
integration: multiple computer programs and systems functioning more or less
autonomously but occasionally – as a result of business rules – bubbling up to
ask for guidance from humans.
Here is a very simplified
example of systems-up integration, in which a human only gets involved in rare
cases:
Example of “Systems-Up” Integration
Humans-down integration works the other way. Consider the common Performance
Review process, an annual ritual at most large companies.
Typically the process begins with the employee composing a self-appraisal – her
or her own judgment of the last year’s work according to previously agreed upon
goals. Then the manager adds an assessment; next-level managers analyze the
performance of the organization; salary guidelines are implemented; and so on.
While the process is largely driven by humans according to their own
timeframes, access to corporate resources – HR systems, payroll, taxation, and
so on – is mandatory.
Procurement is another such
example, equally common, requiring the participation and cooperation of various
systems but ultimately controlled by humans:
Example of “People-Down” Integration
In our world we are
surrounded by process: from mortgage applications to drivers’ license renewals
to income taxes, how many of our interactions with other people and with
computers are tasks within a larger process?
A Late Example
So far we’ve looked at
fairly simplistic business processes; it’s time for something a bit more
complex – and even melodramatic!
Let us for a few moments consider
that most profound human event: death.
Administratively, the act
of expiring is one of the most complex and involved interactions an individual
can have with civil authority. Procuring a death certificate involves
unambiguously identifying the silent body, ascertaining the cause of death,
gathering various demographic information for
statistical purposes. If there is indication of a crime, an inquest or autopsy,
generating their own idiosyncratic volumes of reports,
must be conducted. If a serious illness or potential health threat is
indicated, the Centers for Disease Control in
The death certificate
itself is passed to a number of authorities including public health, the Social
Security Administration, the
In fact, the
administrative context of the death,
that is, its supporting documentation, may involve the collation of any number
of individual, heterogeneous documents, some paper, some electronic, including
the individual’s birth certificate (if one exists), medical forms, criminal
forms, coroners’ reports, and many others. The sheaf of paperwork that the
process carries may vary enormously from one case to the next: and all this, as
you can see, is very different from the purchase order scenario in which the
document context is precisely fixed and well known.
Death Registration, Amendment and
Dissemination Process
(courtesy
Bureau of Health Information, Vital Records Section,
State of
Moreover, given the
enormous variety of human funerary customs, some of these functions must be
accomplished quickly – within 24 hours in some cases. Time, it would seem, is
of the essence -- for everyone, perhaps, except the unwitting initiator of
these processes.
Can such complexity be
modeled and automated?
A New Model for Applications
Yes: but we have to
introduce a new way of thinking about applications and integration.
First, we must note that
the most basic element in all the scenarios we’ve described is a task: not a document, not a form, and
not a program. Paradoxically enough, a task is primarily defined by how it
ends, that is, what is required to declare the task complete and move on to the
next task. String a group of well-defined tasks together:
Receive order
Check credit
Check inventory
Ship product
Update receivables
And you get a business
process, that is, a set of ordered activities that cooperate in order to
perform some larger function, in this case, selling things to customers and
collecting their money.
Now this basic abstraction
works for both the humans-down and systems-up scenarios, although what you do
with tasks is different. Systems-up applications tend, as we mentioned, to be
simpler and more deterministic; tasks are marked as complete by the fact of a
particular return code from some function or subroutine.
In a humans-down scenario,
tasks can take very long times, can be delegated, can be reassigned, can be
postponed, modified, or even ignored; and as often as not a task is declared
complete when someone says it’s done. And
how we say it’s done may be anything from pushing a button on an electronic
form to scribbling “OK” on a paper purchase order.
Indeed, humans often
decide to reopen a task (“Oops, I forgot to append this document,” or “I need
to change something”) which may involve those individuals forcing the workflow
back to a previous state. We are complicated creatures, but messy.
And, unlike computers, we
humans have the notion of “good enough.” If three of the five reviewers of my
article like it – well, that’s good enough, let’s publish it! (Tomorrow it may
be two out of five.) Computers tend to be more binary, more deterministic, and,
let’s admit it, emotionless.
Moreover, experience
informs us that human beings are somewhat resistant to being wrapped in an
envelope of highly structured, highly deterministic systems-up-like process: we
like the help that computers can give us, but because computer software often
demands a high degree of consistency between moving parts, and because we
require large amounts of flexibility, the cost, in dollars or in aggravation,
is often too high. We want automation, but we want to swallow it in small
chunks.
Tasks,
however, appear to be a universal human paradigm, and can be defined and
modeled. Whether those models describe the particular format of a document or
the constraints on a field in a database on the one hand, or simply that a user
may write “OK” to signify the task’s completion -- tasks and processes containing them
can be declared, and thus modeled, tracked, and to varying degrees, automated.
Documented Observations
It’s tempting to say that
all business processes converge upon the collaborative creation of a document, or a set of documents. And in
many cases this is true.
Consider an application to
the Federal Drug Administration for approval of a new drug. Literally hundreds
of professionals contribute information of all kinds – clinical trial results,
statistics and the methodology behind the statistics, the “drug master file”
describing the manufacturing process – and so on. The end result of the process
is, concisely, the finished document, which may run to hundreds of thousands of
pages.
Fascinatingly, one can see
such a document like a modern spreadsheet: as content backed by formulas and external
data, in widely varying formats; as the result of processes, long-running
collaborations with various methodologies for filling it in. All are
interdependent upon one another; a change to any one section of the document
may trigger very significant consequences.
However, other processes
are less centered. As we mentioned, the death certificate scenario involves
possibly dozens of different organizations with many different documents, of
which few and possibly none are standardized. In this case the application is not about documents; documents play an important
supporting role but the larger function is finally about assuring that society
takes the appropriate measures to recognize this change in its status.
Indeed, for small business
the most important tool for communicating with partners is not the computer (horrors!) but the fax machine. Managing faxes,
correlating orders with approvals, proposals with acceptances, negotiating
contracts and home purchases – how many hours do educated people spend standing
around finicky fax machines trying to navigate through busy signals and paper
jams?
So what does all this
mean? How can we make sense of all this and take action upon it?
Process Requires Progress
Our first and most
important conclusion is that the nature
of applications is evolving. The act of writing a document should not be
seen as the application-in-itself, but rather as a (potentially small) part of
a much larger function. Registering a birth or death involves the creation of
many documents, the interaction of many individuals and organizations.
So perhaps we would like
to see on our desktops not individual applications and documents, but the
overall functions or workflows we would like to start, or participate in. I
would like to compose a new expense report, and track it through its various
stages (and not launch a new
spreadsheet document and do Save As expense report!); I will approve or
disapprove the various items members of my team would like to procure, or
suggest alternatives; I want to know if the various performance reviews I’ve
approved fall within salary guidelines for my division, and how many are being
reviewed by my own manager.
The software application is about the business
function: call it a business
process, a workflow, or a collaboration, but the
essence, of people working together and using computers to accomplish a
business task is the grand challenge of our time.
In the early nineties a
famous MIT study suggested that the advent of the personal computer actually lowered individual productivity; with
the passage of time, it’s easy to see why now. For the unconnected PC seduced
users into spending long hours interacting with their own applications, and not
with each other in a coordinated way to advance their business goals.
Our second conclusion is
that while the current fad surrounding services-oriented-architectures (SOA) is
indeed significant, from the standpoint of making applications easier to build,
what those applications are -- how
these services can be aggregated into tasks is a far more important and valuable
exercise. Again: business processes are
the applications of tomorrow.
Thirdly, as we’ve
mentioned, workflows and business processes introduce a notion of temporality
and order to programming which did not exist before. A task must complete
before the next task or tasks can begin; a task may not start, conversely,
until its predecessor task(s) complete (again, if this starts to sound like how
a spreadsheet recalculates, I think the analogy holds). Business processes are constructed according to new models and
paradigms of programming.
Fourthly, how this is
exposed to us quirky and utterly nondeterministic humans is of utmost
importance. The software industry is positively littered with failed companies
whose products have tried to treat us like cogs in machines. The application developer or solutions
provider must be granted both the power to do enforce a systems-up process as
strictly as possible and conversely to make a humans-down application as loose
and flexible as possible – depending on the need.
We are on the verge of an
entirely new and exciting class of software programs. Based upon a new paradigm
of programming and providing a virtual revolution in value and utility, this
new generation of software will transform our computers from glorified
typewriters and browsers into indispensable tools for driving business goals,
for all kinds of organizations and for all levels within them.
Bibliography:
- SWIFT: http://www.swift.com
- Federal Drug Administration, “New Drug
Application Process”, http://www.fda.gov/cder/regulatory/applications/NDA.htm
- State of