|
Click here for the
Guide to Medical Informatics |
Hewlett-Packard Laboratories, Filton Rd. Stoke Gifford, Bristol BS12 6QZ, UK
ewc@hplb.hpl.hp.com
Appeared in P. Barahona, J.P. Christensen (eds), Knowledge and Decisions in Health Telematics - The Next Decade, IOS Press,Amsterdam, (1994), 61-66.
The current research agenda for decision support in health care rests upon two fundamental assumptions:
Assumption 1: Health care workers require assistance with decision making because they either are prone to error or because their efficiency can be improved.
Assumption 2: Such assistance can be provided by computer systems that "mimic or emulate at least part of the processes considered to belong to the human intellect [2]".
These two assumptions have underlined the last two decades of research into artificial intelligence in medicine and medical informatics. Yet after 20 years of effort we are faced with a perplexing result:
Observation 1: The extent of use of such decision support techniques "has not been .. as wide as originally expected and has not come about as fast as imagined.[2]"
My claim will be that we must understand this result deeply before we proceed any further along the research agenda currently before us in academia and industry. Ignoring it or explaining it away will not bring us any closer to our goals.
If we accept these two assumptions and the observation, how do we reconcile them with our belief in the inherent benefit of decision support technologies - how can we explain away our failure to have an impact on health care to date?
Explanation 1: Decision support technology is immature, and more research is required before it is to have significant clinical impact.
The essence of this argument is that the computational problems that need to be solved have not yet yielded to current state of the art technologies of knowledge representation and inference. Thus we must invest further in technology before we can achieve any significant clinical impact with decision support.
This viewpoint can be countered by the successful use of knowledge based systems in other industries, notably financial systems and diagnostic systems for electrical and mechanical components. There are also examples of systems that work well in medical environments. Most of these are small and laboratory-based (see Section 3). But perhaps it is the case that current decision support techniques are successful only in such limited areas and further expansion into complex clinical domains requires further technological progress? For example, building small systems is a far easier task than building and maintaining large systems. I will argue below that wider adoption is limited not by technological maturity, but rather by other "hidden" success factors.
Explanation 2: Decision support technology is now adequately developed, but we lack the appropriate informational infrastructure to support such systems.
This argument is based on the understanding that much of the information needed to drive knowledge based systems will only be available when fully integrated electronic medical records exist. But building roads didn't invent the bicycle. If we cannot demonstrate the success we expect today, why should we expect it to appear spontaneously once we have been given a new information infrastructure? By adopting this position as a shield against current failures, we merely delay the day of reckoning. It is a dangerous position to take without being able to provably assert the value and viability of our proposals for decision support.
Recommendation: We should not be putting all our efforts into developing the infrastructure we believe will be necessary for the generalised adoption of decision support techniques. Rather, we should attempt to put in place pilot sites which demonstrate the successful use of decision support technology on a hospital-wide scale. It is a smaller task to get a computer-based medical record up and running within one hospital than it is to agree on national or international standards. More critically, the knowledge gleaned from setting up such pilot sites will help shape infrastructural standards in the light of decision support requirements. Equally as importantly, we should persist with piloting point solutions that are independent of the electronic medical record, to demonstrate the successful adoption of decision support technologies.
Explanation 3: There is resistance to the use of technology within the health care professions which requires them to be educated to its benefits, and perhaps to be directed to it use by management [6] [10].
We should refrain from interpreting resistance to our systems being introduced as reactionary stubbornness, or due to the lack of sophistication amongst the medical population. Equally we should refrain from demanding that they accept our technology because it is obvious to us that it is of benefit [6]. In fact, clinicians readily adopt new technologies which are of demonstrable clinical benefit (such as NMR or CT), and they readily treat patients with novel drugs once clinical benefit has been established (this is not to say that clinical benefit is the only reason clinicans have for adopting technology - but it is the most defensible one).
The complexity of the medical workplace requires rigorous systems analysis before any decision support system is deployed. For example, one needs to differentiate between the owner of a problem and the user of a system. Thus, a department head who wants costs to come down through better drug usage may readily see the benefit of a system that optimises prescribing behaviour. However, the clinical staff don't share this problem - they are more concerned with treating the patient in front of them now - the intrusion of a new system that does nothing to solve their problems will see that it is rejected.
Recommendation: Clinical reluctance to adopt technology says something important about that technology. We should treat clinical resistance, not as a problem to be overcome, but as a key source of information. It highlights the inadequacies of our proposals, and represents an important opportunity to test our hypotheses. We should encourage formal studies of the way health care workers interact with information technology, and decision support techniques in particular. I shall return to this theme below.
While it is always possible to explain away lack of success, another solution to our difficulty is to question the original observation.
Explanation 4: The observation that decision support techniques have failed to be adopted as widely or rapidly as expected is incorrect. We have simply been looking in the wrong places.
How sure are we that a multitude of expert systems are not in routine use across the world? Is it possible that we have failed to notice many productive knowledge- based systems in medicine because they either are too small to attract political interest, or are not technologically innovative enough to warrant a publication in the academic journals? I have asked this very question on the AIM electronic mailing list, and attempted to collate a list of expert systems in routine clinical use in medicine. At present the list contains approximately 20 systems, culled from repeated requests to the 600 strong global newsletter subscriber base, as well as a request printed in the AAAI newsletter of the AI in Medicine subgroup [12].
If we allow that there must be at least double that number in actual use, given the difficulties of collating such an informal list, are we satisfied with the notion that there are about 40-50 small knowledge based systems operating worldwide in a variety of clinical settings? One is heartened by the fact that we do have an existence proof for the value of decision support techniques in clinical medicine. What is clear upon examining the list is that most systems are small, have been custom built for particular institutions, often as the result of a research project run by an AIM champion, and have a varying impact on clinical outcomes.
Recommendation: I believe it may be a fruitful exercise to look over successfully deployed systems in detail, and by meta-analysis explore why they have been successful, why they have failed to move outside of their home institutions, and to contrast them to the multitude of research systems that have apparently never made it into routine use. However, we may discover that success and failure are often due to the presence of unique factors with each system.
Explanation 5: The observation that decision support techniques have failed to be adopted as widely or rapidly as expected is correct, but misleading. We have built many clinically useful systems, but their value has been incorrectly assessed, resulting in a failure to use them routinely. We have been measuring the wrong things.
The basis of this argument is that we have done ourselves an injustice by defining the wrong set of success criteria for a knowledge based system - and have thus been unable to communicate the value proposition for their continued use to their clinical users.
This is not as bizarre a point of view as one may initially suspect. If a system's success is measured in terms of its diagnostic accuracy, or in terms of the number of people who use it daily, then we may very well be missing the point. Surely what should be measured are clinical outcomes? We are interested in the performance of the man-machine system, not simply the machine. If, after an early high usage rate, clinicians gradually stop using a system, but nevertheless perform at a higher level than they did previously, then the addition of the machine to the health care team surely has been a success? (e.g. [5])
Recommendation: We need to agree to use a definite set of measurable success criteria, that are defined in terms of clinical outcomes [1], rather than simply looking a technical specifications or rates of system usage. Once clinical benefit has been demonstrated, then it is more likely that clinical resistance will turn to acceptance.
So far I have explored explanations that either defend our initial observation or have challenged the observation's validity. I would now like to offer an alternate and perhaps less palatable alternative. It is that our fundamental assumptions are wrong.
Explanation 6: Our assumptions about the decision making needs of health care workers are flawed, as are our assumptions about the methods that we need to support them. Their resistance to this new technology is due to a fundamental mismatch between their genuine needs, and the AIM community's perception of those needs.
Do we really understand user needs? As a group, when we do bother to seek definitions of clinical need prior to system design, we tend to fall into the trap of asking our users - unfortunately users are expert at what they do, not experts in predicting what they need [7]. What we really need is to understand our users better than they do themselves. So, do we really understand the decision making processes we seek to support, and do we understand the ways in which those processes should be supported? How clear is the literature on the cognitive difficulties that clinical workers face, or the loads that the workplace places them under?
For the most part, despite some early efforts in understanding the `classical' diagnostic process in the early history of medical informatics, it is the case that cognitive psychology, and the study of clinical errors have figured poorly in the design and implementation of clinical systems. The research emphasis is on technical rigour and novelty. In many ways, we proceed today with models of clinical performance that are a decade or more out of date, and often inappropriately matched to the domains we seek to build systems for. This is more than saying that we need to excel at systems analysis, or develop good user interfaces to our systems (both of which require an understanding of human needs and tasks) - more pointedly, a fundamental understanding of the cognitive limitations we seek to support is lacking.
For example, there is now evidence that some of the early work on decision making is incorrect. The main difficulty may lie not in human decision bias [8], but in poor situational assessment [9] - in other words the way in which we examine information prior to reaching a decision. If this is true then this might explain why systems that support clinicians by formulating a diagnosis are of little perceived value - what they really need is a way of visualising the clinical situation - selecting an action is the easy part. If we look carefully, we can begin to see confirmation for of view in some of the existing literature - for example setting out a clear paper form for recording the assessment of acute abdominal pain is apparently as valuable as accurate electronic assistance with diagnosis [6] (and more acceptable to users). The reason that so many successful expert systems in medicine sit in clinical laboratories may well be that this is where the match for this technology is best.
Once we have a developed a good understanding of human limitations with clinical tasks, then we can appropriately begin to think about finding solutions. But we must be cautious even here - it does not follow that simply because a human has difficulty with a task that we as technology providers can do anything about it. It has to be the case that our technology actually improves the situation.
Recommendation: We need to catch up with the current cognitive psychological work on human decision making (e.g. [13]), and widen our understanding of the different tasks that face clinicians - diagnosis may be a relatively unimportant task to support after all [4]. Creating a list of "generic tasks" by introspection is less valuable than setting forth into the field and identifying which tasks require support.
We thus need to identify the gaps in our understanding of clinical processes, and set out to study them. Ethnographic analysis of the clinical workplace should be sufficient in many cases to clarify the clinical requirements and feed system design [3]. We should accept with good grace that this may mean that the types of system we currently envisage may turn out to be inappropriate - for example, it may be the case that for assistance with real time monitoring tasks in ICU that there is more clinical benefit in developing concise visualisation techniques than in developing real-time expert systems for diagnostic support.
Where are the gaps in our knowledge? If we look at the current European AIM program for knowledge based systems [11], it does not invest in understanding interface technologies let alone in developing a deep understanding of the decision processes that we intend to support. We are very good at constructing research projects that have a strong technical component, but poor at exploring the human element that it is intended to support. This puts the cart before the horse. Lets understand our problem well before we start to hallucinate solutions.
Back to List of Publications