Question and Answer Session

January 6, 2004 Presentation:
Laboratory Reporting for the Future: Linking Autoverification to the Electronic Medical Record

Welcome to AACC's Expert Access Live Online program.


This month's expert is Robin Felder, PhD, Associate Director of Clinical Chemistry and Toxicology, Director of the Medical Automation Research Center, and Professor of Pathology at the University of Virginia in Charlottesville, Va. View the presentation and direct your questions to Dr. Felder.

AACC would like to extend its most sincere thanks to Bayer HealthCare Diagnostics for making this program possible.

If you wish to learn more about electronic laboratory reporting and other issues in laboratory automation, AACC would like to invite you to attend Laboratory Automation: Smart Strategies for Success, which will be held March 22–23, 2004, in Amsterdam, Holland.

In your opinion, are Efficient Workflow Management difficult to maintain?
Washington, DC

Robin Felder, PhD: As an automation person, I naturally assume that workflow management will be performed by computer. While it is often tedious to input the parameters when one is initially setting up the system, once you have completed this task, then it is very easy to maintain an efficient workflow management system. Keep in mind that pattern recognition expert systems are much easier to maintain than rules based systems.

It would seem that optimizing the value of your archived data and leveraging that information can be a big key in keeping your lab "in the black." Can you recommend any resources (articles, books, etc.) that can help me find ways to to market this data?
Boston, Mass.

Robin Felder, PhD: This individual has hit on a concept that has not caught hold in the clinical laboratory industry. In the past, we have been reporting laboratory data as discrete values in response to an individual diagnostic event. However, there will be great opportunity in the future for selling diagnostic “trends” back to the physician and consumer. For example, in our own research in the Medical Automation Research Center, we are measuring longitudinal health trends in the elderly in their homes using passive sensor technology. We find that we can forecast decline in health more accurately when we take many measurements over days and weeks. Unfortunately, since this is a relatively new field, I am not aware of any good papers, articles, or books on the subject. However, the future looks bright for productivity in this area.

What can / will be the role of artificial neural networks in clinical practice. Do we have to invest in these things and to develop them. Are there hospitals working on this and what are the areas of most interest.
Sittard, the Netherlands

Robin Felder, PhD: As you know, neural networks are one method of using computers to mimic artificial intelligence. As laboratory data becomes more complex, and as we begin to look for longitudinal trends (see previous question), then it will require a computer algorithm to find useful diagnostic information. Some laboratory groups are beginning to experiment with new algorithms to find subtle relationships in data. For example, we see: Moore and Williams, New strategies for identifying gene-gene interactions in hypertension. Ann Med. 2002;34(2):88-95. Review. Thus, molecular diagnosis will be an area of intense interest. The most interesting areas will be in the prediction and early diagnosis of the 5 major chronic diseases: Cardiovascular, diabetes, arthritis, asthma, and cancer.

Dear Dr. Robin, I am interested in knowing the legacy part of the electronic signatures after the online QA correction? Secondly, are there any readymade softwares available for IR (Intelligent or Interpretive Reporting). thanks & regards, Dr. Mukesh Agrawal VIMTA LABS LTD. HYD.
Hyderabad, India

Robin Felder, PhD: I am not entirely clear what you are asking in this question. However, autovalidation software will check and automatically interpret laboratory data and perform a quality check. Then an electronic signature or identification that the data has been autovalidated by computer will be appended. Thus, each physician will know that a human did not participate in the autovalidation process. Autovalidation software is currently available from Beckman Coulter. I am certain that others are available, but I am not aware of specifics at this time. There is also a pattern recognition software tool that will soon be available out of Australia.

Where can I find professionally qualified guidelines/algorithms for autoverification of lab results ? Thanks, Michael Mayer
Jerusalem Israel

Robin Felder, PhD: To my knowledge, there are not any professionally qualified guidelines available at this time. However, an excellent organization, the NNCLS is usually a leader in the development of professional guidelines for the clinical laboratory industry. Please try their email information address:

Is there any commercially available pattern recognition software for autoverification of clinical chemical and hematological data. sincerely, Dr. M. Volmer
Groningen, The Netherlands

Robin Felder, PhD: Yes, there is a commercial package that is in use in Sonic Health Care in Sydney, Australia. If you contact me directly, I can put you in touch with the appropriate individuals.

Where can I learn more about Pattern Recognition
Albany, New York

Robin Felder, PhD: Unfortunately, this method is the newest (and unpublished) method for autovalidation. Please see the previous message for my contact information, and I will see what I can do to get you more information.

The local hospital is beginning the process of establishing an EMR for patients. We are an outpatient laboratory and xray not owned by the hospital and are being asked to participate (or lose business). What HIPAA issues should be addressed at onset?
Waukesha, WI

Robin Felder, PhD: If the hospital is establishing the EMR for their patients, and they are sending their patients to your laboratory, then there are two options for HIPAA compliance. 1. Either the hospital has the patient sign a HIPAA form that includes laboratory data and then has the patient bring that form to the laboratory (or XRAY) for copying and filing. Or 2. Your laboratory has the patient sign a HIPAA compliance form when the services are performed at the outpatient laboratory. In either case, the form will have to indicate what data the patient wishes to allow to be released and to whom it is to be released. Hopefully, your information system vendor will provide a tablet PC or Palm Pilot version of the HIPAA form, with fingerprint signature verification, so that you can avoid all the paperwork.

Do you believe it is a good idea to begin the autoverification process with a very simplistic rule such as: only results that are within normal range and don't present with any delta flags are auto verified?
San Antonio

Robin Felder, PhD: This is an excellent idea that many laboratories have used. It takes some time before laboratory technologists trust an autoverification system. Professional laboratory technologists feel personally responsible for the quality of the data they release, so I certainly agree with their initial lack of trust in a computer algorithm. Thus, start with something simple such as results within a specified reference range before you move on to the complex rules.

With Docs overburdened in lab order entry (e.g. ICD-9s, patient fasting status, etc, etc at an EMR screen in the examining room), do you see a growing need for "front end" decision support? Are there companies (portal or otherwise) focused on providing solutions as more OE information quantity & quality is demanded from time stressed Docs? Very visionary presentation!
Danville, PA

Robin Felder, PhD: Front end filtering and decision support is as important as back end autoverification. I am not aware of any companies working specifically with the laboratory aspects of the EMR entry process simply because EMR companies are having enough difficulty dealing with the many data points that have to be entered to register an encouter. They have not spent a great deal of time specifically on laboratory issues. However, physician order entry systems (what you are calling OE)does accommodate some of the issues regarding specimen type (fasting, 24 hour urine vs spot specimen). I don't think this data is being passed onto the LIS in most cases. Hospitals with "home grown" computer systems have dealt with this issue, but these programs are not commercially available.

Note came via Cerner user group that California was banning autoverification. What's basis or truth about this??
Asheville, NC

Robin Felder, PhD: As with any new technology, there are those who wish to study the problem to assure the public that no harm is being done. I agree completely that an autoverification system should be validated before it is launched. In addition, the algorithms should be rechecked and validated at least every 6 months. Checking of the laboratory's autoverification policies and procedures should be part of a CLIA inspection as well. Thus, I would interpret a ban in California (if it is indeed true) to be more of a delay to study the problem.

How important are user-defined decision rules? It seems that this feature isn't available on every analyzer, but there also seems to be some good software with very useful algortithms that could work with our LIS. Any opinion?
Manchester, UK

Robin Felder, PhD: User-defined decision rules are quite important. In fact, these rules should not really be part of the analytical system but part of the laboratory information system (and ultimately part of the laboratory process control system). My concern with rules that are built into analyzers is that when multiple analyzers are being used in the lab, one must be careful that all the rules are set up the same way on each analyzer. This results in more laboratory system maintenance that can easily be overlooked. My personal opinion is that analyzers should concentrate on producing data. Information should then be produced by a centrally managed autoverification system. Commercially available, or "home grown", software modules can be added to the LIS to perform some of these functions.

It seems to me that there is such a thing as releasing results too fast. For example: a barcoded specimen is put on an instrument and run. There are no previous values on the patient and results are not grossly abnormal. It isn't until the specimen is removed from the instrument that the tech notices something wrong with it (maybe a mislabeling or short draw). With autoverification, this may be of greater concern. How would you set up autoverification to avoid this sort of thing?
fayetteville, ark

Robin Felder, PhD: The issue of mislabeling, short draw, icterus, hemolysis, lipaemia, fibrin clots, etc. are issues that should really be caught in a well designed pre-analytical processor. We use the Tecan Genesis to catch some of these issues. In the future look to smart camera systems that can examine each specimen for issues that even well trained technologists can miss. Autoverification systems might be able to help with some of these issues such as performing a delta check the second time the patient sample appears. Studies that have been conducted on autoverification systems show that the overall error rate of the laboratory decreases.

How are you able to use the Neural Network in your auroverification algorithm if the NN requires that you actually train on the input variables before you use the NN? The other issue is that the NN is a good engineering tool for implementing a program that is already trained, but it isn't a continuous, online learning program. It can be made to be more useful only if an extraction tool for data mining, such as feature extraction, is implemented before training. Larry Bernstein
Brooklyn, NY

Robin Felder, PhD
All autoverification systems are installed and run for several months (several thousand patients) in parallel to human verification before results are ever released without human inspection. Thus, this period can also be used to train the neural network to look for specific patterns. The neural network can also be modified, expanded, and refined for other data mining tasks, as you suggest. In my presentation, I suggest that pattern recognition systems (a form of rules based algorithm)is the better choice for autoverification.

Some jurisdictions do not allow autoverification, e.g., California. How can labs in these areas automate their processes to make them more efficient?
Brea, CA

Robin Felder, PhD: Until states, like California, approve autoverification systems (which is inevitable), a laboratory can use these algorithms to quickly identify problem specimens so that they can be quickly put back into the queue. In addition, once the problems specimens are separated from the specimens without any issues, then the manual verification system will proceed much more efficiently.

Does every permutation for every rule need to be tested in the "Periodic Validation" required by CAP?
Bellingham, Washington

Robin Felder, PhD: The depth to which a laboratory checks for every permutation in their rules system is open to interpretation. This is precisely why a pattern recognition system is so elegant. With a relatively quick glance, anyone can quicly discern if any of the rules have been changed since the last validation session.

In my institution it looks like any discussion of autoverification will have to start with at least some understanding of the middleware “rules” software that governs the autoverification process. For the foreseeable future, our lab will be looking to get the most from our legacy systems (no built-in autoverification), but I understand that you can achieve at least some autoverification functionality from customized--or even off-the-shelf--rules software that is compatible with your LIS/HIS. In addition, I understand that online “middleware libraries” are starting to sprout up. Can you direct me to a few references to find out more about middleware?
Baltimore, MD

Robin Felder, PhD: AACC is now quite active in this field. You can go to the AACC web site ( to find out more about an upcoming audioconference on this topic as well as information about the development of AACC’s own middleware library, which will soon be available on the web site. You can also contact Jean Rhame at AACC for this information (

How has the Autoverification capability of the clinical laboratory information domain been related to both healthcare enterprise information architecture design and the roles and functions of the the EHR in such an enterprise information architecture? How is the clinical laboratorian part of the healthcare enterpise informtion architecture design process?

Robin Felder, PhD: There is still a chasm between the EHR (or EMR)efforts by hospital information technology administration and the needs of the laboratory. Bringing these silos of territorial activity together is what we are calling "medical automation." There is a meeting that addresses these issues to some degree called Tethic (September 2004, Washington D.C.). Hopefully, initiatives such as HL7 and the proliferation of process control software will bring these various efforts together with a focus on patient outcome.

Can you provide more details about implementing autoverification in a clinical laboratory or provide references?
Asheville, NC

Robin Felder, PhD: The following references should prove to be of interest. In addition, Mr. Westgard's web site contains much useful information, especially about establishing a 6 sigma laboratory. Crolla LJ, Westgard JO., Evaluation of rule-based autoverification protocols. Clin Leadersh Manag Rev. 2003 Sep-Oct;17(5):268-72. PMID: 14531220 [PubMed - indexed for MEDLINE] Pearlman ES, Bilello L, Stauffer J, Kamarinos A, Miele R, Wolfert MS. Implications of autoverification for the clinical laboratory. Clin Leadersh Manag Rev. 2002 Jul-Aug;16(4):237-9.

From a systems design point-of-view, where is the best place to store the rules and perform the autoverification? At the analyzer, in a data management system in the lab connected to multiple analyzers, in the LIS, or the HIS?
Brea, CA

Robin Felder, PhD: If you read back through my answers to the previous questions, you will see that I advocate using a process control tool to manage the data in a laboratory. Most laboratory automation systems are managed through a process controller. One of the best process controllers that was available on the market came from Lab Interlink (Omaha, Nebraska)- contact Rod Markin, M.D., Ph.D. at The University of Omaha if you need an expert on this topic. Alternatively, process controllers are available on automation systems from Bayer, Beckman Coulter, Tecan, Roche, and others. I am not aware of a process control software tool that operates without the automation. However, we have been developing one here at the Medical Automation Research Center for several years.

Does 21 CFR Part 11 apply to autoverification and medical records? Which aspects are critical? Traceability? e-Signatures?

Robin Felder, PhD: I am not an expert on 21CFR11. However, my understanding is that it specifies all levels of security, traceability, chain of custody, documentation, and accountability. I also am not aware of the links between 21CFR11 and the EMR. However, this will be next area of study following this session!

Are there published rules based standards for autoverification?
Chicago, IL

Robin Felder, PhD: I am not aware of any rules that have been published. However, as I stated in an early response, the NCCLS is likely to be the leader in this initiative.

Has the server for this session gone down?
Brea, CA

Robin Felder, PhD: I am still working my way through the list......

1)Please give a brief summary of "Auto verification". 2)If this is direct resulting via algorithms, please explain how LIS vendors and LIS managers validate the data stream, e.g., when instrument flags indicate an invalid answer, but the data (and results) are transmitted anyway.
Los Angelees, Ca.

Robin Felder, PhD: Autoverification is a post-analytical computer based intelligent system designed to simplify test interpretation. It is a piece of software that compares the results obtained from each test with a set of rules that dictates what comment to append to each result. The rules are put in the system by each laboratory, and each laboratory is responsible to check that the rules are set up properly. If the instrument flags an invalid answer, then the laboratory could establish a rule that does not allow transmission of the results and requires a technologist to check the specimen. Thus, all the rules are 100% under the control of laboratory professionals.

Is there any work being done in pattern analysis of tests ordered to provide real-time feedback and improve the efficiency of ordering physicians? Thank you.
Chicago, Illinois

Robin Felder, PhD: Checking for test ordering patterns with real-time feedback is one of the future goals of autoverification. For example, physicians that have ordered a glucose on a patient that had a glucose performed 15 minutes before by another physician would be instantly notified of the previous result. However, the infrastructure to accomplish this simple, but effective, feat has not been developed in most hospitals.

Considering passive testing, could you please comment on the potential privacy/confidenciality barriers to its development and implementation?
Rogério Rabelo - Fleury Diagnostics, São Paulo/Brazil

Robin Felder, PhD: In the Medical Automation Research Center system, passive testing is only performed on consenting subjects (or subjects legal guardians if the subject is incapable of making these decisions). For example, in the memory care unit of an eldercare facility we are demonstrating great utility in identifying aberrant behavior that warrants immediate attention from the caregivers. Normally, these issues go unnoticed for hours at a time. Most subjects that participate in our studies indicate that passive monitoring gives them a sense of safety and security.

Do you know the reason why the majority of labs are not using Autoverification?

Robin Felder, PhD: The greatest barriers to broad deployment of autoverification software is the general lack of commercially available products, a lack of knowledge of precisely how to implement the technology, and the general reticence to let go of a task that has been performed manually since the beginning of time (in laboratories).

Gambro Healthcare Laboratory is an independent Laboratory we are currently utlizing E-Signature and has been for a year. We are totally automated, waht aspect of your presentation should we focus on to ensure we following the industry standard. Please advise. Thank you. Murielle Monde-Jean
Fort Lauderdale Florida

Robin Felder, PhD: Your laboratory sounds like one of the leaders in the field. Congratulations! Since there are currently no industry standards that I am aware of, you have to be confident that you are providing the best quality data for your patients that is possible. In the future, standards will be adopted that will provide you with useful guidelines. However, by this time your laboratory will probably be leading us into the next high efficiency trend.

An interesting vision of the future. Is there a clash between the privacy of an individual and passive monitoring as described in your presentation?
Long Beach, CA

Robin Felder, PhD: See the answer to a previous question on this topic. Only consenting adults who feel the need for safety, security, or an extra level of service will adopt this technology.

We are interested in autoverifying the normal CBCs run on our Sysmex XE-2100s based on the "positive" or "negative" flags generated. We have Cerner Millenium. Do you or any viewers have any experience with this approach?
Springfield, Missouri

Robin Felder, PhD: Unfortunately, I have not studied the specifics of the Sysmex flag generator. However, you have two of the most modern systems available and thus you should try to get these two vendors talking to each other. Cerner has been one of the most progressive LIS vendors in the market, and thus they should be very interested in your ideas.

We have been using autoverification of chemistry results for over four years. Most of the ambulatory patient results are normal and released by autoverification. However, less than 30% of the inpatient results are verified by our program without review, due to the large number of grossly abnormal results. At this point we are reluctant to widen the autoverification ranges. Does anybody have greater yield with inpatients? Laszlo Sarkozi
New York

Robin Felder, PhD: I am surprised that you see this many aberrant results. I have heard anecdotally that some inpatient autoverified results are closer to 60%. Either you have a particularly acutely ill population, or as you say, you need to broaden your ranges. You may wish to try contacting William Neely who may be able to help you sort out your issues.

What are some good basic references for the lab contemplating the use of autoverification?
St. Louis, MO

Robin Felder, PhD: Please look back through the answers where I listed a few good references.

Robin, It seems to me the next generation of "smart instruments" are going to require greater connectivity between manufacturer and lab in order to facilitate proactive technical service, automated consumable replenishment, rapid/documented software updates as well as many, many more accomodations. Do you see any manufacturers or areas of the lab today that are fully embracing this advanced vision of real time connectivity between lab and manufacturer?
Pleasanton, CA

Robin Felder, PhD: The discussions I have had with most manufacturers leads me to believe that they see this as a potential important future for laboratory instruments. However, most customers aren't willing to pay for these features because it is difficult to calculate a ROI. Believe it or not, most laboratories still feel slighted if a live repairman doesn't show up on the premises. Furthermore, the technology to implement some of these smart systems is not quite ready for prime time. For example, in a product that I initated called the BIOPHILE the engineers attempted to provide remote software updates and harware repair. We were able to provide some degree of external support, however, it became cost prohibitive to do it in an elegant way. Perhaps in the future......

That wraps up this session of Expert Access Live Online. Thanks to everyone who posted a question for this month's expert, and thanks also to our expert, Dr. Robin Felder. We hope you found today's presentation and Q&A segment to be informative and useful. This Expert Access session, and all previous sessions, are archived on our website in order to serve as a continuing source of education.

AACC would like to again thank Bayer HealthCare Diagnostics for their support of this educational program.

Page Access: