Transcript
Welcome to AACC's Expert Access Live Online program

Our topic this month is Equivalent QC – Some Myths About Internal Monitoring Systems

This month's expert is Jan Krouwer, PhD. View the presentation and direct your questions to our experts.

AACC would like to thank Bayer HealthCare Diagnostics for making this program possible.

How was the graph calculated showing sample sizes for various sigma numbers?
Yonkers, NY

Jan Krouwer, PhD: Given that no defects occur for a specific number of specimens run, the defect rate is zero. However, if this is considered a sample and one wishes to know what would happen if this number of specimens were repeated over and over again, one can calculate a 95% confidence interval which tells one that even though the estimated defect rate is zero, as many defects as “X” may occur when the experiment is repeated, where “X” is the result of the confidence interval calculation. See: Hahn GJ and Meeker WQ. Statistical intervals. A guide for practitioners. Wiley: New York, 1991, p. 104 and also the essay on outliers at http://krouwerconsulting.com/Essays/Outliers.htm

I recently inspected a lab with a blood gas system which was electronically QC'd once per lot (once per month for this lab). The acceptable range for pH was 7.40-7.60. Two things come to mind -- 1) What happens if the QC is out of range? The samples for the past month are irretrievable and cannot be reanalyzed for correction purposes, and 2) What responsibility does the manufacturer have in setting acceptability ranges knowing full well that they are outside reasonably acceptable clinical limits? The FDA and CMS have no definitive guidelines on this, yet they grant license to such methods.
Memphis, TN

Jan Krouwer, PhD: These are important issues and government, manufacturers, and labs all have responsibility. However, as a lab inspector – and I have no knowledge of the regulatory inspection procedure – I would deal with this issue in a way that could have an immediate effect; namely to require the lab to put in place a more effective QA policy. The policy should start with a clinical quality requirement for pH results (and of course all other assays). If manufacturer recommendations won't achieve these goals, then these recommendations shouldn't be used. Blood gas sample stability is measured in hours – and since many of the results are used in ICUs, the treatment decisions may be made as soon as the result is reported so any QA/QC procedure must account for this. Although government and the manufacturer should also be involved, you only have direct immediate influence on the lab. In quality language, the lab is a factor that you can control (hopefully) and government and the manufacturer are “environmental factors” that can't be controlled (in the short term)

When choosing to change to using EQC can cumulative QC data be used to satisfy the 10 or 30 day consecutive testing day requirement?
nj

Jan Krouwer, PhD: If by cumulative you mean historical, then the answer is yes – see, http://www.cms.hhs.gov/clia/6606bk.pdf

The equivalent QC is adding QC testing to serveral of our kits. For example, before the equivalent QC, we ran external controls on the Mono Kit at the time the kit was opened. As long as the internal controls were functioning correctly, we did not run additional external controls. Now, we are running external controls weekly. We implemented this same process for 6 different kits that have internal (not electronic) controls. Is this correct?
Martinsville, VA

Jan Krouwer, PhD: This presentation addresses the rationale for EQC presented by CMS. Questions about interpreting the regulation or a specific lab's regulatory issue is beyond the scope of this presentation. See the 4th slide for references on the regulation.

How can you calculate in clinical chemistry from long-term IQC -variations uncertainties or what portion of estimation of uncertainties is according your meaning in long-term IQC -variations?
Prague, Czech Republic

Jan Krouwer, PhD: Sorry – don't understand the question.

What are the differences between traditional QC and EQC and how could EQC be of benefit to my lab which serves a 30-bed hospital and a 12-specialized clinics center?
Jerusalem, Israel

Jan Krouwer, PhD: EQC allows for less QC to be run. The benefit is less cost but there is a risk of more undetected errors.

We are using the equivalent qc method. Now our lot number has changed. Does another EQC process have to be done on this new QC lot number? Thank you.
Portland, Oregon

Jan Krouwer, PhD: This presentation addresses the rationale for EQC presented by CMS. Questions about interpreting the regulation or a specific lab's regulatory issue is beyond the scope of this presentation. See the 4th slide for references on the regulation.

I follow manufacturer's recommended protocol for my POCT QC frequency. Is CMS trying to say that manufacturer's recommendation (FDA approved) isn't enough? Doesn't that mean I will have to spend more budget dollars to accommodate what is proposed to save me money?
Chicago, Illinois

Jan Krouwer, PhD: This presentation addresses the rationale for EQC presented by CMS. Questions about interpreting the regulation or a specific lab's regulatory issue is beyond the scope of this presentation. See the 4th slide for references on the regulation.

Are you aware of this question from the revised DOH/PA checklist which is---for non-waived test systems with procedural or electronic controls, has the laboratory evaluated control results using 2 levels of external controls for 30 consecutive days to determine the stability of the system? The DOH has clarifed to us that external QC is to be run daily or one may use the equivalent QC option #2--which is validating stability with 30 days of consec QC and then switching to weekly liquid external controls. Would you please comment on this reg? Many Thanks!
Williamsport,PA

Jan Krouwer, PhD: This presentation addresses the rationale for EQC presented by CMS. Questions about interpreting the regulation or a specific lab's regulatory issue is beyond the scope of this presentation. See the 4th slide for references on the regulation.

One problem that you didn't mention is the possibility that a internal monitoring system sensor fails. Wouldn't that cause problems?
Miami, FL

Jan Krouwer, PhD: Sensors provide their results to IMS software. The software is constantly polling the sensor for results. If a sensor fails to provide results because the sensor has failed, the software has routines to deal with this and takes appropriate action often shutting down the system. So this type of problem usually leads to a reliability issue, not a wrong patient result.

I still don't understand how QC is of value for unit use devices, because what is detected in one unit doesn't apply to other units.
Manchester, NH

Jan Krouwer, PhD: There are two ways of looking at error. Say a unit use system has defects that appear in units randomly. QC will not help with this situation. Note however, that analogous situations happen in continuous flow systems. For example, there could be an electrical spike that lasts for one sample, or a clog in the system that lasts for one sample. Again, QC will not help with this situation. BTW, the size of the unit use lot and the number of units tested in QC at the plant (usually destructively), are critical factors in guaranteeing protection for these random defects (assuming that the tests are comprehensive). This also assumes correct statistics! (e.g., use of the hypergeometric distribution). With this testing, one can guarantee quality with respect to this type of defect. As the second error case, assume that a batch of units in a unit use device is – or goes bad and is in customer hands. Lab QC can detect this situation (assuming that the batch nature of the units are preserved – e.g., units are not randomly shipped to different locations). The same is true for a continuous flow device, if a reagent is or goes bad.

The processes we rely on to prevent laboratory error in Nigeria is not up to standard, so are we sure that we can address those errors before they affect patient health or what is the alternative? Ogboi Sonny Johnbull gibsonjb@medinews.com
Zaria-Kaduna,Nigeria

Jan Krouwer, PhD: One of the advantages of working in a lab is that lab errors do not directly affect patient care so there are opportunities to deal with the effects of the errors before they cause harm. You may wish to view last month's presentation by Dr. Michael Astion - Developing a Patient Safety Culture in the Clinical Laboratory. The link is: http://www.aacc.org/access/safety/index.asp Another way to prevent lab errors (as well as any errors that cause patient harm) is to use FMEA / fault trees. This is explained on my web site at http://krouwerconsulting.com/IFTF.htm

Do you agree with the FDA that iQM (IL Gem Premiere 3000) replaces the use of traditional QC? Does this system meet Clia eQC needs (and therefore underly the same restrictions you talk about in your presentation) or does iQM exceed Clia eQC options? why?
Munich, Germany

Jan Krouwer, PhD: Sorry, but as mentioned in one of the slides, this presentation does not discuss specific products nor interpretation of regulations.

What criteria did CMS use to establish their protocols? Slide 32 infers that they did not have any?
Norfolk, VA

Jan Krouwer, PhD: I only have knowledge of CMS's output (the regulation) regarding EQC (e.g., the references on slide 4), not how they got there. I am not suggesting that CMS used no criteria, since anecdotal data are criteria, but I am unaware of published studies.

At the Baltimore CLSI (NCCLS) meeting on QC, Westgard asserted that until quality of assays improves, labs should run more (not less) QC in order to improve quality and to meet quality requirements. In one of your slides you imply that in most cases, running more QC won't help. Could you explain?
Atlanta, Georgia

Jan Krouwer, PhD: I have addressed this issue in an essay on my web site at: http://krouwerconsulting.com/Essays/MoreQC.htm

Can we reduce Quality Control on the BIOSTAR OIA MAX STREP A KIT? Vincennes, IN

Jan Krouwer, PhD: Sorry, but as mentioned in one of the slides, this presentation does not discuss specific products nor interpretation of regulations.

Regarding the diagram showing that QC doesn't detect all errors (slide 29), what would the corresponding diagram look like for IMS?
Chicago, IL

Jan Krouwer, PhD: Good question – before my answer, one reason for the QC diagram. Some people assert that you can get a desired level of quality by running QC (with specific rules). This is not true because as the diagram shows, QC does not detect all errors. Random interferences – IMS may detect a few of these that have unique signals but it won't detect most interferences. This coupled with the lack of detection by QC makes this a big problem for assays. Even though these events may be rare, the consequences can be severe. Short term biases – IMS will detect many of these and is a great benefit of IMS. IMS is always “on” meaning that every sample is examined. BTW, QC could do better for short term biases if QC were run with every patient sample. Long term biases – IMS does less well here although it can detect some errors such as drift. Imprecision – IMS and QC can both detect imprecision that deviates from expectations although the mechanisms of detection are different.

Jan Krouwer, PhD: Good question – before my answer, one reason for the QC diagram. Some people assert that you can get a desired level of quality by running QC (with specific rules). This is not true because as the diagram shows, QC does not detect all errors. Random interferences – IMS may detect a few of these that have unique signals but it won't detect most interferences. This coupled with the lack of detection by QC makes this a big problem for assays. Even though these events may be rare, the consequences can be severe. Short term biases – IMS will detect many of these and is a great benefit of IMS. IMS is always “on” meaning that every sample is examined. BTW, QC could do better for short term biases if QC were run with every patient sample. Long term biases – IMS does less well here although it can detect some errors such as drift. Imprecision – IMS and QC can both detect imprecision that deviates from expectations although the mechanisms of detection are different.

What do you think of the Option 4 Proposal for EQC that was presented at the CLSI/CMS/AdvaMed conf. on EQC in Baltimore March 18 & 18? Will it help to have the manufacturers validate and document their own quality systems so that informed decisions can be made on the type of external QC required?
Ottawa, Ontario, Canada

Jan Krouwer, PhD: I was also at the CLSI meeting where option 4 was discussed. Before addressing the merits of option 4, one needs to ask some basic questions. 1. What is the required quality (e.g., from a clinical standpoint) for each assay? 2. Is that quality being met? 3. If not, how can it be met and at what cost? Unfortunately, in most cases, there is no answer to the first question. The clinical requirements for assays was the subject of a CLSI (formerly NCCLS) subcommittee as well as an ISO / TC212 workgroup. Both projects were cancelled, due to lack of progress. As an aside, everyone always brings up the Clarke grid for glucose, but how many labs use this instead of CLIA limits for glucose? And how many Clarke type grids exist for other assays? For more thoughts on these questions as well as a way to answer question 2, see: http://krouwerconsulting.com/act/act.htm. Whereas this points to a commercial site, ideas relevant to this topic are in the manual, which may be downloaded at no charge. So the bottom line is one must first answer question 1 from above. Nevertheless, I have a few comments about option 4. At one of the CLSI breakout sessions, I got the impression that a bunch of people thought that option 4 meant that manufacturers were ready to do something new with their systems to improve quality. As one of my slides stated, internal monitoring systems have been in systems for over 30 years. Option 4 means proving that internal monitoring systems can largely replace QC, not that new internal monitoring systems will be devised (although of course, the internal monitoring systems are always being improved). Option 4 shifts the burden of accepting EQC as valid, from the lab to the FDA. Is the FDA onboard with this? Do they have the expertise? Do they have the funding? One must answer these questions but it does make sense for a group of experts at FDA to evaluate manufacturer's data, rather than each lab director. BTW, at the CLSI meeting, it was mentioned a few times that the lab director always has the “final word.” Of course, but we still need a rational plan. Option 4 suggests a CLSI (formally NCCLS) subcommittee will prepare a standard on how manufacturers should demonstrate that EQC is valid, using fault trees and FMEAs and evaluations. I have spent about 15 years studying and practicing reliability engineering. Does CLSI have the required expertise to pursue this task? Everyone at CLSI could describe “risk analysis” in some way. How many people at CLSI have actually led a fault tree project for an assay system? Who at CLSI knows what the acronym BIT stands for? Who has ever attended a RAMS meeting or knows what RAMS stands for? Option 4 could be evaluated in at least 2 ways – by modeling or with empirical data. Modeling would be a fault tree / FMEA report, which all manufacturers must do anyway (although an option 4 type of report would require significant modification). One could also validate option 4 not just with models but with data and a large amount of data would be required to prove option 4. Both models and data will provide the best reliability for option 4. Will this be actually done? Having said this, option 4 still represents a reasonable way to answer the question, can one reduce QC frequency and maintain quality, but as the amount of work increases, one will come back to the question, what is the required quality that is needed clinically.

Can you provide your version of the rationales for EQC? How do you think about the feasibility of EQC?
Rockville, Maryland

Jan Krouwer, PhD: This presentation questions the lack of a rationale for EQC and does not try to supply one. The benefit of EQC is lower cost, the risk is more errors. One must weight EQC versus many other quality initiatives and one must have quality goals. The word “preventable” in preventable medical errors can be taken in part to mean “how much does it cost”. So of the many quality initiatives, which ones will provide the best improvement in quality for the least cost.

Could you provide a further esxplanation of slide 16
Boston, Ma

Jan Krouwer, PhD: Slide 16 means that if IMS fails to detect an error that QC catches, this is bad and a reason not to abandon (or reduce) QC. Slide 16 is a 2x2 decision table to answer the question, can IMS catch any error that QC would also catch.

How long do you think it will be before customers begin to demand EQC from manufacturers for large scale instruments with a broad menu where 2-3 levels of QC material are run every day?
NYC,NY

Jan Krouwer, PhD: I don't know the answer to your question, however large scale instruments typically have sophisticated internal monitoring systems, so they are as good a candidate for EQC as there is.

Can you explain the 21.6% failure rate described on slide 23 of the presentation? It seems like the error rate would be closer to 5% - 1 in 20 samples failing QC?
Dallas, TX

Jan Krouwer, PhD: The rate is 5%. The 21.6% is the 95% confidence interval for the 5% rate. One could get a smaller confidence interval (e.g., closer to 5%) by running many more samples.

Do you have any general idea on what would be the best approach for EQC? How the EQC can be validated and implemented in manufacturer and laboratory settings?
Rockville, Maryland

Jan Krouwer, PhD: I am skeptical that EQC could be validated in a lab – way too complicated and too much work. "Option 4" is a reasonable way for manufacturers to validate EQC – this involves applying fault trees and FMEAs and in my opinion data from experiments to validate the fault trees and FMEAs, but the details (e.g., what the FDA requires) still need to be worked out.

You mention that IMS cannot cover all cases of errors because they use models of the measuring process. But doesn't QC material (which can also behave different to blood) have the same problem: not excluding special errors connected with blood samples?
Munich, Germany

Jan Krouwer, PhD: See slides 15 and 28 – QC does suffer from issues such as those on slide 28. Slide 15 shows that running QC does not require as detailed models as IMS. Regarding EQC, the question is whether IMS is redundant to QC. If an issue escapes detection by both IMS and QC, then EQC may still be valid.

How the system drifts can be determined between two EQC procedures. Should EQC be a continuously monitoring sensor system?
Rockville, Maryland

Jan Krouwer, PhD: Internal monitoring systems should play an important role in monitoring system drift and these systems are always “on”. For example, a baseline may be monitored many times between specimens.