Listen to the Clinical Chemistry Podcast


Article

A. Stavelin, P. Hyltoft Petersen, U. Sølvik, and S. Sandberg. External Quality Assessment of Point-of-Care Methods: Model For Combined Assessment of Method Bias and Single-Participant Performance by the Use of Native Patient Samples and Noncommutable Control Material. Clin Chem 2013;59:363-371.

Guest

Dr. Anne Stavelin is a Biomedical Laboratory Scientist and a PhD candidate at the University of Bergen, and at NOKLUS, an EQA organization in Norway.


Transcript

[Download pdf]

Bob Barrett:
This is a podcast from Clinical Chemistry, sponsored by the Department of Laboratory Medicine at Boston Children’s Hospital. I am Bob Barrett.

An optimal External Quality Assessment scheme is characterized by the use of commutable, stable, and homogeneous control materials with the same matrix as patient samples. The target value should be established by a reference method and the control samples should be handled similarly to patient samples. With the expanding use of point-of-care testing at decentralized locations, such as surgical departments, general practitioners’ offices, nursing homes, and pharmacies, there is a need to establish External Quality Assessment in these environments. This process, however, is a challenge due to the large numbers of control samples needed, the possible requirement of different control materials for different point-of-care instruments, and the lack of tradition for quality control systems in these locations.

In the February 2013 issue of Clinical Chemistry, Dr. Anne Stavelin and colleagues describe an External Quality Assessment model that combines an assessment of point- of-care method bias with single participant performance. The authors report that this model should be easy to carry out on a regular basis, and is particularly aimed for constituents where the External Quality Assessment control material is non-commutable. Dr. Stavelin is a Biomedical Laboratory Scientist and a PhD candidate at the University of Bergen, and at NOKLUS, an External Quality Assessment organization in Norway. She is our guest in this podcast.

Doctor, you propose a model to enhance External Quality Assessment (EQA) for monitoring the quality of point-of- care devices, an area of laboratory medicine which needs improvement. Why are traditional EQA for point-of-care methods less useful than your proposed model?

Anne Stavelin:
In traditional EQA schemes it is most common to use lyophilised control materials. Such material is considered non-commutable, meaning that it does not reflect the performance of real patient samples. When non-commutable samples are used, peer group assessment with method specific target values is required, and comparison between methods are therefore not possible.

In other words, you cannot evaluate systematic deviation or bias between methods. This means that an individual laboratory, or indeed a whole method group, could report results with a systematic deviation from the true values that can be assessed as having acceptable performance.

The optimum is to use native or commutable control samples in all EQA schemes, and this is one of the main challenges for all EQA organizers. However, by implementing our proposed models, the schemes become more useful without the need to distribute native samples to all participants. In addition, several different methods can be compared to the same designated comparison methods, meaning that the biases of different methods can be compared.

For the EQA organizers, the proposed model might be simpler and easier to carry out than transitional methods comparison studies, because they will often have limited access to patients. Our model should be easy to carry out on a regular basis.

Bob Barrett:
Can you please give us a short overview of the concept of this model?

Anne Stavelin:
Yeah, the concept is that native patient samples are analyzed, both with a referenced method or a designated comparison method, and at some selected expert primary care centers. The true value is established with the reference or comparison method. Bias of the point-of-care methods can then be estimated.

In the same time period non-commutable control materials are distributed to all EQA participants and to the expert primary care centers. A method-specific target value for the non-commutable control materials for each point-of-care method is established based on the results from the expert centers.

Each single participant result can then be compared with this target value. By using separate quality specifications, combined assessment of both method bias, and single participant performance, is possible. The best case scenario is when both results are within the quality specifications, indicating that the method has no bias and that the participant performed the test correctly.

The worst case scenario is when both results are outside the quality specifications and the results deviate in the same directions. Such a participant will report patients’ result that are greatly over or underestimated.

So the principle of this model is that a selected group of primary care centers establish a bias of the point-of-care method and this information is incorporated in the feedback to the participants in the EQA schemes. As a consequence, the participants get more information about analytical quality of their method. The idea is to accumulate the results from year to year so that we better can address the problem: Is it the method or is it the participant performance that needs to be improved?

Bob Barrett:
And Doctor, regarding the combined assessment, let’s say that a participant uses a method with a bias of +10% and gets a deviating control result of -20% from the method median. That yields a net result of -10% overall. If one looked only at this final result one could think that this might be acceptable. Now, is this is a correct assumption?

Anne Stavelin:
No, this is not correct. It is important to underline that our model is a combined assessment of two different approaches, namely the method bias, which is based on the mean of approximately 100 patient results, and the single participant deviation, which is based only on one result.

Consequently, we don’t think it is correct to numerically add these two results. This is because the method bias is an evaluation of the systematic errors, whereas the single participant deviation is an evaluation of the total error, which includes both random and systematic error.

The quality specifications for these two evaluations are different and the two results should therefore be handled separately. However, we suggest a model where these two results are illustrated in one figure.

In your example, both results are outside their respective quality specifications, meaning that the participant uses a method with unacceptable bias and performs the test incorrectly. Even if the true results deviate in opposite directions, the message to these participants should be to change to a better method and improve their performance.

Bob Barrett:
Is this model also useful for other systems than point-of- care methods?

Anne Stavelin:
Yes, this model is not limited to point-of-care methods, but has potential for all methods where non-commutable materials are used.

Bob Barrett:
Is it possible to use your model for control of individual reagent lot numbers?

Anne Stavelin:
Yes, this is possible. We can evaluate if the lot-to-lot variation using non-commutable control materials is similar to the lot-to-lot variation obtained by native materials. In addition, it is also possible to monitor the lot variation over time.

Bob Barrett:
Now, it’s interesting that the non-commutable control materials show lot-to-lot variation but that the native patient materials did not. How does the EQA organizer and the individual participant deal with this finding?

Anne Stavelin:
I agree that this is a very interesting finding. However we cannot exclude that if more samples were analyzed, we might also have detected lot-to-lot differences using the native patient material. By performing additional experiments we can get more information about this. This should probably be done when there is the suspicion that the lot variation results on native patient materials differs from the results using non-commutable materials.

Regarding the EQA organizers, they should act on the finding of lot-to-lot variation. Information about substantial differences between lot numbers using native materials should be communicated both to the manufacturer and to the participants.

The information to the participant could, for instance, be: Your result deviates from the peer group median, but this is due to a systematic deviation with the lot you are using. However, it is difficult for the participant to deal with this other than changing the lot. This information is on the other hand very useful for the manufacturer.

Bob Barrett:
Well, finally, Doctor, based on your data, where do you think the quality assurance effort for point-of-care INR should be focused?

Anne Stavelin:
Our data indicates that more efforts should be put on method improvement or advising against using poor methods rather than on addressing participant performance.

The results from the EQA using non-commutable materials were very good. More than 90% of the participants got results within the quality specifications. If we only rely on these results, the conclusion is that the point-of-care methods are performing well and the participants believe that they use the good method.

However, the EQA results do not reveal whether or not the point-of-care methods have a systematic error. This is the main reason why we propose this model and why such a model may help in the harmonization process. By using our model, EQA organizers should be able to warn against using poor methods.

Bob Barrett:
Dr. Anne Stavelin is a Biomedical Laboratory Scientist and a PhD candidate at the University of Bergen, and at NOKLUS, an EQA organization in Norway. She has been our guest in this podcast from Chemical Chemistry. I am Bob Barrett. Thanks for listening!