Listen to the Clinical Chemistry Podcast
Article
S. Bennett. Twenty-Eight Grams of Prevention Is Worth a Half Kilogram of Cure. Clin Chem 2013;59:1145-1147.
Guest
Dr. Sterling Bennett is Chair of the Pathology Department at Intermountain Medical Center in Utah..
Transcript
[Download pdf]
Bob Barrett:
This is a podcast from Clinical Chemistry, sponsored by the Department of Laboratory Medicine at Boston Children’s Hospital. I am Bob Barrett.
Maintaining long-term stability of analytical procedures is an important responsibility for clinical laboratories. This process typically includes a comparison of current and new reagent lots through paired measurements of patient or control samples, with defined criteria for acceptance and rejection of the new lot.
A paper in the August 2013 issue of Clinical Chemistry finds that the number of test samples required for adequate lot- to-lot validation protocols is high, and may be prohibitively large, particularly for low volume or complex assays. Today we are joined by the author of an editorial on that paper, Dr. Sterling Bennett. He is Chair of the Pathology Department at Intermountain Medical Center in Utah.
Dr. Bennett, the paper that served as the subject for your editorial described substantial undetected drift in an assay over a five-year period into highly reputable laboratories. Retrospective analysis identified the cause as lot-to-lot inconsistency that went undetected. So which was more important, the lot-to-lot inconsistency or the failure to detect it?
Sterling Bennett:
I would say in this case the failure to detect the inconsistency was actually a greater consequence. Clinicians expect our assays to be consistent over time unless the laboratory communicates otherwise.
Clinicians reasonably expect the laboratory to manage all aspects of their assays, including reagent lot changes, and they count on the laboratory to assure the reliability of test results and to inform them, the clinicians, about the assay’s performance characteristics.
So, if the labs had identified the drift in this particular assay, they would have been able to engage the clinicians in a conversation about the magnitude of the drift and potential ways to minimize the clinical impact.
Bob Barrett:
But you are not suggesting that lot-to-lot inconsistency is unimportant?
Sterling Bennett:
Well, no, of course not. Lot-to-lot consistency is very important, because it enables physicians to develop expertise in the use of the assay, a gestalt about what the results mean, and it also enables long-term trending and management of individual patients.
So in the case of this drifting IGF-1 assay, physicians will have to puzzle over the meaning of the test results when they didn’t match their clinical expectations, and this led to additional referrals and workups that may have been unnecessary if they had known that the assay was drifting.
Bob Barrett:
Lot-to-lot variation is a function of the manufacturing process and is not really under control of the laboratory. What can laboratories do to pressure manufacturers to provide better lot-to-lot consistency?
Sterling Bennett:
Oh, that’s a really good question. I don't want to kick into victim mode here, but to some extent laboratories are at the mercy of the manufacturers. So we, as I imagine most laboratories, have had similar experiences with assay drift that was detected by the clinicians not by our own lab validation procedures. But in those cases when we have detected susceptible assays, we presented our data to the manufacturers, we discussed with them our expectations, and we have changed our own lot validation procedures.
Most of the time we found that the vendors are responsive to that kind of data presentation, but not always; we have sometimes had the response, well, no one else is seeing this, you must be doing something wrong. And then we have to provide more intensive data for them.
So how do we pressure manufacturers? One thing we can do is make our validation procedures more robust and actually reject lots. The manufacturers don't want us to refuse to accept new lots that they have provided to us, because that ends up costing them something.
So if we make it clear to them that we have high expectations for lot-to-lot consistency and that we will actually return what we consider to be defective product that puts as much pressure as anything. We can also be selective about the manufacturers that we use, favoring those with a better track record.
Bob Barrett:
Well, the track record of a manufacturer or a particular assay may be hard to ascertain, right?
Sterling Bennett:
Right, well, for at least two reasons. The first reason is that our commonly used validation protocols are statistically weak in many respects. They don't have the statistical power to reliably detect clinically significant lot-to-lot inconsistency. So the manufacturing deficiencies go undetected.
The second reason is that even when the problems are detected, there isn't a good mechanism for making that information available to other laboratories.
In fact, one of the reasons I accepted the invitation to write this editorial, was out of appreciation for the author's willingness to go public with their experience. I would dare say that virtually every experienced laboratorian has been caught in a similar situation, and astute clinicians have observed the discrepancies between the lab results and the clinical expectations, but very few of us have proactively made that information available to other labs.
We probably haven't even thought about making the information available, and as a community of laboratorians and manufacturers even, could help each other much more by developing effective mechanisms for sharing information about validation failures and post-validation issues.
If we look at commercial aviation, for example, the availability of information about aviation accidents has helped spur improvements in virtually every aspect of that industry, and so extrapolating from that, I believe that validation protocols and assays themselves would be improved if we routinely made public information about the issues that we have with tests.
Bob Barrett:
Well, the authors recommended monitoring patients’ results to identify lot-to-lot inconsistencies relatively quickly and recommend the manufacturers help aggregate data from multiple labs to accelerate the process. What do you think about that?
Sterling Bennett:
Well, these are really good ideas and many labs, including ours, are using aggregated patients’ results as a supplement to traditional quality control, and useful statistics might include means or medians or proportions of abnormal results as the authors presented in their paper. And one of the goals is to recognize assay drift before the clinicians do.
So a few years ago we had an astute clinician who noticed drift in our calcium assay, and when she started seeing high results, she would pick up the phone and give me a call, or give one of my partners a call and say, hey, I am seeing too many high calciums, will you please look into it?
And we would go back and we would mine our database and look at patients’ results and sure enough, we would have an increase in the percentage of positive results. So we set a goal to recognize this before she did and set up a weekly report that looked at our patient median results and trended those over time.
And after we had been running the report a few weeks I got a call from her one morning and she said, I think you have got assay drift again, and I said, well, we will look into it, the report came out a couple of hours later and sure enough the median had spiked that week.
So use of aggregate patient data is a really good idea, but to be valuable it requires large numbers of patients’ results. And so the manufacturers suggested involving the manufacturers in collecting data from multiple laboratories to increase the numbers of results that are analyzed; that seems like a really reasonable recommendation to me, and manufacturers who see the value of that might even be able to differentiate themselves by providing those types of services.
Now, with that being said, post-implementation detection of drift really is insufficient. The goal is to detect problems with the assays prior to even putting them into service. So the FAA referred to reliance on crash investigations to improve aviation as regulating by counting tombstones.
So in a sense using patients’ results to detect drift is regulating by counting tombstones, it’s identifying the problem after the fact. Clearly a higher goal would be to detect inconsistencies before the lot is placed into service or even before the lot is released from the manufacturing process.
Bob Barrett:
Doctor, improving lot validation and lot release protocols can be expensive. In this era of cost containment, how can validation protocols or lot release protocols be improved without dramatically increasing costs?
Sterling Bennett:
Well, the authors in their paper indicated that it would require substantially more samples to increase the power of their lot validation protocols and intimated that increasing to that number would be beyond the resources of their laboratories. I think that's a valid observation, because practically speaking, most laboratories don't have the resources for such expensive validation testing.
But coordinated collaborative efforts of groups of laboratories has the potential of increasing the quality of the validation studies without consuming additional resources. So each laboratory in a group would conduct validation much the way it does today, but by pooling results the power of the statistical tests would be improved and the likelihood of detecting substandard lots would be increased.
Similarly, validation activities might possibly be moved even further upstream by coordinating the activities of manufacturers and laboratories to do a combined lot release, lot validation procedure and thereby avoid the risk of introducing new defective lots into clinical use.
Bob Barrett:
Well, finally doctor, when you titled your editorial, you got some help from old Ben Franklin, why did you go to Ben for this?
Sterling Bennett:
When I read the article I was drawn to it, as I have mentioned, because we have had similar experiences in our lab, and once we become aware of a problem, then we move quickly to squelch the problem, and we refer to that as firefighting. And we all know that in firefighting it’s better to prevent the fire than to fight the fire, and immediately Ben Franklin's adage about “An ounce of prevention is worth a pound of cure” came to mind.
So his expression was actually given in the context of organizing firefighting service in Philadelphia. Now, at that time the handling of flammable materials, coals, and fires themselves was really important, because if a fire in one house got out of control, it could have serious consequences for other dwellings in the same city. So everybody had a collective responsibility to prevent fires.
And similarly, we as laboratorians, along I would say with our manufacturer colleagues, have a collective responsibility to assure assay quality, and I believe that we can do more by sharing information with each other and by combining our efforts and validation. So applying an ounce of prevention so that we can reap many pounds of cure.
Bob Barrett:
Dr. Sterling Bennett is Chair of the Pathology Department at Intermountain Medical Center in Utah. He has been our guest in this podcast from Clinical Chemistry, examining lot- to-lot inconsistencies in the clinical laboratory. I am Bob Barrett. Thanks for listening!