TRANSCRIPT
Download Transcript (pdf)
Slide 1: Title Slide
Hello, my name is Anna K. Füzéry. I am a regional clinical chemist with Alberta Health Services and an Assistant Clinical Professor of Laboratory Medicine at University of Alberta. Welcome to this Pearl of Laboratory Medicine on “Managing Reagent Lot-to-Lot Variability”
Slide 2: A Real World Example
I’d like to begin this Pearl with a real world example. BNP is a small peptide hormone that is used in the diagnosis of congestive heart failure. One of my laboratories received a new lot of reagent for this assay and decided to further evaluate it after noticing a large shift in the quality control (QC) results. The table here shows the results obtained for five patient samples using the current reagent lot and the new reagent lot. As you can see, the differences are quite striking. Should the laboratory accept the new lot and use it for patient testing?
Slide 3: Reagent Lot-to-Lot Variation
Reagent lot-to-lot variation may be defined as a change in the analytical performance of a reagent from one production lot to the next. Here we use the term “lot” to mean a batch of reagent produced by the manufacturer under uniform conditions, and passing as a unit through the same series of processes.
While some lot-to-lot changes only affect QC measurements, others may simultaneously affect patient results. The limited commutability of QC materials with patient samples, however, means that the effects of the new reagent lot on the patient results cannot be predicted from the behavior of the QC measurements. In rare instances, for example, patient results with the new reagent lot may show a significant bias compared to the old lot without a similar effect being observed for the QC.
Reports on lot-to-lot variability seem to be much more common for immunoassays than general chemistry tests, suggesting that the former are more prone to this problem. References 3 through 6 at the end of this presentation provide examples of lot-to-lot variability that have been observed for Insulin-like growth factor 1 (IGF-1), Prostate-specific antigen (PSA) , anti-hepatitis C antibody, and human chorionic gonadotropin (hCG) immunoassays.
Slide 4: Many Causes of Lot-to-Lot Variation
The causes of lot-to-lot variation can be divided into those associated with the manufacturing process, those that are due to inappropriate transport and storage of the reagent, and those that occur as a result of laboratory error. For example, a manufacturer may change the process that it uses to produce a particular reagent antibody and this may inadvertently cause slight alterations in the composition or stability of the reagent.
Slide 5: Regulatory Requirements for the Lab
Manufacturers generally attempt to minimize reagent lot variation through stringent manufacturing processes and by conducting lot-release testing prior to the distribution of a new reagent lot. In turn, most clinical laboratories also perform lot-to-lot validation testing in order to verify manufacturer claims and to ensure consistency in patient results. Historically, laboratory validation practices were highly variable and some laboratories did not do any validation at all. CLIA ’88, however, stipulated that calibration verification must be performed whenever a complete change of reagent occurs, unless the laboratory can demonstrate that changing reagent lot numbers does not affect the range used to report patient test results, and control values are not adversely affected by reagent lot number changes. The enactment of CLIA ’88 in 1992 thus led to the inclusion of reagent lot validation requirements in the accreditation checklists of numerous professional organizations. For example, Standard COM.30450 in the All Common Checklist of the College of American Pathologists (CAP) specifies that new reagent lots and/or shipments must be checked against old reagent lots or with suitable reference material before or concurrently with being placed in service.
Slide 6: New Lot Validation – Approach 1
A typical procedure for validating a new reagent lot in the laboratory involves comparing results for a group of patient samples using both the old and new reagent lots. The first step in this procedure is to establish acceptable performance criteria for the new reagent lot. Most laboratories will define a maximum percent difference compared to the old lot that is based on clinical considerations. For a test with a single, well-defined clinical application, this is relatively straightforward. BNP is a good example of such a test. However, for a test that is used for a variety of different purposes, such as hCG, the selection of an appropriate performance goal is more complex and additional considerations including biological variation, professional recommendations, and the analyzer’s analytical capabilities may be taken into account.
Once acceptable performance criteria have been established, the next step is to choose target analyte concentrations. Ideally, the samples should encompass the reportable range of the assay. In practice, it may be difficult to obtain patient samples with certain analyte concentrations and a smaller measuring interval may be taken to represent the consistency of results over the entire reportable range. At a minimum, a laboratory should evaluate samples with concentrations close to the medical decision limits for the test.
Once acceptable performance criteria and target analyte concentrations have been chosen, the laboratory can go ahead and test 5-20 patient samples with both lots of reagent. The acceptability of the new reagent lot is then decided based on the results of the comparison. It is important to note that testing a larger number of samples provides a higher likelihood of detecting a problem with the new reagent lot but test cost and sample availability may sometimes be prohibitive in performing an extensive comparison.
Slide 7: New Lot Validation – Approach 2
Performing this process for all tests and all new reagent lots is very time consuming and may also be cost-prohibitive for many laboratories. In 2006, Martindale and colleagues published a modified approach that relies on past experience with a test along with the results of QC measurements to decide whether patient comparisons should be performed prior to introducing a new reagent lot.
Martindale and coworkers divide tests into three groups. The first group includes tests that measure very unstable analytes, use highly unstable reagents, are very laborious or time-consuming, or leave little or no specimen after initial analysis. Examples of such tests include ACTH, bile acids, fecal fats, and tissue copper. The recommended approach for this group is to collect four measurements per QC level with the new reagent lot. Troubleshooting follows any out-of-range results or the discovery of QC shifts exceeding one standard deviation.
The second group of tests includes those that rarely show lot-to-lot variation for either QC or patient results. Patient comparisons should only be performed for tests in this group if the initial QC measurements violate error rules or if a persistent shift is detected in the 48 hours following a lot change.
The third group of tests includes those that in the past have demonstrated large lot-to-lot variations in patient results. hCG and troponin are good examples of tests in this group. The recommended approach for this group is to analyze 10 patient samples with the old and new reagent lots regardless of the results of initial QC measurements.
Slide 8: New Lot Validation – Approach 3
Recognizing the importance of having a robust and standardized process for reagent lot validation, the Clinical and Laboratory Standards Institute (CLSI) also released a guidance document in late 2013 for the user evaluation of reagent lot-to-lot variation. The document provides a simple, practical, and statistically sound protocol to evaluate the consistency of patient sample results when a new analytical reagent lot is to replace a reagent lot currently in use. The document is intended for use in the clinical laboratory and is designed to work within the practical limitations that exist in this environment. It provides guidance on establishing acceptable performance criteria, choosing a desired statistical power to detect differences, determining the number of samples to be tested and acceptable sample types, as well as designing a process for evaluation of lot-to-lot differences.
Slide 9: These Approaches Have a Major Flaw
I hope that you can now see that great strides have been made in developing appropriate approaches for validating new reagent lots since the implementation of CLIA’88 in 1992. However, the three approaches I just outlined all have a major limitation – can you guess what it is?
That’s correct! All three approaches have limited power to maintain consistency in patient results over time.
Slide 10: Time-Related Drift in IGF-1 Results
Algeciras-Schimnich and coworkers published an elegant study in 2013 that clearly illustrated this problem. Over the 2-3 years prior to their study, their laboratory was contacted with increasing frequency about reported serum concentrations of IGF-1 that appeared to be spuriously high. In many cases, experienced clinicians reported that the increased IGF-1 concentrations were not associated with signs or symptoms of growth hormone excess. As a result, Algeciras-Schimnich and colleagues decided to reexamine their IGF-1 reagent lot-to-lot comparisons for the past five years. They found an increase in the proportion of IGF-1 results above the upper reference limit and a decrease in the proportion of results below the lower reference limit over time despite having had acceptable results for all of their individual lot-to-lot comparisons.
Slide 11: Moving Averages
In addition to periodically evaluating the cumulative effects of lot-to-lot variation, monitoring patient results over time on the basis of moving averages is an additional means of detecting such long term drifts. The concept of moving averages for the laboratory was first proposed in 1965 by Hoffman and Waid, and monitors in real-time the average patient value for a given analyte.
As a simple illustrative example, consider the prealbumin results noted here. We first calculate a mean using the first five patient results and plot this on a chart. Once a new result becomes available, we move the averaging window by one to include the newest result and to drop the oldest result. The mean is then recalculated and also plotted on the chart. This process is repeated ad infinitum and in real time as results become available. In the absence of systematic error, the average patient value stays close to the historic average. In the example provided here, however, the moving average shows a clear shift that should be investigated if it hasn’t been already identified using traditional QC procedures or reagent lot evaluations.
Moving averages can be performed either at the level of individual laboratories or by the manufacturer through a centralized, real-time repository for patient results. While many laboratories are now implementing patient moving averages, a manufacturer-based patient result repository is yet to be created. This latter system is particularly attractive because it could flag questionable reagent lots within hours after entry into the market and would also benefit laboratories with low testing volumes that don’t benefit from the local implementation of moving averages. For more information on moving averages, consult references 7 through 9.
Slide 12: A Real World Example
In finishing my presentation, I’d like to revisit the real world example that I showed at the beginning of my talk. In my lab, we have not yet implemented moving averages and thus, we rely on patient comparison studies to identify clinically significant shifts associated with new reagent lots. The differences we observed for patient results between our current lot and new lot of reagent were quite striking and on the borderline of acceptability based on our predefined total allowable error criteria. While the clinical management of most patients would not have been altered with this new reagent lot, we remained worried about the contribution of this lot to long term drifts in patient results. In the end, we decided to contact the manufacturer and requested a different lot of reagent.
Keep in mind though that an alternate lot of reagent may not always be immediately available. Some alternate options that you may consider in this case include sending out the testing to a nearby laboratory or removing the observed bias by applying a mathematical factor to the results that are obtained with the new reagent lot.
Slide 13: References
Slide 14: Disclosures
Thank you for joining me on this Pearl of Laboratory Medicine on “Managing Reagent Lot-to-Lot Variability.” My name is Anna Füzéry.