October 2009: Volume 35, Number 10
The Pursuit of Traceability
Lab Groups Push for Greater Harmonization
By Bill Malone
It’s been 30 years since the legendary Norbert Tietz called for establishing a “comprehensive, coherent measurement system” in clinical chemistry that hinged on traceability. Tietz described a hierarchical reference system whereby any lab result could be traced back to a definitive method of undisputed accuracy. Tietz highlighted two benefits of such a system: more accurate results and comparability of results from lab to lab and over time. In the three decades since Tietz’s editorial in Clinical Chemistry, a lot has been accomplished toward this goal, most notably the efforts to standardize cholesterol, glycohemoglobin, and creatinine results. But while European regulators have slowly turned up the pressure on manufacturers to demonstrate traceability for their assays, progress in the U.S. has sometimes stuttered. Now, efforts of regulators, professional associations, and other stakeholders have begun to converge, according to industry observers.
“In clinical chemistry we’ve always had some concept of traceability of our assays, but it’s been somewhat informal,” noted David Armbruster, PhD, scientific affairs manager for Abbott Laboratories. “In the 21st century, we’re moving in the direction of establishing internationally recognized materials and methods that everybody can trace back to so that there’s less variability and method specificity in the clinical lab.”
Laying the Groundwork
Laboratorians are all too familiar with the challenges that can arise when different methods, instruments, and labs produce different results for the same analyte. However, physicians and patients often don’t fully grasp the significance of these variations, leading to a host of problems. It’s this inconsistency that the lab community now hopes to resolve by engaging the concepts of a science known as metrology.
Laboratorians might be less familiar with the concepts of metrology, a distinct but related discipline focused on the science of measurement. “Metrology has been very good about identifying reference methods and reference materials and putting together a formal traceability chain so that you can tie your kit calibrator in your clinical lab back to a reference material and a reference method that are internationally recognized,” explained Armbruster. “The whole idea is that you can then come close to scientific truth rather than a test result that is a relative truth.” Metrology’s obsession with precision lead to the development of a paradigm that seeks to relate all measurements back to a master, absolute definition of a quantity of a given substance using the official SI (International System of Units) units.
Laboratorians are already well acquainted with this paradigm in metrology: well-defined reference materials that serve as the gold standard for measuring an analyte. For these higher order methods and materials to be of value, though, a patient’s test result must be related to the gold standard reference through an unbroken, documented traceability chain. This, say experts, is the essence of traceability (See Fig. 1, below).
Example of Traceability Chain Developed for Serum Cortisol Measurements
Arrows pointing to the left indicate value assignment activity using the procedure; arrows pointing to the right indicate calibration activity using the material.
Source: Clin Chem 2009;55:1067–1075. Reprinted with permission.
Heading efforts to promote such a system is the Joint Committee for Traceability in Laboratory Medicine (JCTLM), an international consortium created in 2002. Sponsored by the International Bureau of Weights and Measures, the International Federation for Clinical Chemistry and Laboratory Medicine, and the International Laboratory Accreditation Cooperative, JCTLM recognizes and publishes reference materials and reference methods for world-wide distribution for calibrator traceability. JCTLM also recognizes and publishes a guide to the reference laboratories that offer these types of reference method procedures. All the information is in a searchable, online database (www.bipm.org/jctlm). This way, manufacturers, labs, or regulators can freely access this information and put together traceability schemes that are approved and acceptable in order to drive harmonization or standardization of assays. Altogether the JCTLM database contains more than 200 certified reference materials (CRMs) for about 130 “measurands” with some 125 reference method methods and procedures.
Many of the reference materials themselves are produced by national bureaus of metrology, like the National Institutes of Standards and Technology (NIST) in the U.S. For example, NIST is responsible for the isotope dilution-mass spectrometry (IDMS) definitive method for cholesterol measurement, as well as its certified-pure cholesterol standard. This method and standard are the linchpins for all cholesterol assays performed in the U.S. Though the CDC cholesterol reference method has served as the practical standard for more than 20 years, this method itself must be periodically assessed against the NIST definitive method. The CDC’s Cholesterol Reference Method Laboratory Network (CRMLN) certifies manufacturers of clinical diagnostic products that measure total cholesterol, HDL-C, and LDL-C.
Part of what’s driving greater visibility of the work of NIST, JCTLM, and other groups in the metrology community is the European Union (EU) In Vitro Diagnostics Directive. As of 2003, this regulatory body required that all manufacturers provide calibrator traceability, when available, for their assay kit calibrators. The directive points to International Organization for Standardization (ISO) guidelines that get even more specific.
U.S. manufacturers have had to step up their efforts to meet the requirements because they produce more than 60% of the devices sold in Europe and they will be blocked from doing business in EU member countries if they do not comply with the directive. “It’s a real necessity for manufacturers to come up with calibrator traceability and to start to adapt to metrology,” said Armbruster. “Europe has driven this, but everyone is grappling with it worldwide. Now we’re able to be more sophisticated and more scientifically correct as we build assays and include traceability chains for calibrators. Anchoring assays to the same reference materials and methods will promote harmonization of laboratory practice around the world. The idea is that it doesn’t matter in which country, what laboratory, or at what time a patient specimen is tested—all results should be universally transferable because we have this metrological infrastructure backing what we do in the clinical lab.”
While the regulatory system in the U.S. is vastly different from that in the EU, FDA officials have voiced strong support for encouraging traceability in new assays and, insofar as their legal framework allows, will be expecting manufacturers to include this kind of information when they file for FDA clearance or approval. In a presentation at the 2009 AACC/CSCC Annual Meeting in Chicago, “FDA and the Quest for Traceability—Moving Regulation into the 21st Century,” Alberto Gutierrez, PhD, the new director of Office of In Vitro Diagnostic Device Evaluation and Safety (OIVD), explained that although the agency wants to push for harmonization, it’s difficult for FDA to make a blanket requirement for traceability like the EU has done.
For class II, or moderate-risk devices, FDA is limited to clearing a new test based on substantial equivalence, comparing it either to a device on the market before May 28, 1976 or to another device cleared since then. “There is a lot of flexibility in the 510(k) process. The bad news is that this is the last thing you would expect to be good for traceability,” said Gutierrez. “[A manufacturer] can choose one that has as bad a bias as theirs has to make theirs look good, and we clear it. So it’s not anchored very well, and you get tests that are all over the place.” Though it is feasible for the FDA to clear a test based on reference methods and materials instead of just comparing it to a predicate test, FDA can’t really force a manufacturer to go this route if they don’t want to, because of the agency’s limits under the law. “To a certain extent, even the efforts we put in don’t match very well with regulations, because we don’t have a good way to force manufacturers to standardize their devices, other than to use the bully pulpit and try to push them along that way,” said Gutierrez. One success the agency has had in this arena are glucose meters, for which a specific, consensus-based ISO standard exists. For these devices, FDA does ask the manufacturer for a comparison back to a lab method.
An area where FDA does have a lot of control is in a test’s labeling. For a handful of tests, such as cholesterol, third-party groups like CRMLN certify a test against a traceable standard. In this case, FDA can make sure that only manufacturers whose tests have been officially certified from this third-party organization can claim so on the label. However, FDA can’t force a company to seek certification. Furthermore, some have voiced concern that the labeling system can actually be a disincentive for progress. If a manufacturer aligns its assay with a traceability scheme, it can be an expensive and time-consuming process to change the label on the assay to reflect the improvement.
In the case of class III, or high-risk devices, which include new tests with no precedent, FDA review requires more data, and is “more inclined to seek information relevant to traceability,” Gutierrez said. However, in this circumstance, the devices are often so new that they don’t yet have traceable methods or materials. Once these tests are on the market, the U.S. system is “somewhat chaotic, in that it relies on proficiency testing to tidy results once they go into the laboratory.”
When it comes to lab-developed tests, Gutierrez painted a bleaker picture. “Quality of labs is variable, standardization is non-existent, and material standards are not available. There is no premarket review, post-market control, and clinical validation is not required. And it’s usually not transparent. This is really the antithesis of what you’d think for traceability,” he said. “Each laboratory has its own tests that they keep pretty close to the vest. Unfortunately, we have seen this practice increase. Even a number of traditional manufacturers have now acquired their own CLIA-certified laboratories and used their lab to put out new tests into the market. Clearly there has been a decision that it is easier to get to market with a CLIA lab than to bring a test to the FDA. This is an issue that we are trying to deal with.”
Despite these challenges, Gutierrez made it clear that FDA has never abandoned the quest for standardization. The agency is a founding member of the Clinical Laboratory Standards Institute and an active member on ISO and the global harmonization task forces. The agency also collaborates with NIST, was an early proponent in the CDC standardization network, and collaborated with the National Cancer Institute and the American Association for Cancer Research on biobanking standardization. However, for FDA to be able to encourage or require traceability and greater standardization, the higher-order reference materials and methods must be air-tight and definitively accepted for an assay by a body such as ISO.
Gutierrez emphasized that while the current FDA review system has been questioned in terms of its impact on quality, at the same time, because of the range of devices FDA reviews and because current regulations are somewhat malleable, “it’s difficult to see how you would do it differently. You could stop everything and start from scratch, but no one has come up with a way that would make sense. I think that a risk-based process works for these devices, and perhaps we can tighten something to improve the system, but it’s unlikely to change in major ways. The science of making things traceable and standardized is very complex. It takes a long time, and the biology is perplexing.”
In addition to the obvious benefits of traceability and greater harmonization of test results, such as more accurate results that are comparable from lab to lab, two other developments in healthcare are now gathering momentum and pushing traceability forward. First, electronic medical records—a major initiative of President Obama’s administration—are built on the idea that a person’s health records can exist independently of a particular hospital or lab, a difficult proposition if lab results aren’t connected to a “comprehensive, coherent measurement system” with uniformity of results, reference intervals, and cutoffs.
Also troubling is the way that a lack of traceability can spoil the much-touted efforts to promote evidence-based laboratory medicine (EBLM). “When you go to the literature, and you start trying to follow the evidence for using a particular analyte to improve healthcare, you’re in a quandary because the results are all over the map,” said Mary Lou Gantzer, PhD, vice president for clinical studies at Siemens Healthcare Diagnostics. “Often, when we try to synthesize the information that’s in the literature, the assays that were used to develop that information are widely disparate in the results they return, making it almost impossible to draw conclusions.”
For lab medicine to be contribute to comparative effectiveness research, also a major priority of the new administration and the focus of millions in new funding to the federal Agency for Healthcare Research and Quality (AHRQ), EBLM needs to be solid. So far AHRQ has had trouble conducting rigorous comparative effectiveness reviews of lab tests because the process requires studies that look at similar patients, across similar care settings, with comparable assays. An assay that doesn’t fit into an overarching traceability scheme will be next to impossible to compare to another when results don’t have a base of consistency (See CLN May, 2009).
Gantzer cited HbA1c as an example of how a lack of standardization can hold back the potential of a test to have a big impact on patient care. Laboratorians have been saying for at least 15 years that HbA1c might be a better way to diagnose diabetes than fasting plasma glucose or an oral glucose tolerance test. “But the problem was that as you looked at studies in the literature, because the assays weren’t standardized, you couldn’t tease out what the cutoff should be for diagnosis,” she explained. “There are a lot of implications that come with diagnosing someone as diabetic, it’s a life-long illness with a lot of concerns associated with it.”
Now, due to the prolonged efforts of the National Glycohemoglobin Standardization Program, along with IFCC and other stakeholders, HbA1c tests have been standardized, with the American Diabetes Association and other groups now considering whether to officially endorse the test for diagnosis of diabetes. “Right there that would be so significant if you could catch diabetes earlier using HbA1c rather than an oral glucose tolerance test or a fasting plasma glucose,” said Gantzer.
These types of successes have inspired Gantzer and other members of the AACC Standardization Task Force to seek higher goals for traceability, harmonization, and when possible, standardization of clinical lab testing. Co-chaired by Gary Myers, PhD, from CDC and Greg Miller, PhD, of the Medical College of Virginia, the task force is organizing a meeting to get stakeholders together and hammer out a way to prioritize which assays should be tackled. The task force aims to focus on the more complex tests for which there is no true reference method available and that can have the most impact on patient care. These assays, referred to as B analytes in the metrology community, include analytes such as troponin, a protein analyte that exists in several different forms and with immunoassays that each recognize a different epitope or antigenic site, Gantzer explained. These types of analytes can be harmonized but not truly standardized the way tests for simpler molecules can, such as glucose. In metrological terms, there is no perfect pure preparation from which to draw the lines of traceability.
Currently, most standardization programs start with someone who has a passion for a particular method and then approaches a group such as IFCC to lead a standardization effort, explained Gantzer. The AACC Standardization Task Force is hoping that the stakeholder meeting, currently slated for 2010, will develop a more systematic approach. “We think it would make more sense for everyone to come together and to look at where there is really a clinical need for harmonization and develop a prioritized list. We also don’t want people duplicating efforts. There’s a lot to be done, and there’s usually not a lot of funding available for harmonization. It’s one of those things where everybody recognizes the importance of it, but it isn’t really one of the things that tends to be at the top of the list for funding,” she said.
Their task won’t be easy. Even though JCTLM and other organizations have paved the way for harmonizing lab tests with the rigorous approach of metrology, most of the simpler, type A analytes have already been catalogued. “While it’s encouraging that the concept is taking root, at the same time, the low-hanging fruit has been plucked,” said Armbruster. “It’s the tough ones, the type B analytes, for which acceptable reference materials and/or methods don’t exist, that’s the problem. The pace of submission to the JCTLM database has naturally slowed down, and it’s going to be a long, slow slog to create and identify those new type B reference materials and methods.”
All hopes for success of the task force’s new efforts center on getting the right people to work together from all settings, Gantzer emphasized. “You really need the clinical lab community, as well as the manufacturers and government, all working closely together in order for this to work. Any one group by themselves is not going to be successful in accomplishing this.”