A Jumpstart for Harmonization
Lab Leaders Meet to Advance a New Agenda
By Bill Malone
When the results of different assays for measuring the same analyte do not agree, clinicians face serious hurdles toward the goal of safely diagnosing and treating patients. Although some tests like cholesterol can be reliably traced back to a reference method and standard reference material, many—especially newer biomarkers—lack a reliable reference measurement procedure. This dilemma is nothing new to laboratorians and manufacturers. In fact, more than 30 years ago stakeholders in the lab community met to address the same issue at a conference sponsored by the Centers for Disease Control and Prevention.
Now, a renewed effort to build a coalition of stakeholders to tackle the problems with harmonization has begun. On October 26 and 27, AACC convened a conference hosted by the National Institute for Standards and Technology (NIST), inviting laboratorians, clinicians, metrologists, manufacturers, and regulatory officials from around the world with expertise in harmonization to step up the pace of progress. The conference organizers charged participants with identifying those analytes most in need of assay harmonization and creating new processes to deal with them.
Moving forward with harmonization is especially critical in an era where evidence-based medicine and practice guidelines increasingly rely on diagnostic tests, emphasized W. Greg Miller, PhD, a co-chair of the conference. “Results from many lab tests are not uniform, and yet clinical societies are publishing recommendations for using lab tests assuming that results are all the same,” he said. “So we know that there is a gap between actual laboratory performance and what our clinical colleagues are expecting from us. What we want to do now is narrow that gap and build a coalition of stakeholders that can develop the technical solutions we need.” Miller is a professor in the department of pathology and director of clinical chemistry at Virginia Commonwealth University in Richmond.
Numerous international organizations, such as the Joint Committee for Traceability in Laboratory Medicine (JCTLM) have worked to foster greater traceability in laboratory medicine and establish well-defined reference materials linked to primary, pure substances that manufacturers can use to standardize their tests. While some notable success stories exist, progress has been slow to develop commutable reference materials, given the complexity of measuring analytes in human sera or plasma. For a large number of lab tests, commutable reference materials that would allow clinicians, laboratorians, and patients to reliably compare results from different laboratories are lacking.
A Growing Dilemma
When test results vary from lab to lab, there is little doubt that the efforts of both clinicians and researchers suffer. Physicians may be hampered in their ability to make correct diagnoses, and the vital ability to compile and compare data from multiple clinical trials is lost, Miller noted.
At the same time, the influx of proteomic and molecular tests has made the classic model of standardization unattainable due to the near impossibility of isolating an absolutely pure substance to which reference methods can link back, such as can be done for a simple molecule like glucose. Moreover, many immunoassays produce results by recognizing different epitopes on the target analyte, and these require a different set of tactics to make different assays more uniform.
Conference participants also considered issues with existing reference materials that don’t work well enough to truly harmonize the various assays on the market. For example, the World Health Organization (WHO) recently developed a panel of reference materials that are highly specific to a subspecies of human chorionic gonadotropin (hCG). Researchers using the WHO panel have investigated clinical conditions and found that some assays will measure one subspecies more than another, and some don’t even recognize some of the subspecies, a typical problem that participants wrestled with. The issue here is “how we identify what we’re really measuring,” said Miller, a problem that can be hard to help physicians understand. “Basically, you tell the physician, here is a result for PSA, for hCG, or for troponin, and the physician expects the numbers to be interchangeable, no matter what lab, what assay.”
In other cases, existing reference materials have simply fallen too far behind current technology to meet today’s needs (See Box, below). “WHO has produced a number of reference materials for hormones in particular, but most of them are 15 to 25 years old,” Miller noted. “At that time they didn’t have recombinant technology, and they didn’t know as much about matrix-related biases and interactions. So a lot of those products—while they were improvements at the time—simply are not good enough today. One goal of this conference is to have a positive influence on the next generation of those types of secondary reference materials.” To deal with this multifaceted problem, the conference featured four breakout groups, each taking on separate aspects of the challenges facing harmonization (See Box, below).
Manufacturers Face Dilemma Without Harmonization
As of 2003, the European Union In Vitro Diagnostics Directive requires that all manufacturers provide calibrator traceability, when available, for their assay kit calibrators in accordance with International Organization for Standardization (ISO) standard 17511. Ideally, this means that each manufacturer’s assay can be traced back to a primary, pure reference material so that it gives a standardized, true result.
However, for many tests, no such pure standard exists, and assays are instead harmonized to a lower level of the traceability chain, for example to a World Health Organization (WHO) material if one is available. If a WHO or other reference is lacking, as is often the case, it means a manufacturer must develop internal reference materials to meet ISO requirements. As a result, many tests do not agree from lab to lab, explained Don Powers, PhD, a consultant on quality management systems in Pittsford, N.Y. Powers was invited to explain the current situation with the ISO standard to the recent AACC-sponsored harmonization conference, Improving Clinical Laboratory Testing through Harmonization: an International Forum, in October, hosted by the U.S. National Institute for Standards and Technology (NIST).
“For non-standardized or non-harmonized analytes, manufacturers are not off the hook. In fact, they’re put in a more difficult position because they have to establish their own way to standardize assays, and many of them just don’t know how to do it,” Powers said. “The problem is that every time an organization like WHO creates a standard material by consensus, when they run out of that and have to create another lot, then it’s a whole new ball game and there is usually a major shift in the values. WHO does try to make materials last a few years, but with some fragile biological materials, it’s not very achievable, and there are often several lots in existence at the same time. This is one of the challenges that this harmonization meeting addresses.”
Molecular tests face even greater and more complex challenges, noted Angela Caliendo, MD, PhD, who explained to conference attendees the challenges in achieving traceable results for nucleic acid-based procedures which were included in the scope of the conference. Caliendo is professor of pathology and laboratory medicine at Emory University School of Medicine in Atlanta, Ga.
AACC, NIST, and other stakeholders involved in the conference are hoping that as the ISO standard begins a periodic review process, the proposals put forward at the conference for dealing with analytes that are difficult to harmonize will help shape any updates to the standard. “The timing works out well here, because then the output of this conference can fit into the revision of the standard,” Powers said. “The manufacturers will be concerned with technical barriers to developing realistic reference materials, but I think they’re all supportive of it, because it’s a lot more work if they’re out there on their own.”
Developing a Game Plan
One of the conference organizers’ main objectives was to develop a process for building agreement among stakeholders to decide which analytes demonstrate the greatest need for harmonization and how resources should be funneled towards top priorities.
To meet these targets, conference attendees agreed on a proposal featuring both an organizational structure to review analytes for harmonization and an objective checklist to help multiple stakeholders weigh in on analytes in a uniform fashion. The proposal centers on a Harmonization Oversight Group that will reside under the auspices of a yet-to-be-determined organizational host. This oversight group will serve as a clearinghouse for collecting and disseminating information, a hub for synthesizing advice from specialty societies and other stakeholders, and as center to connect resources with ongoing harmonization projects, as well as the critical follow-up to see the project through.
A post-conference workgroup will continue efforts to set up the group, while a second workgroup will flesh out the details of the harmonization checklist. The preliminary checklist proposed at the conference includes seven domains, each with a handful of questions aimed at pinpointing the appropriateness of an analyte for harmonization. Domains include questions such as: What is the clinical need for analyte measurement? Are there evidence-based clinical practice guidelines that include the analyte? Is the analyte measured in clinical trials? The conference proposal calls for each set of questions to build up to a numerical score, so analytes may be easily compared and listed by priority. This way, numerous clinical societies and other interested parties can contribute guidance in a consistent way.
Beyond this initial list-making and prioritization process, participants also emphasized that more needs to be done in terms of new assays and those under development so that harmonization becomes a universal goal even in the early stages of characterizing a biomarker. Getting laboratorians to “think harmonization” when investigating a new test can help set the stage for successful and early harmonization, according to Robert Christenson, PhD, chair of the conference break-out group that considered prioritizing analytes. Christenson is professor of pathology and medical and research technology, and director of clinical chemistry laboratories at the University of Maryland Medical School in Baltimore.
Christenson offered an example to illustrate how approaches to biomarker development can either hinder or help an assay’s chances for harmonization. “Imagine that an investigator at a university comes up with a new marker for heart attack,” he said. “She publishes a study using banked samples and indicates that when the enzyme-linked immunosorbent assay reaches a signal of 0.10, the test is positive for myocardial infarction, and that anything below 0.10 is negative. If a company obtains the rights to the test and conducts clinical trials, they may initially use the signal-based cutoff to examine performance; however they are likely to eventually assign quantitative values to the marker at some point. Thus, instead of using a signal of 0.10, they translate this level to a cutoff of, say 10 nanograms per milliliter. The company’s trials include suspected heart attack patients, to assess performance. But the problem then arises, how can you combine and compare datasets before and after the switch from using a signal based cutoff and quantitative [ng/mL] cutoffs without some strategy for harmonization of results? It can get very complicated and, for example, you don’t really know for sure what reagent lot number that the first investigator was using versus the reagents the company used in studies.”
In addition to a tool for prioritizing analytes for harmonization that are already in use in clinical labs, participants suggested that a ‘harmonization checklist’ could be used for new biomarkers by journal editors, similar to the checklist used in the Standards for the Reporting of Diagnostic Accuracy Studies (STARD) initiative. One notion is that before a paper could be accepted for publication, journal reviewers could give feedback to the authors and ask them to consider how the assay could be harmonized for combining data in the future. “There must be a way of connecting all studies that are conducted using the marker, so that you could perform meta-analyses and utilize all the evidence-based medicine tools to build a robust evidence base that will better estimate the true characteristics for a marker,” Christenson said. “It could be very powerful to use a harmonization strategy to monitor biomarker performance as a marker evolves.”
Improving Clinical Laboratory Testing through Harmonization
An International Forum
At the October 26 and 27 conference, participants broke out into workgroups to address four major components of harmonization.
Prioritizing analytes for harmonization
Robert Christenson, PhD, University of Maryland Medical Center, Chair
Stephen Kahn, PhD, Loyola University Health System, Co-Chair
Gap analysis of the current status of harmonization
Catherine Sturgeon, PhD, Royal Infirmary of Edinburgh, Chair
Micha Nübling, PhD, Paul-Ehrlich Institute, Co-Chair
Technical processes to achieve harmonization
John Eckfeldt, MD, PhD, University of Minnesota Medical Center, Chair
David Bunk, PhD, NIST, Co-Chair
Assessing the success of harmonization in labs
Stanley Lo, PhD, Medical College of Wisconsin, Chair
Ralf Schoenbrunner, PhD, Life Technologies, Co-Chair
W. Greg Miller, PhD,
Virginia Commonwealth University
Gary Myers, PhD, AACC
Mary Lou Gantzer, PhD, Siemens Healthcare Diagnostics
Linda Thienpont, PhD, Ghent University
Stephen Kahn, PhD, Loyola University
Ralf Schoenbrunner, PhD, Life Technologies
Richard Flaherty, AACC
Matching Needs with Solutions
Although technical challenges will need to be worked out analyte by analyte as interested parties come forward with resources to tackle them, the conference breakout group assigned with considering technical solutions to harmonization outlined a framework for considering any new harmonization project.
Before a new reference material is developed, the first step should be an assessment study to evaluate the current degree of measurement equivalence and determine whether harmonization is even possible or clinically appropriate, explained Linda Thienpont, Pharm, PhD, a member of the organizing committee of the conference. “It came through very clearly that it is very important that we start by demonstrating the current harmonization status of the assays for a certain measurand. And this should be done by a method comparison with commutable single-donation serum samples,” she said. “In this way you really can show what the current harmonization status is, and how well the different assays correlate with, for example, the all-procedure trimmed mean.” Thienpont is a professor in analytical chemistry and director of a mass spectrometric reference laboratory in the faculty of pharmaceutical sciences at Ghent University in Belgium.
Another seemingly basic but critical step is to reach consensus on what is actually being measured, referred to as the measurand, Thienpont emphasized. “For example, if one wanted to harmonize insulin measurement, it would make a huge difference if one defined human insulin as a unique polypeptide on the basis of its amino acid sequence, versus defining insulin as the molecule or the family of molecules recognized by polyclonal immunoassays. This is why we must come together and form a consensus,” she said. “The difference in this example is that in the second type of definition, it not only includes the unique insulin polypeptide, but potentially also the major insulin fragments. Some scientists insist on using this definition because most of their clinical experience has been acquired with such polyclonal assays. Until we have a consensus on the measurand, it makes no sense that we try to harmonize assays.”
Finally, given the current problems with many WHO reference materials, commutability was underscored as an essential focus. “This means that reference materials must behave with different assays in a similar way as patient samples,” Thienpont said. “That’s the conditio sine qua non for harmonization. In spite of the fact that many assays claim standardization to the same WHO standard, they do not give results that are in agreement with each other, and that is a big problem. That is why we put emphasis on native samples as a vehicle for harmonization.”
Miller reminded the conference attendees that whatever technical solutions were developed, the processes and procedures needed to be realistic and practical. “The emphasis at the conference is on practical solutions, because you could develop a solution that’s so expensive, that despite the fact that it’s good, nobody can actually do it,” he said. “So we need to discuss ways to compromise slightly on the performance expectations and make sure that we’re doing what’s necessary to meet the clinical care requirement, and not go overboard. There can be a tendency among laboratorians to want to make the method just as good as possible. But it only really needs to be as good as the clinical requirement. At the end of the day, we’re looking for practical, implementable solutions.”
Making it Work in the Lab
Since most tests that have been highlighted for harmonization are in vitro diagnostic test kits sold by manufacturers, a lot of the technical efforts will have to be accomplished on the manufacturer’s side of the equation. However, laboratorians using those tests will play an essential role in maintaining harmonization and helping manufacturers implement solutions.
One theme at the conference was the central role of proficiency testing (PT) or external quality assessment (EQA). “PT and EQA data will help monitor for harmonization after it’s been achieved,” said Stanley Lo, PhD, a co-chair of the conference break-out group that considered strategies for sustaining harmonization in labs. “For instance, the College of American Pathologists provides proficiency testing with human material, and using commutable materials allows the operator to determine the status of the method in their laboratory.” Lo is associate professor of pathology, technical director of clinical chemistry, POCT, and biochemical genetics, and director of the Reference Standards Library at the Medical College of Wisconsin in Milwaukee.
According to Lo, it’s up to labs to monitor their PT data and communicate with the manufacturer of a test if they can’t find a fix. “If a lab is having problems with PT and can’t fix it within their own process, they need to ask the manufacturer for help and then try to work together to resolve it.”
Just as in the original harmonization process, commutability will be an issue in maintaining any harmonization scheme, Lo emphasized. His breakout group underscored the fact that manufacturers and clinical labs should participate at least annually in a PT or EQA program that uses commutable materials, although such programs are of limited availability at the present time. Ideally PT and EQA materials will be prepared in agreement with the Clinical and Laboratory Standards Institute Guideline C53-A, “Characterization and Qualification of Commutable Reference Materials for Laboratory Medicine.”
As post-conference workgroups form to formally address the goals and process set out by the October meeting, Miller stressed that AACC is dedicated to seeing the work of the conference bear fruit in the future. “AACC is committed to provide the organizational structure for the follow-up,” he said. “This program will have to be cooperative, and various societies will have to take on leadership roles for each harmonization project, so one of the goals of the conference is developing the sustained cooperation among societies and among countries that we need to maintain the process. We have high hopes.” The organizing committee plans to submit the conference proceedings and recommendations for publication and comment in 2011.