April 2007: Volume 33, Number 4
CDC Ponders Proficiency Testing Updates
Can Revised Regulations Catch Up with New Tests and Analytes?
By Julie McDowell
It’s been 15 years since the CLIA proficiency testing (PT) regulations were established, and with more molecular and genetic technologies entering the diagnostic testing field every year, many in the clinical laboratory community believe that revisions are long overdue. To tackle these issues, the Centers for Disease Control and Prevention (CDC) has begun contemplating both the present PT regulations and their future by assembling an expert workgroup. At its inaugural meeting in January, the Proficiency Testing Workgroup—comprised of representatives from the clinical laboratory community, PT providers, and federal officials—explored some of the key issues. Moving forward, the CDC is calling on laboratorians to speak out on what changes need to be made, as well as whether current PT regulations are accomplishing their goal—to improve the quality of clinical laboratory medicine.
The current regulations, which were written primarily by the CDC, have been in place since 1992. Judy Yost, Director of Division of Laboratory Services for the Centers for Medicare and Medicaid Services (CMS), the agency that oversees CLIA and clinical laboratory operations, believes the time is right for the expert working group to take a critical look at the program. “Now that we’ve had years of experience and the PT providers have gathered a tremendous amount of accumulated data on target values and analytes that are listed in the regulations, it’s probably not a bad idea to step back and see if the values are appropriate, and if the regulations correspond to current technology and test menus,” she said. CMS has two representatives on the PT working group, which also includes officials from the College of American Pathologists (CAP), the American Association of Bioanalysts, The Joint Commission, the New York State Department of Health, as well as Quest Diagnostics (Lyndhurst, N.J.) and Bio-Rad Laboratories (Hercules, Calif.).
In order for labs to be certified by CMS, they must comply with CLIA’s PT regulations. To do so, laboratories need to enroll in an approved program that provides and processes PT specimens. There are currently 14 CMS-approved state and national programs, although a majority of labs participate in the CAP PT program.
Noncompliance can have significant penalties for laboratories. For example, Joint Commission-accredited labs must submit a plan of action within 10 calendar days of notification by the commission of an unsuccessful PT status. If written plans of action do not meet the accrediting organization’s criteria, the laboratory fails the next two consecutive testing events, or has a history of non-consecutive but repeated unsuccessful PT events, then the commission will request that the laboratory cease testing for at least 6 months, explained Margaret Peck, MS, MT (ASCP), Director of The Joint Commission’s Laboratory Accreditation Program, and a member of the PT working group. The laboratory may not resume testing until the criteria for reinstatement are met and the laboratory receives written permission from The Joint Commission to resume testing.
Because it is such an integral part of The Joint Commission’s accreditation process, PT continues to be a priority for the organization. “Our overarching concern regarding proficiency testing, as it relates to a laboratory quality management system, is that three of the top ten compliance issues cited in Joint Commission-surveyed labs in 2006 were related specifically to proficiency testing,” said Peck.
Time to Update Analyte List
One of the major issues with the current regulations is that the list of regulated analytes is lacking—particularly in the area of genetic and molecular testing. This has raised concern from the clinical laboratory community, particularly those labs performing genetic testing, where PT is not required by CMS. “Right now, there are only 83 tests that are listed in the regulations requiring PT, but there are thousands of other tests that labs are performing,” said Yost. “If a lab is performing a test not listed in the regulations, it needs to come up with a method to challenge the accuracy of the test twice a year, or participate in a provider’s formal PT program.”
In order to update the list, however, the CLIA regulations would need to be revised, a task that could be quite formidable. “The list of regulated analytes is getting stale—everyone recognizes that,” said Robert Rej, PhD, Chair of the PT workgroup and Director, Clinical Chemistry, New York State Department of Health. “However, it’s a big hurdle to change regulations.” Consequently, the workgroup is looking at ways to expedite the revision. One option might be for an amendment to CLIA that refers to a listing of analytes drawn up by expert stakeholders. “That way the regulation wouldn’t have to be changed, and it would reference a list that has the potential to be more dynamic and open to further revision, as well as have input from a number of parties,” he explained.
The PT Workgroup—which is part of CDC’s larger effort to define best practices in laboratory medicine—is currently soliciting comments on the effectiveness of current PT programs, and will report their findings to CDC officials by the end of the year (See Boxes, below). At the November 2006 meeting of the Secretary’s Advisory Committee on Genetics, Health, and Society (SACGHS), Yost presented an overview of CMS’s activities related to genetic testing, which included preliminary thoughts related to an action plan for oversight of this testing. CMS is committed to working with professional associations to develop consensus guidelines on molecular and genetic testing, said Yost, adding that the agency plans to work with professional associations, the CDC, and the FDA to develop additional or alternative PT mechanisms for genetic testing. “We also hope to work with CLSI [the Clinical Laboratory Standards Institute] to come up with some professional standards for genetic testing that we could incorporate in to our surveyor guidelines,” she explained. “We decided to take this route—which is nonregulatory and educational—to help labs do the right thing.”
Defining Best Practices for Lab Medicine
The CDC Proficiency Testing Workgroup is part of a broader effort by the agency to evaluate the current and future directions of laboratory medicine, as well as to develop a process to identify best practices. The PT group will prepare one of three reports, expected to be released by the end of the year. The other two reports will describe the current and future state of laboratory medicine and outline a process to identify best practices. “A workgroup has been organized to develop this process,” said Julie Taylor, PhD, a Senior Science Fellow with CDC’s Division of Laboratory Systems. “In the final report, we will not list best practices for laboratory medicine, but we will be sharing the process to define, identify and evaluate them.” Additional information of this project can be found online.
Demand Outpaces Supply for Genetic PT Material
Even though CMS doesn’t require PT for genetic testing, there are a handful of providers offering genetic and molecular PT programs. One of these providers, CAP, requires PT for labs doing genetic testing as part of their accreditation process. However, many laboratorians are frustrated by the inadequate quantity and quality of genetic PT specimens.
“For quantitative BCR-ABL analysis, CAP sent us specimens with an insufficient amount of RNA in the last couple of surveys,” explained Shrihari Kadkol, MD, PhD, Director, Molecular Pathology, Department of Pathology, at the University of Illinois at Chicago. “When we contacted CAP, we were informed that there wasn’t enough of the specimen available for them to send us a replacement.”
Kadkol would also like to see more targeted PT assays from CAP. For example, his lab recently received specimens for lymphoma and leukemia testing from CAP. The instructions require that all the lymphoma or leukemia tests currently performed in the lab, which include clonality, rearrangement, and PCR-based assays, be run on the specimens. “There is a very specific molecular test for a particular type of a leukemia or lymphoma, and when a clinical specimen comes in, we don’t run all of the tests on the same specimen,” said Kadkol. “Just like molecular infectious disease testing, I would like to see molecular hematology testing be more specific and targeted. That’s how we do actual clinical testing, therefore CAP proficiency surveys should mirror that as much as possible.”
In terms of adequate specimen material for emerging tests, CAP officials admit that there is a problem, and say that they are working with test manufacturers and vendors to develop appropriate PT materials for genetic and molecular testing. But when new assays and diagnostic technologies become available, there are economic obstacles to producing PT materials for these tests, according to Paul Valenstein, MD, FCAP, Director of Clinical Microbiology at St. Joseph Mercy Hospital (Ann Arbor, Mich.), and Vice Chair of CAP’s Council of Scientific Affairs.
“There are several challenges that must be overcome before PT can be developed for new genetic assays,” explained Valenstein. “When an assay is first introduced, only a few labs may be performing the test, which means it may not be economical for a provider to develop a PT program.” This economic hurdle is compounded by the fact that CLIA does not require PT for new genetic assays. “To reduce costs to subscribers, the CAP often bundles several molecular assays together into a single PT survey,” he added. “But customers may not understand these compromises and come to the conclusion that CAP is promoting blanket testing.”
Labs Asked to Provide PT Comments
The CDC’s Proficiency Working Group is planning to write a report and is soliciting stakeholder comments on five questions:
- Are current PT programs meeting expectations?
- How programs could be improved?
- Can the educational value of programs be enhanced?
- Are programs keeping up with changes in the field?
- What would be the result of accrediting PT programs to a recognized international standard?
Comments can be submitted via e-mail, or to the following address:
PT Shakeholder Input
2971 Flowers Road South, Suite 230
Atlanta, Ga. 30341
In addition to these economic issues, there are also technical barriers to expanding PT programs to include new technologies, explained Valenstein. For example, several molecular assays that have recently come to market produce a score using a software-driven algorithm that integrates the results of hundreds of individual assays. Examples of these assays include a new class of tests identified by the FDA as in vitro diagnostic multivariate index assays (IVDMIAs). In September, the FDA released draft guidance on IVDMIAs that outlined the agency’s thoughts on regulating these tests as medical devices (CLN, November 2006, March 2007). IVDMIAs, such as the recently-cleared MammaPrint test developed by Agendia (Amersterdam, The Netherlands), use clinical data to empirically identify an algorithm and then employ that algorithm to integrate different data points in order to calculate a patient-specific result. The MammaPrint test uses microarray analysis to study the behavior of certain breast cancer tumor genes and then predicts whether a patient’s existing cancer will metastasize.
“How we adapt PT for these multivariate tests will be a challenge,” said Valenstein. “The results of these assays depend as much on the algorithm that integrates the results as the 10 or 100 or 1,000 individual probes housed on a chip. We need to think about what we are trying to measure with PT—are we trying to measure how well each individual probe performs, or are we trying to look at the accuracy of the end result?”
When developing PT programs for molecular and genetic assays, it’s also important to ensure that the technology is established, so laboratories don’t get ensnared in technical issues that cause erroneous PT results. “We want PT to detect labs that are not proficient at performing a test, so that we can help them do better, but we also don’t want to raise false alarms by citing labs that are perfectly proficient with clinical specimens but don’t appear so on PT due to technical issues with the PT materials,” said Valenstein. “Not every molecular test or assay is sufficiently evolved to allow us to develop robust PT that reliably and quickly separates proficient from the non-proficient laboratories.”
Towards an International Standard?
Another issue that CDC is seeking input on involves holding approved PT programs to an international standard. The proposal has sparked debate, and reveals a more fundamental question—should proficiency testing be a tool to measure accuracy?
“I believe that proficiency testing should be an assessment of accuracy, and to have five, or more, different target values for the same analyte probably doesn’t do as much good as proficiency testing could,” said Rej. “By having, wherever possible, one target value, or relatively few targets, would really help improve the area, so that everyone is aiming for the same result. With the mobile population that we have worldwide, it’s a problem for the laboratory medicine community when someone goes from one physician or facility to another, because the laboratory tests have to needlessly be repeated because there’s a sliding scale of accuracy and that one laboratory’s normal values or reference ranges don’t quite agree with those of another.”
But CAP’s Valenstein points out that in many cases it isn’t cost effective to use PT as a measure of accuracy. “Proficiency testing was designed to measure proficiency,” he said. “It measures whether the staff in a laboratory know how to calibrate and operate a test system. PT wasn’t designed to measure the absolute accuracy of the results. PT materials, in order to make them suitable for shipping, contain stabilizers that affect different machines—and PT results—in different ways. The fact that different test platforms produce different results with PT materials doesn’t mean there is anything amiss with patient testing.”
In addition to these concerns, many observers believe the time simply isn’t right to move towards global standardization. Elissa Passiment, President of the American Society for Clinical Laboratory Science (ASCLS), cites the recent efforts to establish international standards for reporting of glycohemoglobin as a cautionary tale (CLN, October 2006). While the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) has proposed a new reference standard and method of measuring HbA1c, the move has been met with considerable resistance. In addition, Passiment worries that standardizing PT would mean more expense and effort for the PT providers—which would likely be passed along to laboratorians. “We’re not sure that going global at this point would benefit us,” she said. “Not that we are against it in the future, but the benefit of globalization of PT versus the cost is not evident yet.”
For many tests, it simply doesn’t make sense to trace them back to a national standard, according to Valenstein. Take, for example, troponin assays. When a patient is being tested for a myocardial infarction in the emergency department with a troponin assay, the important analytic question is whether the laboratory’s troponin test system is calibrated to the test’s cutoff or reference range for normal myocardium. In other words, the test should indicate whether the patient is or is not having a heart attack. This question is addressed with PT as it is currently constituted, he explained. “There is no value in comparing this patient’s troponin result to those taken at another facility using a different test system,” he said. “Patients don’t go from ER to ER while they are having a heart attack.” Test results associated with chronic conditions are different, however, which is why CAP’s hemoglobin A1c PT tests are accuracy-based and tied to reference methods, he explained.
“There are some tests where good care requires high levels of accuracy in addition to proficiency, because the test tracks a chronic condition and a patient may move from lab to lab over the course of his or her life,” said Valenstein. “But for many assays, harmonization of results to a standard reference method is not relevant. It adds expense without commensurate value. Development of matrix-free, accuracy-based PT materials that are stable enough to ship is expensive, and in some cases, technically impossible.”
An Essential Quality Tool
Whatever the current issues with PT, many in the lab community believe that it has an important role in detecting quality problems in clinical laboratories. But PT is only one tool—among many—that can be useful in improving laboratory quality. “Although proficiency testing for regulated analytes is a regulatory requirement of CLIA, additional drivers of a successful proficiency testing program should be to promote quality assurance and education, to promote excellence in technical performance, to improve poor performance, and to serve as an ongoing competency assessment tool for those performing laboratory testing,” said The Joint Commission’s Peck.
External PT is an adjunct to internal quality management, not a substitute, emphasized Valenstein. In order to further improve laboratory quality, he believes that PT needs to be expanded into the waived testing domain. He cites studies that show tests which are exempt from PT under current regulations have more analytical problems than tests for which PT is required.
“An over-the-counter pregnancy test used at home may seem harmless enough, but what about the physician who clears a pregnant patient for a barium enema based on the result of an incorrectly performed pregnancy test? I am concerned that many patients are not receiving the benefit of PT because it’s not being used in certain settings,” noted Valenstein. “To me, the problem with PT is not what it is; it’s that we don’t have enough of it.”
For More Information
- Judy Yost presented CMS’s Overview of Genetic Testing at the November 2006 meeting of the Secretary’s Advisory Committee on Genetics, Health, and Society. Her presentation is available online.
- For more information about Using Proficiency Testing to Improve the Clinical Laboratory; Approved Guideline—Second Edition (GP27-A2), please visit the Clinical and Laboratory Standards Institute web site.