June 2008: Volume 34, Number 6
CDC Report Assesses the State of Proficiency Testing
What’s Being Done to Turn it Around?
By Phil Kibak
The CLIA regulations that established proficiency testing (PT) as a part of the assessment and accreditation of clinical labs in the U.S. are now more than 15 years old. Today, PT results are recognized as an indication of a lab’s competence, but despite years of experience with PT there have been no comprehensive studies that establish PT as an indicator of the overall quality of laboratory operations. Now, an expert panel assembled by the CDC to examine PT practices has issued its report, “Review of Proficiency Testing Services for Clinical Laboratories in the United States—Final Report of a Technical Working Group,” which assesses PT relative to the objectives of CLIA for all specialties and subspecialties of testing other than cytology. It lists 21 recommendations for improving PT that deal with changes in data collection and analysis, process improvement, process evaluation, and education that would change current laboratory practices.
“If some of these suggestions are adopted by regulating agencies, that could certainly have an impact on PT programs and potentially on labs,” said Connie Laubenthal, Director of the Medical Laboratory Evaluation for the American College of Physicians. “What will likely have the greatest impact will be CMS expanding the PT regulations to include additional analytes.”
But updating the analyte list to reflect current practices is not the only change CMS is considering, noted Judy Yost, MA, MT (ASCP), Director of the Division of Laboratory Services for CMS. “The technologies have changed, and the test menus have changed. We also need to examine the target values and grading schemes that have been in place since 1992 and determine if they are still appropriate or need to be adjusted.” Based on these recommendations, CMS already has contacted the CDC and PT providers to talk about exploring the options later this year.
The Time Is Now
“This review could have been done at any point, but it would have been useless if CMS were not ready to make regulatory changes,” explained D. Joe Boone, PhD, Acting Director of the CDC’s Division of Laboratory Systems, and the project overseer. “The fact is, there needs to be a valid reason for making regulatory changes. CMS recognizes, for example, that some of the tests included in the requirements for PT are not only outdated, but also not widely used anymore. The agency wants to work with us to update the requirements, but what we put in place 20 years ago actually has worked pretty well.”
But, Yost added, it’s not the sort of thing that will result in changes overnight. “There’s a lot of analysis that needs to be done, we need to collect a lot of data about performance, and we need to determine a method to select the tests that are going to be listed. In addition, CMS will work closely with the accrediting organizations, the PT providers, and the CLIAC. The reality is that there’s a tremendous amount of work that needs to be done before we ever set pen to paper.”
What the Panel Examined
The CDC charged the working group with four primary objectives and five main tasks. “We asked the panel members, who represented PT users, PT providers, and representatives of accrediting organizations, to examine PT from a number of different angles,” said Boone. “And not all of the angles we asked them to consider were regulatory in nature. For example, we asked them to evaluate whether the educational content of current PT programs could be improved. We also wanted them to report their findings and opinions on whether PT as it is currently practiced covers the latest technology. There truly was a range of issues.”
Recommendations of the PT Working Group
Although implementing all of the final 21 recommendations presents many challenges, the PT Working Group suggests that CDC, CMS, and the lab community must determine whether the cost is worth the potential benefits. Here are a few highlights.
- CDC should develop a process to collect, consolidate, analyze, and summarize PT-related complaints received by CMS, state health programs, accrediting organizations, and PT providers.
- PT providers should publish scientifically credible PT results regularly in peer-reviewed journals.
- CMS or another party should work with Congress to develop a process to assure that all clinical labs, including those that perform waived tests, participate in PT.
- CMS, perhaps in partnership with CDC and CLIAC, should develop a process to periodically review, update, and publish CLIA PT program requirements, including the list of regulated analytes and allowable limits.
To read the full report, “Review of Proficiency Testing Services for Clinical Laboratories in the United States—Final Report of a Technical Working Group”, go to the Future of Lab Medicine Web site.
Current PT Practices and Limitations
CLIA regulations set forth requirements for PT programs, including a list of 83 regulated analytes or subspecialties. For most of these, regulations stipulate that PT programs must include at least three events annually, each of which must include five samples. A “passing” PT score for tests—other than transfusion medicine, which requires a score of 100%—is defined as 80%, or 4 of 5 test results falling within a specified range of analytical precision.
“PT was initially developed as an educational process, an external way of validating that your internal quality assurance processes were working,” said Laubenthal. “But PT has changed quite a bit since the implementation of CLIA in 1992. It changed to a regulatory tool that became very stressful for the labs—if a lab failed twice in a row it could be told to stop patient testing. And CLIA changed the rules, too. For example, before CLIA a lab could run a test multiple times and get an average reading and use that for quality assessment. But CLIA said you had to run the PT sample only the same number of times you would run the test for a patient, and in a number of cases that’s one time.”
The panel noted in the report that PT in its current form may not be effective in assisting some laboratories in detecting and correcting problems affecting quality in a timely way. Smaller labs and hospitals may lack Internet access, necessitating other, slower ways of reporting PT results, such as mail, courier, or fax. Not only can this delay the reporting of results, but it also increases the possibility of PT failures caused by transcription or other clerical reporting errors.
“Up until very recently there were more labs than you would have thought that lacked Internet access, and this was often by design,” noted Robert Rej, PhD, Director of Clinical Chemistry and Hematology for the Wadsworth Center, New York State Department of Health, and Chair of the PT Working Group. “Certainly there were concerns about infection by computer viruses, but it was more often to protect against hackers breaking in and gaining access to confidential patient records. But I think many of those concerns have now been addressed.”
Some laboratorians worry that PT samples create problems for the lab because they are in tubes that differ from patient samples, have different types of labels, and often must be “flagged” in the laboratory’s electronic information system to avoid creating a patient record.
One of the most conspicuous limitations of PT, according to the expert group, is the exemption of waived testing from PT requirements. “There’s been quite a bit of concern among members of the clinical laboratory community that there are some tests—espe-cially in this domain—that need more scrutiny than they’re getting,” said Boone. “It’s not necessarily that the products perform poorly, but rather a question of whether the people performing the tests have adequate skill. PT not only gives an indication of whether the product performs properly, but also whether the analyst can do the job.”
Improving PT Efficacy
Despite the broad consensus that PT is effective in evaluating and improving laboratory performance, the expert group determined that the industry needs planned, prospective studies that can directly assess the link between PT performance and performance in routine testing of patient samples. Such examinations could track performance of the same group of laboratories over time to identify predictors of PT performance and effective strategies and practices for integrating PT performance into a broad quality management system. These studies also could help distinguish true improvements in PT performance and error rates in routine testing from artificial improvements due to dropouts by poorly performing laboratories or improved skill in test-taking over time.
Specific performance data also could be used to track marginally performing laboratories as well as those that fail. Any new study, the panel pointed out, should determine the reasons for failure. They also suggested requiring laboratories to self-investigate and report specific causes of unacceptable results. Future studies also should investigate links between internal QC and external PT to determine the importance of establishing ranges to predict a laboratory’s performance.
The panel noted, too, that fast turnaround time of PT reports is imperative if the tests reveal a weakness in the laboratory’s operations. Providing results electronically would substantially reduce processing time and cost, and would benefit providers and participants.
“Some PT providers, such as the State of New York, are actively encouraging electronic reporting by PT users and plan eventually to require it. All PT providers should follow this example and move expeditiously toward requiring total electronic reporting,” they wrote. Rej noted that nearly 100% of clinical laboratories in the New York PT program now use electronic submission for all data and already receive electronic return of PT results for most areas of infectious disease testing. The NY program is aiming for 100% electronic return of all PT results within the next 2 years.
In addition, laboratories and PT providers should have a system allowing them to communicate directly with one another. However, the experts realize such a system would be difficult to implement as it would require an intermediate standardized platform.
Educational Value of PT
When optimally employed, PT provides regular, periodic learning experiences for laboratory personnel. Lab directors often use PT samples and PT survey reviews as opportunities to involve, educate, and motivate their clinical staff members. PT also helps reassure lab personnel that their work is accurate and an asset to patient care.
The panel noted that “laboratories must balance the benefits of participating in educational PT and the costs of such activities when they are not required by regulations.” Educational PT represents one of the best means of improving laboratory quality, they added. “PT programs should go beyond the important role they play in the regulation of laboratories. They should also serve as a vehicle to educate and prepare laboratories, not only for new technology and specialized areas of testing, but also for future PT events that will assess their capability to respond to needs that have not yet been clearly defined.”
“But laboratory personnel are stretched further and further and seem to be doing more and more with less and less,” noted Daniel Edson, PhD, President of the American Proficiency Institute (Traverse City, Mich.) and a member of the expert panel. “Everyone agrees education is a good thing, but in a working lab how much time and money can you really put into that?”
PT is rarely addressed in U.S. medical technology classrooms, a practice the panel said should be remedied. They described the experience of students in The Netherlands and how they participated in PT to prove and improve their skills. In this program, students received chemical, environmental, and clinical samples for testing. Their results were methodologically and statistically judged by professional PT organizers. The final results presentation occurred at an annual symposium where awards were presented to the best performing student groups (Fresenius J Anal Chem 2001; 369(7-8):741–743).
The Challenge of New Technologies
In their evaluation, the panel members considered whether PT programs cover the latest technology. “The response to that question was decidedly mixed,” noted Boone. “PT providers will not put out materials where there are insufficient numbers of users. They’ll wait until there are enough users of a certain technology to make it economically worthwhile for them to add PT materials. Molecular genetic testing for rare genetic conditions is a good example, where numbers of users may be too small. However, if you think of it in terms of the technology, particularly with microarrays, which incorporate thousands of tests on one chip, it’s impossible to put every one of those tests under PT scrutiny every time. But a way to work around that is to do random checks of various tests on that chip, and then rotate the tests until all on that chip have been examined.”
Clearly there are growing unmet PT needs where technology is concerned. Gaps in PT availability also exist for tests in which analyte stability is a frequent issue, and for tests of analytes in specific matrices such as hair, saliva, or sweat.
Programs in other nations may provide a partial solution to this problem. Many such programs outside the U. S. keep pace with advancing technologies, the panel reported, and fill gaps in existing domestic PT programs. For example, a program of the Centre de Toxicologie du Quebec (Canada) offers surveys for eight trace metals in blood and serum and 10 metals in urine.
The panel also discussed whether accrediting U.S. PT programs to an international standard would increase the quality or uniformity of domestic programs. Rej explained that the working group was concerned over ISO (the International Organization for Standardization) Guide 43—soon to be ISO 17043—that addresses PT programs. This is the main international consensus document referenced for PT and presents guidance on PT for a broad variety of testing and calibration laboratories.
It also is notable that CAP, for example, has started to adopt an ISO (the International Organization for Standardization) framework for laboratories. Earlier this year, CAP announced a new program to accredit U.S. labs to conform with ISO 15189:2007, an international standard gaining acceptance in other countries. The program will be available to interested medical labs during the fourth quarter of 2008.
“This ISO standard was developed in 2003 and revised in 2007,” said Boone. “It specifies technical and managerial requirements for medical labs. Quite a few of the elements it contains are in CLIA, but organized in a different way. CAP is trying to get this as an alternative to or supplement to the CLIA requirements, but it’s up to CMS as to whether that would be acceptable.”
But in the U. S., complying with CLIA is mandated by law. Labs would have the option of adopting ISO 15189:2007, but compliance with the ISO standards does not exempt labs from CLIA. In addition, the extent to which U.S. PT providers meet ISO management system requirements is currently not known.
“But there’s already a regulatory movement toward this,” said Laubenthal. “The most recent set of quality assurance regulations put out by CMS were actually based on international standards. We’re also seeing a movement in U.S. blood banking toward becoming ISO certified.”
“From a PT provider viewpoint, we don’t see the advantage to this,” said Edson. “We feel we’re already marching to the high standards of CLIA, and we don’t know that an international standard would be better for U.S. labs.”
A difficulty involved with adopting an international standard is that PT providers and PT suppliers would have to implement changes to their current operations to meet the requirements of that standard. A PT provider would need to establish a clearly defined management system, have fully documented standard operating procedures and policies, and would be subject to on-site assessments. While CLIA currently relies upon reviewing documentation to evaluate PT provider compliance, it does not require on-site assessment. PT providers also would have to ensure that their subcontractors meet the same international standard. Increased start-up costs might also be a factor for PT providers to consider, but the panel noted that many organizations that have adopted ISO quality management practices have documented cost savings as a result of increased efficiency, reduced waste, and improved customer satisfaction.
However, the panel also noted that most U.S. PT providers have substantial experience and their technical operations, both in-house and subcontracted, are largely consistent with international requirements. “Broader adoption of an international standard will depend on the degree of compatibility with our statutes,” noted Yost. “Certainly, it will be something we will evaluate during the overall design and development of these regulatory changes.” All changes to the PT regulations will be proposed in a Federal Register Notice of Proposed Rulemaking (NPRM) and comments will be solicited.
A First Step
“The information in this report is preliminary, but it’s important and useful information that will help us in the process of revising PT, along with numerous other resources,” said Yost. “It’s important that folks know we have recognized that the process needs to be updated and that we need to re-evaluate the requirements of the PT providers and for the labs to reflect common practice. But we have a lot of work to do before we can use any of the information in the report.”
Certainly, some of the recommendations—especially those that don’t require regulatory intervention—can be completed sooner than others, and there are a number of specific actions in the report, such as developing ways to provide faster turnaround time, that the PT providers can take up on their own without any intervention from CMS, noted Rej. “What I would also really like to see is more effort on the part of PT providers to mine their database and publish reports in the peer-reviewed literature to show the effectiveness or non-effectiveness of PT in a particular discipline or with a specific analyte. It would be of immense value to the laboratory community to illustrate trends or show actual data backing up the position that, say, one particular technology gives more variable results than another. Overall this document will be the focal point for all future discussions on the best directions for PT.”
But there always are limitations when a group this small reviews a field as large as PT, explained Boone. “The people on the panel offered their expert opinions, because in many cases we had no data. But their opinions and recommendations are appropriate. It largely is a matter of how feasible the changes appear to be and the effort that may be required to implement them. All of the recommendations may not be feasible for implementation at the present time, but just about all of them could be done.”