Listen to the JALMTalk Podcast
Huub H van Rossum and Daan van den Broek. Ten-Month Evaluation of the Routine Application of Patient Moving Average for Real-Time Quality Control in a Hospital Setting. J Appl Lab Med 2020;5:1184-93.
Dr. Huub van Rossum is a specialist in laboratory medicine at the Netherlands Cancer Institute.
Hello, and welcome to this edition of “JALM Talk” from The Journal of Applied Laboratory Medicine, a publication of the American Association for Clinical Chemistry. I’m your host, Randye Kaye.
Clinical laboratories have long used external quality control products to monitor analytical assay performance. More recently, there has been increasing interest in the use of patient-based, real-time quality control. One common approach is known as moving averages, which uses real-time averaging patient results to detect shifts and trends in analytical performance. A Working Group of the International Federation of Clinical Chemistry and Laboratory Medicine’s Committee on Analytical Quality has been commissioned to promote the adoption of patient-based, real-time quality control techniques. However, these techniques can be difficult to implement and few real-world studies and experiences have been reported to date. An original article in the November 2020 issue of JALM describes a ten-month evaluation study of the use of moving averages quality control for 16 routine clinical chemistry and hematology tests in a hospital laboratory. The first author of the article is Dr. Huub van Rossum. Dr. van Rossum works as a specialist in laboratory medicine at the Netherlands Cancer Institute. He started his company, Huvaros, that markets MA Generator, an application that supports the laboratory-specific optimization and validation of patient-based, real-time quality control. Dr. van Rossum is our guest for this podcast.
Welcome, Dr. van Rossum. It can be difficult to get started with moving averages in the clinical lab. How did your lab start the process, and why?
Huub van Rossum:
Well, at my former job, I had some experience with working on patient moving averages and in that lab, we actually worked on this new kind of method to get the settings and to implement it in the lab. And it was sort of the starting point at my new lab in the Netherlands Cancer Institute. So, I had some background on the topic, but I totally agree that starting to work on the topic, it’s quite difficult. Even though it seems simple, there are quite a lot of challenges and the first challenge is probably, how do I get my settings?
So, if you look at a moving average procedure, there are several parts that need to be configured. So, the first part is which data, which result, is included in the algorithm. And some settings have to be determined, for instance, which results are included based on result or based on basically any criteria can be used, to include results in the moving average procedure. The second part is the algorithm itself. There are several options that you can choose. So, you can choose for instance a mean calculation, a median, an exponentially weighted moving average procedure, or a Bull’s algorithm. And within each calculation type, there is a variable that needs to be selected, like are you going to use the mean of the last five results, or the last ten, or the last twenty-five results. And the final part of the procedure are the control limits. So, these limits, when exceeded give the alarm, and they are also critical in terms of the performance of the moving average.
All these choices have to be made and from a historical perspective, this has been done on more or less a trial and error procedure, but using new insights and new methods, more like a simulation method, these settings can now be based on the true performance of these moving averages. So, we used those kinds of approaches to get the settings.
All right. Thank you. So, you had a lot of options. How did you define your moving average settings for each test?
Huub van Rossum:
Yes, so we used a method which is available online, which is available in the so-called Moving Average Generator Program, and in that method you upload a historical set of data from the lab and the method does sort of simulations. So, the first thing it does, it determines the control limits based on the manageable number of alarms, and the second step it does is that it does a lot of error detection simulations based on your own laboratory data. So, it adds error and it determines how long it takes to be detected, plus all these results in the so-called bias detecting curves, a validation chart, which is actually the readout of your error detection performance. So, using these graphs, we were able to compare the different moving average procedures, and we could choose the moving average procedure that had the best error detection performance through our hospital.
All right. Thank you. Now, your study actually evaluated this process for ten months. So, within that time period, what were your major findings?
Huub van Rossum:
Yes, so we have used the historical training set to implement it, and then of course, it’s an interesting question to see if those settings are working in practice in a perspective manner. So, the first goal was to check if it’s feasible in routine clinical practice. So, we have implemented in routine practice and, therefore, it’s required, when alarm was generated, it requires 24/7 working up of the alarm, and then also to investigate the cause of the error and whether the alarm was detecting true error or whether it was a false alarm. So, that was the first requirement of the study, to check whether the requirement of a manageable number of alarms was met, but also to see if the moving average QC could detect clinically-relevant errors.
So, overall, would you say that the implementation was successful, and would you recommend that other clinical labs start using moving averages to monitor quality?
Huub van Rossum:
Yes, so I think it does take some time and effort to put them into your lab. So, as said, you need to study them, because it’s essential to have suitable procedures because if you get too many alarms, you get alarm fatigue, it doesn’t work. If you don’t get any alarm or you don’t have an insight whether it helps you, whether it provides relevant error detection, it might be useless. So, I think that’s quite a challenge for labs to get proper settings and set things that they know are providing them with relevant error detection. So, luckily these online tools are now available to help them with that.
The second step is we had to configure our settings in our list, because we used our list to manage the moving average in daily practice, and so we generated this special screen that presents real-time the alarms. We configured specific graphs, so we don’t normally use these beginning graphs for presenting this QC, but we use this accuracy graph. So, if a moving average is in control, it’s in the green area. If it’s outside, it’s red, so it’s easy to understand for technicians. So, we had to configure all these software features.
The next thing we did is that we designed a lab protocol, and this is really important for the technicians to understand what needs to be done when alarm occurs and also, the protocol determines which moving average alarms that actually detect an error are truly acknowledged. So, for instance, the first thing we do when we get a moving average alarm is run a normal QC because it gives quite fast insight in whether an error occurred and the size of the error. The second step was that we repeated the analysis of four patient samples. So, for instance if we had a moving average alarm which exceeded the upper control limits, we look for higher results and when it exceeded the lower limit, we look for lower test results. We repeated them, because repeating these results enabled us to detect temporary assay failure, but also allows us to detect smaller errors which are clinically relevant but hard to detect using internal QC, and the final step was that we reviewed all results to see if there was another reason for the alarm, for instance one patient with extreme results, or if we see a pre-analytical cause of a sample with an extreme result.
So, these were all designs and then of course, we needed to train the technicians for all these procedures. So, all technicians were trained to be able to perform these moving average alarm workups, and then we of course, when we started to use it in practice, we switched on the on button and then the system truly generated alarms. So, to get this system in your lab, it does require some efforts.
I see. So, after all that, and finally, what do you now envision for the future of patient-based quality control techniques in the clinical lab?
Huub van Rossum:
Of course, implementation takes time, but we have implemented, I think it was, two years ago now, and it’s still running on the back and it doesn’t cost a lot of time at this point. So, implementation is an investment, but now we have real-time QC running on the back. So, even now in the Netherlands, now it’s in the evening, and I don’t have to worry that something goes wrong for a lot of critical tests because I know moving average is running. So, if it produces a result, moving average QC is generated and it alarms when—I know it will alarm when a relevant error occurs for many tests.
So, with that in mind, I do think it’s really, really valuable, for at least our setting, but I also think it’s quite valuable for a lot of other settings. So, we are a rather small hospital. We are the Netherlands Cancer Institute so we have a lot of cancer patients, really sick patients and, therefore, we also have a lot of pathological results, and these conditions are actually not favorable for using moving averages. So, if you have a larger number of tests and with less pathological results, I think the error detection performance increases a lot and so for many labs, I think moving average allows them to get a really interesting additional quality control instrument, that they can really benefit from—and it can help them to improve the quality assurance, but also to enable to design more cost-effective and efficient QC procedures and maybe reducing the number of statistical quality control measures, for instance.
That was Dr. Huub van Rossum from the Netherlands Cancer Institute describing the article from the November 2020 issue of JALM entitled, “Ten-Month Evaluation of the Routine Application of Patient Moving Average for Real-Time Quality Control in a Hospital Setting.” Thanks for tuning into this episode of JALM Talk. See you next time, and don’t forget to submit something for us to talk about.