Setting Analytical Quality Goals with Biological Variation Data

SLIDES

TRANSCRIPT

Slide 1: Title Slide

Hello, my name is Paul Johnson. I am an Associate Professor at Upstate Medical University in Syracuse, New York. Welcome to this Pearl of Laboratory Medicine on “Setting Analytical Quality Goals with Biological Variation Data”

Slide 2: Method performance validation, concept of total error

Clinical chemists are familiar with the need to establish or verify analytical performance prior to implementing a test for patient care use. These components include Precision, Bias, and Total Method Error (or “TME”). TME is then compared with Total Allowable Error (abbreviated TEa, or “TAE” in this presentation). When TME is less than TAE, the new test method is usually considered to have met the necessary requirements for patient care use.

Slide 3: Concept of Total Method Error (TME)

TME is calculated from Bias and standard deviation (SD) values obtained from method comparison experiments.
The figure shown illustrates a positive bias with the new test method as compared to the true value, established by a reference standard or other comparative method. Imprecision follows a Gaussian distribution, which is calculated using z-value at desired probability level. The traditional calculation as shown for TME = Bias + (1.65 x SD), where 1.65 is the one- sided z-value for 5% probability of a normal distribution curve.
Published biological variation estimates for a test analyte will allow us to apply specific cut- off values to evaluate these components of method error, as illustrated in the figure.

Slide 4: Establishing TAE goals

Establishing analytical quality goals for TAE has a rich history. Briefly, these may be based on legal requirements, as for example CLIA laws in the United States; or it may be set by
providers of proficiency testing (PT) / external quality assessment schemes (EQAS), which is the case in many regions of the world. As it has become evident that biological variation (“BV”) ranges differ across test analytes, and that improved estimates of BV are readily accessible, there is renewed attention on establishing TAE goals using BV data.
- Addressing this topic of how clinical laboratorians can use published biological variation data for setting analytical quality goals is the focus of this Pearl.
- For additional background on BV, as well as use of calculations like the reference change value (RCV) and Index of Individuality, listeners are referred to a September 2012 Pearl on this subject by Danni Li.

Slide 5: Terms to describe biological variation data

As the name suggests, biological variation is a source of variance in laboratory test results. These data are commonly expressed as percent coefficient of variation (CV) or relative standard deviation (RSD). Both terms are equivalent and are calculated by first converting variance to SD, dividing SD by the Mean value of the data set, and then multiplying by 100% to express as percentage value. BV data presented in this Pearl are expressed in % CV terms.
There are two components of importance: First, the between-subject component: CV-G, which can be viewed as “group” variation. Second, the within-subject component: CV-I, which can be viewed as “individual” variation. These terms and subscripts can vary across publications, although today the generally agreed-upon nomenclature is CV-G and CV-I.

Slide 6: Two components of BV: CV-G, CV-I

The two components of biological variation, CV-G and CV-I, can be visualized in this figure.
Shown in this example are 8 unique samples, collected at different times, on 12 different subjects (labeled “A through L”). Data are shown with individual values (open red-colored circles), subject mean value (blue square), and range box to show the minimum and maximum test values for each subject. Of note, all test samples were obtained from a healthy/disease-free cohort of subjects.
Between-subject variability, CV-G, is visualized in the horizontal direction based on variation across subject mean values. Vertical boxes help illustrate the within-subject variation, CV-I.

Slide 7: Biological variation database

The most recent biological variation estimates and database are provided online by the European Federation of Clinical Chemistry and Laboratory Medicine, or EFLM.
On its website, the EFLM provide a meta-analysis of published papers on BV to obtain CV-I and CV-G estimates.
The Table in this slide was created for selected chemistry analytes by using the median values for CV-I and CV-G provided on the EFLM website, along with the number of
publications (N) used to generate the final estimates. It is these numbers that we will be using to set analytical quality goals.

Slide 8: Setting “Desirable” Limits

I’ve expanded the previous BV data Table to now include ‘Desirable limits’ for imprecision (I), Bias (B), and total error (TE) all expressed as percentage values. In the next few slides, I will go through each of the calculations used to obtain these numbers.

Slide 9: Calculating “Desirable” Imprecision Goal

The first of our calculations is setting the desirable imprecision goal, which will be based on CV-I estimate for the test biomarker.
The listener will recall that CV-I represents a distribution of repeated test values taken from the same subject. Ideally, CV-I is a “pure” estimate of normal human biology only.
In practice, analytical variation (CV-A) of the test method used to generate the result contributes variation to the CV-I estimate. Because of this relationship, it has been
recommended that the desirable analytical imprecision goal should be less than one-half of within-subject biological variation.

Slide 10: Analytical imprecision adds variability to within-subject variation

The justification for the previous desirable goal for imprecision is based on the direct contribution of analytical imprecision to within-subject biological variation.
This can be calculated using the sum of variances rule.
The left-side panel shows the combined CV-A and CV-I values, summed as variance terms, and denoted CV-I(T). The formula is SQRT (CV-I^2 + CV-A^2).
From this formula, it is clear that the percentage variability added to the true test result increases as random analytical variation (CV-A) increases, and vice-versa.
It will be useful to imagine a fixed average test value, based on BV, for a person when thinking about the impact of analytical imprecision on the test result.
For example, when CV-A approaches 0% (ideal), the person’s true test result from a single measurement has uncertainty due only to within-subject BV. By contrast, what happens when CV-A is exactly equal to one-half (50%) of CV-I at the desirable goal?
Substituting 50% CV-I (“0.5 CV-I”) for CV-A into the equation shown on the left now gives a total CV-I value equal to the square root of 1.25 CV-I, which is 1.12 or 12% additional variation to the true test result because of analytical imprecision (CV-A).
This same equation can be applied to any ratio value of CV-A to CV-I.

Slide 11: Total biological variation

Shown in this figure is sum of squares rule for total biological variation, which is the combination of both within-subject BV and between-subject BV. This equation will be useful in setting the desirable bias goal.

Slide 12: Calculating “Desirable” Bias Goal

In order to set desirable method bias goal, we will use the equation that sums both components of BV, as just described.
It has been recommended that any method bias should be less than one-fourth, or 0.25, of total BV. We can think of “bias” as a difference between people which is why both CV-I and CV-G values are included in the calculation.
Since reference intervals for a clinical test are determined based on biological grouping of individuals, it is reasonable to expect that any method bias could shift the proportion of individuals correctly classified as true positive and true negatives.
The factor 0.25, used to set the maximum “desirable” ratio of method bias to total BV, is a practical effort to minimize false positives and negative that may occur when method bias is present.
Callum Fraser and colleagues have summarized that a positive bias ratio of 0.25 (at the desirable goal), would cause 4.4 % of healthy individuals to be classified as having an abnormal value outside the reference limit (versus the expected 2.5%).

Slide 13: Total Allowable Error Goals

Lastly, we can derive total allowable error (TAE) by combining the two previous equations. A standard calculation for TAE is sum of 1.65 x Imprecision plus Bias. The 1.65 multiplier is the one-sided 95% significance probability level.
If setting TAE limits to 99% probability, 2.33 will be used as the multiplying factor (instead of 1.65).

Slide 14: Alanine aminotransferase (ALT) test, as example

Returning to our original table, let’s look at Alanine Aminotransferase (ALT) test as an example. The CV-I and CV-G estimates for ALT are based on 14 total publications.
To get desirable I = 0.5 x 9.6 (or 9.6/2) = 4.8%, Bias = 0.25 factor x SQRT (9.6^2 + 28.0^2) = 7.4%. Lastly TE = 1.65 x 4.8 (imprecision) + 7.4 (bias) = 15.3%
Expressing these limits and BV data in percent CV terms is helpful because they apply equally to any unit of measure, whether conventional U.S. units or international units.
When needed, these percentage values can be back-converted to a specific unit of measure for the test analyte through simple mathematical calculations.

Slide 15: Additional performance criteria for bias and imprecision

So far we have looked at establishing the “Desirable” cut-offs, which really represents a balance between the optimal cut-off and minimum acceptable cut-off.
Illustrated in this figure is a relative comparison of three levels of performance criteria, which differ only in the multiplier factor used. Rest of calculations are the same.
For some test analytes, achieving optimal level cut-off may be extremely difficult because of its BV. Nonetheless, it is expected that acceptable test methods will perform at least as well as the minimum cut-off values for it to be put into clinical use for patient care.

Slide 16: Comparison of optimal, desirable, and minimal goals

Returning back to our ALT example, now included are the multiplier factors for all three performance levels. The desirable cut-offs for I and B are same as previously shown.
Notice for the ALT test that the 4.8% desirable analytical imprecision goal is reduced to 2.4% to meet the optimal goal, but increases up to 7.2% to achieve minimal (or minimum)
acceptable performance. Similar calculations apply for bias goals, and these can be extended to show total error, as discussed earlier.

Slide 17: ALT method comparison data

Let’s assume we are interested in establishing a new instrument method for ALT enzyme, designated “Instrument B”. Method comparison experiments are performed in the
laboratory.
Mean bias of Instrument B is determined by comparing it to an established ALT method, “Instrument A”. In this example, bias equals 2.3 U/L, or 8.0% relative to the mean value of the two instruments (29 U/L).
Total imprecision, determined from separate precision experiments, equals CV of 6.5%. Using the standard calculation for TME, Instrument B has total method error of 18.8%.

Slide 18: Evaluating method performance

Lastly, let’s take the test method’s imprecision, Bias, and TME data to compare with calculated limits based on BV.
Shown in the table are comparison of total allowable error based on biological variation desirable and minimum limits. The method has not met desirable goals because all 3 values are greater than the desirable cut-off limits. However, all three calculated values for this ALT test method are less than the minimal cut-off goals, so we can state it has passed these standards.
By way of additional comparison, the error limit for ALT enzyme is 20% based on CLIA proficiency testing criteria (in the United States). Whether such limits are greater than, less than, or equal to biological variation goals depends on the specific test analyte.
Improved alignment of performance limits across clinical tests may occur if the biological variation-based approach is adopted by external quality assessment providers or used within the legal regulatory environment.

Slide 19: References

Miller WG, Myers GL, Ashwood ER, Killeen AA, Wang E, Ehlers GW, et al. State of the art in trueness and interlaboratory harmonization for 10 analytes in general clinical chemistry. Arch Path Lab Med 2008;132:838-46.
Jones GRD, Albarede S, Kesseler D, MacKenzie F, Mammen J, Pedersen M, et al. Analytical performance specifications for external quality assessment - definitions and descriptions. Clin Chem Lab Med 2017;55:949-55.
Fraser CG, Hyltoft Petersen P, Libeer JC, Ricos C. Proposals for setting generally applicable quality goals solely based on biology. Ann Clin Biochem 1997;34 (Pt 1):8-12.
Fraser CG. Quality specifications. In: Biological variation: From principles to practice, Vol. 1: the Association for Diagnostics & Laboratory Medicine (formerly AACC) Press; 2001. p. 29-66.
European federation of clinical chemistry and laboratory medicine (EFLM). Biological variation database.
https://biologicalvariation.eu/bv_specifications/measurand# (Accessed 11 July 2019).
Jhang JS, Sireci AN, Kratz A. Postanalysis: Medical decision making. In: McPherson: Henry's clinical diagnosis and management by laboratory methods. 22nd Ed.: Saunders; 2011. p. 80-90.

Slide 20: Disclosures

Slide 21: Thank You from www.TraineeCouncil.org

Thank you for joining me on this Pearl of Laboratory Medicine “Setting Analytical Quality Goals with Biological Variation Data”