The third annual ADLM data analytics competition is cohosted by ADLM's Data Analytics Steering Committee, Health Equity & Access Division, and Informatics Division, and by the Washington University Section of Pathology Informatics. A representative of the winning team will be invited to speak at ADLM 2024 with travel support and free meeting registration included.
The Challenge: Develop a Tool for Analyzing Fairness in Laboratory Testing
This year's competition aims to develop novel tools for monitoring fairness in laboratory testing. Competitors will be provided with a real de-identified clinical dataset documenting peripartum urine drug screening in the labor and delivery department of a single institution. The challenge is to create a dashboard that analyzes and visualizes fairness using these data. Solutions will be judged based upon functionality and adherence to best practices (see Scoring, below). Extra points will be awarded to teams that go beyond the dataset provided by engaging local stakeholders and applying their tool to analyze fairness in their own laboratory practice.
Context
Fairness in Laboratory Testing
Laboratory testing carries benefits and unintended harms that are distributed unequally among patients. Patients belonging to vulnerable subgroups often bear a disproportionate burden of harms while enjoying fewer benefits. Inequities in laboratory medicine may be underappreciated due to limited real time monitoring in practice.
Peripartum Urine Drug Screening
Peripartum Urine Drug Screening (UDS) is an example of laboratory testing with benefits and unintended harms.
Benefits: Detection of specific compounds can be highly informative and guide immediate management decisions, as in the case of opiate positivity leading to treatment of neonatal abstinence syndrome. Additional benefits include the opportunity to provide counseling and support cessation efforts. In other cases the clinical benefit is unclear, as in the detection of isolated cannabis positivity.
Harms: Peripartum UDS testing can undermine the mother's trust in the clinical team and lead to reporting to Child Protective Services (CPS), which can be traumatic and harmful to the new family.
A recent retrospective study performed at Washington University School of Medicine identified history of isolated marijuana use (IMU) as the most frequently cited indication for peripartum UDS1. The study demonstrated that 99% of the positive UDS ordered for this indication were positive for cannabis only. The vast majority (89.8%) of mothers with a positive UDS ordered for IMU were reported to CPS. Of those mothers reported to CPS, 20.9% were White and 79.1% were Black despite only about 50% of mothers identifying as Black and positivity for a nonprescribed substance other than marijuana being 3 times more likely among White mothers.
To try to address these inequities in UDS harms, a quality improvement initiative was developed. This initiative included revising the labor and delivery policies to prohibit ordering UDS for an indication of IMU. To promote accountability to these policy changes, an electronic question was added to the peripartum UDS order requiring the clinician select an appropriate testing indication.
Your task is to create a dashboard that can evaluate the impact of this quality improvement initiative by analyzing and visualizing the fairness of peripartum UDS testing over time.
Scoring
Competitors will be evaluated based on the functionality of their tool, their adherence to best practices for application development, and their ability to deploy their tool to address fairness in their local laboratory practice. Scoring will be performed by expert reviewers using a standardized scoring rubric and blinded to the competitor's identity.
Functionality (45 points)
There are many ways to conceptualize and quantify fairness2. Points will be awarded for each concept that is correctly implemented. Additional functionality points will be awarded for quality of visualization and user experience.
Implementation of Fairness Concepts (20 points)
- Demographic parity (4 points)
- Equalized odds (4 points)
- Predictive parity (4 points)
- Equal outcomes (4 points)
- Other (4 points)
Quality of Visualizations (10 points)
- Impactful visuals (2 points)
- Clean and appealing visualizations (2 points)
- Axes and plots labeled (2 points)
- Appropriate use of colors (2 points)
- Appropriately sized fonts (2 points)
Statistics (5 points)
- Confidence intervals provided for point estimates (2 points)
- Formal hypothesis testing (3 points)
User Experience (10 points)
- Intuitive navigation (5 points)
- Interactivity (5 points)
Local Activation (20 points)
Competitors are highly encouraged to identify local stakeholders that would utilize a fairness monitor in practice. Points will be awarded for running the analytical tool on a local dataset. Please provide a screenshot of your local analysis (without PHI) and a short letter from the end user documenting their engagement. Please summarize key findings from local activation.
- Local testing data analyzed for fairness (10 points)
- End-user engages with fairness dashboard running on local data (5 points)
- Describe insight provided by local analysis in 500 words or fewer (5 points)
Best Practices (12 points)
Best practices in software development help to optimize quality, maintainability, and reusability. Note that source code and commit histories are required to be eligible for points in this section.
Readability (4 points)
- Code is clearly and cleanly commented (1 point)
- Code is simple and not bloated (1 point)
- Code utilizes a clear and consistent naming convention (1 point)
- Code is organized into hierarchy of modular functional units (1 point)
Reusability (2 points)
- Code utilizes functionalized or object oriented programming (1 point)
- Code can be configured to new dataset without modification of source code (1 point)
Version Control (2 points)
- Version control system used to track development (1 point)
- Commits are modular, logical, and appropriately scoped (1 point)
Documentation and Deployment (4 points)
- Usage notes provided (1 point)
- Dependencies are defined (1 point)
- Virtualized or containerized environment used (1 point)
- Tool accessible through web hosting (1 point)
Significance
The tools developed in this competition could be used more broadly to help laboratories to identify unfairness in laboratory testing and monitor quality improvement projects aimed to promote equity in laboratory practice.
Timeline
January 15, 2024: Competition begins
May 15, 2024: Competition ends
June 15, 2024: Announcement of winning team
July 29, 2024 (anticipated): Presentation from winning team at ADLM 2024
How to Participate
Sign Up for a GitHub Account
- Navigate to https://github.com
- Click "Sign up"
- Follow the prompts to create your personal account
Fork the Competition Repository
- Navigate to https://github.com/WUSM-LGM-Informatics-Section/adlm-fairlabs-competition
- Click "Fork"
- Select "Create a new fork"
- Set your GitHub account as the owner (default)
- Click "Create fork"
Build Your Solution
- Clone the forked repository with the command
git clone https://github.com/myGitHubUsername/adlm-fairlabs-competition.git
Note: Replace myGitHubUsername with your GitHub handle
- Make a folder in the cloned repository with your team name
- Build your solution within your team folder
Submit Your Solution Via a Pull Request
- Navigate to https://github.com/myGitHubUsername/adlm-fairlabs-competition
Note: Replace myGitHubUsername with your GitHub handle
- Click "Contribute"
- Select "Open pull request"
- Click "Create pull request"
Note: we will review your pull request to ensure that it contains everything needed to score your submission
Important: You must submit your complete solution by May 15, 2024, to be eligible to win the competition!
Need Help?
If you are unfamiliar with GitHub, need help getting starting, or have other questions, please email [email protected] for assistance.
References
- 1.Rubin A, Zhong L, Nacke L, Woolfolk C, Raghuraman N, Carter E, Kelly J. Urine drug screening for isolated marijuana use in labor and delivery units. Obstet Gynecol 2022;140:607–9. https://doi.org/10.1097/AOG.0000000000004930
- 2. Azimi V, Zaydman MA. Optimizing equity: working towards fair machine learning algorithms in laboratory medicine. J Appl Lab Med 2023;8:113–28. https://doi.org/10.1093/jalm/jfac085