Comparative validation of open-source artificial intelligence models for fracture detection using the MURA dataset

Introduction

Artificial intelligence (AI) in medicine has evolved from conceptual machine learning frameworks to clinically integrated diagnostic systems (1). Early theoretical foundations described supervised learning models capable of extracting structured patterns from high-dimensional medical data (1). In radiology, convolutional neural networks (CNNs) have demonstrated diagnostic performance approaching radiologist-level accuracy across multiple imaging domains (2).

Musculoskeletal imaging presents a particularly compelling application of AI. The MURA (Musculoskeletal Radiographs) dataset, developed by Stanford University, is a large-scale benchmark consisting of clinically acquired and expert-labeled upper-extremity musculoskeletal radiographs for abnormality detection research (3). Subsequent studies demonstrated that deep CNN architectures such as DenseNet, VGG, and ResNet achieve high diagnostic performance when trained on MURA (4). However, external validation remains limited, and concerns about generalizability and applicability of models persist (4). Systematic reviews have reported pooled AI fracture detection sensitivities ranging from 0.81–0.94 and specificities 0.83–0.92 (5,6). Nevertheless, many studies rely on proprietary algorithms, limiting reproducibility and access in resource-limited settings.

Recent ensemble deep learning approaches further improved fracture classification performance, reaching accuracies above 92% in MURA humerus studies (6). Hybrid detection-classification pipelines combining CNNs with feature-matching techniques [e.g., Speeded-Up Robust Features (SURF)] have demonstrated high localization accuracy in computed tomography (CT) imaging (7).

Despite promising performance metrics, concerns remain regarding dataset bias, reference standards, and lack of prospective validation. In settings where radiology expertise and computational resources are limited, open-source AI models may offer practical deployment advantages in the setting of musculoskeletal imaging interpretation. Ultimately, safe and reliable fracture detection tools have the potential to increase accessibility, reduce diagnostic delays, and better support healthcare infrastructure in resource-limited environments, leading to improved patient outcomes in musculoskeletal care.

In light of these considerations, this study evaluates two publicly accessible GitHub-based fracture detection models: YOLOv7-BoneFractureDetection and BoneFractureClassification. YOLOv7 is a real-time object detection architecture based on CNNs, designed to identify and localize features of interest within images. In the context of fracture detection, it produces bounding box predictions with associated confidence scores. In contrast, the BoneFractureClassification model is a classification-based deep learning model that outputs a binary prediction (fracture vs. no fracture) along with a probability score, without explicit localization.

Both models are publicly available through open-source repositories; however, there is limited published evidence of independent external validation in peer-reviewed literature. As such, their generalizability and clinical robustness remain uncertain.

The evaluated models’ training datasets differ substantially from the MURA dataset. The YOLOv7-BoneFractureDetection model was trained on the GRAZPEDWRI-DX dataset, which consists of pediatric wrist radiographs with bounding box annotations for fracture detection. While this dataset is clinically relevant, it is limited to wrist injuries and does not cover the broader range of anatomical regions included in MURA. In contrast, the BoneFractureClassification model was trained on a publicly available X-ray dataset sourced from Kaggle for binary fracture classification. This dataset is not a standardized benchmark and lacks detailed documentation regarding patient population, anatomical distribution, and annotation methodology. As a result, neither model was trained on the MURA dataset or an equivalent multi-region musculoskeletal dataset. This discrepancy in training data may introduce domain shift and influence model performance when applied to MURA, and should be considered when interpreting the results.

The MURA dataset was selected for this study due to its widespread use as a benchmark dataset for musculoskeletal radiograph analysis. It contains expert-annotated radiographs across multiple upper-extremity anatomical regions, including the humerus, elbow, forearm, wrist, hand, and finger. Its standardized structure and public availability make it well-suited for reproducible and comparative evaluation of fracture detection models. Compared with smaller or more specialized fracture datasets, MURA provides greater diversity and has been extensively used in prior deep learning studies, facilitating contextual comparison with existing literature.

Methods

Study design and dataset

We conducted a comparative validation study of two open-source deep learning models for binary fracture detection, using radiographs from the publicly-available MURA dataset. The MURA dataset comprises upper-extremity musculoskeletal radiographs across multiple anatomical regions, including the humerus, elbow, forearm, wrist, hand, and finger. This dataset reflects a heterogeneous range of abnormalities, from obvious displaced fractures to more subtle nondisplaced findings, as defined by the original dataset annotations. We used a total of 100 radiographs for validation, with 50 labeled as fracture-positive and 50 as fracture-negative. To preserve the independence of observations and avoid bias from correlated samples, a maximum of one radiograph was included per patient ID, even when multiple images or views were available for that patient. No additional stratification by anatomical region or fracture type was applied and ground-truth labels provided by MURA without further adjudication.

Sample size considerations

Given that the study was designed as a preliminary feasibility assessment rather than a definitive hypothesis-testing trial, no formal prospective power calculation was performed before dataset selection. Statistical uncertainty was addressed by reporting 95% confidence intervals (CIs) for performance metrics and by using paired statistical testing for model comparison. As such, the findings should be interpreted as preliminary and hypothesis-generating, with larger studies, more rigorously powered studies needed for robust validation.

Statistical comparison of model performance

McNemar’s test was performed to compare the paired classification performance of YOLOv7 and the BoneFractureClassification model.

Model implementation and inference pipeline

Both models were deployed locally using custom code written in Python (version 3.12). We developed a lightweight web interface using the Streamlit library to facilitate image upload, model inference, and structured logging of predictions. We processed all images (e.g. resizing, normalizing) according to the predefined inference pipelines of the each models. The code used for data processing, model inference, and performance evaluation for each model is publicly available at: https://github.com/mdciri/YOLOv7-Bone-Fracture-Detection and https://github.com/MohtashamMurshid/BoneFractureClassification.

The BoneFractureClassification (4) model was trained for classification, so each radiographic image passed through the model produced a probability value between 0 and 1. We used a threshold of 0.5 to determine if the model predicted a fracture (P>0.5) or not (P<0.5). The YOLOv7-BoneFractureDetection (8) model was trained to produce one or more bounding-box label per image, with each associated with a confidence score. In order to derive an image-level classification, detections were aggregated such that a radiographic image was labeled as “fracture” if at least one bounding-box detection was present above the model’s default confidence threshold of 0.20. If no such detections were identified, the radiographic image was labeled as “non-fracture”.

This study focused exclusively on image-level classification performance. Although YOLOv7 provides localization outputs, fracture localization performance was not formally evaluated, either quantitatively or qualitatively.

Performance evaluation

Model predictions were compared against ground-truth labels to construct confusion matrices, including true positives, false positives, true negatives, and false negatives. Based on these, standard performance metrics were computed, including sensitivity (recall), specificity, precision [positive predictive value (PPV)], accuracy, and F1 score.

In addition to classification metrics, model confidence was evaluated by calculating the average confidence score for correctly identified fracture cases (true positives), providing an estimate of model certainty in positive detections.

To assess statistical uncertainty, 95% CIs were calculated for key performance metrics. Paired comparisons between models were performed using McNemar’s test to evaluate whether differences in classification performance were statistically significant.

Statistical tools and implementation

All analyses were conducted in Python (version 3.12). Data handling and preprocessing were performed using Pandas, numerical computations were carried out using NumPy, and performance metrics were computed using Scikit-learn.

Both models were evaluated using their default pretrained parameters without additional fine-tuning, reflecting a real-world deployment scenario using readily available open-source tools.

The code used for data processing, model inference, and performance evaluation is publicly available at: https://github.com/mdciri/YOLOv7-Bone-Fracture-Detection and https://github.com/MohtashamMurshid/BoneFractureClassification.

Ethical considerations

The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The MURA dataset contains de-identified radiographs. No institutional review board approval was required for secondary analysis of publicly available data.

Results

Based on 100 radiographs (50 fracture, 50 non-fracture), diagnostic performance metrics were computed using the definitions provided in the Methods section.

YOLOv7:

• Sensitivity (Seᵧ) =0.72;
• Specificity (Spᵧ) =0.92;
• Precision (PPVᵧ) =0.90;
• Accuracy (Accᵧ) =0.82;
• F1ᵧ =0.80;
• AUCᵧ =0.89;
• Mean TP confidence (C̄ₜₚ,ᵧ) =0.784.

From these values:

Seγ=TPγTPγ+FNγ=0.72[1]

Spγ=TNγTNγ+FPγ=0.92[2]

The relatively high Spᵧ and PPVᵧ indicate strong false-positive control.

BoneFractureClassification:

• Sensitivity (Seᵦ) =0.88;
• Specificity (Spᵦ) =0.64;
• Precision (PPVᵦ) =0.71;
• Accuracy (Accᵦ) =0.76;
• F1ᵦ =0.786;
• AUCᵦ =0.85;
• Mean TP confidence (C̄ₜₚ,ᵦ) =0.727.

  $$S{e}_{\beta }=\frac{T{P}_{\beta }}{T{P}_{\beta }+F{N}_{\beta }}=0.88$$[3]

  $$S{p}_{\beta }=\frac{T{N}_{\beta }}{T{N}_{\beta }+F{P}_{\beta }}=0.64$$[4]

Higher Seᵦ reflects improved fracture capture, but lower Spᵦ increased false positives. Please see Table 1.

Statistical comparison of model performance

McNemar’s test was performed to compare the classification performance of YOLOv7 and the BoneFractureClassification model. The results indicated that the difference in performance between the two models was not statistically significant] (P=0.55). The number of discordant pairs where YOLOv7 was correct and the BoneFractureClassification model was incorrect was 7, whereas the reverse occurred in 4 cases. These findings suggest that, despite differences in sensitivity and specificity, neither model demonstrated statistically superior overall classification performance on this dataset.

ROC analysis

Receiver operating characteristic (ROC) analysis was performed using model confidence scores. The YOLOv7 model achieved an area under the receiver operating characteristic curve (AUC) of 0.894 (95% CI: 0.738–0.961), while the BoneFractureClassification model achieved an AUC of 0.854 (95% CI: 0.782–0.946). ROC curves for both models are shown in Figure 1.

Figure 1 Receiver operating characteristic curves for YOLOv7 and BoneFractureClassification on the MURA dataset. AUC, area under the receiver operating characteristic curve; CI, confidence interval; MURA, Musculoskeletal Radiographs.

Calibration curves were also generated using quantile based binning to improve stability in the setting of limited sample size. For YOLOv7, the available confidence-score dataset included 50 cases, whereas the available BoneFractureClassification confidence-score dataset included 31 cases. Accordingly, calibration results should be interpreted cautiously, particularly for the BoneFractureClassification model. The corresponding Brier scores were 0.146 for YOLOv7 and 0.145 for BoneFractureClassification. Calibration curves are shown Figure 2.

Figure 2 Calibration curves for YOLOv7 and BoneFractureClassification on the MURA dataset using quantile binning. MURA, Musculoskeletal Radiographs.

Discussion

Context within existing literature

Deep learning has demonstrated strong performance in fracture detection across multiple imaging modalities (1-3). The MURA benchmark study reported radiologist-comparable abnormality detection using deep CNN architectures (3). Subsequent investigations employing the MURA dataset, including DenseNet and transfer-learning models, achieved accuracies ranging from 83–92% (9).

More recently, ensemble deep-learning approaches further improved classification performance, with one study reporting a classification accuracy of 92.96% in humeral fracture detection (10). However, many high-performing systems rely on proprietary optimization or multi-model ensembles, which may limit transparency and replicability.

Beyond radiology, ResNet-based fracture classification in CT imaging achieved accuracy as high as 98% in calcaneal fracture detection (11). That study also demonstrated the importance of preprocessing steps such as image resizing, normalization, and architecture depth in enhancing performance. Their observation that deeper residual networks outperform serial VGG networks align with the principle that increased architectural depth improves hierarchical feature extraction (11).

Systematic reviews of AI-based fracture detection consistently report pooled sensitivities and specificities exceeding 0.85, but also highlight important limitations, including spectrum bias and limited external validation (5,6). Elkohail et al. reported pooled sensitivities and specificities between 0.85–0.95, while highlighting generalizability limitations and implementation barriers in the real world (5).

Within this context, as seen in Table 1: our YOLOv7 model demonstrated strong confirmatory performance, with Spᵧ =0.92 and PPVᵧ =0.90, aligning with performance thresholds reported in prior systematic analyses for diagnostic applications (9,12). Simultaneously, BoneFractureClassification’s Seᵦ =0.88 is consistent with triage-oriented AI applications that prioritize sensitivity and recall over precision to minimize the number of missed fractures at the time of initial evaluation.

Clinical workflow implications

Diagnostic AI systems can be conceptualized in two roles:

• Confirmatory support
  High specificity and PPV reduce false positives, minimizing unnecessary imaging and specialist referrals. YOLOv7’s Spᵧp=0.92 suggests suitability for confirmatory workflows.
• Triage screening
  High sensitivity ensures fewer missed fractures. BoneFractureClassification’s SeeF=0.88 may support early triage, especially where radiologists are unavailable.

This dual-use distinction mirrors prior AI workflow models in orthopedics and surgical innovation (5).

Reproducibility and open-source considerations

Unlike many proprietary systems, both evaluated models are publicly deployable, this addresses concerns regarding transparency and accessibility as emphasized in prior orthopedic AI literature (5-7,12). Open-source availability enhances reproducibility and allows low-resource institutions to implement and independently assess these tools.

A key limitation of this study is the relatively small sample size (n=100), which may reduce the statistical reliability and generalizability of the findings. Performance metrics such as sensitivity and specificity may vary depending on the specific sample selected, and the observed results may not fully represent model performance across the broader MURA dataset or other clinical populations.

To better characterize this uncertainty, we incorporated 95% CIs for the principal performance metrics and performed paired statistical testing using McNemar’s test. Nevertheless, the findings should still be interpreted cautiously, as a small sample may limit the stability of performance estimates and reduce the ability to detect meaningful differences between models.

Accordingly, the present findings should be interpreted as preliminary and hypothesis-generating. Larger studies, including full-dataset evaluations and multi-center external validation, are required to establish robust and generalizable conclusions regarding model performance.

Conclusions

Open-source AI models demonstrate clinically meaningful fracture detection performance using the MURA dataset.

YOLOv7 appears to be better suited for diagnostic support (Spᵧ =0.92), while BoneFractureClassification model demonstrates higher sensitivity (Seᵦ =0.88), which may be more useful for initial triage or screening. In the future, further multi-center, radiologist-compared validation are needed before these AI-tools can be integrated into our clinical practice and can help enhance healthcare systems where radiology expertise is limited.

Acknowledgments

None.

Footnote

Peer Review File: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-2026-1-0041/prf

Funding: None.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jmai.amegroups.com/article/view/10.21037/jmai-2026-1-0041/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The MURA dataset contains de-identified radiographs. No institutional review board approval was required for secondary analysis of publicly available data.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Deo RC. Machine Learning in Medicine. Circulation 2015;132:1920-30. [Crossref] [PubMed]
2. Han XG, Tian W. Artificial intelligence in orthopedic surgery: current state and future perspective. Chin Med J (Engl) 2019;132:2521-3. [Crossref] [PubMed]
3. Rajpurkar P, Irvin J, Bagul A, et al. MURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographs. Stanford Machine Learning Group, 2017. Available online: https://stanfordmlgroup.github.io/competitions/mura/
4. Murshid M. BoneFractureClassification. Available online: https://github.com/MohtashamMurshid/BoneFractureClassification
5. Elkohail A, Soffar A, Paul A, et al. Artificial Intelligence in Bone Fracture Detection: A Review of Evidence, Limitations, and Clinical Integration. Cureus 2025;17:e97674. [Crossref] [PubMed]
6. Kutbi M. Artificial Intelligence-Based Applications for Bone Fracture Detection Using Medical Images: A Systematic Review. Diagnostics (Basel) 2024;14:1879. [Crossref] [PubMed]
7. Magan AA, Kayani B, Chang JS, et al. Artificial intelligence and surgical innovation: lower limb arthroplasty. Br J Hosp Med (Lond) 2020;81:1-7. [Crossref] [PubMed]
8. Ciriello M. YOLOv7 Bone Fracture Detection. Available online: https://github.com/mdciri/YOLOv7-Bone-Fracture-Detection
9. Chada G. Machine Learning Models for Abnormality Detection in Musculoskeletal Radiographs. Reports 2019;2:26.
10. Tahir A, Saadia A, Khan K, et al. Enhancing diagnosis: ensemble deep-learning model for fracture detection using X-ray images. Clin Radiol 2024;79:e1394-402. [Crossref] [PubMed]
11. Pranata YD, Wang KC, Wang JC, et al. Deep learning and SURF for automated classification and detection of calcaneus fractures in CT images. Comput Methods Programs Biomed 2019;171:27-37. [Crossref] [PubMed]
12. Lisacek-Kiosoglous AB, Powling AS, Fontalis A, et al. Artificial intelligence in orthopaedic surgery. Bone Joint Res 2023;12:447-54. [Crossref] [PubMed]