Qualitative American Heart Association plot of late gadolinium enhancement with mortality and ventricular arrhythmia prediction using artificial intelligence

Introduction

Background

Contrast-enhanced cardiac magnetic resonance (CMR) imaging with gadolinium-based contrast agents (GBCAs) has revolutionised the differential diagnosis of ischaemic and non-ischaemic cardiomyopathy and the assessment of acute myocardial infarction (MI) and its complications since its inception in 1984 (1). The introduction of inversion recovery (IR) techniques in the late 1990s further enhanced contrast-to-noise ratio and paved the way for late gadolinium enhancement (LGE) imaging (2,3).

LGE imaging not only aids in the diagnosis but also offers valuable prognostic information. Extensive research has demonstrated that the extent of hyper-enhancement on a segmental basis, assessed visually, correlates with recovery of myocardial function post-infarction, both in acute and chronic settings (4,5). Additionally, infarct size in the acute phase and the extent of myocardial viability in the chronic phase are independent predictors of prognosis (6).

Currently, clinical practice relies heavily on visual interpretation of LGE images following established guidelines, such as those provided by the American Heart Association (AHA) (7,8) and the Society for Cardiovascular Magnetic Resonance (SCMR). These guidelines recommend visually comparing LGE images with cine and perfusion images, if available, to accurately categorise ischaemia and viability and to estimate the transmural extent of LGE within each segment (9).

In parallel, the advancement of artificial intelligence (AI) and neural networks development has made significant strides in the medical field over the past decade. Image classification, facilitated by deep neural networks (DNNs) like convolutional neural networks (CNNs), has emerged as a powerful application of AI. CNNs have demonstrated remarkable performance in image analysis tasks since their introduction in 2012 (10).

Rationale and objectives

The qualitative assessment of LGE images by CMR based on AHA segmentation remains dependent on expert readers and has not been automated. Despite the importance of scar signal intensity variation, texture, MI scar transmurality, and location in CMR reporting for diagnostic and prognostic purposes, identifying the presence or absence of myocardial LGE can be challenging due to subtle pathology or image quality issues (such as sequence type, artefacts, contrast timing, and selection of IR timing) (11).

Even basic binary assessment of LGE presence can significantly influence the final diagnosis and patient management plan. However, this process requires significant time and expertise, as CMR readers need to review numerous images in multiple sequence, including long axis and short axis views.

While the prognostic value of LGE images and their association with clinical outcomes are well-established, the direct relationship between image pixels and outcomes remains unclear.

The objective of this study is to utilise AI to address these challenges by:

• Automating the AHA-based qualitative LGE plot and comparing the performance to expert CMR readers.
• Establishing a link between LGE image data and clinical health records to predict outcomes.

By leveraging AI techniques using neural networks, this study aims to streamline the interpretation of LGE images, enhance diagnostic accuracy, and facilitate the integration of imaging data with clinical outcomes for improved patient management. We present this article in accordance with the STARD reporting checklist (available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-94/rc).

Methods

Study design and population

This retrospective observational study was conducted on a large cohort of patients who underwent stress perfusion CMR imaging for the evaluation of suspected or known coronary artery disease (CAD). The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the King’s College London Research Ethics Committee under reference number 20/ES/0005 and individual consent for this retrospective analysis was waived.

The study population was randomly derived from a single centre, specifically St Thomas’ Hospital at King’s College London, and data were collected between April 2011 and March 2021. Only completed studies with comprehensive reports and available images were included in the analysis.

The primary outcome of interest was all-cause mortality, and data on mortality events were collected for the entire study population. The end of the follow-up period was defined as the date of data collection, which was August 20, 2021, when all events had been recorded and accounted for.

Data extraction

Clinical data

The clinical data for this study were extracted using CogStack, a healthcare application framework designed for extracting information from unstructured data sources such as electronic health records (EHRs). EHRs often contain a wealth of information in various formats, including text fields, Word documents, PDFs, and images, which can be challenging to access programmatically. CogStack facilitates the extraction, harmonisation, and processing of this unstructured data, enabling various applications including natural language processing (NLP) (12).

In this study, CogStack was utilised to extract baseline characteristics and clinical risk factors from the EHRs. Baseline characteristics such as age and gender, as well as the occurrence of all-cause mortality, were extracted using standard Elasticsearch queries from structured datasets. Clinical risk factors, on the other hand, were extracted using NLP models that had been trained specifically for this purpose, as detailed in a previous publication (13).

The clinical variables included in the analysis were age (expressed as a continuous variable) and several categorical variables: gender, chronic kidney disease (CKD), hypertension (HTN), heart failure, smoking history, dyslipidaemia, diabetes mellitus (DM), and cerebrovascular accident (CVA). Each categorical variable was expressed as a binary variable indicating the presence or absence of the respective condition.

Image data

Contrast-enhanced CMR images were obtained using a standard contrast agent, gadobutrol (Gadovist, Bayer AG, Leverkusen, Germany), administered at a dose of 0.0075 mmol/kg. Image acquisition was performed 10–20 minutes after contrast administration using Siemens Healthineers or Philips scanners, with both 3 T and 1.5 T field strengths.

The extracted images included 3 long-axis views covering 2-chamber, 3-chamber, and 4-chamber orientations, along with 10 short-axis views covering most or the whole of the myocardium. The majority of LGE images were acquired using white blood (WB) sequences.

Unique identification numbers (IDs) were used to link each image series with the corresponding clinical data. This comprehensive approach allowed for the integration of clinical and imaging data to investigate associations with all-cause mortality in patients undergoing stress perfusion CMR studies.

Neural networks building

Defining labels

Positive LGE segments were annotated based on the AHA 17-segment model, with each segment labelled as positive or negative for LGE. Binary labels were used for all-cause mortality and ventricular arrhythmia (ventricular tachycardia and ventricular fibrillation) prediction. Ground truth was considered to be level 3 CMR readers reporting the studies.

AHA plot classifier

CNN architecture was used for training the AHA plot classifier. Different CNN architectures were tested and trained from scratch, including LeNet with two convolutional layers followed by two fully connected layers (14), AlexNet with eight convolutional layers followed by three fully connected layers (15), VGG19 with 16 convolutional layers and two fully connected layer (16), ResNet50 with 50 layers including convolutional layers organised in residual blocks, batch normalisation layers, activation layers and fully connected layers (17) and GoogleNet with three inception blocks, max pooling layers, and average poolying layers (18). Two training approaches were tested: multi-label classifier and cluster of binary classifiers. Data extraction and neural networks choices are depicted in Figure 1.

Figure 1 A diagram showing data extraction and the training process for AHA classifiers. CNN, convolutional neural networks; AHA, American Heart Association; LGE, late gadolinium enhancement.

The best performing architecture, determined based on validation precision/recall curve or F1 score, was selected for further training. Data was split into 70% for training, 15% for validation, and 15% for testing. Images were resized to 224×224 pixels, and all frames were stacked for each case with an input shape of (224, 224, 13).

Binary cross-entropy loss function was used, and Adam optimiser with a learning rate of 0.001 was chosen. Early stopping was implemented based on the best F1 score achieved in the validation set.

Hybrid neural network (HNN)

HNN was developed to integrate both image and clinical data for predicting mortality and ventricular arrhythmia. CNN architecture was used to extract features from stress perfusion and late gadolinium images, and a multi-layer perceptron (MLP) was used to extract features from clinical variables. Features extracted from both data types were concatenated and passed through Dense layers for final binary prediction. Different CNN architectures were tested similar to AHA classifiers approach. Clinical variables included: age, gender, CKD, HTN, heart failure, smoking history, dyslipidaemia, DM and CVA.

Statistical analysis

Categorical variables were expressed using numbers and percentages, while continuous variables were expressed as means and standard deviations. Follow-up duration was calculated as the mean time to all-cause mortality event, excluding cases without events and those with durations shorter than the interval between the CMR date and the collection date.

The population was stratified into three age subgroups, as cardiovascular disease (CVD) risks differ across adult age groups (19): <65 years, 65–75 years, and >75 years. Differences in baseline characteristics, clinical risk factors, and CMR data among all subgroups were tested using the Chi-square test for categorical variables and one-way analysis of variance (ANOVA) for continuous variables. A P value of <0.05 was considered statistically significant.

For binary classifiers, class weights were utilised during training and calculated as:

classweights=1/n×N/2[1]

where n represents the number of examples per class and N denotes the sample size. Class weights were incorporated into model training to address class imbalance.

Performance metrics for the models included accuracy, recall, precision, area under the curve (AUC), and F1 score. The total AUC was calculated as the micro-average value. Precision, recall and F1 score were calculated for optimal threshold value provided by the receiver operating characteristic (ROC) curve. Information gain for mortality and ventricular arrhythmia predictors was assessed using multivariate regression.

All analyses were conducted using the Python programming language, version 3.10, with the Tensorflow library for model building and training.

Results

Baseline characteristics

All baseline characteristics, CMR data and clinical risk factors are shown in Table 1.

The total number of cases included was 2,740, with 218 patients experiencing positive mortality events (8%). The mean follow-up period was calculated at 1,090 days. Males outnumbered females (63% vs. 37%) in the study, and there were more cases performed on 3 T field strength CMR scanners than on 1.5 T scanners (66% vs. 34%).

The total number of cases with at least one AHA segment positive for LGE was 823 (30%), among which 111 (13%) experienced mortality events, and 84 (10%) had ventricular arrhythmia events. When dividing the population into three age subgroups, the older population aged over 75 years had a higher percentage of positive stress perfusion, as depicted in Figure 2.

Figure 2 Categorical barplot showing different age groups with gender categories and comparison based on positive LGE (with 95% confidence intervals). LGE, late gadolinium enhancement.

There was variability in the number of positive LGE cases based on each AHA segment. The most commonly positive segment was segment 5 (basal inferolateral segment), with 458 positive cases, while the least commonly positive segment was segment 1 (basal anterior segment), with 98 cases.

When assessing all segments combined, the most common cases were those classified as normal studies, with each AHA segment having a score of 0 (1,661 cases, 60.6%). Conversely, the least common cases were those with 16 out of 17 positive segments (0 cases). A breakdown of all individual statistics is depicted in Figure 3.

Figure 3 Categorical barplots showing different AHA segments with the number of positive cases in each category (top) and the number of cases with positive segments combined (bottom). LGE, late gadolinium enhancement; AHA, American Heart Association.

Neural networks training

The best performing neural network was ResNet50. The multi-label classifier demonstrated fair performance (Figure S1) (AUC: 64%), whereas the cluster classifier did not yield any predictions (Figure S2) (AUC: 53%, P<0.001), as illustrated in Figure 4.

Figure 4 Micro-average ROC curve for AHA cluster binary classifiers and multi-label classifier. ROC, receiver operating characteristic; AHA, American Heart Association; AUC, area under the curve.

Multivariate regression analysis revealed the following important predictors for mortality, listed in order of importance: age, left ventricular ejection fraction (LVEF), CKD, HTN, gender, and heart failure. The mortality HNN achieved a satisfactory performance (Figure S3) with an AUC of 77%, as depicted in Figure 5.

Figure 5 Predictors information gain for mortality (left), and AUC for mortality HNN model (right). CVA, cerebrovascular accident; DM, diabetes mellitus; HTN, hypertension; CKD, chronic kidney disease; LV, left ventricular; AUC, area under the curve; HNN, hybrid neural network.

Similarly, multivariate regression analysis identified the key predictors for ventricular arrhythmia, ranked in order of importance as follows: age, LVEF, CKD, HTN, gender, heart failure, smoking, dyslipidaemia, and MI. Similarly, the ventricular arrhythmia HNN achieved a satisfactory performance (Figure S4) with an AUC of 75%, as depicted in Figure 6.

Figure 6 Predictors information gain for ventricular arrhythmia (left), and AUC for ventricular arrhythmia HNN model (right). CVA, cerebrovascular accident; DM, diabetes mellitus; HTN, hypertension; CKD, chronic kidney disease; LV, left ventricular; AUC, area under the curve; HNN, hybrid neural network.

Performance metrics for all networks in shown in Table 2. A diagram of full pipeline and one case test with left anterior descending (LAD) infarct and ventricular arrhythmia is shown in Figure 7.

Figure 7 A diagram showing the full pipeline for qualitative AHA LGE map with a case example of LAD infarct with previous VA. LGE, late gadolinium enhancement; HTN, hypertension; DM, diabetes mellitus; CKD, chronic kidney disease; HF, heart failure; CVA, cerebrovascular accident; FC, fully connected; VA, ventricular arrhythmia; HNN, hybrid neural network; MLP, multi-layer perceptron; LAD, left anterior descending; AHA, American Heart Association.

Discussion

CMR imaging offers a rapid and reliable means of assessing myocardial tissue characteristics and viability, demonstrating high accuracy in detecting MI and fibrosis. However, visual interpretation can be time-consuming, and the accuracy of results relies on factors such as the operator’s training and experience, the LGE sequence utilised, vendor type, and image quality (11). This study highlights the feasibility of AI-based image classification for LGE images based on AHA segmentation, with an average AUC reported at 64%.

Despite encountering some LGE legacy data with poor image quality, CNN classifiers managed to learn certain features and classify some images correctly. Notably, the heavy class imbalance, with around 60.6% of cases featuring 17 normal segments, presents a significant challenge in training models to classify the remaining 39.4% of cases into 17 other classes. From our knowledge, no previous papers have described LGE image classification based on AHA model. Future work could benefit from more balanced data sets containing a greater number of positive LGE cases to enhance model performance.

In addition to its utility in MI and fibrosis detection, CMR with LGE imaging is emerging as a potentially transformative technique for patients presenting with symptoms and signs of acute MI but with angiographically unobstructed coronary arteries (20). It stands as the preferred diagnostic test in such cases, carrying crucial therapeutic implications. A precise diagnosis facilitates appropriate counselling, insurance coverage, outpatient follow-up management, and future risk stratification. In this study, all types of LGE were scored as binary values. With a more stratified and balanced dataset, AI models could be trained to classify different types of LGE, further enhancing patient management.

The emergence of neural networks has made a profound impact on the medical field, with a noticeable surge in clinical AI studies in recent years, indicating their immense potential in extracting features from large datasets. There is also room for integrating various AI techniques within the same pipeline to achieve optimal performance. This study demonstrates the feasibility of training different neural networks for distinct tasks, such as classification using CNN and outcome prediction using HNN, and integrating both networks into a single pipeline.

The utilisation of automated pipelines in clinical workflows offers several advantages, including reproducible results and increased efficiency. Manual interpretation of LGE from CMR images can be time-consuming, with expert readers often spending several minutes, or even longer in challenging cases, reviewing images frame by frame until confident in their findings. In contrast, automated AI pipelines generate AHA plots within seconds. With further enhancements in performance, such pipelines have the potential to streamline clinical workflows by replacing repetitive tasks, thereby improving efficiency.

Limitations

The primary limitation of this study stems from the predominantly retrospective nature of data collection, primarily relying on legacy LGE images of limited quality. This compromises model performance during testing. Additionally, the substantial class imbalance, with a majority of cases featuring normal segments, presents challenges for accurately predicting positive abnormal segments in external validation datasets.

Another limitation concerns the predominance of images acquired using 3 T field strength and scanners from Siemens or Philips vendors. Generalising the model to different field strengths or scanner vendors should be approached cautiously.

A notable proportion of cases were excluded from the baseline sample due to non-diagnostic image quality. While newer high-resolution sequences have replaced some legacy LGE CMR sequences, the models may not have been adequately tested on such data.

The variable follow-up period introduces complexity, with some patients having shorter follow-up durations compared to others, depending on their scan date. This variability could potentially impact analysis, particularly when using predictive binary models. Future research efforts should prioritise utilising data from randomised controlled trials (RCTs) to enhance the reliability of outcome predictions.

Conclusions

The interpretation of LGE images demands a high level of expertise, and advancements in CMR sequences have notably improved the reliability of qualitative reporting.

Despite being trained on low-quality legacy images and heavily imbalanced datasets, AI-based image classification using AHA segmentation has achieved a commendable level of interpretation. Further optimisation with better stratified datasets holds promise for enhanced performance. Automation of CMR image reporting using this approach has the potential to significantly improve the efficiency of clinical workflows.

The novel approach of predicting mortality and ventricular arrhythmia using HNN on mixed data types, coupled with qualitative AHA plots, presents a promising avenue for clinical applications. The integration of these predictive models into a fully automated classification and prediction pipeline offers considerable potential for improving patient care and outcomes.

Acknowledgments

We are grateful to London AI Centre for their support in data extraction and curation.

Funding: This work was supported by NIHR MedTech Co-operative for Cardiovascular Disease at Guy’s and St Thomas’ NHS Foundation Trust; Wellcome/EPSRC Centre for Medical Engineering (No. WT 203148/Z/16/Z); and the Wellcome Trust Innovator Award (No. 222678/Z/21/Z).

Footnote

Reporting Checklist: The authors have completed the STARD reporting checklist. Available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-94/rc

Data Sharing Statement: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-94/dss

Peer Review File: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-94/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-94/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the King’s College London Research Ethics Committee under reference number 20/ES/0005 and individual consent for this retrospective analysis was waived.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Wesbey GE, Higgins CB, McNamara MT, et al. Effect of gadolinium-DTPA on the magnetic relaxation times of normal and infarcted myocardium. Radiology 1984;153:165-9. [Crossref] [PubMed]
2. Kim RJ, Chen EL, Lima JA, et al. Myocardial Gd-DTPA kinetics determine MRI contrast enhancement and reflect the extent and severity of myocardial injury after acute reperfused infarction. Circulation 1996;94:3318-26. [Crossref] [PubMed]
3. Kellman P, Arai AE. Cardiac imaging techniques for physicians: late enhancement. J Magn Reson Imaging 2012;36:529-42. [Crossref] [PubMed]
4. Kim RJ, Wu E, Rafael A, et al. The use of contrast-enhanced magnetic resonance imaging to identify reversible myocardial dysfunction. N Engl J Med 2000;343:1445-53. [Crossref] [PubMed]
5. Pegg TJ, Selvanayagam JB, Jennifer J, et al. Prediction of global left ventricular functional recovery in patients with heart failure undergoing surgical revascularisation, based on late gadolinium enhancement cardiovascular magnetic resonance. J Cardiovasc Magn Reson 2010;12:56. [Crossref] [PubMed]
6. de Waha S, Eitel I, Desch S, et al. Prognosis after ST-elevation myocardial infarction: a study on cardiac magnetic resonance imaging versus clinical routine. Trials 2014;15:249. [Crossref] [PubMed]
7. Selvadurai BSN, Puntmann VO, Bluemke DA, et al. Definition of Left Ventricular Segments for Cardiac Magnetic Resonance Imaging. JACC Cardiovasc Imaging 2018;11:926-8. [Crossref] [PubMed]
8. Shaw LJ, Berman DS, Picard MH, et al. Comparative definitions for moderate-severe ischemia in stress nuclear, echocardiography, and magnetic resonance imaging. JACC Cardiovasc Imaging 2014;7:593-604. [Crossref] [PubMed]
9. Kim RJ, Shah DJ, Judd RM. How we perform delayed enhancement imaging. J Cardiovasc Magn Reson 2003;5:505-14. [Crossref] [PubMed]
10. Rawat W, Wang Z. Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review. Neural Comput 2017;29:2352-449. [Crossref] [PubMed]
11. Hundley WG, Bluemke DA, Bogaert J, et al. Society for Cardiovascular Magnetic Resonance (SCMR) guidelines for reporting cardiovascular magnetic resonance examinations. J Cardiovasc Magn Reson 2022;24:29. [Crossref] [PubMed]
12. About CogStack. Available online: https://cogstack.org/about/
13. Alskaf E, Frey SM, Scannell CM, et al. Machine learning outcome prediction using stress perfusion cardiac magnetic resonance reports and natural language processing of electronic health records. Inform Med Unlocked 2024;44:101418. [Crossref] [PubMed]
14. LeNet Architecture: A Complete Guide. Available online: https://www.kaggle.com/code/blurredmachine/lenet-architecture-a-complete-guide
15. AlexNet Architecture: A Complete Guide. Available online: https://www.kaggle.com/code/blurredmachine/alexnet-architecture-a-complete-guide
16. SimonyanKZissermanA.Very Deep Convolutional Networks for Large-Scale Image Recognition.axrXiv 2014. arXiv:1409.1556.
17. HeKZhangXRenSDeep Residual Learning for Image Recognition.axrXiv 2015. axrXiv:1512.03385.
18. SzegedyCLiuWJiaYGoing Deeper with Convolutions.axrXiv 2014. arXiv:1409.4842.
19. Rodgers JL, Jones J, Bolleddu SI, et al. Cardiovascular Risks Associated with Gender and Aging. J Cardiovasc Dev Dis 2019;6:19. [Crossref] [PubMed]
20. Pathik B, Raman B, Mohd Amin NH, et al. Troponin-positive chest pain with unobstructed coronary arteries: incremental diagnostic value of cardiovascular magnetic resonance imaging. Eur Heart J Cardiovasc Imaging 2016;17:1146-52. [Crossref] [PubMed]