Multimodal ECG Abnormalities Classification Approach Based on Anamnesis Patient Data and Signal Integration

Abstract

Automated interpretation of 12-lead electrocardiogram (ECG) images offers critical value as a clinical decision support tool, enabling rapid and accurate patient triage in fastpaced emergency environments. However, existing classification models often rely solely on visual data, ignoring the essential patient history utilized by human cardiologists. This paper proposes a novel multimodal deep learning architecture that integrates raw static ECG images with baseline cardiovascular risk factors (e.g., age, sex, blood pressure) to optimize diagnostic precision. the model employs a Contrastive Language-Image Pretraining (CLIP) backbone, utilizing cardiologist reports as an auxiliary supervisory signal during training to extract complex morphological features without requiring manual annotations. During inference, patient clinical metadata generates an attention mask that dynamically scales the extracted visual embeddings. This early-fusion gating mechanism balances information across modalities, enabling the model to adjust its visual processing based on each patient’s risk profile. Evaluated on ten highly imbalanced cardiovascular abnormalities from the MIMIC-IV dataset using Focal Loss, the proposed model achieves a weighted average F1-score of 75.8%, overall AUC of 95.23%, Accuracy of 93.50%, Precision of 75.72% and Recall of 76.15%, establishing a highly competitive benchmark against current state-of-the-art multi-label classifiers. Additionally, the integration of clinical context significantly improved the predictive confidence for critical ischemic events, boosting the detection rate for Acute Myocardial Infarction (AMI) by over 66% compared to an image-only baseline. These results demonstrate that patient clinical context is an indispensable prior, effectively transitioning theoretical ECG classifiers into robust, safety-first automated triage tools.