Training an AI model for publication type classification: a case study on alopecia areata

Introduction

Background

The rapid growth of scientific publications in recent decades has led to an overwhelming volume of research available across major databases like PubMed, Scopus, and Web of Science. In 2021 alone, an estimated 2.5 million articles were published worldwide, creating a vast repository of knowledge (1). PubMed Central (PMC) alone currently houses over 35 million citations, making this database an essential resource for researchers, clinicians, and academics (2). However, the increasing volume of publications poses significant challenges for navigating and extracting meaningful insights, particularly given the diverse types of scientific literature and their varying levels of evidence.

Scientific publications come in different forms, each serving a unique role within the research ecosystem. The most common types include meta-analyses, systematic reviews, clinical trials, case reports, and studies. Meta-analyses and systematic reviews provide the highest levels of evidence by synthesizing results from multiple studies, offering comprehensive insights into specific clinical questions (3). Clinical trials, especially randomized controlled trials (RCTs), generate primary data that assess the efficacy and safety of medical interventions (4). Case reports, while lower in the evidence hierarchy, document rare or novel medical cases and are invaluable for identifying new conditions (3). Preclinical studies, which often involve laboratory or animal research, play a critical role in the early stages of drug development, helping to establish safety and biological activity before advancing to human trials (5).

Understanding the differences between these publication types is crucial for drawing accurate conclusions, evaluating the strength of evidence, and identifying linguistic patterns specific to each type of research. For instance, meta-analyses typically focus on statistical language, while case reports may be more descriptive and patient-centered. Despite these critical distinctions, current publication databases often fail to reflect them. In PMC, a large set of articles are categorized under multiple publication types, with a significant portion specifically classified as research articles (6).

This complex classification hampers the discovery of relevant studies and reduces easy accessibility to scientific documentation. To improve the utility of these databases, there is a pressing need to reclassify the vast number of articles currently labeled as research articles. By refining the classification system, researchers and clinicians would be able to more effectively filter high-quality evidence, ultimately improving the documentation and its application in research and clinical practice. Reclassifying these articles could significantly reduce difficulties in accessing the literature, as well as diminish the workload and cognitive effort required to repeat a clear classification at a later stage, ensuring that the vast amount of published data is used optimally to advance knowledge and inform decision-making.

Rationale and knowledge gap

Alopecia areata (AA) is an autoimmune condition characterized by sudden, patchy hair loss, affecting individuals across all ages, genders, and ethnicities (7). The estimated lifetime prevalence of AA is 0.10% globally—0.12% in adults and 0.03% in children (7). Although its exact cause remains unknown, factors such as genetic predisposition, autoimmune comorbidities (e.g., thyroid disease, vitiligo), and environmental triggers like stress or viral infections are thought to contribute to its development (8). The psychological toll of AA is considerable. Patients often struggle with self-image, which can lead to social withdrawal, anxiety, and depression. The unpredictable nature of the disease only exacerbates emotional distress, making effective treatment all the more crucial (9). Current treatments primarily aim to stimulate hair regrowth, including topical corticosteroids, minoxidil, and immunotherapy. However, responses are variable, with some patients achieving regrowth while others experience persistent hair loss or relapses. Recent progress in small molecule therapies—particularly Janus kinase (JAK) inhibitors—has introduced new possibilities by targeting the immune dysregulation underlying AA. Agents such as tofacitinib (Xeljanz) and ruxolitinib (Jakafi) have demonstrated encouraging results in clinical trials (10).

As AA is a rare condition, any published information may hold value. Given the complex classification already mentioned, which also affects publications related to AA, there is a need to improve their accessibility. At the same time, semantic models are playing an increasingly important role in biomedical data structuring and interpretation. Yet, their performance varies by publication type—for instance, Natural Language Processing (NLP) models for patient detection tend to perform better on clinical trials and case reports than on preclinical studies. This variability underscores the need for more precise article classification to enhance both data accessibility and model reliability.

To our knowledge, only one preprint has addressed the classification of publication types using AI (11). That study introduced a PubMedBERT-based model trained on over 1.2 million articles to automatically tag publication types and study designs. The model outperformed prior approaches, with performance gains attributed to techniques such as undersampling and feature verbalization. Nonetheless, the authors noted that further improvements could be achieved through full-text processing and enhanced model interpretability.

Objective

The aim of this study is to assess the impact of AI-driven solutions on refining the categorization of scientific literature, with the goal of enhancing research efficiency and improving clinical applications.

Methods

Model training

The aim of the AI model we trained was to classify scientific publications into 10 types: abstracts, case reports, clinical trials, guidelines and protocols, meta-analyses, observational studies, other, reviews, studies, and systematic reviews. The model is based on BiomedBERT, a 110-million-parameter biomedical model designed for processing biomedical literature. The pretrained model was fine-tuned through a targeted training process on a curated dataset of 27,437 biomedical articles, with additional optimization using domain-specific examples to enhance annotation performance and ensure accuracy across diverse biomedical text types and clinical contexts. Our internal dataset was used to benchmark and fine-tune model performance against a rule-based baseline. The classification pipeline is continuously improved, with updates to publication typologies and category refinements introduced every 12 months.

Distinctions between publication types were based on general characteristics, such as the presence of a dedicated case presentation section for case reports and a forest plot image for meta-analyses, with additional attention to specific mentions of elements like cross-sectional clinical trials, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) presentations, and others. Given that the model only processes 512 tokens at a time, a strategic selection of text was employed to maximize model performance, ensuring key information was highlighted effectively.

Data retrieval for the AA use case

We conducted a comprehensive literature search on PMC to identify articles mentioning “alopecia areata” anywhere in their full text within a specified timeframe (01 January 2022 to 20 October 2024), using a rule-based approach, publication types were extracted from the source and organized according to a custom-defined hierarchy (guidelines > case reports > meta-analysis > systematic review > review > observational study > clinical trial > study > other > research article).

Application on AA publications

For classifying publications related to AA treatments, we utilized the eXtensible Markup Language (XML) format of each article, enabling efficient classification by the model. The GPU-deployed and optimized model achieves a processing rate of approximately 6,018 articles per minute. A descriptive analysis was performed, and chi-square tests were used to analyze correlations between categorical variables. These statistical tests were conducted using the platform socscistatistics.com.

Statistical analysis

Descriptive statistics, including frequencies and proportions, were used to summarize the classification of article types and journal sources. To compare the distribution of publication types between the AI-based and rule-based classification systems, changes in proportions were calculated and presented as percentage point differences. Associations between categorical variables, such as agreement between journal-reported publication types and those assigned by the AI model, were assessed using chi-square tests. Statistical significance was set at P<0.05. All analyses were performed using the online platform socscistatistics.com.

The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. Since this study does not include human subjects or personal data, Declaration of Helsinki is not applicable.

Results

Model validation

ArcaScience’s information technology (IT) and technical teams curated a diverse dataset of 27,437 training articles, 1,734 test articles, and 1,474 validation articles to evaluate the performance of our AI classification model. The evaluation was based on a golden index—a manually curated dataset in which each article was independently reviewed by two annotators, with disagreements resolved by a third senior reviewer to ensure labeling accuracy. After this curation process, the model demonstrated strong performance metrics, achieving an accuracy of 0.8379, an F1 score of 0.8385, a precision of 0.8366, and a recall of 0.8626.

Descriptive analysis

A total of 3,019 articles mentioning AA were identified within the specified timeframe, originating from 732 different journals. Among these, Frontiers in Immunology contributed 121 articles, International Journal of Molecular Sciences 118 articles, Cureus 117 articles, Clinical, Cosmetic and Investigational Dermatology 77 articles, and JAAD Case Reports 67 articles. Publication types were classified based on PubMed’s categories; however, since articles can be assigned multiple types, we applied a custom hierarchy (guidelines > case reports > meta-analysis > systematic review > review > observational study > clinical trial > study > other > research article) to maintain consistent categorization. Table 1 presents a detailed breakdown, showing that approximately 50.45% of articles were categorized as research articles, followed by reviews at 27.36% and case reports at 11.6%.

Additionally, Table 1 compares publication type classifications between an AI model and a rule-based approach used by PMC. The largest category, “review”, accounted for 27.66% of articles, with a slight increase of +0.3%. “Observational study” experienced the most notable growth at +13.68%, while “clinical trial” rose by +6.5%. Categories such as “case reports” and “meta-analysis” also showed significant increases, indicating broader recognition and more comprehensive classification by the AI model.

Key journals and the classification of publication types

An analysis comparing journal classifications with model results revealed patterns in agreement (“same”) and disagreement (“different”). Frontiers in Immunology reported 121 articles, with 77 aligning with the model’s classifications (“same”) and 44 differing (“different”). International Journal of Molecular Sciences contributed 118 articles, with 66 “same” and 52 “different”. Similarly, Cureus reported 117 articles, split into 62 “same” and 55 “different”. Clinical, Cosmetic and Investigational Dermatology had 77 articles, with 39 matching the model results and 38 differing. Journal of the American Academy of Dermatology (JAAD) published 100 articles, of which 63 were “same” and 37 “different”. In total, 307 articles aligned with the model results, while 226 did not, across 533 publications. The chi-square statistic indicates no significant relationship between journal and the alignment of classification types with model results (P>0.05).

Discussion

Key findings

A critical finding was the detection of case reports, which are important despite their lower rank in the hierarchy of evidence. These reports provide crucial insights into rare or novel disease presentations, off-label drug use, and potential adverse reactions. The AI model identified 432 case reports, representing a 2.72% increase over the 350 identified by the rule-based approach. Undetected case reports can obscure important patient outcomes that may not be captured in more rigorous study types. Observational studies were also detected more accurately, with the AI identifying 13.68% more than the rule-based system, demonstrating the model’s ability to highlight real-world evidence critical for rare diseases like AA.

Research has consistently shown that case reports, while not high-level evidence, significantly contribute to early detection of rare adverse events and novel therapeutic approaches (12). Observational studies provide invaluable insights into long-term outcomes and real-world efficacy, aligning with findings in other chronic conditions where traditional RCTs may not capture the variability of patient responses (13).

The increase in the categorization of “study” and “clinical trial” publications is particularly significant, as these types encompass essential research that provides quantifiable and scientifically validated information related to AA. Studies, including both preclinical in vivo and in vitro research, play a crucial role in understanding the underlying mechanisms of the disease, offering insights into potential therapeutic targets. Clinical trials, on the other hand, provide direct evidence on the efficacy and safety of treatments, offering data that can lead to tangible advances in patient care. As such, these categories are vital for advancing scientific knowledge and facilitating the development of evidence-based therapies for AA.

Strengths and limitations

The AI-driven model demonstrated superior accuracy in identifying nuanced study types such as case reports and observational studies. However, a key limitation lies in the structure of PMC’s classification system, which lists all possible publication types for an article without establishing a hierarchy. Comparing these with a single classification provided by the AI model creates an inherent bias, especially when the rule-based approach relies on author-provided classifications rather than the PubMed metadata itself. This disparity underscores the challenge of achieving an equitable comparison and highlights the need for standardized benchmarks in categorization.

In terms of AI model performance, while the overall results are promising, with an accuracy of 0.8379, an F1 score of 0.8385, precision of 0.8366, and recall of 0.8626, there are notable limitations. One such limitation is the confusion between “study” and “observational study”, which resulted in a 14% misclassification rate between the two. Similarly, a 11% misclassification rate occurred between “study” and “clinical trial”. Another significant limitation is the model’s mono-classification bias, where many articles could easily fit into more than one category, leading to a loss of nuanced categorization. This suggests that further refinement is needed to address multi-class assignments and improve the model’s ability to classify articles with overlapping characteristics.

Finally, even though this study highlights the impact of the AI classification model in AA, more studies should consider its impact on a broader scope.

Comparison with similar research

To our knowledge, except the preprint paper mentioned above (11), no current work has focused on reclassifying publications from public data sources such as companies or startups. Moreover, querying “publication type classification” in PMC does not yield relevant results related to articles discussing the classification of PMC, its rules, or any attempts to reclassify it. This suggests that the area of reclassification of publication types in public databases remains underexplored.

Explanations of findings

The discrepancies between AI and rule-based approaches may stem from AI’s ability to detect nuanced patterns and classify studies based on broader contextual understanding. The higher identification rates for case reports and observational studies suggest that AI can mitigate underrepresentation issues and enhance the diversity of evidence included in systematic reviews.

Implications and actions needed

Integrating AI-driven classification systems can ensure a more comprehensive evidence base, easier access to data and improved performance of NLP models. AI’s role in systematic reviews could improve evidence-based medicine by increasing the research efficiency and recall of relevant information, particularly in complex therapeutic areas like AA. Future steps should include refining AI algorithms to minimize biases, incorporating clinician and researcher feedback, and expanding its application to broader medical fields (14,15).

Conclusions

In conclusion, the use of AI for data classification represents a transformative approach to improving evidence generation in medical research. By refining the categorization of scientific publications, AI enables researchers and clinicians to more easily identify high-quality studies, ultimately enhancing clinical decision-making. As demonstrated in the analysis of AA literature, AI-driven solutions can overcome the limitations of traditional rule-based approaches, reducing misclassification and ensuring that valuable evidence, such as case reports and observational studies, is not overlooked. The continued advancement of AI technologies promises to significantly improve the efficiency and accuracy of evidence synthesis, leading to more personalized treatment strategies and better patient outcomes.

Acknowledgments

The article was written with the assistance of ChatGPT-4, a language model developed by OpenAI.

Footnote

Peer Review File: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-455/prf

Funding: None.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jmai.amegroups.com/article/view/10.21037/jmai-24-455/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. Since this study does not include human subjects or personal data, Declaration of Helsinki is not applicable.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. White M. Sample size in quantitative instrument validation studies: A systematic review of articles published in Scopus, 2021. Heliyon 2022;8:e12223. [Crossref] [PubMed]
2. NCBI's Health resources. Available online: https://www.ncbi.nlm.nih.gov/home/health/
3. Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg 2011;128:305-10. [Crossref] [PubMed]
4. Friedman LM, Furberg CD, DeMets DL, et al. Introduction to clinical trials. Fundamentals of clinical trials. Cham: Springer; 2015:1-23.
5. Sena ES, Currie GL, McCann SK, et al. Systematic reviews and meta-analysis of preclinical studies: why perform them and how to appraise them critically. J Cereb Blood Flow Metab 2014;34:737-42. [Crossref] [PubMed]
6. ArcaScience, Data Science Department. Available online: https://www.arcascience.ai/
7. Jeon JJ, Jung SW, Kim YH, et al. Global, regional and national epidemiology of alopecia areata: a systematic review and modelling study. Br J Dermatol 2024;191:325-35. [Crossref] [PubMed]
8. McDonagh AJ, Tazi-Ahnini R. Epidemiology and genetics of alopecia areata. Clin Exp Dermatol 2002;27:405-9. [Crossref] [PubMed]
9. Mesinkovska N, Craiglow B, Ball SG, et al. The Invisible Impact of a Visible Disease: Psychosocial Impact of Alopecia Areata. Dermatol Ther (Heidelb) 2023;13:1503-15. [Crossref] [PubMed]
10. Rudnicka L, Arenbergerova M, Grimalt R, et al. European expert consensus statement on the systemic treatment of alopecia areata. J Eur Acad Dermatol Venereol 2024;38:687-94. [Crossref] [PubMed]
11. Menke JD, Kilicoglu H, Smalheiser NR. Publication Type Tagging using Transformer Models and Multi-Label Classification. AMIA Annu Symp Proc 2025;2024:818-27. [Crossref] [PubMed]
12. Carey JC. The importance of case reports in advancing scientific knowledge of rare diseases. Adv Exp Med Biol 2010;686:77-86. [Crossref] [PubMed]
13. Gershon AS, Jafarzadeh SR, Wilson KC, et al. Clinical Knowledge from Observational Studies. Everything You Wanted to Know but Were Afraid to Ask. Am J Respir Crit Care Med 2018;198:859-67. [Crossref] [PubMed]
14. Pillay TS, Topcu Dİ, Yenice S. Harnessing AI for enhanced evidence-based laboratory medicine (EBLM). Clin Chim Acta 2025;569:120181. [Crossref] [PubMed]
15. Riaz IB, Naqvi SAA, Hasan B, et al. Future of Evidence Synthesis: Automated, Living, and Interactive Systematic Reviews and Meta-analyses. Mayo Clin Proc Digit Health 2024;2:361-5. [Crossref] [PubMed]