Novel sentiment positivity index: natural language processing of online patient forums for atopic dermatitis

Introduction

Background

Atopic dermatitis (AD) is a chronic inflammatory skin condition affecting 12.6% of children and 10.2% of adults in the United States (1). The cutaneous manifestations of AD and its associated symptoms of pruritus, pain, and sleep disturbance have profound effects on a patient’s self-perception of physical and mental health (2).

The Dermatology Life Quality Index (DLQI) is a 10-question survey with a 1-week recall period and is a consensus-approved standard metric for AD in clinical trials (3). The DLQI marked a major milestone as it recognized the importance of including patient perspective in efficacy measures of treatment for AD.

Online forums have become a popular resource for the exchange of information and discussion about chronic skin disease, with dramatic growth in subscribership during the coronavirus disease 2019 (COVID-19) pandemic (4). Participants can write comments in response to question prompts submitted by others. The posts tend to involve personal narratives discussing the emotional and daily life impacts of AD and treatments (5,6).

Rationale and knowledge gap

Unlike questionnaire-based evaluations, anonymous online discussions encourage honest and extensive dialogue. The unstructured narrative posts may be harnessed to avoid potential bias limitations of the DLQI. Understanding patient sentiment from online forums regarding AD treatment options may serve as a new avenue for analysis of the overall sentiment of a treatment.

The online form Reddit provides an abundance of online discussion, accessibility and ease of retrieval, and the anonymous nature of the comments. Analysis of other skin disease-related Reddit online forums has shown that there is a robust discussion covering all aspects of the disease, including emotional impact, management options including pharmacotherapy, discussion of side effects, and influence on the quality of life (QoL) (7). While NLP has been used in other medical disease states in a limited fashion, harnessing this to quantify perceptions and satisfaction with therapies remains understudied (8).

Objective

This study aims to assess the perception of different treatments for AD by developing a sentiment positivity index (SPI) of treatment-specific comments made to the Reddit AD forum.

Methods

Software and packages

All operations were conducted with Python 3 and its libraries within a Google Colab environment to allow efficient analysis of large data sets without requiring specialized local computing resources. The extraction of social media posts was done with Pushshift.io, an open-source Reddit application programming interface (API) (9). Natural language processing (NLP) was done using DistilBERT (10). This model was chosen as a practical NLP model that balances performance and computational efficiency. The specific software used was: Python: 3.10.5 (python libraries: PyTorch: 2.0.1, NumPy: 1.23.0, Pandas: 1.5.1, HuggingFace Transformers 4.25), language model DistilBERT “distilbert-base-uncased-finetuned-sst-2-english”.

Data source

The Reddit forum on AD, accessed through reddit.com/r/eczema, was the primary source for narrative evaluations of treatments. The 5-year study period was from November 1, 2017 to December 31, 2022. This time range was chosen as there was a substantial growth in the number of forum subscribers from 8,600 to 60,482 participants, which allowed for a reasonable amount of data within the time frame and sufficient discussion on recent therapies (11).

Data collection and filtering

The Pushshift.io API was queried for all 322,460 posts, including primary submissions and comments during the 5-year study period made to reddit.com/r/eczema. Duplicate posts (n=5,300), defined as those made by the same author with identical post content, were identified by software analysis and subsequently removed. The posts were then filtered to exclusively include those mentioning either the generic or brand name equivalent of a single medication used in the treatment of AD (Table 1). Posts mentioning two or more medications (n=3,326) were excluded from this study to reduce ambiguity and maintain the isolation of effects. Following the filtering process, 28,159 posts were available for analysis.

Demographics

Information pertaining to average post word length and number of unique post authors were obtained. Posts were also searched and tallied for mentions of words indicating female sex including “Woman, Female, Lady, Girl, Gal, Sister, Sis”, for words indicating male sex including “Man, Guy, Brother, Bro, Boy, Male”, and for words indicating discussion of children including “Baby, Kid, Daughter, Son, Child, Toddler, Infant, Kiddo, Youth, Junior”.

Data preprocessing

The length of each post was truncated to the first 600 characters to optimize the efficiency of model execution. Posts were then stripped of any special characters, links to websites, emojis, extra spaces, and punctuation, and converted to lowercase. Posts were sorted to their respective medication which included both brand name and generic search terms (Table 1). To address potential bias inherent in pre-trained NLP models, the mention of the medication’s name in each post was substituted with the term “medication”.

Model selection

Due to favorable accuracy rates reported with transformer-based NLP, consideration was given to using Bidirectional Encoder Representations from Transformers (BERT) (12). With the large volume of posts to be analyzed, preference was given to DistilBERT due to its advantages, including a lighter memory footprint and faster inference speed with minimal compromise on accuracy. Like BERT, DistilBERT is pre-trained on a corpus of 11,038 unpublished books and Wikipedia articles (10).

With the data largely being comprised of colloquial communication typical of social media content, it was elected to use a DistilBERT model fine-tuned to the Stanford Sentiment Treebank (SST) database. This premade model fine-tuned on a corpus of 215,154 phrases derived from movie reviews and is implemented to assess whether a given text expresses positive or negative sentiment and provide an associated confidence score for its prediction (10). The use of this model without further fine-tuning has been demonstrated previously for analysis of alopecia areata forums (13). While it is not trained specifically on medical data, it has been widely used for general sentiment classification and performs well in conversational texts such as online discussions. We did not include domain specific fine tuning because discussion among patients was casual in tone, and therefore best assessed by the default model. Its smaller size allowed efficient processing of a large number of posts that were included in the study.

Model execution

The DistilBERT model fine-tuned to SST was queried with each set of posts associated with a treatment to determine the number of posts that were associated with positive sentiment and negative sentiment. The algorithm was forced to positive or negative sentiment and rounded up to positive sentiment for scores 0.50 or above.

Ratio calculation

SPI was calculated as the percentage of treatment-related posts expressing positive sentiment relative to the total number of posts regarding the same therapy. Treatment perceived more positively by patients results in a higher SPI, and treatments with a less positive perception result in a lower SPI.

Statistical analysis

95% confidence intervals (CIs) were estimated using binomial methods. The 95% CIs for proportions were calculated using the Wilson score interval method, as implemented via the binconf function in the Hmisc package for R. This method was selected for its robust coverage properties across various sample sizes and proportions.

Treatments with less than 10 posts were excluded from the analysis. The threshold of at least 10 posts for inclusion was established a priori to ensure a minimum level of statistical stability for the point estimates and to prevent the inclusion of highly volatile data points and wide 95% CIs that lack both clinical and statistical utility. The analysis was done using R4.2.2.

Ethical considerations

The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The Mayo Clinic Institutional Review Board determined this study to be exempt from ethics approval.

Results

On the online AD forum Reddit, out of the 322,460 posts made during the study period, 11,535 posts mentioned children, 5,431 posts mentioned males, and 2,588 posts mentioned women. There were 39,623 unique post authors and the average post was 50 words long, for the subset of posts specifically related to AD treatments: there were 28,159 posts, with 1,373 mentioning children, 462 mentioning men, 328 mentioning women. For AD treatments, there were 12,439 unique post authors with an average post length of 92 words, or 688 characters, and 38.51% of posts were truncated for analysis.

The most frequently discussed treatments were dupilumab (36.68%), topical tacrolimus (17.49%), and oral antihistamines (11.94%). The least discussed treatment options included topical tofacitinib (0.00%), topical tapinoraf (0.00%), and oral (systemic) tofacitinib (0.06%).

The SPI for all treatments varied from 0.116 to 0.365, with a mean of 0.221. The treatments with the highest SPI were the Janus kinase (JAK) inhibitors: upadacitinib (0.365; 95% CI: 0.264–0.479), topical ruxolitinib (0.324; 95% CI: 0.243–0.417), and baricitinib (0.290; 95% CI: 0.161–0.466). The treatments with the lowest SPI were topical corticosteroids (TCS), specifically Class V corticosteroids (0.116; 95% CI: 0.066–0.196), Class VI corticosteroids (0.128; 95% CI: 0.087–0.185) (Table 2, Figure 1).

Figure 1 Forest plot of estimated SPI and 95% CI by treatment. Vertical axis: AD treatment options with online forum commentary. Horizontal axis: SPI with 95% CI for each treatment option. Treatment perceived more positively by patients results in a higher SPI, and treatments with less positive perception result in a lower SPI. AD, atopic dermatitis; CI, confidence interval; SPI, sentiment positivity index.

JAK inhibitors had lower overall numbers of posts with 159 total posts for systemic JAK inhibitors (abrocitinib, barcitinib, tofacitinib, upadacitinib) and 108 total posts for topical JAK inhibitors (ruxolitinib), resulting in relatively wider CIs than other classes (0.074–0.304; 0.161–0.466; 0.066–0.430; 0.264–0.479; and 0.243–0.417 respectively). Azathioprine also had a smaller total number of posts (n=22) with a wider CI (0.047–0.333) but had a less positive SPI (0.136).

Discussion

Key findings

This study introduces a novel approach using NLP of an online AD forum to develop a SPI. Most notably, SPI of JAK inhibitors were the most positive of all therapies analyzed. The least positive SPI results were for low-to-medium potency Class V–VI TCS.

Comparison with similar research

Metanalyses of clinic trials have yielded estimates of DLQI of topical and systemic treatment options for AD. As these results are compared to SPI results in the current study, there are some consistencies and notably some discrepancies identified. Specifically, among systemic medications, DLQI improved the most with upadacitinib, followed by phototherapy, dupilumab, and tralokinumab (14,15). This is consistent with this study’s findings based on SPI. However, one study found that cyclosporine and methotrexate were associated with highly improved DLQI, while the SPI for these therapies demonstrated lower positivity, which may reflect patient concerns about the safety profiles of cyclosporine and methotrexate (15).

Among topical treatments, the most improvement with DLQI occurred with ruxolitinib, Class V TSCs, and tacrolimus with the least improvement occurring with other topical therapies including Class III TSCs, crisaborole, and pimecrolimus (16). SPI scores for ruxolitinib showed higher relative positivity and thus consistent with DLQI findings. However, all classes of TSCs were associated with low SPI while topical crisaborole was associated with a higher SPI. It is possible the higher SPI for crisaborole is due to its favorable accessibility, as it was available before many of the other topical medications, and has better accessibility (16).

The differences in the assessments between DLQI and SPI could possibly be explained by differences in sampling strategy. TSCs have long been used as first-line treatments along with moisturizers and trigger avoidance in the management of AD, with numerous studies validating their effectiveness (17). However, these treatments are time-consuming, and treatment failure can easily occur without strict compliance, thus studies have shown that this treatment regimen can inflict a significant negative QoL impact (1,18). Additionally, there is prevalent TSC phobia and discussion over topical steroid withdrawal on online forums which may contribute a lower SPI for these therapies (19-21).

Treatment Satisfaction Questionnaire for Medication (TSQM) is another standardized questionnaire that evaluates patient satisfaction with their therapy. In a 2023 study, patients with more severe AD had decreased TSQM scores. The most commonly used treatments in this patient population were oral antihistamines and TCS (22). This correlates with the findings of our study which found lower positivity SPI for low to moderate potency TCS. In a study looking at baricitinib treatment satisfaction by patients, the TSQM showed favorable scores, particularly in the domain of convenience. Our study similarly reflects more positive SPI for baricitinib (23).

A recent study evaluating ChatGPT compared to traditional NLP systems suggested that ChatGPT outperformed in the analysis of patient sentiment (24). Interestingly, upadacitinib and dupilumab had higher portions of positive comments, which is similar to the findings in our study with these treatments associated with a more positive SPI. In contrast, a strength of our study is that we examined many more treatment options available to patients.

Explanations of findings, strengths and limitations

While the DLQI has many advantages including standardization, ease of use, and rapid assessment with broad acceptance which allows for cross-study comparison, several areas for improvement have been identified (25). Studies cite concerns with limited emotional scope, lack of consideration of treatment side effects, lack of anonymity in survey-based studies, and the likelihood of recruitment during active flares adding bias to answers (2,26-28). Of growing concern, the high rates of “not relevant” option selection in the DLQI may lead to falsely deflated scores which underestimate QoL impact and may indicate content validity bias (26).

A strength of this study is that SPI is based on anonymous free text testimonials about treatments, potentially serving as a supplement to address gaps inherent to the DLQI. On an online discussion forum, participants are free to discuss any aspect that is important to them and are not constrained by the domains screened in standardized questionnaires, allowing for the capture of additional struggles that individuals may be experiencing. Commentary from the online format, compared to the clinical setting, may allow for a broader range of AD severities to be represented, including milder disease presentation that may not seek medical care. With anonymous posting and no predetermined recall period, experiences with AD treatments are expected to be driven by thoughtful reflection over recent salient events. There are limitations that should also be considered. The SPI decision algorithm was forced to classify every post as one expressing either positive or negative sentiment, removing the possibility of “no response” or “not relevant”. This binary classification can be favorable in supplanting a limitation within the DLQI; however, it could also potentially misclassify neutral responses as positive or negative or oversimplify complex medical narratives. The binary nature could not provide contextual analysis in this initial study. For example, the posts were not further analyzed to identify which aspect of the therapy was perceived positively or negatively such as treatment efficacy, side effects, convenience, or cost concerns. Lastly, since binary positive or negative sentiment categorization was required, posts that reviewed multiple medications were excluded from this study. However, this specific subset could be an important area for future study as they may provide comparative information.

The NLP model employed was not specifically trained or tested with AD forum data, and thus there may be inaccuracies with sentiment classification. The SST dataset of DistilBERT is trained on movie reviews and has not been validated on medical discussion forums. It is unknown if this may impact the classification of medical posts due to differences in vocabulary or nuanced discussion inherent to medical conditions. Duplicate posts with identical author and post content were removed, but near-duplicates or edited posts may have been included in the final analysis.

On average, SPI for all treatment discussions demonstrated more negative than positive comments, suggesting a possible selection bias as individuals reporting negative experiences with treatment may be more likely to post on an online discussion. Furthermore, published data has demonstrated that negativity in Reddit threads impacts popularity, and specifically in medicine discussions that expressions of anger and anxiety increase thread popularity (29). It is also possible that the overall negative sentiment is a reflection of the chronicity and challenges associated with AD therapy and potentially a gap in current therapeutic options that can address the multidimensional concerns of patients.

The sample from the online discussion forum may not be fully representative of patients with AD. Online forum discussion users are likely to be younger and have more access to technology than the general global population. Specifically for Reddit, posting is anonymous, but it is known that users are more likely to be young and male; specifically, 58% of Reddit users are between the ages of 18 and 34 years and 57% are male (30). By the nature of this study, patients or caregivers who do not have access to a computer with internet or a smartphone would be excluded from this analysis as they would not have access to Reddit. It is also unknown what proportion of authors posting on the forum are adult patients versus caretakers of others with AD as the data extracted from social media limited our ability to analyze patient profiles, which has the potential to confound interpretation and clinical applicability.

SPI is dependent on organic discussion, thus, newer treatments such as JAK inhibitors had less discussion on the forum, leading to larger associated CIs in the SPI calculation. Specifically, treatment access bias may be impacting the newer JAK inhibitors which had the highest SPI, but the smallest sample sizes and widest CIs. Patients accessing JAK inhibitors likely reflect a different patient population, as they are more likely to have previously failed other therapies, may have more severe disease, and are more likely to have access to specialty medical care in higher-income countries, which may limit global generalizability.

Implications and actions needed

NLP to develop SPI emerges as a promising tool for understanding the patient perspective of AD treatment. Future research should first validate SPI and confirm clinical relevance. Next steps could evaluate for longitudinal changes over time, consider subset analyses from demographic data, expanded modeling to allow for neutral response category, and evaluation with varying NLP models may be warranted to identify best approaches.

Expanding this methodology to other diseases and other languages that have substantial online discussions could offer valuable insights into the overall perception by patients of their treatment options. This can help clinicians improve patient satisfaction and gain a more global perspective to treatment perceptions and be better prepared to address patient questions and concerns regarding treatment options.

Conclusions

Using NLP of online forum commentary to create the SPI opens a window of opportunity for clinicians to better understand the patient perceptions of therapeutic treatment options. This approach serves as a model to apply to other medical conditions on a global scale.

Acknowledgments

None.

Footnote

Peer Review File: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-2025-1-235/prf

Funding: None.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://jmai.amegroups.com/article/view/10.21037/jmai-2025-1-235/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The Mayo Clinic Institutional Review Board determined this study to be exempt from ethics approval.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Blome C, Radtke MA, Eissing L, et al. Quality of Life in Patients with Atopic Dermatitis: Disease Burden, Measurement, and Treatment Benefit. Am J Clin Dermatol 2016;17:163-9. [Crossref] [PubMed]
2. Birdi G, Cooke R, Knibb RC. Impact of atopic dermatitis on quality of life in adults: a systematic review and meta-analysis. Int J Dermatol 2020;59:e75-91. [Crossref] [PubMed]
3. Finlay AY, Khan GK. Dermatology Life Quality Index (DLQI)--a simple practical measure for routine clinical use. Clin Exp Dermatol 1994;19:210-6. [Crossref] [PubMed]
4. Joly-Chevrier M, Aly S, Bahous K, et al. Atopic Dermatitis Patient Needs Assessed through the Largest Online Patient Community: A Cross-Sectional Reddit Analysis. J Cutan Med Surg 2024;28:75-7. [Crossref] [PubMed]
5. Silverberg JI, Feldman SR, Smith Begolka W, et al. Patient perspectives of atopic dermatitis: comparative analysis of terminology in social media and scientific literature, identified by a systematic literature review. J Eur Acad Dermatol Venereol 2022;36:1980-90. [Crossref] [PubMed]
6. Voillot P, Riche B, Portafax M, et al. Social Media Platforms Listening Study on Atopic Dermatitis: Quantitative and Qualitative Findings. J Med Internet Res 2022;24:e31140. [Crossref] [PubMed]
7. Rajalingam K, Levin N, Marques O, et al. Treatment options and emotional well-being in patients with rosacea: An unsupervised machine learning analysis of over 200,000 posts. JAAD Int 2023;13:172-8. [Crossref] [PubMed]
8. Rajalingam K, Johansen PM, Levin NJ, et al. Natural language processing of online support group postings reveals patients' perspectives on strategies for managing psoriasis. Dermatol Reports 2024;16:9824. [Crossref] [PubMed]
9. Baumgartner J, Zannettou S, Keegan B, et al. The Pushshift Reddit Dataset. Proceedings of the International AAAI Conference on Web and Social Media 2020;14:830-9.
10. Sanh V, Debut L, Chaumond J, et al. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv 2020. arXiv.1910.01108.
11. Subreddit Statistics for r/eczema. Available online: https://subredditstats.com/r/eczema
12. Kumar S, Roy PP, Dogra DP, et al. A comprehensive review on sentiment analysis: tasks, approaches and applications. arXiv 2023. arXiv.2311.11250.
13. Jueng J, Bhupalam V, Su A, et al. Using artificial intelligence to analyze publicly available social media posts to understand patient perspectives toward specific treatments of alopecia areata. JAAD Int 2022;9:102-4. [Crossref] [PubMed]
14. Silverberg JI, Thyssen JP, Fahrbach K, et al. Comparative efficacy and safety of systemic therapies used in moderate-to-severe atopic dermatitis: a systematic literature review and network meta-analysis. J Eur Acad Dermatol Venereol 2021;35:1797-810. [Crossref] [PubMed]
15. Chu AWL, Wong MM, Rayner DG, et al. Systemic treatments for atopic dermatitis (eczema): Systematic review and network meta-analysis of randomized trials. J Allergy Clin Immunol 2023;152:1470-92. [Crossref] [PubMed]
16. Chu DK, Chu AWL, Rayner DG, et al. Topical treatments for atopic dermatitis (eczema): Systematic review and network meta-analysis of randomized trials. J Allergy Clin Immunol 2023;152:1493-519. [Crossref] [PubMed]
17. Atherton DJ. Topical corticosteroids in atopic dermatitis. BMJ 2003;327:942-3. [Crossref] [PubMed]
18. Tier HL, Balogh EA, Bashyam AM, et al. Tolerability of and Adherence to Topical Treatments in Atopic Dermatitis: A Narrative Review. Dermatol Ther (Heidelb) 2021;11:415-31. [Crossref] [PubMed]
19. Li AW, Yin ES, Antaya RJ. Topical Corticosteroid Phobia in Atopic Dermatitis: A Systematic Review. JAMA Dermatol 2017;153:1036-42. [Crossref] [PubMed]
20. Bowe S, Masterson S, Murray G, et al. Topical steroid withdrawal through the lens of social media. Clin Exp Dermatol 2022;47:1554-7. [Crossref] [PubMed]
21. Teasdale EJ, Muller I, Santer M. Carers' views of topical corticosteroid use in childhood eczema: a qualitative study of online discussion forums. Br J Dermatol 2017;176:1500-7. [Crossref] [PubMed]
22. Bacci ED, Correll JR, Pierce EJ, et al. Burden of adult atopic dermatitis and unmet needs with existing therapies. J Dermatolog Treat 2023;34:2202288. [Crossref] [PubMed]
23. Augustin M, Nosbaum A, Werfel T, et al. Treatment satisfaction of adult patients with moderate-to-severe atopic dermatitis treated with baricitinib in France, Germany, and the United Kingdom: results from a cross-sectional international patient survey. J Dermatolog Treat 2023;34:2276047. [Crossref] [PubMed]
24. Cummins JA, Yu J. Identifying Patient Sentiment in Atopic Dermatitis Treatment: Large Language Model Approach. JMIR Form Res 2026;10:e78054. [Crossref] [PubMed]
25. Twiss J, Meads DM, Preston EP, et al. Can we rely on the Dermatology Life Quality Index as a measure of the impact of psoriasis or atopic dermatitis? J Invest Dermatol 2012;132:76-84. [Crossref] [PubMed]
26. Langenbruch A, Radtke MA, Gutknecht M, et al. Does the Dermatology Life Quality Index (DLQI) underestimate the disease-specific burden of psoriasis patients? J Eur Acad Dermatol Venereol 2019;33:123-7. [Crossref] [PubMed]
27. Balieva FN, Finlay AY, Kupfer J, et al. The Role of Therapy in Impairing Quality of Life in Dermatological Patients: A Multinational Study. Acta Derm Venereol 2018;98:563-9. [Crossref] [PubMed]
28. Dalgard FJ, Gieler U, Tomas-Aragones L, et al. The psychological burden of skin diseases: a cross-sectional multicenter study among dermatological out-patients in 13 European countries. J Invest Dermatol 2015;135:984-91. [Crossref] [PubMed]
29. Abouei M, Kordzadeh N, Ghasemaghaei M, et al. The impact of emotional expressions on the popularity of discussion threads: evidence from Reddit. Internet Research 2025; Epub ahead of print. [Crossref]
30. Proferes N, Jones N, Gilbert S, et al. Studying Reddit: A Systematic Overview of Disciplines, Approaches, Methods, and Ethics. Social Media and Society 2021;7:205630512110190.