Development and evaluation of planned missing data designs for clinical randomized controlled trials generated by the METRIK framework

Introduction

Background

The clinical randomized controlled trial (RCT) is the gold-standard approach for evaluating the efficacy of a medical intervention (e.g., drug). The trial is designed to statistically establish efficacy based on a single outcome, also called the primary endpoint (1), examples of which include overall survival in cancer trials (2) or changes in physiological measures or quality-of-life scores in patients with heart disease (3). Besides the primary outcome, trial investigators monitor many additional exploratory outcomes to generate new hypotheses through post hoc analyses (4), given the intense implementation effort and investment made to conduct the RCT (5); however, doing so increases trial expenses [e.g., by $1.7M USD (5)] without immediate return since they are non-essential to the RCT’s objective (4). Consequently, trial designers have devised a couple of strategies to reduce trial expenses incurred by these exploratory measures. One strategy implements a governance committee that simplifies the data collection protocol by omitting extraneous outcomes (6) at the cost of undermining their prospective scientific value. Another strategy implements a planned missing design (PMD), which removes the collection of certain measurements at certain timepoints rather than removing entire outcomes from the study protocol. Specifically, the random sampling design (RSD) (7) is a type of PMD that randomly samples each measurement across timepoints and subjects. The random sampling of measurements enables inference methods to provide valid estimates of statistical parameters (8,9). RSD variants have been applied across clinical studies, for example, for investigating discrepancies between pediatric patient and parent-reported responses on lengthy questionnaires (10) and for analyzing patterns of change from longitudinal studies (8).

Rationale and knowledge gap

While PMDs based on random sampling are widely used, given their simplicity and statistical properties, they have low efficiency since they do not leverage the correlational structure within the data to inform PMD construction. For example, prior work demonstrated that optimizing the PMD could substantially reduce the data collection budget (e.g., by up to 75%) while matching the statistical properties (e.g., the variance level) of estimates measured under an ad hoc PMD based on random sampling (11). Therefore, it is essential to develop a method that can optimize the PMD’s efficiency based on the dataset properties since we expect that many exploratory outcomes will be correlated overtime and across variables based on prior evidence (12). However, there is no algorithm that constructs a dataset-specific optimal PMD suitable for exploratory outcomes. The algorithm presented in (11) optimized the PMD with respect to a pre-specified statistical parameter of interest (e.g., the variance of the mean slope parameter in a linear growth curve model); therefore, this approach does not extend to the setting of generating PMDs across exploratory outcomes since there can be numerous types of unspecified statistical analyses [e.g., effects of interactions (1,13), effects of nonlinear terms (14)] that could be explored through post hoc analyses.

Objective

The objective of our study is to present a new solution we developed that creates a dataset-specific PMD intended to generalize to unspecified downstream statistical tasks and provide evidence establishing its capacity. Our framework is called METRIK (Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing). Specifically, METRIK learns the PMD by modeling it as a differentiable function and optimizing it for the task of imputation using gradient-based search to generate candidates followed by a selection strategy that compares the generated candidates against baseline PMDs to find high-quality candidates. The candidate PMDs are fit on data collected from an internal pilot study implementing the complete data collection protocol for the RCT of interest. We have empirically validated the superiority of METRIK’s PMDs compared to a design based on random sampling across several downstream statistical tasks and real-world RCT datasets. We present this article in accordance with the TRIPOD reporting checklist (available at https://jmai.amegroups.com/article/view/10.21037/jmai-2025-123/rc).

Methods

Framework description

METRIK is a framework for learning a PMD for an RCT. Its flowchart is shown in Figure 1. First, METRIK requires the user to conduct an internal pilot RCT (1) under the complete data collection protocol p_c, which is a binary matrix of size n_ttimepoints × n_m metrics, that encodes whether a measurement for a specific timepoint-metric pair is collected. Given the data collected from the pilot, D_pilot, METRIK then creates a PMD-imputer pair (pmd_sol, m_sol) that satisfies the user’s constraints that include the target budget or efficiency level e_target, which is the desired fraction of measurements under the complete protocol that should be omitted under the PMD, and eligibility mask E, which is a matrix encoding that determines which measurements are eligible to be omitted. These would include outcomes that are essential to the statistical analysis defining the RCT questions, including the primary outcome, baseline features (e.g., demographics, medical history, etc.), key secondary outcomes, and safety-related endpoints (1,15). The generated PMD, pmd_sol, is applied to the remaining study population of the RCT (N − n_pilot) and the correspondingly omitted measures are imputed using m_sol. Finally, the data from the pilot and remaining RCT study are merged to yield the final dataset.

Figure 1 Flowchart of the METRIK framework. The flowchart demonstrates the METRIK pipeline. First, METRIK implements a pilot RCT conducted under the complete protocol p_c(white = measurements that are collected; gray = measurements that are not collected). METRIK uses the pilot dataset D_pilot to create a PMD-imputer pair that satisfies the user’s constraints, including the target efficiency e_target and eligibility mask E (white = eligible for omission; gray = not eligible). The generated PMD pmd_sol (black = omitted, white = retained) is applied to the remaining RCT population, and omitted measurements are imputed with m_sol. The “+” symbol denotes the merging of the datasets to yield the final dataset D_final. METRIK, Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing; PMD, planned missing design; RCT, randomized controlled trial.

Next, we provide more details underlying each step of the framework.

Conduct pilot study

To learn a suitable PMD-imputer pair, METRIK requires a small amount of data to be collected under the full data collection protocol p_c. Since the PMD is a subset of this data collection protocol, METRIK collects the data using an internal pilot RCT of size n_pilotsubjects (1,16), whose data form a subset of the final RCT dataset of size N subjects. This means that the pilot study’s data originates from the same source as the Phase-3 trial, thereby ensuring comparability across key dimensions like eligibility criteria and collection timeframes. Conducting an internal pilot study avoids collecting additional data beyond the final population size N while providing a path forward for optimizing the data collection protocol for the rest of the RCT (16,17).

Generate the PMD-imputer pair

To learn a suitable PMD-imputer pair, METRIK implements a novel search strategy. First, METRIK generates candidate PMD-imputer pairs by learning an initial set of imputation models for various efficiency levels under the random sampling strategy and then using these imputation models to learn diverse PMD-imputer pairs by optimizing the PMD for the imputation model under different hyperparameter settings. Then, METRIK chooses from among the candidates based on their performance.

Next, we describe each step in more detail.

Initializing imputers

METRIK learns initial imputers using the Multivariate Time-Series Imputation with Transformers (MTSIT) framework (18), given its state-of-the-art performance on tabular datasets (see Appendix 1.1 for details). The flowchart for this step is shown in Figure 2, with detailed pseudocode given in Algorithm 1 in Figure 3. To apply MTSIT, METRIK first pre-processes the pilot study dataset, D_pilot, (Algorithm 1, lines 1-6) by filling in natively missing data, which is encoded by S∈{0,1}npilot×nt×nm, a tensor indicating which measurements were collected per subject (this includes measurements not collected under protocol p_c). METRIK does this to train the model since the model cannot handle null entries; hence, for simplicity, METRIK uses mean-based imputation as a placeholder. After imputation, METRIK performs min-max normalization per feature and divides the dataset into training and validation sets at the patient level.

Figure 2 Flowchart of the initial imputer learning step. First, the pilot dataset D_pilot is pre-processed (e.g., by filling in natively missing data) based on the subject mask tensor S [white (gray) indicates measurements that are (not) collected]. For each efficiency e, METRIK trains an imputer model m_e using MTSIT and masks sampled with the PMD generator pmd_RSD(e) , where the sampled masks are subject to the eligibility constraints enforced by E. The resulting models are stored in M_ref. METRIK, Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing; MTSIT, Multivariate Time-Series Imputation with Transformers; PMD, planned missing design; RSD, random sampling design.

Figure 3 The algorithm for the imputer initialization step in METRIK. METRIK, Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing; MTSIT, Multivariate Time-Series Imputation with Transformers.

After pre-processing the data, METRIK applies MTSIT to train imputers for different efficiencies e spanning the range given by 0 and 100% (Algorithm 1, lines 7-13). To train an imputer m_e for some efficiency e, it samples a mask per subject under the RSD, given by function pmd_RSD(e), where the probability of some measurement being omitted or masked out is given by a Bernoulli distribution parameterized by efficiency level e. Measurements not eligible for masking, as determined by the user-specified constraint matrix E, have 0 masking probability. During training, losses are only evaluated over masked elements that do not correspond to natively missing data (determined by S) to avoid learning from entries that lack ground truth. After training, METRIK stores the initial imputation models along with the RSD function in set M_ref.

Generating candidate PMD-imputer pairs

Next, METRIK generates a candidate set of new PMD-imputer pairs, as shown in Figure 4. Detailed pseudocode is given by Algorithm 2 in Figure 5. To do this, METRIK uses gradient-based search to identify good candidates, given its computational advantage relative to other search algorithms (19). Specifically, to learn a candidate pair, denoted by PMD pmd* and imputer m*, METRIK first initializes a new imputation model using the weights of some reference model m_e and prepends a differentiable binary mask layer (20) to the model (see Appendix 1.2 for details). METRIK initializes this layer according to the distribution used to generate an RSD of efficiency e to mitigate effects from distribution shifts during training (Algorithm 2, lines 5-7). The model with the masking layer is trained on D_pilotusing the MTSIT algorithm, where the PMD is modeled by the distribution parameterized by the mask layer. Furthermore, to control the mask’s efficiency, METRIK incorporates the mask sparsity loss, which is weighted against the imputation objective by λ_mw. To learn a valid PMD, METRIK zeroes out gradients for ineligible measurements under the user constraint set by E (Algorithm 2, line 8). Using this generation algorithm, METRIK learns diverse candidates by sampling different hyperparameter configurations defined by the choice of initial imputer m_e, mask sparsity weight λ_mw, and learning rate η (Algorithm 2, lines 2-4). The candidate pairs are stored in set M*.

Figure 4 Flowchart of the generation of candidate PMD-imputer pairs. METRIK first initializes a new imputer (m*) using weights from a reference imputer (m_e), then prepends a differentiable mask layer (pmd*). The mask layer is initialized using the distribution under the random sampling design generator [i.e., pmd_RSD(e)]. The mask layer and imputer are trained on the pilot dataset D_pilot using MTSIT (with learning rate η ) and the mask sparsity loss, which is weighted by λ_mw. In addition, the mask learning is subject to the constraints determined by the eligibility matrix E. Generated PMD-imputer pairs are stored in m*. METRIK, Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing; MTSIT, Multivariate Time-Series Imputation with Transformers; PMD, planned missing design; RSD, random sampling design.

Figure 5 The algorithm for generating candidate PMD-imputer pairs in METRIK. METRIK, Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing; MTSIT, Multivariate Time-Series Imputation with Transformers; PMD, planned missing design.

Selecting the PMD-imputer pair

METRIK identifies the best PMD-imputer pair of desired efficiency e_targetusing a novel search process, which is shown in Figure 6. Detailed pseudocode is given by Algorithm 3 in Figure 7. Given the reference and candidate PMD-imputer pairs stored in sets M_refand M*, respectively, METRIK chooses between each candidate pair (pmd*, m*) and each reference pair (pmd_ref, m_ref) based on their performance, which is estimated using the pilot data. METRIK determines performance using a suitable measure, such as normalized root-mean-square deviation (nRMSD) (21), which is given by Eq. [1]. nRMSD averages the squared error across masked measurements indexed by the indicator 1_ijm, which indicates if the measurement associated with subject i at timepoint j for metric m has been masked out. The squared error compares the model prediction y^ijm against the ground truth yijm and is normalized to the range of the metric defined by y_m, the matrix of measurements across subjects and timepoints for metric m. The equation presented here differs from others (22) in that it normalizes across subjects to avoid dividing by zero and averages across all metrics to report a single score.

nRMSD=∑(i,j,m)∈ns×nt×nmⅡijm[(y^ijm−yijm)2max(ym)−min(ym)]∑(i,j,m)∈ns×nt×nmⅡijm[1]

Figure 6 Flowchart of the selection of the PMD-imputer pair. METRIK compares candidate pairs (pmd*, m*) against reference pairs (pmd_ref, m_ref) using the pilot validation data (D_pilot). Candidates with higher efficiency (e* > e_ref) and lower imputation error based on upper and lower limits of the confidence intervals (pupper*<pref,lower) are considered eligible. If no solution exists, the reference pairs are returned. Otherwise, METRIK prunes candidate set (Meligible∗) to select the best-performing solutions. The final solution (pmd_sol, m_sol) is chosen from the solution set (m_sol) based on the desired efficiency (e_target). METRIK, Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing; PMD, planned missing design.

Figure 7 The algorithm for selecting the final PMD-imputer pair in METRIK. METRIK, Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing; PMD, planned missing design.

To obtain robust performance estimates, given the small size of the validation split of D_pilot, METRIK draws many samples from each PMD to calculate the error and calculates confidence intervals over the errors using bootstraps (23). Then, METRIK identifies eligible candidates as pairs with higher efficiency (i.e., e* > e_ref) and lower imputation error (i.e., pupper*<pref,lower) compared to the reference pair. For the latter criterion, METRIK uses the upper and lower limits of the confidence intervals for the candidate and reference pairs, respectively, to identify solutions that generalize. If no eligible candidates exist, METRIK selects the reference PMD-imputer pair (Algorithm 3, lines 3-11). Otherwise, when multiple eligible candidates exist, METRIK prunes the set by identifying candidates with comparable efficiency and selecting the one with the lowest imputation error, yielding the set M_{eligible,pruned}*. The identified solutions are stored in M_sol(Algorithm 3, lines 15-31). METRIK repeats this selection process for each reference pair in M_refto yield the final solution set M_sol. Finally, since METRIK has generated solutions spanning efficiencies between 0% and 100%, METRIK returns the PMD-imputer pair given by (pmd_sol, m_sol) whose efficiency is closest to e_target(Algorithm 3, lines 33-39).

Apply PMD-imputer pair

Given the selected PMD-imputer pair (pmd_sol, m_sol), the trial investigator applies this PMD to the remaining number of subjects (N − n_pilot) in the study population. Once the data for these subjects have been collected under pmd_sol, the measurements omitted from pmd_solare imputed using m_solto yield the dataset D^RCT. The final dataset is obtained by concatenating the pilot study data with this dataset.

Experimental setup

Next, we describe the datasets and performance measures used for our experiments. Implementation details (e.g., hyperparameter selection) are given in Appendix 2.

Datasets

We evaluate METRIK on three real-world clinical RCT datasets, which we obtained from the National Institute of Neurological Disorders and Stroke (NINDS) (24). We received approval from NINDS to use the datasets for research purposes. These RCTs are Phase-2/Phase-3 trials that compare the effect of 1–2 experimental drugs against a control condition or standard-of-care in treating various neurological disorders. Specifically, these include the following RCTs: “A Phase 2 Randomized, Double-blind, Placebo-controlled Study to Evaluate the Safety, Tolerability and Activity of Ibudilast (MN-166) in Subjects With Progressive Multiple Sclerosis” (NN102) (25), “A Multicenter, Double-blind, Parallel Group, Placebo Controlled Study of Creatine in Subjects with Treated Parkinson’s Disease Long Term Study” (LS1) (26), and “Clinical Trial Ceftriaxone in Subjects With Amyotrophic Lateral Sclerosis” (CEF) (27). For each RCT, we extract a convenience sample from the dataset (details of this procedure can be found in Appendix 3). We consider only continuous features to simplify our evaluation setup. Furthermore, due to the large size of the CEF and LS1 datasets, we sampled the set of features included in each dataset. The characteristics of the final datasets are reported in Table 1, which includes the number of timepoints n_tand metrics n_m to demonstrate the scale of data collection. The table also reports the total number of measurements eligible for masking, which is a subset of the possible n_t× n_m measurements because the data are sampled irregularly and because the primary metric and baseline measurements are ineligible for masking, given that they are essential for the RCT’s primary objective. Finally, the total number of subjects in the dataset is given by n_s. n_pilotsubjects are used by METRIK while the remaining subjects are used for testing.

Performance measures

To demonstrate the framework’s utility, we first characterize its ability to generate new and diverse PMDs. Specifically, we report the total number of solutions generated under METRIK and the percentage of solutions that are new or distinct relative to the baseline (described ahead). To measure diversity, we report the size of the range in efficiencies of the newly generated solutions.

Next, we report statistical performance over the newly generated PMD-imputer pairs across different sampling budgets in the 5–95% range. Specifically, we calculate the imputation performance over the PMD-masked elements using nRMSD. In addition, to obtain denoised estimates of imputation performance, we report the nRMSD over 10K masked elements obtained by sampling masks under each PMD and applying them to the data samples.

We also perform downstream statistical analyses using the imputed datasets (i.e., the imputed test dataset merged with the pilot dataset). Specifically, we use Generalized Estimating Equations (GEE), a common type of statistical model used for calculating population-average treatment effects from longitudinal data, given its ability to account for correlations and robustness to misspecification of covariance structure (28-30). We consider several common statistical analysis tasks to test the framework’s generalizability, including testing for the effect of the treatment and the effect of treatment-time interactions (29). Specifically, we fit a model per metric, where for each model, we exclude other post-treatment variables to avoid biasing the analysis per standard guidance (14,31) and also exclude baseline predictors to test the model’s sensitivity to imputed measures. In addition, to handle natively missing data, we apply a common strategy in which the model is fit using only observed outcomes (32-34). Following examples from prior work (35), the form of the treatment effect model is given by Eq. [2] and the form of the time-interaction effects model is given by Eq. [3], where Y is the metric value, X is the treatment indicator, t is the time index, and model parameters (intercepts and coefficients) are given by β_*, where β_treat corresponds to the treatment effect and β_treat×timecorresponds to the effect of the interaction between treatment and time.

Y∼βtreatX+β0+β1t[2]

Y∼βtreat×timeXt+β0+β1X+β2t[3]

Besides calculating the parameters of interest, i.e., β_treat and β_treat×time, the GEE model also calculates their variances, i.e., Var(β_treat) and Var(β_treat×time). We compare the parameter estimates and their variances calculated using the PMD-imputed dataset against the estimates calculated using the ground truth dataset. We measure the error using absolute percentage error (36). To test the effect of the imputations on parameter estimates, we assume that the PMD-imputed dataset has the same natively missing data patterns as the ground truth dataset so that the GEE models are fit on corresponding pairs of imputed and ground truth measurements. To yield a single error measure from error estimates across the different metrics, we pool the absolute percentage errors by taking their median (36) [i.e., this is the median absolute percentage error (MAPE)], as shown in Eq. [4].

MAPE=Median[|ypred,f−ytrue,f||ytrue,f+ε|]f∈F×100[4]

In addition to calculating absolute performance levels, we compare the performance measures under METRIK against those obtained under some baseline random sampling longitudinal designs (8,11), which include the RSD and wave design (WD). The WD is a type of longitudinal study design that randomly samples timepoints only and has been used for studying longitudinal patterns efficiently as well as removing bias throughout data collection. When comparing against each baseline, we use the corresponding PMD-imputer pairs as the reference set in the METRIK algorithm.

To obtain robust performance estimates, we characterize their distribution using five-fold cross validation per dataset and visualize results using scatterplots and boxplots and report their summary statistics [i.e., the median and interquartile range (IQR)]. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. We received approval from NINDS to use the datasets for research purposes. IRB approval and patient consent are not applicable to this study.

Results

First, we demonstrate that METRIK generates many diverse PMD-imputer pairs that are distinct from the baseline solutions. We capture this tendency by characterizing the distribution in the number of generated solutions, the percentage of solutions that are new or distinct from each baseline, and the range in the efficiency of the newly generated solutions. Higher numbers for the percentage of new solutions and range of efficiency are better since they indicate a large breadth of unique solutions that cover different efficiencies. For ease of interpretation, we group the percentages into ranges that define solution quality: 0–25% is poor, 26–49% is moderate, 50–74% is good, and 75–100% is excellent. The results are shown in Figure 8 for the RSD baseline. For example, on the NN102 dataset, METRIK generates a median of 29 solutions (IQR: 8) (Figure 8A) with a median of 79% of solutions being new (IQR: 10%) (Figure 8B) and the statistical mode being ‘excellent’. The range in efficiency among the newly generated solutions is a median of 60% (IQR: 5%) (Figure 8C), with the statistical mode being ‘good’. Comparable statistics are seen across the other datasets, as reflected in the remaining boxplots in Figure 8, where the statistical mode for all datasets is classified as ‘good’ or ‘excellent’ for the percentage of new solutions, and as ‘moderate’ or ‘good’ for the efficiency range. When comparing METRIK’s solutions with the WD baseline, similar trends emerge: the mode for percentage of new solutions consistently falls under ‘excellent,’ while the mode for efficiency range is either ‘moderate’ or ‘good’ (Table 2).

Figure 8 Distribution in solution characteristics. (A) Number of solutions, (B) percent of new solutions, and (C) efficiency range of the solution space under METRIK when the reference set is obtained under the RSD (shown for each dataset). METRIK, Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing; RSD, random sampling design. CEF, phase 3 ceftriaxone trial in amyotrophic lateral sclerosis; LS1, long-term creatine trial in Parkinson’s disease; NN102, phase 2 trial of ibudilast in progressive multiple sclerosis.

Next, we demonstrate that the newly generated PMD-imputer pairs under METRIK have superior performance to the baseline ones across imputation and downstream statistical tasks. First, we establish this on the imputation task since the solutions were optimized for this task. As shown in Figure 9, the imputation error under the newly generated solutions is generally lower than the error under the RSD-generated solutions across different efficiencies and datasets. For example, on the NN102 dataset, all of the baseline PMD-imputer pairs have nRMSDs above 0.20 while METRIK’s newly generated pairs have nRMSDs no worse than 0.20 (Figure 9A). The median improvement (reduction) in the nRMSD across efficiencies is 0.017 (IQR: 0.007) (Figure 9B). Similar levels of reduction are observed across other datasets (Table 3: RSD column) and when compared against the WD baseline (Table 3: WD column). The improvements in test imputation performance demonstrate that METRIK’s selection strategy identifies PMD-imputer pairs that generalize despite being selected on a small validation dataset.

Figure 9 Comparison of imputation performance between METRIK and the RSD baseline on NN102. (A) nRMSD as a function of PMD efficiency under METRIK vs. the RSD baseline. (B) The distribution of the performance difference (METRIK minus baseline), with negative scores indicating that METRIK has lower nRMSD. METRIK, Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing; NN102, phase 2 trial of ibudilast in progressive multiple sclerosis; nRMSD, normalized root-mean-square deviation; PMD, planned missing design; RSD, random sampling design.

Next, we demonstrate that solutions under METRIK generalize to downstream statistical tasks for which they were not optimized. We demonstrate this by evaluating the error in estimates of sample statistical parameters (i.e., average and variance of treatment effect and treatment-time interactions) against ground truth estimates. Across a range of efficiencies, METRIK’s newly generated solutions have MAPEs that are generally lower or competitive with those under the baselines. This is shown for the NN102 dataset when compared against the RSD baseline in Figure 10. For example, the MAPE in the treatment effect coefficient typically lies below 10% for efficiencies less than 30% under METRIK’s newly generated solutions while the MAPE typically lies above 10% for the baseline solutions (Figure 10A). The reduction in the MAPE across efficiencies is a median of 7% (IQR: 4%), with more than 75% of METRIK’s solutions reducing the MAPE by more than 5% (Figure 10B). A small fraction (4%) of solutions emerge that increase the MAPE, with the maximum increase being 1.4%; these solutions tend to have high efficiencies (>50%) (Figure 10A). Reductions in MAPE are also observed across other statistical parameters of interest as captured by the summary statistics in Table 4 (first row), although for the variance estimates, the performance improvement is lower. For example, on NN102, the median reduction in MAPE for the variance of the treatment effect is 3% (IQR: 4%); this is because there exists a subset of solutions (23%) that increase MAPE by up to 10% (Figure 10C,10D), and these solutions tend to have high efficiency (>60%). Similar trends are observed across other datasets (see remaining rows of Table 4) and in comparisons against the WD baseline (Table 5). The overall performance improvement under METRIK’s generated solutions across diverse statistical evaluation tasks demonstrates that improvements on imputation generalize to improvements on unseen downstream statistical tasks.

Figure 10 Comparison of MAPE between METRIK solutions and the RSD baseline across some statistical parameters. (A,C) MAPE as a function of PMD efficiency. (B,D) The distribution of the performance difference (METRIK minus baseline), with negative scores indicating that METRIK has lower MAPE. Results by statistical parameter (A,B) treatment effect coefficient (β_treat) and (C,D) variance of treatment effect coefficient (Var(β_treat)). MAPE, median absolute percentage error; METRIK, Measurement EfficienT Randomized Controlled Trials using Transformers with Input MasKing; NN102, phase 2 trial of ibudilast in progressive multiple sclerosis; PMD, planned missing design.

Discussion

Key findings and explanation of findings

Our experimental findings demonstrate that METRIK learns PMDs that generalize better to diverse downstream statistical tasks compared to baseline random sampling strategies. Specifically, we showed across three real-world RCT datasets that METRIK can generate a large, diverse set of solutions that have lower imputation error and estimation error across standard statistical estimation tasks than solutions based on random sampling across a range of efficiencies. There were some exceptions, with a minor fraction of solutions not generalizing well (i.e., they had worse imputation and/or downstream statistical performance); this set of solutions tended to be ones with high efficiencies (>60%). This could be because the selected PMD may be skewed towards omitting entire variables (i.e., timepoints), which could make statistical parameter estimation more difficult in contrast to estimation under the RSD. In addition, the performance reduction observed under METRIK was less for variance parameters; this could be because of the variance added by the imputation. Overall, our findings validate the core principles of METRIK: our algorithm efficiently generates numerous strong candidates using a novel differentiable architecture, identifies promising solutions through imputation performance comparisons with baseline PMD-imputer pairs, and demonstrates that optimizing for imputation improvements leads to enhanced downstream parameter estimation.

Strengths and limitations

METRIK’s strengths arise from its ability to efficiently explore the vast, high-dimensional search space enabled by RCTs through a novel differentiable architecture, and by leveraging existing baseline PMDs to identify superior solutions. Our study demonstrates METRIK’s performance advantage over various current random sampling strategies (8,11), with results validated across multiple real-world RCT datasets. However, there are important limitations in our approach and study design that could restrict its applicability in other RCT contexts, which we discuss below.

Implementation challenges

METRIK assumes that trial investigators can implement an internal pilot study. While this approach does not increase overall costs compared to starting with a full trial protocol, the sequential design can extend trial duration (17), potentially increasing expenses in settings where pilot studies are not standard (37). In addition, pilot studies may not be practical or feasible for trials with outcomes that take a long time to measure (e.g., time-to-event outcomes such as mortality). Trial designers should therefore assess the feasibility of conducting a pilot based on outcome characteristics and anticipated cost savings through the selected PMD before implementing METRIK. Furthermore, METRIK is only compatible with tabular datasets and cannot impute unstructured data such as text or images. METRIK also assumes that adherence patterns observed during the pilot study will persist, even though altering data collection schedules can affect adherence, and future work should address how to model uncertainties in missing data patterns. Beyond implementation issues, ethical considerations may arise due to regulatory reasons [e.g., safety-related events need to be monitored accurately to determine if trials should be terminated (1)] or lack of patient consent (38). Also, data collection protocols may need to be justified (39), which could present challenges, particularly when METRIK’s outputs lack interpretability. In these cases, trial designers must incorporate such constraints when learning the PMD or during data collection.

Generalizability of study findings

Several aspects of our study warrant further expansion to assess generalizability. Our current assessments focus on common downstream statistical analyses—such as treatment effect estimation and variance using GEE models—but METRIK could also be evaluated within other analytic frameworks, such as covariate adjustment or subgroup analysis; it could also be extended to handle categorical variables. Since these analyses still rely on the imputed variables, we anticipate similar performance improvements under METRIK, though this requires confirmation. In addition, our experiments used GEE models applied only to observed outcomes, but modern methods like multiple imputation (34) are being applied to account for uncertainty in missing data. However, applying multiple imputation to METRIK-imputed results may not perform well since multiple imputation is sensitive to data quality and variable interactions (40). Mediation analyses, which assess intervention mechanisms by conditioning on certain post-treatment variables, were not addressed, as they are less common due to potential bias (41). If mediation analyses are a focus, METRIK’s solutions may not always generalize, since their optimization could deprioritize key variables required for mediation. To address this, we recommend excluding these variables from PMD learning.

Another limitation concerns the type of datasets examined. We evaluated METRIK using real-world clinical RCTs in neurological disorders (NINDs) due to public data availability, but future studies should explore METRIK’s application in other medical areas such as oncology or immunology (42), as variations in data types and missingness patterns across domains could impact its efficacy. For example, some clinical trials for cancer may not collect patient-reported outcomes (43), unlike those for neurological disorders (44), potentially affecting METRIK’s generalizability. Also, our study focused on Phase-3 trials as these are commonly implemented as RCTs (45), but METRIK may not be suitable for all trial types. For instance, pragmatic trials—common for conditions such as diabetes (46)—use simpler data protocols, making METRIK less necessary. The utility of METRIK in Phase-2 trials also remains to be evaluated, as these studies are typically smaller and shorter than Phase-3, thus potentially offering lower returns on investment (15). While our work has focused on clinical RCTs due to well-established data burden issues (15), METRIK could also be applied to non-clinical domains, and future research should examine its suitability for RCTs in fields such as agriculture or education.

Comparison with similar research

Our framework presents the first approach to learning a PMD-imputer that is specifically optimized for a given dataset and can generalize across unspecified downstream statistical tasks. This sets it apart from current methods, which typically rely on random sampling to generate PMDs (8,11)—an approach we have shown performs suboptimally across sampled RCTs. Although existing frameworks for learning PMDs do exist, they generally suffer from significant limitations: some only optimize the PMD for a pre-specified statistical parameter via brute-force search over random PMDs (11), while others employ search algorithms that become impractical in the high-dimensional PMD spaces characteristic of RCTs (47,48). In contrast, METRIK efficiently explores these high-dimensional search spaces, enabling the generation of high-quality PMDs that are tailored to the dataset. Importantly, our method does not require investigators to pre-specify the statistical parameter of interest. This flexibility introduces a new capability: investigators can design PMDs that are efficient for data collection across multiple exploratory outcomes, supporting post hoc analyses without imposing significant additional financial costs.

Implications and actions needed

Based on our methodology and experimental findings, the effective use of METRIK is most apparent in clinical RCTs where:

• Researchers aim to collect a wide range of exploratory, tabularly-encoded outcomes while minimizing associated data collection costs.
• A pilot study prior to a Phase 3 trial is feasible within the project’s resources.
• Post hoc analyses will employ standard statistical methods similar to those evaluated in our METRIK experiments, focusing on exploratory variables.
• Trial designers have the flexibility to implement any data collection protocol generated by METRIK, with minimal restrictions on omitting data points (for instance, few regulatory constraints limiting omissions).
• To expand METRIK’s applicability beyond these scenarios, several steps should be considered:
• Integrate METRIK with contemporary statistical approaches for various analyses, such as incorporating multiple imputation for handling native missingness (34) and formally evaluating its performance on alternative statistical tasks like mediation analysis.
• Explore METRIK’s utility in other clinical research contexts and assess the feasibility and benefit of extending support to additional data modalities, such as imaging or unstructured clinical text, leveraging recent advancements in relevant architectures (49,50).

Conclusions

In conclusion, we have introduced METRIK, a novel framework that, for the first time, designs PMDs that yield substantially less biased estimates for any statistical parameter of interest compared to prior art, thereby ensuring that the data collected under the PMD will be useful for unspecified future post hoc analyses while making data collection efficient. METRIK achieves this by generating candidate PMDs and associated imputation models using a differentiable framework and selecting among candidates based on imputation performance obtained on an internal pilot RCT dataset implementing the full data collection protocol. Results across imputation and several downstream statistical tasks across real-world RCT datasets demonstrate that METRIK is effective at discovering PMD-imputer pairs with substantially better statistical performance (i.e., with median reductions in nRMSD and MAPE ranging up to 0.03 and 36%, respectively, across datasets) compared to random-sampling-based baselines. By constructing high-quality PMD-imputer pairs, METRIK obtains estimates on nonprimary outcomes, thereby preserving the scientific, exploratory value of the study while offering study designers an opportunity to strategically invest the savings into higher-priority clinical studies.

Acknowledgments

This work was performed using computing resources from Princeton Research Computing.

This research was based on data from NINDS obtained from its Archived Clinical Research Dataset website. The CEF dataset came from the Clinical Trial Ceftriaxone in Subjects with Amyotrophic Lateral Sclerosis, conducted under principal investigator (PI) Merit E. Cudkowicz, MD, under Grant No. U01NS049640-02. The LS1 dataset came from A Multicenter, Double-blind, Parallel Group, Placebo Controlled Study of Creatine in Subjects with Treated Parkinson’s Disease Long Term Study, conducted under PI Karl Kieburtz, MD, MPH, under Grant No. U01NS43128 NET-PD. The NN102 dataset came from A Phase 2 Randomized, Double-blind, Placebo-controlled Study to Evaluate the Safety, Tolerability and Activity of Ibudilast (MN-166) in Subjects with Progressive Multiple Sclerosis, conducted under PI Robert J. Fox, MD, under Grant No. 1U01NS082329-01A1.

Footnote

Reporting Checklist: The authors have completed the TRIPOD reporting checklist. Available at https://jmai.amegroups.com/article/view/10.21037/jmai-2025-123/rc

Data Sharing Statement: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-2025-123/dss

Peer Review File: Available at https://jmai.amegroups.com/article/view/10.21037/jmai-2025-123/prf

Funding: This work was supported by the National Science Foundation (Grant No. CNS-1907381).

Conflicts of Interest: Both authors have completed the ICMJE uniform disclosure form (available at https://jmai.amegroups.com/article/view/10.21037/jmai-2025-123/coif). Both authors report that a patent is filed for this work. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. We received approval from NINDS to use the datasets for research purposes. IRB approval and patient consent are not applicable to this study.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Friedman LM, Furberg CD, DeMets DL, et al. Fundamentals of clinical trials. Springer; 2015.
2. US Food and Drug Administration. Clinical trial endpoints for the approval of cancer drugs and biologics: Guidance for industry. Oncology Center of Excellence, Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER); 2018. Available online: https://www.fda.gov/media/71195/download
3. US Food and Drug Administration. Multiple endpoints in clinical trials: Guidance for industry. Center for Drug Evaluation and Research (CDER); 2022. Available online: https://www.fda.gov/media/167530/download
4. Moyé L. What can we do about exploratory analyses in clinical trials? Contemp Clin Trials 2015;45:302-10. [Crossref] [PubMed]
5. Getz K, Campo R. Drug development study designs have reached the danger zone. Expert Rev Clin Pharmacol 2013;6:589-91. [Crossref] [PubMed]
6. Getz KA, Kim J, Stergiopoulos S, et al. New Governance Mechanisms to Optimize Protocol Design. Ther Innov Regul Sci 2013;47:651-5. [Crossref] [PubMed]
7. Raghunathan TE, Grizzle JE. A split questionnaire survey design. J Am Stat Assoc 1995;90:54-63.
8. Wu W, Jia F. Applying planned missingness designs to longitudinal panel studies in developmental science: An overview. New Dir Child Adolesc Dev 2021;2021:35-63. [Crossref] [PubMed]
9. Van Buuren S. Flexible imputation of missing data. 2nd ed. CRC Press; 2018.
10. Rioux C, Lewin A, Odejimi OA, et al. Reflection on modern methods: planned missing data designs for epidemiological research. Int J Epidemiol 2020;49:1702-11. [Crossref] [PubMed]
11. Wu W, Jia F, Rhemtulla M, et al. Search for efficient complete and planned missing data designs for analysis of change. Behav Res Methods 2016;48:1047-61. [Crossref] [PubMed]
12. Kavelaars X, Mulder J, Kaptein M. Decision-making with multiple correlated binary outcomes in clinical trials. Stat Methods Med Res 2020;29:3265-77. [Crossref] [PubMed]
13. Cotter J, Schmiege S, Moss A, et al. How to Interact With Interactions: What Clinicians Should Know About Statistical Interactions. Hosp Pediatr 2023;13:e319-23. [Crossref] [PubMed]
14. US Food and Drug Administration. Adjusting for covariates in randomized clinical trials for drugs and biological products: Guidance for industry. Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER), Oncology Center of Excellence (OCE); 2023. Available online: https://www.fda.gov/media/148910/download
15. Getz KA, Stergiopoulos S, Marlborough M, et al. Quantifying the magnitude and cost of collecting extraneous protocol data. Am J Ther 2015;22:117-24. [Crossref] [PubMed]
16. Teare MD, Dimairo M, Shephard N, et al. Sample size requirements to estimate key design parameters from external pilot randomised controlled trials: a simulation study. Trials 2014;15:264. [Crossref] [PubMed]
17. Avery KN, Williamson PR, Gamble C, et al. Informing efficient randomised controlled trials: exploration of challenges in developing progression criteria for internal pilot studies. BMJ Open 2017;7:e013537. [Crossref] [PubMed]
18. Yıldız AY, Koç E, Koç A. Multivariate time series imputation with Transformers. IEEE Signal Process Lett 2022;29:2517-21.
19. Zingg DW, Nemec M, Pulliam TH. A comparative evaluation of genetic and gradient-based algorithms applied to aerodynamic optimization. Eur J Comput Mech 2008;17:103-26.
20. Csordás R, van Steenkiste S, Schmidhuber J. Are neural nets modular? Inspecting functional modularity through differentiable weight masks. In: Proceedings of the 9th International Conference on Learning Representations (ICLR); 2021.
21. Liu M, Li S, Yuan H, et al. Handling missing values in healthcare data: A systematic review of deep learning-based imputation techniques. Artif Intell Med 2023;142:102587. [Crossref] [PubMed]
22. Luo Y. Evaluating the state of the art in missing data imputation for clinical data. Brief Bioinform 2022;23:bbab489. [Crossref] [PubMed]
23. Rosner B. Fundamentals of biostatistics. Cengage Learning; 2016.
24. National Institute of Neurologic Disorder and Stroke. Archived clinical research datasets. 2024. Available online: https://www.ninds.nih.gov/current-research/research-funded-ninds/ clinical-research/archived-clinical-research-datasets
25. Fox RJ, Coffey CS, Conwit R, et al. Phase 2 Trial of Ibudilast in Progressive Multiple Sclerosis. N Engl J Med 2018;379:846-55. [Crossref] [PubMed]
26. Writing Group for the NINDS Exploratory Trials in Parkinson Disease (NET-PD) Investigators. Effect of creatine monohydrate on clinical progression in patients with Parkinson disease: a randomized clinical trial. JAMA 2015;313:584-93. [Crossref] [PubMed]
27. Cudkowicz ME, Titus S, Kearney M, et al. Safety and efficacy of ceftriaxone for amyotrophic lateral sclerosis: a multi-stage, randomised, double-blind, placebo-controlled trial. Lancet Neurol 2014;13:1083-91. [Crossref] [PubMed]
28. Ballinger GA. Using generalized estimating equations for longitudinal data analysis. Organ Res Methods 2004;7:127-50.
29. Schober P, Vetter TR. Repeated measures designs and analysis of longitudinal data: If at first you do not succeed—try, try again. Anesth Analg 2018;127:569-75. [Crossref] [PubMed]
30. Plantinga A, Wilson K. Generalized estimating equations and mixed-effects models for longitudinal data analysis. University of Washington; 2023. Available online: https://si.biostat.washington.edu/institutes/siscer/CR2507
31. Montgomery JM, Nyhan B, Torres M. How conditioning on posttreatment variables can ruin your experiment and what to do about it. Am J Pol Sci 2018;62:760-75.
32. Lin G, Rodriguez RN. Weighted methods for analyzing missing data with the GEE procedure. SAS Institute Inc.; 2014. Available online: https://support.sas.com/resources/papers/proceedings14/SAS166-2014.pdf
33. Bell ML, Fiero M, Horton NJ, et al. Handling missing data in RCTs; a review of the top medical journals. BMC Med Res Methodol 2014;14:118. [Crossref] [PubMed]
34. Medcalf E, Turner RM, Espinoza D, et al. Addressing missing outcome data in randomised controlled trials: A methodological scoping review. Contemp Clin Trials 2024;143:107602. [Crossref] [PubMed]
35. Twisk J, Bosman L, Hoekstra T, et al. Different ways to estimate treatment effects in randomised controlled trials. Contemp Clin Trials Commun 2018;10:80-5. [Crossref] [PubMed]
36. Scikit-learn.org. Metrics and scoring: quantifying the quality of predictions. Mean absolute percentage error. Available online: https://scikit-learn.org/stable/modules/model_evaluation.html#mean-absolute-percentage-error
37. Herbert E, Julious SA, Goodacre S. Progression criteria in trials with an internal pilot: an audit of publicly funded randomised controlled trials. Trials 2019;20:493. [Crossref] [PubMed]
38. Committee on Strategies for Responsible Sharing of Clinical Trial Data; Board on Health Sciences Policy; Institute of Medicine. The clinical trial life cycle and when to share data. In: Sharing clinical trial data: Maximizing benefits, minimizing risk. Washington (DC): National Academies Press (US); 2015.
39. U.S. Food and Drug Administration. E6(R2) good clinical practice: Integrated addendum to ICH E6(R1). 2018. Available online: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/e6r2-good-clinical-practice-integrated-addendum-ich-e6r1
40. Chaput-Langlois S, Stickley ZL, Little TD, et al. Multiple imputation when variables exceed observations: An overview of challenges and solutions. Collabra Psychol 2024;10:92993.
41. Allem JP. Challenges to Mediation Analysis From Experimental Designs. Nicotine Tob Res 2017;19:1120-1. [Crossref] [PubMed]
42. Crowley E, Treweek S, Banister K, et al. Using systematic data categorisation to quantify the types of data collected in clinical trials: the DataCat project. Trials 2020;21:535. [Crossref] [PubMed]
43. Patel K, Ivanov A, Jocelyn T, et al. Patient-Reported Outcomes in Phase 3 Clinical Trials for Blood Cancers: A Systematic Review. JAMA Netw Open 2024;7:e2414425. [Crossref] [PubMed]
44. Saver JL, Warach S, Janis S, et al. Standardizing the structure of stroke clinical and epidemiologic research data: the National Institute of Neurological Disorders and Stroke (NINDS) Stroke Common Data Element (CDE) project. Stroke 2012;43:967-73. [Crossref] [PubMed]
45. Stolberg HO, Norman G, Trop I. Randomized controlled trials. AJR Am J Roentgenol 2004;183:1539-44. [Crossref] [PubMed]
46. Salive ME. Pragmatic clinical trials: An update on NIA-funded real-world research to improve patient care. National Institute on Aging Blog. 2021 Sep 1. Available online: https://www.nia.nih.gov/research/blog/2021/09/pragmatic-clinical-trials-update-nia-funded-real-world-research-improve
47. Adigüzel F, Wedel M. Split questionnaire design for massive surveys. J Mark Res 2008;45:608-17.
48. Imbriano P. Methods for improving efficiency of planned missing data designs [dissertation]. University of Michigan; 2018.
49. Luo S. A survey on multimodal deep learning for image synthesis: Applications, methods, datasets, evaluation metrics, and results comparison. In: Proceedings of the 5th International Conference on Innovation in Artificial Intelligence; 2021;108-20.
50. Fang X, Xu W, Tan FA, et al. Large language models (LLMs) on tabular data: Prediction, generation, and understanding—a survey. arXiv 2024. arXiv:2402.17944.