The prognostic model was developed using a consecutive series of patients with acute ischemic stroke who were admitted to Seoul National University Bundang Hospital between January 1, 2004 and March 31, 2008. Patients hospitalized within 12 hours of stroke onset and showing relevant ischemic lesions on an initial diffusion-weighted magnetic resonance imaging were enrolled using a prospective stroke registry database.14 Of the 960 consecutive patients, 2 were excluded because of inadequate clinical information; therefore, 958 patients were enrolled in the model development cohort. All patients were included for developing the safety prediction model, whereas only 912 patients were included for the global outcome prediction model after excluding 48 patients whose outcome data were unavailable.

A modified Rankin scale score (mRS) score of 0-2 (independence in activities of daily living) was used as a global outcome variable. The scores were obtained prospectively 3 months after stroke onset by a telephone interview as part of the quality-of-care monitoring and improvement program for previously hospitalized stroke patients in the participating institutions. Dedicated, trained stroke nurses were responsible for assessing the mRS.

The safety outcome variable was the occurrence of sHT, defined as any neurologic deterioration accompanied by hemorrhagic transformation on brain imaging as well as that considered to be caused by hemorrhagic transformation based on clinical judgment.15 Neurologic deterioration was defined operationally as worsening of ≥2 points of the National Institutes of Health Stroke Scale (NIHSS) score, ≥1 point on the motor items of the NIHSS, ≥1 point on the level of consciousness NIHSS items, or the presence of any new neurologic symptoms or signs that were not thought to be due to nonstroke causes, according to the definition used in prior studies.16,17

An internal validation of the prognostic model was based on 1,000 bootstrap replicates. Bootstrapping was used to estimate the optimism-corrected model performance estimates.

After the prognostic model was established, it was validated externally using patient data collected from April 2008 through September 2009 in 5 university hospitals or regional stroke centers participating in the Clinical Research Center for Stroke (CRCS). The CRCS continually collects uniform registry data on all stroke patients hospitalized at participating centers through a web-based database, since March 2007, as a prospective multicenter stroke register.18 From this registry database, 954 patients who were hospitalized within 12 hours of onset and who showed relevant ischemic lesions on an initial diffusion-weighted magnetic resonance imaging (MRI) were identified and included in the external validation of the safety model. For the global outcome model, however, data from only 897 patients were used as an external validation cohort after excluding 57 patients because of missing information. A post-hoc external validation was also performed using 7,448 patients from January 2011 to March 2014 from CRCS database to examine an applicability of the updated model to recent stroke patients.

The institutional review boards from all participating centers approved the collection of clinical information, without the need for patient's consent, to the registry database whose purpose was for monitoring and improving quality-of-care of stroke patients, based on the anonymization of patient information, minimal risk to participants, and the retrospective nature of the study. Additional approval was obtained to use the registry database and to continue the collection of data through the review of medical records specifically for this study.

Comparisons between the development and external validation cohorts were performed using the Wilcoxon rank-sum test and chi-square test for continuous and categorical variables, respectively. The non-parametric Wilcoxon rank-sum test was used in the analysis of initial NIHSS or onset to treatment time as they were non-normally distributed. The predictive value of parameters associated with outcomes in the development cohort was analyzed using a logistic regression model. Two-sided P values of <0.05 were considered the minimum level of statistical significance.

A description of the model development process is as follows.

1) Selection of potential predictors. Predictors needed to be preselected to avoid an increase in type I errors and an overfitting of the prognostic model. In this study, potential predictors were decided using 3 steps: (1) performing a systematic review, (2) checking their availability in the registry database, and (3) having discussions with 6 stroke neurologists who participated in the CRCS to develop a consensus for potential predictors. As a result, 18 and 15 predictors were chosen for the global and safety outcome models, respectively. The 18 predictors for the global outcome model were age, gender, previous mRS, initial NIHSS score, previous stroke history, hypertension, diabetes, hyperlipidemia, atrial fibrillation, previous transient ischemic attack, prior use of antiplatelet drugs, prior use of antihypertensive drugs, prior use of statins, prior use of glucose, initial systolic blood pressure (SBP), thrombolysis, onset to treatment time (i.e., time between onset of symptoms and arrival at the hospital), and lacune. For the safety outcome model, the prior use of an anticoagulant drug variable was included in the model, whereas previous mRS, atrial fibrillation, previous transient ischemic attack, and prior use of antihypertensive drugs were excluded. In this study, lacune in the acute phase after cerebral infarction, i.e., within 48 hours from disease onset, was defined in two ways by stroke physicians, depending on whether the patient had undergone a brain MRI. The first definition included patients who underwent a brain MRI for the diagnosis of penetrating artery infarction of the basal ganglia, corona radiate, thalamus, or pons without a defined cardioembolic source, showing a single lesion with the largest diameter of ≤20 mm in an axial diffusion-weighted image slice. The second definition included patients who did not undergo a brain MRI, but had typical lacunar syndromes and no territorial or embolic infarction in brain CT scans. While intravenous thrombolysis was defined as the intravenous injection of recombinant tissue plasminogen activator, intra-arterial thrombolysis consisted of interventional approaches carried out on the relevant arteries either by applying thrombolytic agents via the intra-arterial route or mechanical thrombectomy using intra-arterial devices, without intravenous thrombolysis. Whereas combined thrombolysis implied initial intravenous thrombolysis followed by subsequent intra-arterial thrombolysis.

2) Evaluation of predictor effects. A difference of -2 log-likelihood (-2 LL) between models with and without the predictors was used to evaluate the effect of potential predictors, taking into account that the predictors with high -2 LL had a greater influence on the outcome than those with a low -2 LL.

3) Prognostic model fitting procedure. The existence of multicollinearity among predictors and assumption of linearity of an event's logit on continuous predictors were checked before developing the model. In the global outcome model, interactions between thrombolysis and other variables were included. However, for the safety outcome model, interactions between thrombolysis and other variables were found to be negligible and hence were not included. In addition, the event per predictor variable was less than 10 for the safety outcome model, and this was another reason to exclude the interactions from the safety model. To develop a final prognostic model for the global outcome, a fast backward elimination method using the Akaike information criterion (AIC) was implemented. For the safety model, however, because of the possibility of decreased predictive power from the so-called estimation bias that occurred because of small event per predictor variable, the Lasso method was used.19,20

4) Internal and external validations. Internal and external validations were performed, and bootstrapping was used for the former.

5) Statistics. Discrimination statistics such as C-statistics (equivalent to the area under the receiver operating characteristic curve) were calculated to indicate how well an entire model was matched with observed values. C-statistics >0.80 were considered acceptable values. Model calibration was assessed by the Hosmer-Lemeshow test and by the plots comparing predicted versus observed probability of outcome. Analyses were performed using R-project 2.11.1 (package "rms" version 4.11).

6) Model update. For the practical application of the model, model updating was necessary to increase its predictive power. This study used a logistic calibration method to update the calibration intercept and slope based on the external validation results.21

To develop the global outcome model, the difference of -2 LL between the model with and without predictors was used to assess the effect of predictors. Among the 18 potential predictors chosen a priori, variables of age, previous mRS, initial NIHSS score, previous stroke, diabetes mellitus, history of statin use, thrombolysis, and lacune influenced a good functional outcome with high predictor effects (P<0.1) (Table 2).

Results of fast backward elimination logistic regression analysis, interaction terms with thrombolysis, and nonlinear terms for SBP and initial NIHSS score are shown in Table 3. In the model, a squared term of the initial NIHSS score, along with its linear term were added, whereas SBP was modeled with a restricted cubic spline function with 4 knots.

Discriminative ability of the developed model turned out to be satisfactory with a C-statistic of 0.89 (95% confidence interval [CI], 0.87-0.91; Figure 1A). The Hosmer-Lemeshow test also showed a high degree of goodness of fit (P=0.52). Internal and external validation results showed high optimism-corrected C-statistics of 0.87 (95% CI, 0.83-0.90) and 0.82 (95% CI, 0.79-0.85; Figure 1A), respectively.

Compared with the global outcome model, initial NIHSS score, thrombolysis, and lacune variables were found to influence sHT, among the 15 variables selected initially (Table 2). Using the Lasso method, age, initial NIHSS score, thrombolysis, onset to treatment time, SBP, and glucose were selected as predictors for the safety model (Table 3). Lacune was not selected as a variable for the full model because no sHT occurred in patients with lacune. The safety model showed a high C-statistic (0.84; 95% CI, 0.79-0.88; Figure 1B) and a satisfactory goodness of fit using the Hosmer-Lemeshow test (P=0.27).

The external validation C-statistic was still high (0.82; 95% CI, 0.77-0.86; Figure 1B) and a calibration slope was 0.96 (Figure 2B). A good model fitting was also observed after the external validation was performed using the Hosmer-Lemeshow test (P=0.20).

For the global outcome, the optimism-corrected calibration slopes after the internal and external validations were 0.90 and 0.68, respectively (Figure 2A, upper left panel). As a result, a method of updating both the calibration intercept and slope was needed to increase the predictive power of the global outcome model to apply the model to the new population. After the update, the calibration graph showed that a calibration intercept and slope approached 0 and 1, respectively (Figure 2A, upper right panel). The regression coefficients of the updated model are presented in Table 3.

The deviated intercept and slope of the safety model were also recalibrated to enhance the model's performance. This updating process resulted in a calibration graph with a calibration intercept and slope approaching 0 and 1, respectively (Figure 2B, bottom panels). The regression coefficients in the updated safety model are presented in Table 3.

The thrombolytic modalities were categorized as follows: intravenous alone, intraarterial alone, and combined (intravenous and intraarterial) thrombolysis. As seen in Table 4, the number of patients within each of the thrombolysis modalities was not sufficiently large in our model development dataset. However, in the sensitivity analysis, C-statistics of the global outcome model were calculated for each of these modalities, after excluding patients treated with other modalities, from the external validation cohort, and all were >80% (Figure 3A). For the safety model, C-statistics for each thrombolytic modality were also >80% (Figure 3B). Moreover, the post-hoc external validation of the updated model including the recent patient dataset showed a high C-statistics (0.85; 95% CI, 0.84-0.86), and the calibration slope approached 1 for the global outcome (1.09; Supplementary Figure 1). We further assessed the performance of the updated model with a dataset of 5,757 patients who were admitted within 6 hours of stroke onset by carrying out another post-hoc external validation. The validation results were still satisfactory, showing a C-statistic for this subgroup of 0.84 (95% CI, 0.83-0.85) and a calibration slope of 1.04. This result indicated that, for patients who were admitted after 6 hours onset, the external reliability of the updated model was also maintained.