The ICUSI (Spanish acronym for INR controls, emergency department referral, bleeding and stroke, and heparin injections) questionnaire was designed based on the opinion of an expert panel of study investigators and was conceived as a screening tool to detect patients treated with VKAs with inadequate anticoagulant therapy management in the last 6 months. Initially, the ICUSI questionnaire included 5 questions, with 2–3 possible answers: (1) INR determinations more than once per month (No=0 points; Yes, one month=1 point; Yes, more than one month=2 points), (2) Occasional heparin injections due to low INR levels (No=0 points; Yes, once=1 point; Yes, more than once=2 points), (3) Referred to emergency department due to high INR levels (vitamin K administration) (No=0 points; Yes=1 point), (4) Bleeding requiring visit to the emergency department (No=0 points; Yes=1 point) and (5) Stroke during this period (No=0 points; Yes=1 point).

The sample size was determined based on (a) the number of questions included in the ICUSI questionnaire, (b) the minimum size for obtaining stable multivariate estimations, and (c) the representativeness of the sample on a geographic context.

Categorical variables were summarised as frequencies and percentages. Continuous variables were tested for normality by the Kolmogorov–Smirnov test and presented as mean±standard deviation (SD) or median and interquartile range (IQR), as appropriate.

The Pearson Chi-squared test was used to compare proportions. Differences between continuous and categorical variables were assessed using the Mann–Whitney U test, and correlations between continuous variables were assessed by the Pearson's correlation coefficient.

In order to measure the internal consistency and homogeneity of the ICUSI questionnaire, Cronbach's alpha coefficient was estimated. Confirmatory factor analysis was performed, using polychoric estimation of correlations and the weighted least squares mean and variance (WLSMV) adjusted estimation method, and assuming a one-dimensional structure (Table 1 of the supplementary data).

A logistic regression model was performed to determine the association between the components of the ICUSI score and the risk of TTR < 65%. Relations between predictor variables and with the dichotomous criteria were assessed using odds ratio Chi-square and Monte-Carlo estimates of significance, in order to avoid problems due to small cell frequencies.

Once the optimal logistic model was obtained, a more suitable scoring procedure was proposed to obtain ICUSI scores, to avoid the use of decimal weights and the exponential function. Anchoring the absence of each indicator at the value of 0 (as it is done in the logistic model) consecutive integer scalars were proposed for each increasing level of severity in the indicators. The simplified scoring method was tested against the optimal logistic scoring, ensuring that a similar scaling ranking was preserved (without inversions).

Receiver operating characteristic (ROC) curve was applied to evaluate the predictive ability (expressed as c-index) of the ICUSI score. The Youden index was used to determine the score with the best combination of sensitivity and specificity, in order to establish a cut-off value.

A P value < .05 was accepted as statistically significant. Statistical analyses were performed using IBM SPSS v. 25.0 (SPSS, Inc., Chicago, IL, USA), Mplus v. 7.3 (Muhén & Muthén, Los Angeles, CA, USA) and MedCalc v. 16.4.3 (MedCalc Software bvba, Ostend, Belgium) for Windows.

An average of 9 INR determinations per patient (range 4–15) was recorded. The mean period since the patient started VKA therapy was 7.5±2.2 months and the mean INR determinations per month was 1.5±0.5 (range 0.4–5.5determinations/month).

Regarding the quality of anticoagulation, the mean TTR according to the method of Rosendaal was 62.2%±20.3%. Of note, more than half of patients (427, 52.5%) had a TTR < 65%. When the quality of anticoagulation was assessed by the proportion of INRs in range, the mean TTR was of 61.7±20.4%. A total of 366 (45.0%) patients showed a TTR < 60%. The Pearson correlation index showed a high direct correlation between both methods (r=0.752, P<.001).

All patients completed the ICUSI questionnaire. Regarding the question about the need for occasional heparin injections in the last 6 months, most (82.9%) patients referred that they did not require them, whereas 7.9% required them on one occasion and 9.2% required them more than one occasion. Concerning the INRs determinations frequency, 34.6% of patients needed only one determination per month during the last 6 months, whereas 46.5% of patients needed more than once per month in one occasion and 18.9% needed more than once per month in several occasions. Finally, 2.7% of patients referred that they attended to the emergency room due to high INRs, 4.6% suffered an active bleeding and 0.9% suffered a stroke/systemic embolism in the last 6 months, which indicates the low prevalence of these events in our population.

Taking the above data into account, the mean ICUSI score was 1.19±1.17, with a minimum of 0 points (31.0%) and a maximum of 6 points (0.1%). The mean score in patients with TTR < 65% was 1.58±1.264 whereas the mean score in patients with TTR ≥ 65% was 0.74±0.845 (P<.001). The TTR negatively correlated with the ICUSI score (r=−0.410, P<.001) i.e., higher scores in the ICUSI questionnaire were related to lower TTR by the Rosendaal method.

Using logistic regression models, we investigated the association of each question of the ICUSI with the risk of TTR < 65% by the Rosendaal method. The first 3 questions (about heparin injections, INRs determinations frequency and attendance/hospitalizations due to high INRs) yielded significant coefficients (P<.05) in the univariate analysis whereas non-significant coefficients were shown by the questions about bleeding (OR 1.71, 95%CI, 0.86–3.41; P=.128) and stroke/systemic embolism (OR 5.49, 95%CI, 0.66–45.78; P=.116) events. Given that the coefficient for the bleeding events was not significant in the univariate model (albeit it is clinical relevant), this variable was not included into the multivariate analysis and thus, it was excluded from the ICUSI questionnaire. Similarly, stroke/systemic embolism did not achieve significance in the multivariate analysis. However, after the judgment of the study investigators and given the clinical significance that the importance that suffering stroke/systemic embolism events has for the management of the anticoagulant therapy regardless of optimal or poor TTR, this question was retained in the final model (Table 2). Thus, the final version of the ICUSI questionnaire included 4 questions (Table 3).

Given the low prevalence, in our sample, of haemorrhagic events (4.6%) and embolic events (0.9%), the statistical power of these variables may have been compromised. Haemorrhagic events were related to the scaling measurement given by Rosendaal TTR, grouped in ten score intervals (χ2=19.6, degree of freedom [df]=9, P=.022), but haemorrhagic events were also related to other predictors (heparin injections: χ2=18.4, df=2, P<.001; INR control frequency: χ2=11.03, df=2, P=.004; hospitalisation by high INR: χ2=60.5, df=1, P<.001), limiting its discriminant power in the multivariate model. On the other hand, embolic events did not exhibit such multicollinearity, being related only to INR control frequency (χ2=9.3, df=2, P=.010).

Finally, the predictive ability of the ICUSI questionnaire for the prediction of TTR < 65% was moderate, with a significant c-index of 0.707 (95%CI, 0.670–0.740; P<.001). Different cut-off points were investigated by comparing sensitivities and specificities as is shown in Table 4. Thus, an ICUSI score of 1 yielded a sensitivity of 80.8% and a specificity of 45.6%, whereas an ICUSI score of 2 yielded a sensitivity of 43.2% and a specificity of 88.1%.

There are some limitations that we should acknowledge. First, the ICUSI questionnaire has not been developed by investigating variables associated with TTR, but the opinion of experts based on the most common reasons and observations referred by patients in the daily clinical practice. However, this study reflects a population representative of Spain and similar to those of all other observational studies carried out in our country. Second, to date there is no prospective external validation and we consider that it is necessary for the introduction of this tool in the real-life. Finally, it is important to note that the questionnaire was filled in as per the responses provided by patients and thus it based on subjective information.