Developing an evidence-based framework for grading and assessment of clinical predictive tools
When selecting clinical predictive tools, for implementation at the clinical practice or for recommendation in clinical practice guidelines, clinicians and healthcare professionals are challenged with an overwhelming and ever-growing number of tools, most of which have never been implemented or assessed for comparative effectiveness. To overcome this major challenge, this PhD research aimed at developing, validating, applying, and evaluating a new framework for evidence-based grading and assessment of predictive tools used for clinical decision support; abbreviated as the GRASP framework. The aim of the framework is to provide clinicians with a standardised, evidence-based grading system to support their search for, comparison, and selection of effective tools. The framework grades predictive tools based on the critical appraisal of the published evidence across three dimensions: 1) Phase of evaluation; 2) Level of evidence; and 3) Direction of evidence. The final grade of a tool is based on the highest phase of evaluation, supported by the highest level of positive evidence, or mixed evidence that supports positive conclusion.
After the development, the framework was validated, and its evaluation criteria were updated, through the feedback of a wide international groups of eighty-one experts. On a five-points Likert scale, experts overall strongly agreed to GRASP evaluation criteria (4.35). Sixty-four respondents provided recommendations to open-ended questions regarding adding, removing, or changing evaluation criteria. Forty-three respondents suggested the potential effect should be higher than the usability. Accordingly, GRASP concept and its detailed report were updated. The interrater reliability of the updated framework, to assign grades to predictive tools by independent users, was also evaluated, and the framework was found reliable, useful, and easy to use.
Using the validated and updated version, the framework was applied to grade fourteen paediatric head injury predictive tools. The highest-grade tool is PECARN; the only tool evaluated in post-implementation impact studies. PECARN and CHALICE were evaluated for their potential effect on healthcare, while the remaining twelve tools were only evaluated for predictive performance. Three tools; CATCH, NEXUS II, and Palchak, were externally validated. Three tools; Haydel, Atabaki, and Buchanich, were only internally validated. The remaining six tools; Da Dalt, Greenes, Klemetti, Quayle, Dietrich, and Güzel did not show sufficient internal validity for use in clinical practice. Accordingly, the best tools, the most validated in the literature and/or implemented in the clinical practice, were assigned the highest grades. The assigned grades were correlated with the quality of the tools’ development studies, the experience and credibility of their authors, and the support by well-funded research programs. Furthermore, the impact of using the framework on clinicians and healthcare professionals’ decisions in selecting predictive tools was evaluated.
In a conclusion, the GRASP framework represents a high-level approach to provide clinicians and healthcare professionals with an evidence-based and comprehensive, yet simple and feasible, method to evaluate, compare, and select predictive tools. The GRASP framework is not intended to be applied by clinicians; grading, assessing, and then selecting predictive tools. Rather it is designed to be applied by researchers and then tools with their assigned grades are presented to clinical guideline developers and decision-making clinicians to support selecting and recommending predictive tools.
The GRASP framework is not meant to be prescriptive. A lower-grade tool could be preferred, by clinicians and healthcare professionals, to improve certain clinical or healthcare outcomes that are not supported by a higher-grade tool. For example, a tool that has been discussed for its positive potential effect to improve patient safety could be preferred, according to the clinicians and their objectives, to another tool that has been discussed for its positive post-implementation impact on reducing costs of care, predicting the same clinical condition or course of disease. Selecting predictive tools using the GRASP framework depends on the target objectives and planned priorities, where the framework provides clinicians and healthcare professionals with detailed reports on predictive tools to support their evidence-based evaluation and informed decision making in evaluating, comparing, and selecting clinical predictive tools.
