American Journal of Critical Care. 2009;18: 118-121 doi:10.4037/ajcc2009368
Copyright © 2009 by the American Association of Critical-Care Nurses.
CE Article
Interrater Variability of a Severity-of-Illness Score in Critically Ill Adults
By
April D. Miller, PharmD,
Pam Branson, RN, MSN,
P. Shane Winstead, PharmD and
David M. Hiestand, MD, PhD.
April D. Miller is a clinical assistant professor in the South Carolina College of Pharmacy, University of South Carolina Campus, Columbia. At the time of the study, Dr Miller was a critical care pharmacy resident in the Department of Pharmacy Services at University of Kentucky Health-Care, Lexington. Pam Bransonis a nursing patient care facilitator and David M. Hiestandis an assistant professor of medicine and pediatrics in the Division of Pulmonary, Critical Care, and Sleep Medicine at University of Kentucky HealthCare, Lexington. P. Shane Winsteadis a pharmacy specialist in the medical intensive care unit at University of Kentucky HealthCare, Lexington.
Corresponding author: April D. Miller, PharmD. South Carolina College of Pharmacy-USC Campus, Coker Life Sciences, 715 Sumter St, Columbia, SC 29208 (e-mail: millerad{at}sccp.sc.edu).
 |
Abstract
|
|---|
Background Critically ill patients often require sedation and analgesia. Scales have been developed to provide clinicians with sedation targets. Daily interruption of continuous infusions of sedatives and sedation protocols improve patients’ outcomes. However, perceived instability of a patient’s condition can prevent implementation of appropriate sedation targets and daily interruption of sedation.
Objective To evaluate the interrater variability of a severity-of-illness score developed to help nurses determine patient-specific sedation targets and identify candidates for daily interruption of sedation.
Methods The severity-of-illness score was implemented as part of an institutional protocol, and bedside nurses in the medical intensive care unit were taught how to determine and use the score. Bedside nurses recorded the score daily in patients’ medical records. For study purposes, a study nurse who made rounds with the medical team and a pulmonary/critical care fellow physician also independently determined the score.
Results A total of 38 assessments of severity-of-illness scores in 10 different patients were made during the study period. For the 24 assessments made by all 3 observers, the 
coefficient for agreement for the severity-of-illness score was 0.58.
Conclusions The severity-of-illness score had good interrater variability as a tool for determining sedation targets and identifying candidates for daily interruption of sedation. Future study on how use of the score affects sedative dosing and outcomes is needed.
Notice to CE enrollees:A closed-book, multiple-choice examination following this article tests your understanding of the following objectives:- Identify potential problems for critically ill patients whose sedation and analgesia are not properly managed.
- Examine severity-of-illness score and its potential for use in the critical care area.
- Evaluate this study for its value in clinical practice.
To read this article and take the CE test online, visit www.ajcconline.org and click "CE Articles in This Issue." No CE test fee for AACN members.
|
Critically ill patients often require sedation and analgesia to promote comfort, prevent delirium and agitation, and limit recall of the illness. The Society of Critical Care Medicine has issued guidelines for the use of sedative and analgesic agents for prolonged sedation in critically ill patients.1 Those guidelines provide recommendations related to the protocols associated with administration of sedatives. A cornerstone of sedation practices is frequent use of subjective scales to assess agitation or sedation and target prespecified end points.
The ideal scale has not been established; however, it is recommended that institutions choose a scale and educate health care personnel on use of the scale to facilitate communication about sedation goals. In addition to subjective scales, algorithms that allow bedside nurses to titrate sedative dosages improve outcomes. In a prospective controlled study, patients whose sedative dosages were titrated by bedside nurses using a preestablished algorithm had fewer days of mechanical ventilation than did a control group whose sedative dosages were titrated by physicians (4.4 days in the algorithm group vs 10.3 days in the control group, P = .01).2
For patients receiving mechanical ventilation who require continuous infusions of sedatives, daily interruption of the sedation can reduce length of stay in the intensive care unit and days of mechanical ventilation.3 Sedation protocols often include recommendations for implementing daily interruption of sedation.
Despite widespread implementation of protocols that include establishing sedation goals, assessing patients by using sedation scales, and interrupting sedation daily, compliance with protocols is variable. An Australian institutional survey4 of staff after implementation of a sedation protocol highlighted the need for additional tools and protocols to facilitate decision making. This need was manifested by a mean score of 3.29 on a visual analog scale where 0 represented agreement. In an exploratory study into nurses’ perceptions of sedation management, Walker and Gillen5 found that target levels of sedation should be assessed for each patient. Objective tools to help determine sedation goals are not available, however. Walker and Gillen also found that communication between physicians and nurses was clear only 55% of the time. Standardized criteria for determining goals of sedation can serve as an interdisciplinary communication tool. Observation at University of Kentucky HealthCare in Lexington also indicated that one reason for variable compliance with protocols is nurses’ perception of patients’ unstable condition. These assessments are often based on subjective information and limited objective data. Although other objective criteria such as scores on the Acute Physiology and Chronic Health Evaluation can be used to determine severity of illness, such scores are cumbersome to use at the bedside for routine patient care.
| The severity-of-illness score was intended to help identify sedation targets and patients for sedation interruption.
|
The severity-of-illness (SOI) score was developed with objective criteria to help bedside nurses determine appropriate sedation targets and identify patients who are candidates for daily interruption of sedation (see Figure
). This score was developed through an interdisciplinary effort that included physicians, nurses, and pharmacists. The scale ranges from 1 to 3, with 1 assigned to the severely critically ill, 2 assigned to critically ill patients, and 3 assigned to patients in the convalescent phase of critical illness. The criteria for each class were determined on the basis of clinical factors associated with readiness to be weaned from mechanical ventilation. Criteria for category 1 include clinical features associated with severe illness, and patients in this class should be considered too ill to warrant removal of sedation. Criteria in category 2 include clinical features associated with critical illness for which routine cessations of sedation are appropriate in anticipation of future weaning from mechanical ventilation. Finally, criteria in category 3 include clinical features associated with immediate readiness for weaning. The target Riker score and recommendations for daily sedative interruption that correspond to the SOI score are implemented for each patient. Patients are assigned the SOI score that corresponds to the most severely ill category for which they meet at least 1 criterion for that score. For example, a patient whose hemodynamic status is stable who requires a positive end-expiratory pressure of 10 cm H2O would be assigned a score of 2. The purpose of this score is to ensure that patients are not oversedated or under-sedated and that continuous infusions of sedatives are interrupted daily depending on the stability of the patient’s condition.
| Standardized criteria for sedation improves interdisciplinary communication.
|
|
Methods
|
|---|
As part of development of a ventilator and sedation protocol, assessment of the SOI score was implemented for patients receiving mechanical ventilation and prolonged sedation (>24 hours) who were admitted to the medical critical care service at the University of Kentucky HealthCare in Lexington. The Riker Sedation-Agitation Scale is used throughout the institution to assess patients’ sedation6 (Table 1). For patients receiving sedatives, the SOI score is determined and recorded daily by the patient’s nurse as part of the newly implemented protocol. In addition, attending and fellow physicians observe the nurses’ recorded scores, and input on patients’ sedation is provided as needed. The medical critical care service averages approximately 100 admissions per month and operates within a 450-bed academic medical center. Before full-scale implementation of the SOI score, this study was conducted with the primary objective of evaluating the interrater variability of the score.
The study was approved by the institutional review board and carried out in accordance with the ethical standards established by the Helsinki Declaration. Informed consent was obtained from nurses caring for patients in the medical intensive care unit who were willing to participate in the study. As part of routine education during 1 week, 13 patient care nurses and the fellow physician were taught how to determine the SOI score and were asked to record the score for the patients they were caring for as part of the nursing care record. These SOI scores were collected by study personnel each morning before patient care rounds. All scores were collected blindly and without knowledge of the nurse who determined the score. To determine interrater variability, 2 additional observers independently assigned an SOI score to patients. A study nurse involved in the initial development of the score made rounds daily with the team and independently determined the SOI score during morning patient care rounds. A pulmonary/ critical care fellow physician providing care for the patients also independently assessed the patient’s SOI score during morning patient care rounds. After a month-long study period, all scores were then compiled by one investigator (A.M.) for data analysis.
A coefficient was estimated to evaluate the reliability of the SOI score.7 This statistic measures the agreement between the 2 concurrent measures of the same type. Although the scores were determined for the same patient, independence between these assessments is assumed because the score is expected to change with time.
|
Results
|
|---|
During the month-long study period, SOI scores were determined for 10 different patients for a total of 37 assessments at different times. Twenty-four assessments were determined by all 3 observers, and 13 assessments were determined by 2 of the 3 observers. In total, 104 assessments were made with 5 assessments yielding a score of 1, thirty-eight assessments yielding a score of 2, and 61 assessments yielding a score of 3. For the 24 assessments made by all 3 observers, the coefficient for the SOI score was 0.58. Percent agreement, coefficients, and 95% confidence intervals for comparisons between each observer are shown in Table 2. Nurses’ assessments agreed 10% more of the time than did the nurses’ with the physician’s assessments; however, those differences were not statistically significant. Overall, these results indicated good interrater variability for the SOI score.
|
Discussion
|
|---|
According to these data, the SOI score has good interrater variability. Therefore, assessments of the SOI score by bedside nurses can be used to determine sedation goals and to decide which patients meet the criteria for daily interruption of continuous infusions of sedatives.
The SOI score is intended as a supplement to existing scales for assessment of sedation. The ultimate goal of the SOI score is to provide nurses who are directly caring for patients with an objective means of determining sedation goals and criteria for daily interruption of sedation. We attempted to maximize the external validity of the study by having bedside nurses determine the score as part of their daily patient care activities in a manner identical to the actual use of the score. An additional strength of our measurements of validity is the interdisciplinary assessments of the score. Agreement between physicians and nurses on a patient’s SOI score enhances communication. After wide-scale implementation of this score, bedside nurses can quickly determine and implement sedation goals for patients.
| The SOI score supplements existing scales for assessment of sedation.
|
Limitations of this study include the small sample sizes of both personnel determining the score and patients assessed, as well as the potential bias of bedside nurses in score determination. The limitation of small sample size is manifested by the wide confidence intervals associated with the coefficients. In addition, the physician’s and the nurses’ assessments differed; although the difference was not statistically significant, it could potentially affect compliance with the SOI score protocol when implemented. Score determination by a patient’s nurse might be biased because of nurses’ participation in patient care rounds. Although the SOI score was not specifically discussed during rounds, a patient’s sedation and goals are often addressed by the medical team. Future directions for the implementation of the SOI score include additional study of the reliability of the score in surgical critical care patients, and a prospective evaluation of the effect of the score on sedative dosing, days of mechanical ventilation, and length of stay in the hospital and in the intensive care unit.
In summary, the SOI score has good interrater variability, making it a useful tool for communication among health care professionals. It offers bedside nurses a means to guide sedative dosing for their patients.
| The SOI score demonstrates good interrater variability.
|
eLetters
Now that you’ve read the article, create or contribute to an online discussion on this topic. Visit www.ajcconline.org and click "Respond to This Article" in either the full-text or PDF view of the article.
FINANCIAL DISCLOSURES
None reported.
To purchase electronic or print reprints, contact The InnoVision Group, 101 Columbia, Aliso Viejo, CA 92656. Phone, (800) 899-1712 or (949) 362-2050 (ext 532); fax, (949) 362-2049; e-mail, reprints{at}aacn.org.
|
REFERENCES
|
|---|
- Jacobi J, Fraser GL, Coursin DB, et al. Clinical practice guidelines for the sustained use of sedatives and analgesics in the critically ill adult. Crit Care Med. 2002;30(1):119–141.[CrossRef][Medline]
- De Jonghe B, Bastuji-Garin S, Fangio P, et al. Sedation algorithm in critically ill patients without acute brain injury. Crit Care Med. 2005;33(1):120–127.[CrossRef][Medline]
- Kress JP, Pohlman AS, O’Connor MF, Hall JB. Daily interruption of sedative infusions in critically ill patients undergoing mechanical ventilation. N Engl J Med. 2000;342(20):1471–1477.[Abstract/Free Full Text]
- Rose RL, Bucknall T. Staff perceptions on the use of a sedation protocol in the intensive care setting. Aust Crit Care. 2004;17(4):151–159.[CrossRef][Medline]
- Walker N, Gillen P. Investigating nurses’ perceptions of their role in managing sedation in intensive care: an exploratory study. Intensive Crit Care Nurs. 2006;22(6):338–345.[CrossRef][Medline]
- Riker RR, Picard JT, Fraser GL. Prospective evaluation of the Sedation-Agitation Scale for adult critically ill patients. Crit Care Med. 1999;27(7):1325–1329.[CrossRef][Medline]
- Fleiss J. Statistical Methods for Rates and Proportions. 2nd ed. New York, NY: John Wiley & Sons; 1981.
Top
Abstract
Methods
