| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Corresponding author: Dr Mike Tweed, Department of Medicine, School of Medicine and Health Sciences, University of Otago, Wellington, PO Pox 7343, Wellington 6242 New Zealand (e-mail: mike.tweed{at}otago.ac.nz).
 |
Abstract |
|---|
 Top Abstract MethodsResultsDiscussionConclusionReferences |
|---|
Objective To assess intensive care nurses’ knowledge of pressure ulcers and the impact of an educational program on knowledge levels.
Methods A knowledge assessment test was developed. A cohort of registered nurses in a tertiary referral hospital in New Zealand had knowledge assessed 3 times: before an educational program, within 2 weeks after the program, and 20 weeks later. Multivariate analysis was performed to determine if attributes such as length of time since qualifying or level of intensive care unit experience were associated with test scores. The content and results of the assessment test were evaluated.
Results Completion of the educational program resulted in improved levels of knowledge. Mean scores on the assessment test were 84% at baseline and 89% following the educational program. The mean baseline score did not differ significantly from the mean 20-week follow-up score of 85%. No association was detected between demographic data and test scores. Content validity and standard setting were verified by using a variety of methods.
Conclusion Levels of knowledge to prevent and manage pressure ulcers were good initially and improved with an educational program, but soon returned to baseline.
|
Notice to CE enrollees: A closed-book, multiple-choice examination following this article tests your understanding of the following objectives:
|
Studies14,16–24 of nurses’ knowledge of pressure ulcer prediction, prevention, and management have had conflicting results; knowledge was good in some studies and poor in others. These studies were limited by a lack of clarity of content, differences in scoring, and differences in standard setting of the assessment tests.
Educational programs can have positive effects. In one study,25 nurses who reported attendance at a pressure ulcer educational program within the preceding year scored significantly higher on a knowledge questionnaire than did nurses who did not. Educational programs can improve decision making,26 and informed decision making can reduce the incidence and prevalence of pressure ulcers.27,28 In a study24 of the effect of an educational program for registered nurses on knowledge of pressure ulcer risk and prevention, data were collected immediately before and after the program. Scores on knowledge assessment improved significantly after the program for reporting numbers of risk factors, use of the risk assessment tool subscales, and use of more preventive strategies.
The aim of our study was to develop an assessment tool to determine the effect of an educational program on ICU nurses’ knowledge of pressure ulcers.
|
Methods |
|---|
|
TopAbstractMethodsResultsDiscussionConclusionReferences |
|---|
Setting and Sample
The study was conducted in the 12-bed ICU of Wellington Public Hospital, Capital and Coast District Health Board, Wellington, New Zealand, a regional and tertiary referral university teaching hospital. Patients in the ICU represent a wide variety of specialties, including general surgery and medicine, cardiac surgery, neurosurgery, and pediatrics. All qualified registered nurses working within the ICU were invited to participate in the study.
Educational Program
The educational program used was adapted from one developed and used by Prentice29 and followed the Australian Wound Management Association guidelines30 for the prediction and prevention of pressure ulcers. The program was presented by a single investigator (C.T.) to small groups of nursing staff during a 2-week period. An interactive format based on the oral presentation and discussion of 112 slides, lasting approximately 3 hours, was used. Key areas addressed were guideline methods, epidemiology of pressure ulcers in Australia and New Zealand, etiology and pathophysiology, risk factors and risk assessment, pressure ulcer staging, equipment to prevent and manage pressure ulcers, and documentation.
| Nurses hold the most responsibility for prevention and management of pressure ulcers.
|
Knowledge Test
Written tests were developed to collect demographic information and to assess nurses’ knowledge of pressure ulcers. Because knowledge was assessed on 3 separate occasions, in order to maintain consistency, 3 separate but broadly similar knowledge tests were developed. The format was initially based on that used by Prentice,29 but was changed considerably in content and layout after review by an expert group.
Each test consisted of 11 different, but similar, selected (multiple choice) and constructed (short answer) questions on specifics of pressure ulcer prediction, prevention, and management. The question formats were judged appropriate to the purpose and content of the test.31,32 The total score possible for each test was 27.
A panel of 8 international experts was used in the development of the tests and the standard setting. The experts included members of the European Pressure Ulcer Advisory Panel, the Australian Wound Management Association, and 2 experienced clinical nurse specialists from New Zealand. A modified Delphi technique was used,33 with 2 rounds of consultation with feedback between each of the experts. All communications with the expert group took place by e-mail.
A pilot study with 7 nurses from a step-down unit at Wellington Public Hospital was conducted to assess usability of the test and the scoring scheme. The step-down unit was chosen for a practical reason: the nurses in the ICU under study were therefore not exposed to the test questions, avoiding a potential source of bias. Feedback was obtained on a variety of aspects of the test, including clarity and layout, possible ambiguities, time taken to complete the test, and coding of answers for data analysis. On the basis of this feedback, 30 minutes was allowed for taking the actual tests used in the study.
| Scores on the knowledge test increased after the education session, but returned to baseline in 20 weeks.
|
Immediately before participating in the educational program, the ICU nursing staff completed the first (baseline) test. For the second and third tests, given within 2 weeks and 20 weeks after the session, respectively, the same cohort of nurses was used that was used for the first test. Any newly appointed ICU nurses and nurses in ICU managerial positions who had previously viewed the tests were not included in the study. Nurses were asked to enter their staff identification numbers on the test response sheets, enabling paired analysis while maintaining confidentiality. We had no means of identifying individual staff members on the basis of the identification numbers.
A score of 1 was allocated for each correct answer. If the answer was incorrect or the nurse did not answer the question, the score was 0. The possible score for each question was highlighted within the test, enabling nurses to be aware of the scoring scheme. One investigator (C.T.) marked all the tests.
A judgmental procedure34 was used to set the standard; that is, to determine the test score indicative of a minimum acceptable level of knowledge of pressure ulcer prevention. The same group of experts who provided input into the test development set the standard (ie, the passing score). Members of the expert group were requested to determine what a nurse with a minimum level of competency should score on each question and hence the entire test. The standard setting was done before the ICU nurses took the 3 tests.
The passing score was verified during the marking process by using the borderline group (BG)35 and the borderline regression (BR)36 methods. For the BG method, the borderline was defined as test responses that could not be categorized as pass (greater than the minimum acceptable level of knowledge) or fail (less than the minimum acceptable level of knowledge). For the BR method, the anchored point on the scale, which was equivalent to the passing score, was at the minimum acceptable level of knowledge.
Data Analysis
The mean of the scores from expert group opinion was calculated as the passing score. Comparisons between passing scores on the 3 tests were analyzed by using the Friedman test37 to ensure that the 3 tests did not differ significantly.
BG medians were analyzed by using the Kruskal-Wallis test.37 For the BR method, the standard error of the score equivalent to the cut point on the global scale (5/10) was calculated and used.37
The test scores were analyzed by using multivariate analysis37 to consider the following nurse attributes: designation, time since qualifying (grouped in 3-year blocks), major qualifications, length of time worked in ICU (grouped in 3-year blocks), and self-reported previous attendance at an educational program on pressure ulcers within the previous 2 years compared with no educational program.
The standard error of measurement (SEM) was calculated by using the Cronbach 
and the standard deviation of scores.38
Mean scores were calculated for all 3 tests. For test 2 and test 3, differences from baseline were calculated for the cohort of nurses as a whole (unpaired t test) and for individuals matched from their identification number (paired t test).37
The pass rate of each of the 3 groups was calculated and was expressed as a percentage of the total. Differences between pass rates, with 95% confidence intervals, were calculated from the standard error of proportion.37
Questions about important attributes can be predictive of outcomes, and this characteristic is a fundamental aspect of content validation.39 A pass-fail categorization for each test was made on the basis of the actual score achieved by each nurse compared with the passing score set by the expert group. Evaluation of the questions included determining whether or not each of the questions was predictive of the pass-fail categorization, the global score used for BR analysis, and the pass-borderline-fail categorization. These determinations were made by using binary, linear, and ordinal logistic regression analysis, respectively.37,40
|
Results |
|---|
|
TopAbstractMethodsResultsDiscussionConclusionReferences |
|---|
. Of 75 eligible ICU registered nurses, 62 (83%) attended a presentation of the educational program. All nursing staff who attended the educational program completed the first test (baseline). Of the 62 who completed the first test, 38 (61%) completed the second test (within 2 weeks after the program), and 29 (47%) completed the third test (20 weeks after the program).
|
Of the 8 experts, 6 gave an opinion of what score a nurse with a minimum acceptable level of knowledge of pressure ulcers should achieve. The passing score for each of the tests did not differ significantly (P = .50), and a single passing score of 76% was generated for all 3 tests.
The passing scores determined by using the verification methods for each of the tests did not differ significantly (P = .66); the passing score was 76% by the BG method and 69% by the BR method. The standard error of measurement of the scores varied between 5.9% and 7.2% (Table 2).
|
). Paired data were available for 31 nurses; their mean score of 89% on the second test was also higher than their baseline score of 86% (paired t test, P =.01). The pass rate for the test taken within 2 weeks after the educational program was 7% higher than the rate for the baseline test, but the difference in the 2 rates was not significant (95% confidence interval, –1.9% to +16%). Similar patterns were seen for the verification pass scores (Table 2).
For all nurses, the mean score of 85% on the 20-week follow-up test did not differ significantly from the mean score of 84% at baseline (unpaired t test, P = .83). Paired data were available for 21 nurses; their mean scores were 86% at baseline and 85% at 20-week follow-up (paired t test, P = .60). The pass rate for the 20-week follow-up test did not differ from the pass rate for the baseline test. Similar patterns were seen for the verification pass scores (Table 2).
Content Verification
Content validity was further verified by the analysis of whether or not each question was predictive of outcomes. The analysis encompassed the pass-fail decision based on expert judgment, global score, and pass-borderline-fail categorization.
For the baseline test, none of the 11 questions was a significant predictor of the pass-fail decision; 5 questions were significant predictors of the anchored global score used for the BR method. The scores of 50 nurses were categorized as pass, 8 as borderline, and 4 as fail. On regression analysis, no question was a significant predictor of this categorization.
For the test taken within 2 weeks after the educational program, none of the 11 questions was a significant predictor of the pass-fail decision; 10 questions were significant predictors of the anchored global score used for the BR method. The scores of 36 nurses were categorized as pass, 2 as borderline, and 0 as fail. On regression analysis, no question was a significant predictor of this categorization.
For the 20-week follow-up test, none of the 11 questions was a significant predictor of the pass-fail decision. Regarding the anchored global score used for the BR method, all respondents answered 2 questions identically, so no analysis was possible. Of the 9 remaining questions, 8 were predictors of the global score. The scores of 24 nurses were categorized as pass, 4 as borderline, and 1 as fail. On the ordinal regression analysis, 2 questions were significant predictors of categorization.
Association of Demographic Data With Test Scores
We found no association between year of qualification, designation, length of time in the ICU, or self-reporting of additional education on pressure ulcers and outcome scores in any of the 3 tests.
On all 3 tests, nurses were asked to indicate if they had attended any educational programs on pressure ulcers in the preceding 2 years. Results were 12% (7 of 59 nurses) at baseline, 37% (14 of 38 nurses) for the test within 2 weeks after our educational program, and 45% (13 of 29 nurses) at the 20-week follow-up. All respondents in the second and third tests had attended our educational program.
|
Discussion |
|---|
|
TopAbstractMethodsResultsDiscussionConclusionReferences |
|---|
Attendance at the educational program was high, and subsequent completion rates for the first test accounted for 83% (62 of 75) of all identified staff. This good response rate reduces the likelihood of nonresponder bias,41,42 and, although our methods differed markedly from those of other investigators, compares favorably with response rates of 62%16 and 71%.23 For the second and third tests, response rates decreased to 61% and 47%, respectively, of the original 62 staff members. This decrease probably occurred because the requests to complete the tests were via letter rather than via in-person interaction, as was the case for the first test. This method also meant that the nurses could have accessed resource material while completing the second and third tests.
The level of self-reported ICU experience that nursing staff had was considerable: a mean of 82 months (range, 1–360 months). This high level of experience may explain the relative high levels of knowledge of pressure ulcer prevention and management, although only a few nurses (11%) reported that they had attended any educational program on pressure ulcers in the preceding 2 years. All staff members who completed the second and third tests attended our educational session, yet only 37% and 45%, respectively, acknowledged that they had received any education on this subject. What nurses report, consider relevant, or remember may differ from what actually happened or what affects their practice.
Our scoring scheme differs from the schemes used in other research in which knowledge assessment tests were used. In an assessment technique initially developed by Halfens and Eggink21 and subsequently modified by Panagiotopoulou and Kerr,23 a 4-point Likert scale was used. Maylor and Torrance20 used a similar 4-point Likert scale. These methods are used to describe knowledge rather than to decide if an individual’s performance is greater or less than an adequate defining threshold (written communications, R. Halfens, PhD, July 2006, M. Maylor, PhD, July 2006).
We found no differences in knowledge levels between staff members with different levels of experience, qualifications, and seniority. These findings support the results of some studies25,43; other researchers19 have reported that the more senior a position a nurse has, the more knowledge the nurse is likely to have. However, nurses who had been in their current position for more than 5 years were less knowledgeable than nurses whose appointments were more recent.20 The inability of our knowledge assessment test to indicate differences between these attributes may not be a failing of the test itself, as highlighted by the variation in the results of other studies.
| No difference in knowledge level was found related to nurses’ experience, qualifications, or seniority.
|
Results from the first test suggested that existing knowledge levels were good; the mean score was 84%, which was equivalent to 90% of the nurses having at least a minimum level of knowledge of pressure ulcers. These results are consistent with those of other studies16,20,23 in which levels of nurses’ knowledge were also good. However, in some studies,21,24,43,44 nurses’ knowledge was poor. The discrepancies between study outcomes are likely to be due to different assessment methods used, with different nurse cohorts and lack of definition of the threshold score determined acceptable. Examples include the study by Pieper and Mattern,43 where 90% of the cohort of ICU nurses had to answer a question correctly for the cohort to be considered knowledgeable in that subject area, and the study by Mockridge and Anthony,19 who did not expect nurses to score more than 50%, without reference as to what that score meant.
In our study, the use of 3 different tests rather than any changes in nurses’ knowledge might account for the differences in scores and pass rates for the 3 tests. We deemed 3 different tests necessary because using the same test on 3 occasions during a relatively short time (baseline, 2 weeks, and 20 weeks) would constitute prior exposure to the test questions, and then differences in scores on the second and third tests might be due to knowledge of the questions rather than to knowledge of pressure ulcer prevention and management. Equivalence across the 3 tests was ensured by review by the expert panel and confirmation of no differences between standard scores on tests.
Formal methods for standard setting such as expert opinion, the BG method,36 and the BR method37 have not previously been described in the context of clinical nursing practice. These methods are widely used for criterion-referenced standard setting in undergraduate and postgraduate health care assessments.35,36,45–48 We used these 3 different methods to define the score equivalent to an acceptable level of knowledge and then to verify the score. Discrepancy between methods has been noted previously36,45,47,48 and can be attributed to the fact that standard setting relies on human judgment and thus variation is inevitable.49 Despite these discrepancies, the pattern of a nonsignificant increase in pass rate for the test given within 2 weeks after the educational program and then a return to the baseline pass rate at the 20-week follow-up was the same for each method of standard setting.
| Although nurses had high levels of knowledge of pressure ulcer prevention, that knowledge may not be reflected in clinical practice.
|
The scores on the third test were the similar to the scores at baseline. The reasons for this decrease are probably multifactorial; prevention and management of pressure ulcers may not be perceived as a high priority by either individual nurses or the institution, because of the complex illnesses and life-threatening situations of ICU patients. Our results highlight that frequent reenforcement of learning is required to prevent knowledge decay.50
Evaluation of an assessment technique is required because the evaluation either supports or refutes the meaning or interpretation given to the outcome of the assessment.51 No assessment technique is able to measure all facets of what can be termed clinical competence,52 and limitations are inevitable. We used a variety of methods to establish content representation of the tests in our study. A modified Delphi technique33 involving a panel of 8 international experts in pressure ulcers was used in the development phase of the tests. After the test scores were determined, regression modeling indicated that most questions were important to the threshold global score. The low number of questions predictive of the pass-fail decision or pass-borderline-fail categorization probably reflects the lack of variance in these decisions and categorizations with high numbers of passing scores.
In other studies in which knowledge of pressure ulcers also was assessed, content representation was not always described. Some investigators added to an existing scale on the basis of their own opinion and the literature,16,20 or the opinion of an expert.44 Other researchers23,53 used a previously developed scale and also established content by use of expert panels.
Although the cohort of nurses in our study had high levels of knowledge, this finding does not necessarily indicate that this knowledge is reflected in clinical practice. Without observations of nurses actually working, making assumptions about what level of care is being delivered is difficult. Studies in both nursing54,55 and medical56 education have shown that acceptable performance requires not only knowledge, but ability to use that knowledge. In order to determine both of these parameters, assessment of both knowledge and performance is required. A major concern is that whatever knowledge nurses have is not always translated into practice.55 We found only a single study24 in which nurses’ knowledge, as indicated by a test, was compared with how they actually performed. However, as the author acknowledges, the study had considerable limitations; the data were not assessed for reliability or validity and patient outcomes were not included.
Other limitations of our study are that the test at baseline was completed under observation, whereas the second and third tests were not. Use of nonobserved tests often results in missing data and poor response rates, making interpretation of results challenging.42 In addition, respondents might not have answered in the same way if the test taking had been observed. The poor response rate, lack of variation in scores, and high pass rate suggest that the combination of nurses, test, and education program used in this study was underpowered.
Knowledge improved after the educational program, with an increase in mean score. However, rather than an improvement in the cohort as a whole, this finding may have only represented improvement in those nurses whose knowledge was already greater than the minimum threshold; in other words, nurses who were already proficient got better and those not yet proficient did not improve. This possibility is unlikely, however, because the cohort score, paired scores, and the improvement in pass rate, although not statistically significant, occurred with all the methods used to determine a passing score.
The sample in our study was of all trained nursing staff within a single ICU. In order to improve the generalizability of our findings, both knowledge assessment and observation of practice could be undertaken in more than a single ICU, in more than a single speciality within the same hospital, and in different hospitals.
The standard errors of measurement, across the assessments, were relatively constant at 5.9% to 7.2%. When combined with the error from the threshold score, this degree of error would be too great for making high-stakes (progress limiting) decisions.57 This error most likely occurred because the nurses in the cohort all scored relatively high and thus variance was small.38 Reducing this error by increasing the number and diversity of questions in the test, increasing the number of expert reviewers, developing a larger pool of questions and using pretesting, and selecting questions on psychometric properties were all beyond the scope of the study.
|
Conclusion |
|---|
|
TopAbstractMethodsResultsDiscussionConclusionReferences |
|---|
This article is followed by an AJCC Patient Care Page on page 348.
Huntleigh Healthcare (NZ) paid for the printing of the knowledge assessment tests. No other grant or financial assistance was received for the study.
Now that you’ve read the article, create or contribute to an online discussion about this topic using eLetters. Just visit www.ajcconline.org and click "Respond to This Article" in either the full-text or PDF view of the article.
To learn more about preventing pressure ulcers in intensive care units, visit http://ccn.aacnjournals.org and read the article by Tracy Ann Pasek et al, "Skin Care Team in the Pediatric Intensive Care Unit: A Model for Excellence" (Critical Care Nurse, April 2008).
|
REFERENCES |
|---|
|
TopAbstractMethodsResultsDiscussionConclusionReferences |
|---|
Related articles in AJCC:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
