Discussion #6
Read the accompanying article and respond to the prompts below.
1. Evaluate the Results and Discussion sections of the accompanying article and critique how the following were addressed. Do not copy and paste from the article. Do not simply respond Yes or No. Describe in your own words how the researchers demonstrated that these elements were addressed.
a) First, check the beginning of the article. What is the hypothesis and/or research question(s)?
b) Now, look at the Results section. Are the results of each of the hypothesis(es) and/or research questions presented?
c) Discuss whether the results are presented objectively.
d) If tables or figures are used, do they meet the following standards?
* They supplement and economize the text
*They have precise titles and headings
*They are not repetitious of the text
e) Now move to the Discussion section. Are the results interpreted in light of the hypotheses, research questions, sampling, data collection, instruments, and other steps that preceded the results?
f) How does the researcher identify the study’s weaknesses (threats to validity) and limitations? Are solutions proposed to address these issues?
g) How does the researcher discuss the study’s clinical relevance and/or implications for nursing?
h) Is generalizability discussed? If so, do these generalizations fall within the scope of the findings?
i) Are any recommendations for future research stated or implied? If so, what are they?
2. Discuss whether the strength of the evidence supports a change in current practice. Support your conclusions.
3. What is your cosmic question? (This should be based on chapter of the week. Pose a research question on the Results or Discussion section of a research article.)
EBP Research
Effect of an E-mental Health Approach to Workers’
Health Surveillance versus Control Group on Work
Functioning of Hospital Employees: A Cluster-RCT
Sarah M. Ketelaar1*, Karen Nieuwenhuijsen1, Fania R. Gärtner1, Linda Bolier2, Odile Smeets2,
Judith K. Sluiter1
1 Coronel Institute of Occupational Health, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands, 2 Innovation Center of Mental Health and
Technology (I.COM), Trimbos Institute, Netherlands Institute of Mental Health and Addiction, Utrecht, The Netherland
s
Abstract
Objective: To evaluate an e-mental health (EMH) approach to workers’ health surveillance (WHS) targeting work functioning
(WF) and mental health (MH) of healthcare professionals in a randomised controlled trial
.
Methods: Nurses and allied health professionals (N = 1140) were cluster-randomised at ward level to the intervention (IG) or
control group (CG). The intervention consisted of two parts: (a) online screening and personalised feedback on impaired WF
and MH, followed by (b) a tailored offer of self-help EMH interventions. CG received none of these parts. Primary outcom
e
was impaired WF (Nurses Work Functioning Questionnaire), assessed at baseline and after three and six months. Analyses
were performed in the positively screened subgroup (i) and in all participants (ii).
Results: Participation rate at baseline was 32% (NIG = 178; NCG = 188). Eighty-two percent screened positive for at least mild
impairments in WF and/or MH (NIG = 139; NCG = 161). All IG-participants (N = 178) received part (a) of the intervention, nine
participants (all positively screened, 6%) followed an EMH intervention to at least some extent. Regarding the subgroup of
positively screened participants (i), both IG and CG improved over time regarding WF (non-significant between-group
difference). After six months, 36% of positively screened IG-participants (18/50) had a relevant WF improvement compared
to baseline, versus 28% (32/115) of positively screened CG-participants (non-significant difference). In the complete sample
(ii), IG and CG improved over time but IG further improved between three and six months while CG did not (significant
interaction effect).
Conclusions: In our study with a full compliance rate of 6% and substantial drop-out leading to a small and underpowered
sample, we could not demonstrate that an EMH-approach to WHS is more effective to improve WF and MH than a control
group. The effect found in the complete sample of participants is not easily interpreted. Reported results may be useful for
future meta-analytic work.
Trial Registration: Dutch Trial Register NTR2786 http://www.trialregister.nl
Citation: Ketelaar SM, Nieuwenhuijsen K, Gärtner FR, Bolier L, Smeets O, et al. (2013) Effect of an E-mental Health Approach to Workers’ Health Surveillance versus
Control Group on Work Functioning of Hospital Employees: A Cluster-RCT. PLoS ONE 8(9): e72546. doi:10.1371/journal.pone.0072546
Editor: Jim van Os, Maastricht University Medical Centre, The Netherlands
Received January 11, 2013; Accepted July 11, 2013; Published September 12, 201
3
Copyright: � 2013 Ketelaar et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits
unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The Mental Vitality @ Work trial was co-financed by a grant from the Dutch Foundation Institute Gak (URL: http://www.instituutgak.nl, PrevBGZ/project
D) and a grant from The Netherlands Organisation for Health Research and Development (ZonMW) (URL: http://www.zonmw.nl/en, grant number 208010001).
The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing Interests: The e-mental health interventions used in this study are stand-alone interventions which were developed by the Trimbos Institute at an
earlier stage. However, neither the authors working at the Trimbos Institute (LB and OS) nor the institute itself derive financial income from the interventions. The
other authors have declared that no competing interests exist.
* E-mail: S.M.Ketelaar@amc.uva.nl
Introduction
Nurses have a high risk of developing common mental health
complaints, such as distress, depression, and anxiety [1–3].
Impaired mental health of employees in healthcare occupations
can have serious adverse effects, endangering the health and safety
of not only themselves but also their patients. A study by Gärtner
and colleagues found that impaired mental health in nurses and
allied health professionals affects several aspects of their work
functioning, including cognitive aspects (e.g. staying alert) and
causing incidents at work [4]. Another study by Letvak and
colleagues showed that depression in nurses was associated with
presenteeism, which is in turn associated with patient falls,
medication errors, and lower self-reported quality of care [5].
Adding to this, increased levels of psychological distress, even in a
mild form, have been found to be associated with an increased
likelihood of obtaining a disability pension in later life [6]. To
sustain nurses’ and allied health professionals’ mental health and
to enable them to remain healthy and well-functioning in their
profession until retirement age, it is crucial to periodically screen
PLOS ONE | www.plosone.org 1 September 2013 | Volume 8 | Issue 9 | e72546
these employees and provide interventions to improve their mental
health and work functioning.
A potentially promising method for the early detection of
impaired mental health and subsequent treatment in nurses and
allied health professionals, is offering a mental module for workers’
health surveillance (WHS). Although attention has been paid to
the occupational hazards of healthcare employees [7], WHS
targeting work functioning and mental health of nurses and allied
health professionals has, to our knowledge, not been reported
before.
WHS is an important component of occupational healthcare
[8]. It is a means to implement preventive action by identifying
and treating health complaints relevant to work, and it should be
an essential component of programmes aimed at the protection of
employees [9]. In the Netherlands, it has three aims: 1) to prevent
the onset, recurrence, or worsening of work-related diseases, 2) to
monitor and promote work-related health, and 3) to monitor and
improve work functioning and employability [10]. It can be used
to periodically monitor employees’ health and work functioning to
detect impairments early and to bring timely interventions into
action to prevent further impairment. It is recommended to apply
a job-specific assessment, to allow for tailoring of interventions to
the specific detected work functioning impairments as fitting as
possible [11]. In this study, we detect early signs of impaired
mental health and impaired work functioning in nurses and allied
health professionals, and offer interventions using an e-mental
health approach.
E-mental health (EMH) is the use of information and
communication technology, and in particular the many tech-
nologies related to the Internet, to support and improve mental
health [12]. Applying EMH might be a useful and feasible
approach to perform a mental module for WHS. Online
screening is a practical and efficient method to screen for self-
reported impaired work functioning and impaired mental
health. Furthermore, EMH offers possibilities regarding the
subsequent interventions. Ritterband and colleagues defined
Internet interventions as typically focused on behavioral issues,
aiming to institute behavior change and subsequent symptom
improvement, usually self-paced, interactive, and tailored to the
user, and making use of the multimedia format offered by the
Internet [13]. EMH interventions exist which target a wide
variety of common mental disorders such as depression, anxiety,
panic, phobias, and various addictions. Unguided self-help
EMH interventions have been found to have positive outcomes
for a variety of mental health aspects (e.g. Warmerdam et al.
[14]; Farrer et al. [15]; Riper et al. [16]; Blankers et al. [17];
Billings et al. [18]), although to our knowledge their effects on
work functioning have not been studied in a specific working
population such as nurses and allied health professionals.
Moreover, EMH interventions have thus far only been offered
as stand-alone interventions for a specific mental health
complaint. In our study, we offer a choice of EMH interven-
tions, tailored to the specific complaints as indicated by the
individual’s screening results.
In this paper, we study the effect of an EMH-approach to
WHS targeting work functioning and mental health of hospital-
employed nurses and allied health professionals, on their work
functioning, distress, work-related fatigue, posttraumatic stress,
and work ability in a cluster-randomized controlled trial. We
hypothesized that WHS, consisting of online screening on
impaired work functioning and impaired mental health
followed by personalised feedback and a tailored offer of self-
help EMH interventions, will improve work functioning and
mental health.
Methods
The protocol for this trial and supporting CONSORT checklist
are available as supporting information; see Checklist S1 and
Protocol S1 (http://www.biomedcentral.com/1471-2458/11/
290).
Ethics statement
The Medical Ethics Committee of the Academic Medical
Center Amsterdam approved this study (for approved protocol see
Protocol S2). All participants gave their written informed consent
before taking part.
Study design
The study was designed as a cluster-randomised trial with block
randomisation carried out at ward level. In order to guarantee
allocation concealment, randomisation was performed by one
researcher (KN) who was not involved in the practical recruitment
of employees, using the computer software program Nquery
Advisor with a block size of three. The complete trial included two
intervention groups and one control group [19]. The present study
compared one of the intervention groups, the e-mental health
approach (EMH-approach) group, to the control group. The other
intervention group consisted of an invitation for a preventive
consultation with an occupational physician. A pre-randomisation
procedure with incomplete-double-consent design was applied
[20], meaning that individuals were only informed about their own
group.
Outcome measures were obtained from all participants at
baseline (March 2011) and follow-up measures were obtained
three and six months after baseline.
The design, conduct and reporting of this study adhere to the
Consolidated Standards of Reporting Trials guidelines [21,22].
Details of the study design are reported elsewhere [19]. The trial
registration number of the study is NTR2786 (Dutch Trial
Register: http://www.trialregister.nl).
Participants
The study population of the complete trial was formed by all
nurses, including surgical nurses and anaesthetic nurses, and allied
health professionals (such as physiotherapists and radiotherapists
)
employed at one academic hospital in the Netherlands (N = 1731).
Nurses and allied health professionals form two large groups of
hospital employees, and many of their work demands and work
conditions are similar. Since it regarded a preventive study,
participants were included if they were not, or were not expecting
to be on sick leave for more than two weeks at baseline.
All eligible employees were invited for participation in the study.
To detect a clinically significant effect (effect size$.33), while
conducting the tests with alpha = .05 (two-tailed) and power = .80,
and allowing for possible cluster effects and loss to-follow-up, the
minimum required sample size was 718 participants for the
complete trial [19]. After randomization at ward level (N = 86), 29
wards with 579 employees were assigned to the EMH-approach
group and 29 wards with 561 employees to the control group
(Figure 1).
Procedure
In March 2011, potential participants received an invitation by
e-mail to fill out the online baseline questionnaire which could be
filled out at any time during six weeks. It was possible to
discontinue the questionnaire and complete it after logging in
again. Three reminders were sent, as well as an information letter
to their home address. Those who had completed the baseline
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 2 September 2013 | Volume 8 | Issue 9 | e72546
questionnaire were invited to fill out the follow-up questionnaires
three and six months after baseline.
Intervention
E-mental health approach group. At baseline, participants
in the EMH-approach group were screened on the following
aspects (for details see Table 1 and Gärtner and colleagues [19]):
impaired work functioning, distress, work-related fatigue, risky
drinking behaviour, depression including suicide risk, anxiety,
panic disorder, and posttraumatic stress. Participants received
personalised feedback on their screening results immediately after
filling out the baseline questionnaire, both onscreen and in an e-
mail.
The personalised feedback was followed by an invitation for a
tailored offer of self-help EMH interventions, on the basis of an
algorithm based on the specific symptoms and the work-
relatedness of the symptoms (available as supporting information,
see Algorithm S1). Participants were mostly offered a choice of two
or three EMH interventions to leave room for personal
preferences. Participants who screened negative on all mental
health complaints were invited to follow an EMH intervention
aimed at enhancing and retaining their mental fitness.
The EMH interventions used in this study are self-help
interventions on the Internet aimed at reducing specific mental
health complaints or enhancing wellbeing. The interventions are
mainly based on the principles of cognitive behavioural therapy
and combine a variety of aspects, e.g. providing information and
advice, weekly assignments, the option of keeping a diary and a
forum to get in contact with others who have similar complaints.
The EMH interventions were developed as stand-alone interven-
tions by the Trimbos Institute (Netherlands Institute of Mental
Figure 1. Flow of participants through the trial.
doi:10.1371/journal.pone.0072546.g001
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 3 September 2013 | Volume 8 | Issue 9 | e72546
Health and Addiction) at an earlier stage. The following EMH
interventions were used in the study:
– Psyfit [23]: aimed at enhancing mental fitness. Also applied for
healthy participants.
– Strong at work [24]: aimed at gaining insight into work stress and
learning skills to cope with it.
– Colour your Life [25]: aimed at tackling depressive symptoms.
– Don’t Panic Online [26]: aimed at reducing panic symptoms for
subclinical and mild cases of panic disorder.
– Drinking Less [16]: aimed at reducing risky drinking behaviour.
Psyfit was found to be effective in decreasing symptoms of
depression and anxiety and improving well-being and vitality [27].
Warmerdam and colleagues [14] showed that Colour your Life
resulted in significantly lower depression and anxiety scores
compared to a waiting-list control group and to significantly
higher quality of life scores. The number of participants showing
clinically relevant change regarding depression after 12 weeks was
significantly higher. Spek and colleagues also found a significantly
larger improvement in depressive symptoms compared to a
waiting-list control group [28,29]. Drinking Less resulted in more
participants who reduced their drinking successfully to within
guideline norms, and a significantly larger decrease in mean
weekly alcohol consumption than a control group [16].
In case of positive screening on impaired work functioning
(regardless of their mental health status), participants received an
onscreen educational leaflet on how to improve their work
functioning (available upon request).
Control group. Participants in the control group filled out
the same baseline questionnaire as the EMH-approach group, but
did not receive an intervention, and thus no screening results
either. However, they were informed that they would receive
personalised feedback and a tailored offer of self-help EMH
interventions after six months, following the six months follow-up
questionnaire.
Measures
All outcomes were measured at baseline and at three and si
x
months follow-up.
Primary outcome. The primary outcome of this study was
impaired work functioning, measured with the total score of the
Nurses Work Functioning Questionnaire (NWFQ) [4]. This
questionnaire has been developed to assess impaired work
functioning in nurses and allied health professionals. In the
screening phase, all seven of the original subscales were used.
Participants scored either green, orange or red on each subscale.
A red score on one or more subscales and/or three or more
orange scores led to case identification of impaired work
functioning (i.e. scoring above cut-off point on impaired work
functioning) [19].
Only six of the seven original NWFQ subscales were used for
the outcome measure, in contrast to what was described in the
trial’s design study [19], because the reproducibility of the impaired
decision-making subscale was found to be poor [30]. The total score
on the NWFQ was calculated with the 47 items of the remaining
six subscales, with a total score range of 0–100, a higher score
indicating more severely impaired work functioning.
The difference between the EMH-approach group and the
control group regarding impaired work functioning was investi-
gated using the continuous outcome and the percentage of
individuals who had improved relevantly at follow-up [31].
Secondary outcomes. The secondary outcomes included
distress, work-related fatigue, posttraumatic stress, and work
ability.
Distress was measured with the distress subscale of the Four-
Dimensional Symptoms Questionnaire (4DSQ) [32,33]. The 16-
item questionnaire uses a 5-point response scale (0 = no, 4 = very
often) and has a total score range of 0–32, a higher score indicating
a higher level of distress (cut-off point $11 [34]).
Work-related fatigue after working time was measured using the
need for recovery subscale of the Dutch Questionnaire on the
Experience and Evaluation of Work (QEEW) [35]. The 11-item
questionnaire with dichotomous response categories (yes, no) has a
total score range of 0–11 and a standardized score range of 0–100,
a higher score indicating a higher level of work-related fatigue (cut-
off point $54.5 [36]).
Posttraumatic stress was measured with the Dutch version of the
Impact of Event Scale [37,38]. The 15 items can be answered on a
4-point response scale (0 = not at all, 3 = often). Total scores range
from 0–75, a higher score indicating a higher level of posttrau-
matic stress (cut-off point $26 [39]).
Work ability was assessed with the first item of the Work Ability
Index (WAI) [40]. This item concerns the evaluation of current
work ability compared to their lifetime best on an 11-point scale
Table 1. Screening instruments and cut-off points.
Aspect Instrument Cut-off point
Impaired work functioning Nurses Work Functioning Questionnaire (7 subscales) [4] Red score on $1 subscales and/or orange
score on $3 subscales [19]
Distress Four-Dimensional Symptoms Questionnaire, distress subscale [32,33] Total score$11 [34]
Work-related fatigue Need for recovery subscale of the Dutch Questionnaire on the Experience
and Evaluation of Work [35]
Standardised total score$54.5 [36]
Risky drinking behaviour AUDIT-C [49] Total score$5 for men, $4 for women [50]
Depression Brief Symptom Inventory, depression subscale [51] Mean score$0.42 [52]
(Suicide risk) (One item from Brief Symptom Inventory, depression subscale [51]) (Score$3 on 0–4 scale)
Anxiety Brief Symptom Inventory, anxiety subscale [51] Mean score$0.42 [52]
Panic disorder Patient Health Questionnaire [53], only assessed for participants identified
as having anxiety complaints
2 answers affirmative on the first 4 items
plus 4 symptoms affirmative on the
following 11 items [54]
Posttraumatic stress Dutch translation of the Impact of Event Scale [37,38] Total score$26 [39]
doi:10.1371/journal.pone.0072546.t001
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 4 September 2013 | Volume 8 | Issue 9 | e72546
(0 = completely unable to work, 10 = work ability at its best), a higher score
indicating a higher level of work ability.
Statistical analyses
All participants who completed the baseline questionnaire and
who screened positive on impaired work functioning and/or
impaired mental health (the targeted sample) were analysed, as the
work functioning and mental health of these participants could be
expected to change due to the intervention. However, since this
was not pre-specified in the trial registration, the analyses were also
performed with the total sample of participants (i.e. all partici-
pants, regardless of their screening results).
To describe participants, we used the following demographics:
sex, age, occupation, specialization (yes/no), years of working
experience, working hours per week, and type of contract.
Additionally, the number of participants scoring above cut-off
point for impaired work functioning and mental health complaints
were calculated.
The analyses were performed at the level of the individual
employee, according to the intention-to-treat principle. The
significance level was set at a = .05. All analyses were carried out
using the statistical package IBM SPSS Statistics 19.
Drop-out analysis. A drop-out analysis was performed to
detect whether dropping out of the trial was related to the
primary outcome impaired work functioning, and to identify
potential predictive variables of drop-out. Dropping out of the
trial was defined as completing the baseline and three months
follow-up questionnaires, but not the six months follow-up
questionnaire; or completing the baseline questionnaire, but
none of the follow-up questionnaires. Differences between drop-
outs and non drop-outs in impaired work functioning over time
in both separate groups were explored in graphs. If different
patterns of the effect after three months were detected, a Mann-
Whitney U test was performed to test the significance of the
differences. In the event of statistically significant differences, a
multiple logistic regression analysis was performed with drop-
out as the dependent variable, to identify potential predictive
variables for drop-out. Screening positive on mental health
complaints at baseline (yes/no) and age were included as the
independent variables, as we expected that these two aspects
might be related to dropping out of the trial. If the multiple
logistic regression analysis showed one or both of these aspects
to have a statistically significant effect on drop-out, they were
included as a covariate in the effect analyses [41].
Effect analysis. To analyse the differences over time
between the EMH-approach group and the control group on
each outcome, Linear Mixed Models (LMM) were applied. If the
assumption of a normal distribution of residuals was not met, a
log-transformation was used for the LMM and the median and
range were used to describe the outcome. Otherwise, the mean
and standard deviation were used to describe the outcome.
For each outcome, the scores at three and six months follow-up
were included as dependent variables in the LMM, while the
baseline score was included as a covariate. The main effects of
group and time of measurement, and the interaction of group*time of
measurement were included as fixed effects in the model. Ward (the
cluster level) and subject (the individual level) were included as
random effects; however if the cluster level did not have a
statistically significant effect, it was considered negligible and was
therefore excluded from the model. The effects of interest were the
main effect of group (interpreted as the difference between the
groups from baseline to six months follow-up) and the interaction
effect of group*time of measurement (interpreted as the difference
between the groups from three to six months follow-up).
For all outcomes in the positively screened subgroup, we
calculated Cohen’s d [42] by determining the mean difference
between the baseline score and the score at follow-up, divided by
the pooled standard deviation. For Cohen’s d, a score of 0.2 to 0.5
can be considered a small effect, 0.5 to 0.8 a medium effect, and
greater than 0.8 a large effect [42].
Additionally, the relative change scores of individuals on
impaired work functioning after three and after six months of
follow-up compared to their baseline score were calculated.
Individuals with a relative improvement on their NWFQ total
score of 40% or more, which is the minimal important change
(MIC) value of the NWFQ total scale [31], were defined as
relevantly improved. The percentages of individuals who had
improved relevantly in each group were compared using a Fisher’s
exact test, for both three months and six months follow-up.
Results
Participant flow
Figure 1 presents the flow of participants through the trial. From
March 15
th
until April 26
th
, 423 employees (37%) started on the
baseline questionnaire. Of those, 366 (32% of invited employees)
were eligible for participation, 178 (31%) in the EMH-approach
group and 188 (34%) in the control group. In the EMH-approach
group, 80 participants (45%) were lost to follow-up, compared to
33 participants (18%) in the control group. Reasons for withdrawal
were not assessed. Fifty-six participants (31%) in the EMH-
approach group and 126 participants (67%) in the control group
completed all three questionnaires.
Analyses were performed on the participants who screened
positive (primary outcome: EMH N = 75, 54%; control N = 131,
81%), and additionally on all participants (primary outcome:
EMH N = 98, 55%; control N = 155, 82%) who had participated
in at least one follow-up.
Twenty-two participants (17 positively screened) logged into
Psyfit, seven logged into Strong at work, four logged into Colour your
Life, and no-one logged into Don’t Panic Online or Drinking Less. Nine
participants (all positively screened) followed an intervention to at
least some extent (Psyfit: 6, Colour your Life: 3).
Study population at baseline
As shown in Table 2, the study groups were quite similar
regarding demographic and occupational characteristics. The
majority of participants were female and employed as a nurse.
Participants in the EMH-approach group had a younger average
age of 37, compared to 42 in the control group. The participants
worked an average of 31 hours per week and most of them had a
permanent position in the hospital. Around 4/5
th
of participants
screened positive on work functioning impairments and/or
impaired mental health, more participants in the control group
(N = 161, 86%) than in the EMH-approach group (N = 139, 78%).
Drop-out analysis
Graphs in which the scores of drop-outs and non drop-outs on
the primary outcome were compared, showed that in both groups
drop-outs had a worse score on impaired work functioning (EMH
baseline median = 13, 3 mn follow-up median = 12; C baseline
median = 14, 3 mn follow-up median = 11) than non drop-outs
(EMH baseline median = 9, 3 mn follow-up median = 8; C
baseline median = 12, 3 mn follow-up median = 8) at baseline
and three months follow-up. A Mann-Whitney U test identified
that these differences were statistically significant in the EMH-
approach group (baseline U = 4.688, p = .01; 3 mn follow-up
U = 970, p = .04) and in the entire group of participants (baseline
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 5 September 2013 | Volume 8 | Issue 9 | e72546
T
a
b
le
2
.
P
a
rt
ic
i
p
a
n
t
c
h
a
ra
c
te
ri
st
ic
s
a
t
b
a
se
li
n
e
fo
r
t
h
e
E
M
H
-a
p
p
ro
a
c
h
g
ro
u
p
a
n
d
th
e
c
o
n
tr
o
l
g
r
o
u
p
.
T
o
t
a
l
s
a
m
p
le
P
o
s
it
iv
e
ly
s
c
r
e
e
n
e
d
s
a
m
p
le
E
M
H
-a
p
p
ro
a
c
h
(
N
=
1
7
8
)
C
o
n
tr
o
l
(N
=
1
8
8
)
E
M
H
-a
p
p
ro
a
c
h
(N
=
1
3
9
)
C
o
n
tr
o
l
(N
=
1
6
1
)
V
a
ri
a
b
le
n
(%
)
n
(%
)
n
(%
)
n
(%
)
S
e
x
F
e
m
a
le
1
4
7
(
8
3
)
1
4
5
(7
7
)
1
1
3
(8
1
)
1
2
6
(7
8
)
A
g
e
in
y
e
a
rs
(m
e
a
n
(S
D
))
3
7
(
1
2
)
4
2
(1
1
)
3
8
(1
2
)
4
2
(1
2
)
O
cc
u
p
a
ti
o
n
N
u
r
s
e
1
2
9
(7
3
)
1
3
4
(7
1
)
9
9
(
7
1
)
1
1
5
(7
1
)
N
u
rs
e
p
ra
c
ti
ti
o
n
e
r
1
1
(6
)
2
2
(1
2
)
7
(5
)
1
8
(1
1
)
S
u
rg
ic
a
l
n
u
rs
e
0
(0
)
5
(3
)
0
(0
)
5
(3
)
A
n
e
st
h
e
ti
c
n
u
rs
e
0
(0
)
0
(
0
)
0
(0
)
0
(0
)
A
ll
i
e
d
h
e
a
lt
h
p
ro
fe
ss
io
n
a
l
3
8
(2
1
)
2
7
(1
4
)
3
3
(2
4
)
2
3
(1
4
)
N
u
rs
in
g
sp
e
ci
a
li
za
ti
o
n
Y
e
s
7
4
(5
7
)
8
6
(6
4
)
5
7
(5
8
)
7
5
(6
5
)
Y
e
a
rs
o
f
e
xp
e
ri
e
n
ce
(m
e
a
n
(S
D
))
1
0
(1
0
)
1
1
(1
0
)
1
1
(1
0
)
1
1
(1
0
)
W
o
r
k
in
g
h
o
u
rs
p
e
r
w
e
e
k
a
cc
o
rd
in
g
to
co
n
tr
a
ct
(
m
e
a
n
(S
D
))
3
1
(5
)
3
1
(6
)
3
1
(5
)
3
1
(6
)
T
yp
e
o
f
co
n
tr
a
ct
P
e
rm
a
n
e
n
t
p
o
si
ti
o
n
1
6
0
(9
1
)
1
7
4
(9
3
)
1
2
5
(9
1
)
1
5
0
(9
4
)
F
ix
e
d
-t
e
rm
c
o
n
t
r
a
c
t
1
3
(7
)
1
2
(
6
)
1
1
(8
)
9
(6
)
T
e
m
p
o
ra
ry
e
m
p
lo
y
m
e
n
t
0
(0
)
0
(0
)
0
(0
)
0
(0
)
O
th
e
r
3
(2
)
1
(1
)
1
(1
)
1
(1
)
Im
p
a
ir
e
d
w
o
rk
fu
n
ct
io
n
in
g
*
(a
b
o
v
e
c
u
t-
o
ff
)
W
o
rk
fu
n
c
ti
o
n
in
g
im
p
a
i
r
m
e
n
ts
1
0
7
(6
0
)
1
3
1
(7
0
)
1
0
7
(7
7
)
1
3
1
(8
1
)
Im
p
a
ir
e
d
m
e
n
ta
l
h
e
a
lt
h
(
a
b
o
v
e
cu
t-
o
ff
)
Im
p
a
ir
e
d
m
e
n
ta
l
h
e
a
lt
h
(a
b
o
v
e
c
u
t-
o
ff
o
f
o
n
e
o
r
m
o
re
o
f
th
e
si
x
m
e
n
ta
l
h
e
a
lt
h
a
sp
e
c
ts
)
1
0
9
(6
1
)
1
1
9
(6
3
)
1
0
9
(7
8
)
1
1
9
(7
4
)
D
is
tr
e
ss
4
1
(2
3
)
4
8
(2
6
)
4
1
(3
0
)
4
8
(
3
0
)
W
o
rk
re
la
te
d
fa
ti
g
u
e
6
1
(
3
4
)
6
5
(3
5
)
6
1
(
4
4
)
6
5
(4
0
)
P
o
st
tr
a
u
m
a
ti
c
st
re
ss
2
4
(1
4
)
1
9
(1
0
)
2
4
(1
7
)
1
9
(1
2
)
S
cr
e
e
n
e
d
p
o
si
ti
v
e
o
n
im
p
a
ir
e
d
w
o
rk
fu
n
ct
io
n
in
g
a
n
d
/o
r
im
p
a
ir
e
d
m
e
n
ta
l
h
e
a
lt
h
1
3
9
(7
8
)
1
6
1
(8
6
)
1
3
9
(1
0
0
)
1
6
1
(1
0
0
)
*N
o
te
:
W
o
rk
fu
n
c
ti
o
n
in
g
is
p
re
se
n
te
d
h
e
re
in
c
lu
d
in
g
th
e
su
b
sc
a
le
im
p
a
ir
e
d
d
e
ci
si
o
n
-m
a
k
in
g
,
a
s
it
w
a
s
in
c
lu
d
e
d
i
n
th
e
sc
re
e
n
in
g
a
t
b
a
se
li
n
e
.
d
o
i:1
0
.1
3
7
1
/j
o
u
rn
a
l.p
o
n
e
.0
0
7
2
5
4
6
.t
0
0
2
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 6 September 2013 | Volume 8 | Issue 9 | e72546
U = 19.202,5, p = .01; 3 mn follow-up U = 5.079,5, p = .01). In a
subsequent logistic regression analysis, age was identified as a
statistically significant predictor of drop-out in the entire group
(p = .02, younger participants had a higher chance of drop-out),
but screening positive for mental health complaints at baseline
(yes/no) was not (p = .16). Therefore, age was included as a
covariate in the effect analyses.
Intervention effects
The results in Table 3 refer to the group of participants who
screened positive for impaired work functioning and/or mental
health impairments at baseline. The relative frequency of
participants who scored above cut-off point on the outcome
measures and the mean and median scores (in case of a non-
normal distribution) are presented for baseline and both follow-up
points, as well as the results of the LMM analyses.
Since the random effect of ward (the cluster level) was not
statistically significant in any of the analyses, it was excluded from
the model in the LMM analyses.
Impaired work functioning (primary outcome). The
EMH-approach group and the control group improved to a
similar degree between baseline and three months follow-up. The
EMH-approach group improved further between three and six
months, while the control group remained at approximately the
same level. As shown in Table 3, in the LMM analysis of impaired
work functioning in the positively screened sample of participants,
no statistically significant difference between the EMH-approach
group and the control group was identified (main effect of group
p = .77; interaction effect of group*time p = .28). The effect size
estimate after three and six months was comparably low in both
groups.
In the LMM analysis of the total sample of participants (data
not shown in table), no significant effect of group was found
(p = .68), but a significant interaction effect of group*time was
found (p = .04), suggesting there to be a different pattern of scores
on impaired work functioning from three to six months follow-up
between the EMH-approach group and the control group. A
closer look at the median scores on impaired work functioning
revealed that both groups improved to a similar degree between
baseline and three months follow-up, and that the EMH-approach
group further improved between three and six months follow-up
while the control group slightly deteriorated.
In Table 4, the percentages of individual employees with a
relevant improvement on work functioning after three and after six
months compared to their baseline score are shown. After three
months, in the positively screened sample as well as the total
sample, roughly the same percentage of participants in both
groups had improved relevantly regarding work functioning
compared to their baseline score. After six months, more
participants in the EMH-approach group than in the control
group had improved relevantly compared to baseline, in both the
positively screened sample (EMH 36%; control 28%) and the total
sample (EMH 40%; control 30%). However, these differences
were not statistically significant (p = .36 and p = .21, respectively).
Secondary outcomes. As shown in Table 3, both groups
improved over time regarding distress, work-related fatigue, and
posttraumatic stress, with the largest improvement between
baseline and three months of follow-up. On distress and work-
related fatigue, the EMH-approach group had a larger overall
improvement than the control group (non-significant). In the
LMM analyses on distress, work-related fatigue, posttraumatic
stress, and work ability in the positively screened sample, no
statistically significant differences were found between the EMH-
approach group and the control group (main effect of group
.36#p#.62; interaction effect of group*time .12#p#.83). Effect
sizes in both groups were fairly similar, small to non-relevant.
In the LMM analyses on the secondary outcomes in the total
sample of participants (data not shown in table), no significant
differences were found between the EMH-approach group and the
control group either (.31#p#.97).
Discussion
The results of our study suggest that an e-mental health (EMH)
approach of workers’ health surveillance (WHS), consisting of
online screening on impaired work functioning and impaired
mental health followed by personalised feedback and a tailored
offer of self-help EMH interventions, shows no significant
improvement in impaired work functioning, distress, work-related
fatigue, posttraumatic stress, and work ability to a larger extent
than a control group. Compliance to the EMH interventions was
low, which impedes drawing a conclusion about the effect of this
part of the intervention. Screening and personalised feedback was
received by all participants in the intervention group. Although the
study had insufficient power, the low effect sizes do not give reason
to expect a relevant effect of screening and feedback. The
outcomes may be of value for future meta-analytic work.
One third of the employees who were invited, participated in
the study. Of these participants, more than 80% screened positive
for at least mild impairments in work functioning and/or mental
health. Both the intervention group and the control group
improved over time on work functioning, distress, work-related
fatigue, and posttraumatic stress, with no statistically significant
difference between the groups. However, when including all
participants in the analyses and not only those who had screened
positive on impairments at baseline, the work functioning of the
EMH-approach group showed a significantly different pattern
compared to the control group, as the EMH-approach group
further improved between three and six months after baseline
while the control group did not. After six months, a relevant
improvement of work functioning was found for 36% of positively
screened participants in the intervention group and 28% in the
control group, but the difference between the groups was non-
significant.
Interpretation of results
First of all, our study had a high percentage of participants who
screened positive for at least mild impairments. This included
screening positive on impairments in work functioning, on one or
more mental health complaints or both. In choosing our cut-off
points, we aimed for high sensitivity, since we did not want to miss
participants who might need help. The cut-off points that we
applied for the mental health complaints were all validated.
However, high sensitivity generally comes at the expense of high
specificity, which might have led to higher numbers of false
positives in our study. We formulated the online feedback mildly,
careful not to speak of diagnosis or mental health problems, to prevent
incorrect interpretation. Additionally, the relatively high number
of screening instruments might have led to a high overall
percentage of participants who screened positive for at least one
of the screeners.
Our intervention consisted of two parts. First, the participants in
the intervention group underwent online screening on impaired
work functioning and impaired mental health, followed by
personalised feedback on their screening results. Subsequently,
they were offered a tailored offer of EMH interventions. In
addition, participants with impaired work functioning received an
onscreen educational leaflet on how to improve their work
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 7 September 2013 | Volume 8 | Issue 9 | e72546
T
a
b
le
3
.
D
e
sc
ri
p
ti
v
e
s
a
n
d
a
n
a
ly
si
s
re
su
l
t
s
o
n
p
ri
m
a
ry
a
n
d
se
c
o
n
d
a
ry
o
u
tc
o
m
e
s
o
f
th
e
p
o
si
ti
v
e
ly
sc
re
e
n
e
d
sa
m
p
le
a
t
b
a
se
li
n
e
,
3
a
n
d
6
m
o
n
th
s
fo
ll
o
w
-u
p
.
E
-m
e
n
ta
l
h
e
a
lt
h
a
p
p
ro
a
c
h
g
ro
u
p
C
o
n
tr
o
l
g
ro
u
p
p
-v
a
lu
e
(L
M
M
a
n
a
ly
s
e
s
)*
*
R
e
la
ti
v
e
fr
e
q
u
e
n
c
y
a
b
o
v
e
c
u
t-
o
ff
(%
)
M
e
d
ia
n
(r
a
n
g
e
)
M
e
a
n
(S
D
)
E
ff
e
c
t
s
iz
e
(9
5
%
C
I)
R
e
la
ti
v
e
fr
e
q
u
e
n
c
y
a
b
o
v
e
c
u
t-
o
ff
(%
)
M
e
d
ia
n
(r
a
n
g
e
)
M
e
a
n
(S
D
)
E
ff
e
c
t
s
iz
e
(9
5
%
C
I)
g
ro
u
p
g
ro
u
p
*
ti
m
e
P
ri
m
a
ry
o
u
tc
o
m
e
Im
p
a
ir
e
d
w
o
rk
fu
n
ct
io
n
in
g
b
9
1
/1
3
9
(6
6
)
1
4
(0
–
5
6
)
1
1
0
/1
6
1
(6
8
)
1
4
(0
–
5
4
)
.7
7
1
.2
8
3
(
N
W
F
Q
0
–
1
0
0
)*
3
m
n
3
3
/6
2
(5
3
)
1
0
(0
–
3
9
)
.1
9
(2
.1
6
–
.5
5
)
6
1
/1
2
4
(4
9
)
1
0
(0
–
3
8
)
.2
6
(.
0
1
–
.5
1
)
6
m
n
1
9
/5
2
(3
7
)
8
(0
–
4
1
)
.1
6
(2
.2
2
–
.5
5
)
6
0
/1
1
6
(5
2
)
1
0
(0
–
4
4
)
.
2
4
(
2
.0
1
–
.5
0
)
S
e
c
o
n
d
a
ry
o
u
tc
o
m
e
s
D
is
tr
e
ss
b
4
1
/1
3
9
(3
0
)
7
(0
–
3
2
)
4
8
/1
6
1
(3
0
)
6
(0
–
3
2
)
.5
9
2
.8
2
8
(4
D
S
Q
,
0
–
3
2
)
3
m
n
9
/6
1
(1
5
)
4
(0
–
2
9
)
.2
0
(2
.1
6
–
.5
6
)
2
5
/1
2
3
(2
0
)
5
(0
–
2
9
)
.2
6
(.
0
1
–
.5
1
)
6
m
n
1
0
/5
2
(1
9
)
5
(0
–
2
9
)
.0
7
(2
.3
2
–
.4
5
)
2
6
/1
1
6
(
2
2
)
5
(0
–
3
0
)
.1
4
(2
.1
1
–
.4
0
)
W
o
rk
-r
e
la
te
d
fa
ti
g
u
e
b
6
1
/1
3
9
(4
4
)
4
4
(2
8
)
6
5
/1
6
1
(4
0
)
3
9
(3
0
)
.6
1
7
.7
3
2
(Q
E
E
W
,
0
–
1
0
0
)
3
m
n
2
2
/6
1
(3
6
)
3
6
(3
1
)
.1
6
(2
.2
0
–
.5
2
)
4
2
/1
2
3
(3
4
)
3
5
(3
0
)
.1
2
(2
.1
3
–
.3
7
)
6
m
n
1
4
/5
2
(2
7
)
3
4
(3
0
)
.0
2
(2
.3
6
–
.4
1
)
3
9
/1
1
6
(3
4
)
3
7
(3
1
)
.0
2
(2
.2
4
–
.2
7
)
P
o
st
tr
a
u
m
a
ti
c
st
re
ss
b
2
4
/1
3
9
(1
7
)
3
(0
–
7
1
)
1
9
/1
6
1
(1
2
)
3
(0
–
4
8
)
.3
5
7
.1
2
4
(I
E
S
,
0
–
7
5
)
3
m
n
1
0
/6
1
(1
6
)
1
(0
–
4
8
)
.0
7
(2
.2
9
–
.4
2
)
1
3
/1
2
2
(1
1
)
0
(0
–
6
2
)
.3
1
(.
0
5
–
.5
6
)
6
m
n
5
/5
1
(1
0
)
0
(0
–
3
1
)
.2
4
(2
.1
5
–
.6
3
)
9
/1
1
6
(8
)
0
(0
–
4
8
)
.2
6
(.
0
0
–
.5
1
)
W
o
rk
a
b
il
it
y
b
7
(1
)
8
(2
)
.4
8
3
.5
5
2
(W
A
I,
0
–
1
0
)
3
m
n
8
(1
)
.1
4
(2
.2
1
–
.5
0
)
8
(1
)
2
.0
1
(2
.2
6
–
.2
5
)
6
m
n
8
(2
)
.0
5
(2
.3
4
–
.4
4
)
8
(1
)
.0
1
(2
.2
5
–
.2
6
)
*N
o
te
:
N
W
F
Q
to
ta
l
sc
o
re
s
w
e
re
c
a
lc
u
la
te
d
w
it
h
o
u
t
th
e
su
b
sc
a
le
im
p
a
ir
e
d
d
e
ci
si
o
n
-m
a
k
in
g
.
**
N
u
m
b
e
r
a
n
a
ly
se
d
in
E
M
H
-a
p
p
ro
a
c
h
g
ro
u
p
:
N
=
7
5
(i
m
p
a
ir
e
d
w
o
rk
fu
n
c
ti
o
n
in
g
),
N
=
7
4
(d
is
tr
e
ss
a
n
d
w
o
rk
-r
e
la
te
d
fa
ti
g
u
e
),
N
=
7
3
(p
o
st
tr
a
u
m
a
ti
c
st
re
ss
a
n
d
w
o
rk
a
b
il
it
y
);
n
u
m
b
e
rs
a
n
a
ly
se
d
in
C
o
n
tr
o
l
g
ro
u
p
:
N
=
1
3
1
(a
ll
o
u
tc
o
m
e
s)
.
b
,
b
a
se
li
n
e
;
3
m
n
,
fo
ll
o
w
-u
p
a
ft
e
r
3
m
o
n
th
s;
6
m
n
,
fo
ll
o
w
-u
p
a
ft
e
r
6
m
o
n
th
s.
d
o
i:1
0
.1
3
7
1
/j
o
u
rn
a
l.p
o
n
e
.0
0
7
2
5
4
6
.t
0
0
3
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 8 September 2013 | Volume 8 | Issue 9 | e72546
functioning. Two scenarios might explain our not finding an effect
of the intervention: programme failure and theory failure [43]: the
intervention was not carried out as intended (programme failure),
or the intervention is not effective (theory failure).
The process evaluation that was carried out alongside this
randomised controlled trial [44] offers some information on
potential programme failure. The personalised feedback was
received by all participants in the intervention group, since it
appeared onscreen immediately after filling out the baseline
questionnaire and was sent to the participants’ e-mail address
automatically. The onscreen educational leaflet on how to
improve work functioning was also sent automatically to
participants with impaired work functioning. However, the
compliance to the subsequently offered self-help EMH interven-
tions was low. Only 28 participants logged into an EMH
intervention, and 6% (N = 9) of the positively screened participants
in the intervention group started an EMH intervention to at least
some extent. Regarding the second part of the intervention (the
EMH interventions), program failure may therefore have
occurred.
Participants offered no explanation why they did not follow an
EMH intervention [44]. Three explanations are conceivable.
Firstly, there is a reported trend in the literature of a low
perception of need for mental health interventions. Lexis and
colleagues found that 43% of employees who were identified with
mild to severe depressive complaints, did not report to experience
health complaints themselves [45]. Codony and colleagues found
that merely a third of those who had a mental disorder in the past
12 months, had a perceived need of mental healthcare [46]. Since
our study regarded a preventive setting and we chose for high
sensitivity in our screening, perhaps the perceived need of our
participants was insufficient to motivate them to log into and
follow an EMH intervention. Secondly, some of the participants
(N = 9) reported problems with logging into the interventions, due
to technical problems and/or inadequate computer skills, which
might have posed a problem for more participants. A third
explanation is that the channelling from the personalised feedback
towards the EMH interventions might not have been attractive
enough to encourage participants to follow an EMH intervention.
The possibility of theory failure should also be considered. The
intervention consisted of screening and personalised feedback on
screening results including channelling towards EMH interven-
tions, an onscreen educational leaflet on how to improve work
functioning (if applicable), and following the EMH
interventions.
Most of the EMH interventions that were used in our study have
been found effective to reduce symptoms of impaired mental
health in previous research [14,16,27–29], supporting our
hypothesis that an EMH-approach to WHS, including EMH
interventions (if complied to), might be effective in improving
mental health and improving work functioning. However, it
should be noted that for these previous studies, most participants
had actively responded to advertisements targeting people who
wanted to work on their depressive symptoms or their mental
fitness. Therefore, these participants actively sought help and
improvement through EMH interventions. This differs consider-
ably from our setting, as our participants took part in WHS
targeting work functioning and mental health and might not have
been as much aware that they would be offered EMH
interventions.
However, since the intervention was not carried out as intended,
we cannot conclude that the complete EMH-approach to WHS
targeting work functioning and mental health of healthcare
employees is ineffective. Moreover, we found that when looking
at the total sample of participants, both groups improved over
time, but the EMH-approach group continued to improve
between three and six months after baseline while the control
group slightly deteriorated in this time interval. Possibly, we were
able to find a significant effect in this total sample, because the
number of participants was higher in this group and the analysis
was therefore better powered to find existing differences. Since the
EMH interventions themselves were hardly followed, this suggests
that the other elements of the complete EMH-approach – possibly
increasing awareness – might have had some (delayed) effect on
work functioning. However, the results in this total sample of
participants are not easily interpreted, since only the personalised
feedback was received by all participants in the intervention group,
and the observed effect did not occur until later in time.
Limitations
Several limitations of our study can be noted. First of all, we did
not meet our required sample size for sufficient power, set at 189
participants in each group who completed participation. This
increases the chance of finding non-significant p-values despite
trends for differences. The data show that, regarding impaired
work functioning, a higher percentage of participants in the EMH-
approach group than in the control group improved to a relevant
degree compared to their own baseline scores, but this difference
was not statistically significant, which might have been a result of
insufficient power. However, the observed effect sizes were very
small, and in most cases were fairly similar between the groups.
A second limitation of our study was the fairly high and selective
drop-out rate of participants, especially in the intervention group.
Drop-outs had higher scores on impaired work functioning at
baseline and three months follow-up than participants who did not
drop out of the trial. We do not know why this occurred, since we
did not assess reasons for drop-out. We received mixed reactions
to the personalised but automatic feedback on screening results
and the for some participants unexpected offer to follow an EMH
Table 4. Participants whose work functioning had improved with at least the minimal important change at 3 and 6 months follow-
up compared to baseline: descriptives and analysis results.
EMH-approach Control group p-value
Relative frequency (%) Relative frequency (%) (Fisher’s exact test)
Positively screened sample 3 mn 18/60 (30%) 37/123 (30%) 1.000
6 mn 18/50 (36%) 32/115 (28%) .357
Total sample 3 mn 24/80 (30%) 46/142 (32%) .765
6 mn 27/68 (40%) 40/134 (30%) .206
3 mn, follow-up after 3 months; 6 mn, follow-up after 6 months.
doi:10.1371/journal.pone.0072546.t004
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 9 September 2013 | Volume 8 | Issue 9 | e72546
intervention. We suppose this might have led to resistance and the
higher drop-out in the intervention group. The high and selective
drop-out may have introduced bias, although we have no way of
knowing in which direction this possible bias occurred.
Thirdly, as discussed before, the compliance to the offered
EMH interventions was low, complicating studying the effect of
the complete EMH-approach to WHS.
Lastly, we studied the effects of the EMH-approach in a group
of positively screened participants, regardless of what they
screened positive for. Since not everyone screened positive for
every impairment of complaints and the offered intervention was
tailored to each individual, it might not be reasonable to expect an
improvement for every impairment or complaint if examining the
total group.
Implications for practice and further research
Our study confirms that preventive actions are essential for
nurses and allied health professionals, since we identified that
more than 80% of participants show at least some level of
impaired work functioning and/or symptoms of mental health
problems.
We endeavoured to improve work functioning and mental
health through online screening, personalised feedback, and a
subsequent tailored offer of self-help EMH interventions. We think
that targeting work functioning is an important approach, as the
ultimate goal of occupational healthcare is to keep employees
functioning well and as healthy as possible. However, we were
unsuccessful in studying the EMH-approach, because very few
participants followed an EMH intervention to at least some
degree. Therefore, we recommend further research on two
aspects. First, it is essential to identify the specific needs and
wishes that nurses and allied health professionals have regarding
their work related health and to study how they want to be
supported to stay healthy and well-functioning at work. Possibly, a
more comprehensive WHS including important other factors of
their work, such as physical aspects (e.g. musculoskeletal
complaints), would increase their interest and participation.
Secondly, it should be investigated whether EMH interventions
are suitable and acceptable for a WHS setting for nurses and allied
health professionals, and if they would prefer some degree of
contact with a healthcare provider. It is recommended to explore
the possibility of ‘‘blended care’’, i.e. combining an offer of an
EMH intervention with several coaching sessions. Moreover, it
could be useful to apply elements of persuasive design to
encourage employees to follow an EMH intervention [47,48].
Supporting Information
Algorithm S1 Algorithm for determining the specific
choice of e-mental health interventions.
(PDF)
Checklist S1 CONSORT Checklist.
(DOC)
Protocol S1 Study protocol as published in BMC Public
Health (http://www.biomedcentral.com/1471-2458/11/290).
(PDF)
Protocol S2 Study protocol as approved by ethics
committee.
(PDF)
Acknowledgments
We thank Eva Fischer (Innovation Center of Mental Health & Technology,
Trimbos Institute, Netherlands Institute of Mental Health and Addiction,
Utrecht, the Netherlands) for her contribution in the design of the study.
Data sharing: data are available on request.
Author Contributions
Conceived and designed the experiments: SMK KN FRG LB OS JKS.
Performed the experiments: SMK FRG. Analyzed the data: SMK KN
FRG JKS. Wrote the paper: SMK KN FRG LB OS JKS.
References
1. Letvak S, Ruhm CJ, McCoy T (2012) Depression in hospital-employed nurses.
Clin Nurse Spec 26: 177–182.
2. Hammig O, Brauchli R, Bauer G (2012) Effort-reward and work-life imbalance,
general stress and burnout among employees of a large public hospital in
Switzerland. Swiss Med Wkly 142.
3. Adriaenssens J, de Gucht V, Maes S (2012) The impact of traumatic events on
emergency room nurses: Findings from a questionnaire survey. Int J Nurs Stud
49: 1411–1422.
4. Gärtner FR, Nieuwenhuijsen K, van Dijk FJ, Sluiter JK (2012) Impaired work
functioning due to common mental disorders in nurses and allied health
professionals: the Nurses Work Functioning Questionnaire. Int Arch Occup
Environ Health 85: 125–138.
5. Letvak SA, Ruhm CJ, Gupta SN (2012) Nurses’ presenteeism and its effects on
self-reported quality of care and costs. Am J Nurs 112: 30–38.
6. Rai D, Kosidou K, Lundberg M, Araya R, Lewis G, et al. (2012) Psychological
distress and risk of long-term disability: population-based longitudinal study.
J Epidemiol Community Health 66: 586–592.
7. Hood J, Larranaga M (2007) Employee health surveillance in the health care
industry. AAOHN J 55: 423–431.
8. Koh D, Aw TC (2003) Surveillance in occupational health. Occup Environ Med
60: 705–710.
9. International Labour Organization (1998) Technical and ethical guidelines for
workers’ health surveillance. Geneva: ILO.
10. NVAB (2005) Leidraad Preventief Medisch Onderzoek [Guidance Workers’
Health Surveillance]. Utrecht, the Netherlands: NVAB [Netherlands Society of
Occupational Medicine].
11. Sluiter JK (2006) High-demand jobs: age-related diversity in work ability? Appl
Ergon 37: 429–440.
12. Riper H, Smit F, van der Zanden R, Conijn B, Kramer J, et al. (2007) E-mental
health. High tech, high touch, high trust. Utrecht: Trimbos-instituut.
13. Ritterband LM, Gonder-Frederick LA, Cox DJ, Clifton AD, West RW, et al.
(2003) Internet Interventions: In Review, In Use, and Into the Future. Prof
Psychol Res Pr 34: 527–534.
14. Warmerdam L, van Straten A, Twisk J, Riper H, Cuijpers P (2008) Internet-
based treatment for adults with depressive symptoms: randomized controlled
trial. J Med Internet Res 10: e44.
15. Farrer L, Christensen H, Griffiths KM, Mackinnon A (2011) Internet-based
CBT for depression with and without telephone tracking in a national helpline:
randomised controlled trial. PLoS One 6: e28099.
16. Riper H, Kramer J, Smit F, Conijn B, Schippers G, et al. (2008) Web-based self-
help for problem drinkers: a pragmatic randomized trial. Addiction 103: 218–
227.
17. Blankers M, Koeter MW, Schippers GM (2011) Internet therapy versus internet
self-help versus no treatment for problematic alcohol use: A randomized
controlled trial. J Consult Clin Psychol 79: 330–341.
18. Billings DW, Cook RF, Hendrickson A, Dove DC (2008) A web-based approach
to managing stress and mood disorders in the workforce. J Occup Environ Med
50: 960–968.
19. Gärtner FR, Ketelaar SM, Smeets O, Bolier L, Fischer E, et al. (2011) The
Mental Vitality @ Work study: design of a randomized controlled trial on the
effect of a workers’ health surveillance mental module for nurses and allied
health professionals. BMC Public Health 11: 290.
20. Schellings R, Kessels AG, ter Riet G, Sturmans F, Widdershoven GA, et al.
(2009) Indications and requirements for the use of prerandomization. J Clin
Epidemiol 62: 393–399.
21. Campbell MK, Elbourne DR, Altman DG (2004) CONSORT statement:
extension to cluster randomised trials. BMJ 328: 702–708.
22. Schulz KF, Altman DG, Moher D (2010) CONSORT 2010 statement: Updated
guidelines for reporting parallel group randomised trials. J Pharmacol
Pharmacother 1: 100–107.
23. Bolier L, Haverman M, Kramer J, Boon B, Smit F, et al. (2012) Internet-based
intervention to promote mental fitness in mildly depressed adults: design of a
randomized controlled trial. JMIR Res Protoc 1: e2.
24. Cursus ‘Sterk op het werk’ [Course Strong at Work]. Available: http://www.
loketgezondleven.nl/i-database/interventies/c/13907. Accessed: December 6,
2012.
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 10 September 2013 | Volume 8 | Issue 9 | e72546
25. de Graaf LE, Gerhards SA, Arntz A, Riper H, Metsemakers JF, et al. (2009)
Clinical effectiveness of online computerised cognitive-behavioural therapy
without support for depression in primary care: randomised trial. Br J Psychiatry
195: 73–80.
26. van Ballegooijen W, Riper H, van Straten A, Kramer J, Conijn B, et al. (2011) The
effects of an Internet based self-help course for reducing panic symptoms – Don’t
Panic Online: study protocol for a randomised controlled trial. Trials 12: 75.
27. Bolier L, Haverman M, Bohlmeijer E (2010) Psyfit online – Gerandomiseerde
effectstudie naar een e-mental fitness zelfhulpprogramma – Intern rapport [Psyfit
online – Randomized effect study on an online mental fitness self-help program –
Internal report]. Utrecht: Trimbos-instituut.
28. Spek V, Nyklicek I, Smits N, Cuijpers P, Riper H, et al. (2007) Internet-based
cognitive behavioural therapy for subthreshold depression in people over 50
years old: a randomized controlled clinical trial. Psychol Med 37: 1797–1806.
29. Spek V, Cuijpers P, Nyklicek I, Smits N, Riper H, et al. (2008) One-year follow-
up results of a randomized controlled clinical trial on internet-based cognitive
behavioural therapy for subthreshold depression in people over 50 years. Psychol
Med 38: 635–639.
30. Gärtner FR, Nieuwenhuijsen K, van Dijk FJ, Sluiter JK (2011) Psychometric
properties of the Nurses Work Functioning Questionnaire (NWFQ). PLoS One
6: e26565.
31. Gärtner FR, Nieuwenhuijsen K, van Dijk FJ, Sluiter JK (2012) Interpretability of
change in the Nurses Work Functioning Questionnaire: minimal important
change and smallest detectable change. J Clin Epidemiol 65: 1337–1347.
32. Terluin B (1998) De Vierdimensionele Klachtenlijst (4DKL) in de huisart-
spraktijk [The Four Dimensional Symptom Questionnaire (4DSQ)]. De
Psycholoog 33: 18–24.
33. Terluin B, van Marwijk HW, Ader HJ, de Vet HC, Penninx BW, et al. (2006)
The Four-Dimensional Symptom Questionnaire (4DSQ): a validation study of a
multidimensional self-report questionnaire to assess distress, depression, anxiety
and somatization. BMC Psychiatry 6: 34.
34. van Rhenen W, van Dijk FJ, Schaufeli WB, Blonk RW (2008) Distress or no
distress, that’s the question: A cutoff point for distress in a working population.
J Occup Med Toxicol 3: 3.
35. van Veldhoven M, Meijman TF (1994) Het meten van een psychosociale
arbeidsbelasting met een vragenlijst: de vragenlijst beleving en beoordeling van
arbeid (VBBA) [The measurement of psychosocial job demands with a
questionnaire: the questionnaire on the experience and evaluation of work
(QEEW)]. Amsterdam: Dutch Institute for Working Conditions.
36. Broersen JPJ, Fortuin RJ, Dijkstra L, van Veldhoven M, Prins J (2004) Monitor
Arboconvenanten: kengetallen en grenswaarden [Monitor occupational health
and safety: key indicators and limits]. TBV 12: 100–104.
37. Brom D, Kleber RJ (1985) De Schok Verwerkings Lijst [The Impact of Event
Scale]. Nederlands Tijdschrift voor de Psychologie 40: 164–168.
38. Horowitz M, Wilner N, Alvarez W (1979) Impact of Event Scale: a measure of
subjective stress. Psychosom Med 41: 209–218.
39. van der Ploeg E, Kleber RJ (2003) Acute and chronic job stressors among
ambulance personnel: predictors of health symptoms. Occup Environ Med 60
Suppl 1: i40–i46.
40. Tuomi K, Ilmarinen J, Jahkola A, Katajarinne L, Tulkki A (1998) Work Ability
Index. Helsinki: Finnish Institute of Occupational Health.
41. DeSouza CM, Legedza AT, Sankoh AJ (2009) An overview of practical
approaches for handling missing data in clinical trials. J Biopharm Stat 19:
1055–1073.
42. Cohen J (1988) Statistical power analysis for the behavioral sciences. 2nd edn.
Hillsdale, N.J.: Lawrence Erlbaum Associates.
43. Kristensen TS (2005) Intervention studies in occupational epidemiology. Occup
Environ Med 62: 205–210.
44. Ketelaar SM, Gärtner FR, Bolier L, Smeets O, Nieuwenhuijsen K, et al. (2013)
Mental Vitality @ Work – A Workers’ Health Surveillance mental module for
nurses and allied health care professionals. Process evaluation of a randomized
controlled trial. J Occup Environ Med 55: 563–571.
45. Lexis MA, Jansen NW, Stevens FC, van Amelsvoort LG, Kant I (2010)
Experience of health complaints and help seeking behavior in employees
screened for depressive complaints and risk of future sickness absence. J Occup
Rehabil 20: 537–546.
46. Codony M, Alonso J, Almansa J, Bernert S, de Girolamo G, et al. (2009)
Perceived need for mental health care and service use among adults in Western
Europe: results of the ESEMeD project. Psychiatr Serv 60: 1051–1058.
47. Fogg BJ (2009) A Behavior Model for Persuasive Design. Available: http://
bjfogg.com/fbm_files/page4_1 . Accessed: December 6, 2012.
48. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE (2012) Persuasive
system design does matter: a systematic review of adherence to web-based
interventions. J Med Internet Res 14: e152.
49. Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA (1998) The
AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening
test for problem drinking. Ambulatory Care Quality Improvement Project
(ACQUIP). Alcohol Use Disorders Identification Test. Arch Intern Med 158:
1789–1795.
50. Gual A, Segura L, Contel M, Heather N, Colom J (2002) Audit-3 and audit-4:
effectiveness of two short forms of the alcohol use disorders identification test.
Alcohol Alcohol 37: 591–596.
51. de Beurs E (2006) Brief Symptom Inventory (BSI): Handleiding. Leiden, the
Netherlands: Pits Publishers.
52. de Beurs E, Zitman FG (2006) De Brief Symptom Inventory (BSI): De
betrouwbaarheid en validiteit van een handzaam alternatief voor de SCL-90.
[The Brief Symptom Inventory (BSI): The reliability and validity of a brief
alternative of the SCL-90]. Maandblad Geestelijke Volksgezondheid 61: 120–
141.
53. Spitzer RL, Kroenke K, Williams JB (1999) Validation and utility of a self-report
version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation
of Mental Disorders. Patient Health Questionnaire. JAMA 282: 1737–1744.
54. Lowe B, Grafe K, Zipfel S, Spitzer RL, Herrmann-Lingen C, et al. (2003)
Detecting panic disorder in medical and psychosomatic outpatients: comparative
validation of the Hospital Anxiety and Depression Scale, the Patient Health
Questionnaire, a screening question, and physicians’ diagnosis. J Psychosom Res
55: 515–519.
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 11 September 2013 | Volume 8 | Issue 9 | e72546
© 2013 Ketelaar et al. This is an open-access article distributed under the
terms of the Creative Commons Attribution License:
https://creativecommons.org/licenses/by/4.0/ (the “License”), which permits
unrestricted use, distribution, and reproduction in any medium, provided the
original author and source are credited. Notwithstanding the ProQuest Terms
and Conditions, you may use this content in accordance with the terms of the
License.
