Using the conceptual framework of shared decision-making and evidence-based practice, a web portal was developed to serve as a generic (non disease-specific) tailored intervention to improve the lay public's health literacy skills.
To evaluate the effects of the web portal compared to no intervention in a real-life setting.
A pragmatic randomised controlled parallel trial using simple randomisation of 96 parents who had children aged <4 years. Parents were allocated to receive either access to the portal or no intervention, and assigned three tasks to perform over a three-week period. These included a searching task, a critical appraisal task, and reporting on perceptions about participation. Data were collected from March through June 2011.
Use of the web portal was found to improve attitudes towards searching for health information. This variable was identified as the most important predictor of intention to search in both samples. Participants considered the web portal to have good usability, usefulness, and credibility. The intervention group showed slight increases in the use of evidence-based information, critical appraisal skills, and participation compared to the group receiving no intervention, but these differences were not statistically significant.
Despite the fact that the study was underpowered, we found that the web portal may have a positive effect on attitudes towards searching for health information. Furthermore, participants considered the web portal to be a relevant tool. It is important to continue experimenting with web-based resources in order to increase user participation in health care decision-making.
Citation: Austvoll-Dahlgren A, Bjørndal A, Odgaard-Jensen J, Helseth S (2012) Evaluation of a Web Portal for Improving Public Access to Evidence-Based Health Information and Health Literacy Skills: A Pragmatic Trial. PLoS ONE 7(5): e37715. https://doi.org/10.1371/journal.pone.0037715
Editor: Marcelo Hermes-Lima, Universidade de Brasília, Brazil
Received: September 20, 2011; Accepted: April 25, 2012; Published: May 31, 2012
Copyright: © 2012 Austvoll-Dahlgren et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The project was funded by Oslo University College. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
The active involvement of healthcare users (hereafter referred to as ‘users’) is argued in respect for individual autonomy, as a critical component of sustainable health and healthcare , , and as central to evidence-based practice . However, effective participation is dependent on access to research-based information, and skills that would enable users to obtain, understand, evaluate, and act upon the information available . Such health literacy skills include basic reading and numeracy (functional literacy), as well as critical and social skills , . Health literacy is described by the World Health Organization as the main desired outcome of health education , as an asset in itself, and as a public health issue , . Through health literacy, it is argued that people are able to take better control of their own lives and health, including the personal, social and environmental determinants of health , , . In a systematic review of the evidence, low health literacy levels were associated with poorer health, increased health care utilisation, inappropriate drug use, and the low uptake of disease prevention services (such as vaccinations) .
Although access to health information has been improved greatly by new information technologies, evidence-based information is not readily available to the lay public , , . Studies have found that users may be overwhelmed and frustrated by the vast amount of information available and unsure about who or what they should trust , . Moreover, people struggle to understand and critically appraise health information, do not effectively check the accuracy of health information they find and overrate the trustworthiness of such information , , , , , . Specifically, people are unfamiliar with the principles of medical and health related research and concepts such as randomisation, risk, uncertainty, and causality , , , , , . Research has also shown that many people are sub-optimally involved in decision making, unaware of their rights or of treatment alternatives, and uncertain about what they need to ask their health care provider , , , , .
Theoretical and empirical studies suggest that the development of initiatives targeting critical and interactive skills among users is needed, and that these efforts should be evaluated in order to inform practice , , . Essential skills includes basic reading, writing and numeracy skills (functional or fundamental literacy), but also critical and social literacy skills, including scientific literacy, civic literacy and cultural literacy .
Coulter and Ellins' comprehensive review of the evidence indicates that interactive online interventions may be effective strategies for health education , , . This method of learning is associated with high levels of user satisfaction  and may also be more cost-effective and flexible compared to traditional methods of health education . The content can be also be easily updated and made available to all.
Inspired by these findings we developed a web portal, with the aim of improving the public's access to evidence-based health information and health literacy skills. What health literacy skills really entail has been conceptualised in many different ways. We used the multi-dimensional model formulated by Zarcadoolas and colleagues which contains four central domains: fundamental literacy (reading, writing, speaking and working with numbers), science literacy (understanding and using science and technology), civic literacy (skills and abilities that enables awareness, participation and involvement), and cultural literacy (skills and abilities to recognise, understand and use beliefs, customs, world-views and social identities) . The web portal was designed from a public health perspective to target both healthy people as well as patients, and to be used either independently or in consultation with health providers. The web portal was intended to be used by those who are interested in searching for health information, who would like to know more about medical and health-related information, or need support in decisions related to health. Its content and key intervention targets were informed by extensive literature searches as well as explorative pre-studies with input from people within our target audiences, including focus groups and a questionnaire study based on the Theory of Planned Behaviour (TPB) , , , . Three key barriers to obtaining information were identified; not knowing where to find reliable and relevant information, the inability to understand and critically appraise health information and the inability to exchange information in consultations . The content of the web portal was tailored to address these barriers by facilitating specific domains of health literacy through a choice of evidence based strategies . The web portal was developed within the conceptual framework of the shared-decision making model and evidence based practice, encouraging participation and emphasising the importance of that decisions should be based on the best available evidence , . Using illustrations of typical healthcare topics, the web portal focused on how research is conducted and why this is important rather than just reporting conclusions and expert interpretations. Generic and non-disease specific in focus, the web portal was designed to be applicable to a range of healthcare decisions and settings, and included three facilitators or tool-sets to address each of the main barriers to obtaining information:
- Access to medical and health-related research databases, an introduction to research methods, the principles of science (based on the steps of the evidence based practice model) and levels of evidence synthesis , .
- A checklist for critically assessing health information (DISCERN)  and information about why critical assessment is important.
- A checklist for consultations with health care providers  and information about what decision making related to treatment and screening entails.
The development and content of the web portal is described in more detail in another paper , or can be viewed online at www.sunnskepsis.no. An overview of the targeted barriers, the content of the intervention, the hypothesised corresponding health literacy domains targeted are presented in table 1.
The objective of this study was to evaluate the effects of this web portal intervention compared to no intervention in a real life setting on:
- - Beliefs about searching for health information and overall activation (participation).
- - Searching for research-based information and the development of critical appraisal skills.
In addition, we also wanted to get feedback from the participants on their satisfaction with the web portal.
The study was a pragmatic parallel-trial in which one group received access to the portal and its tools (the intervention group) while the other group received no intervention (the control group).
Participants and recruitment.
Our intention was to include typical users for the web portal in the participant sample. In addition, we wished to increase the probability that the portal would be used by participants in association with visits to health professionals during the trial. Parents with children under the age of 4-years of age were therefore targeted. At this life stage, parents are typically having many questions about treating and preventing health problems. They are also healthcare users with the highest number of health visits per year both for themselves (a mean number of visits per year of 4. 6), and for their children (mean number of visits of 3) . Such parents are also statistically more likely to search for health information online .
Sample size justification.
Sample size calculations should be based on assumptions about expected underlying population event rate and minimum detectable difference in means (and standard deviation of the response) in previous studied populations . Few studies, however, have targeted the publics' health literacy skills including domains other than functional literacy. Thus, for the outcomes included in this trial we had very little previous experience to rely on. Considering that the intervention was passive in nature, we anticipated the effects to be modest. We assumed a conservative minimum detectable difference of means (amounting to a one point difference on seven point scale with a standard deviation of 2,1) on the outcome ‘beliefs about searching’. These assumptions were supported by the means and standard deviations observed in the piloting phase and in the validation study of the TPB questionnaire . Furthermore, we based our sample size calculations on a power of 0.80, a level of significance of 0.95 and the use of a two-sided t-test for statistical analysis. Hence, the required sample size was estimated as 140 persons.
Sources and methods of recruitment.
Information about the study was distributed at maternity and child health centres, in online advertisements on social media networks, on Internet sites such as Google, and discussion forums for parents. Those who were interested and wished to participate were directed to a recruitment web page.
Informed consent and inclusion of participants.
People who expressed an interest in participating received information about the study, were asked to give written consent to participation, and directed to an online questionnaire for inclusion criteria screening. Participants were excluded if anyone else in their household was already participating in the study (to ensure that participants were blinded and to protect against potential sample contamination) and if they did not have children aged <4-years. If a participant did not meet these inclusion criteria, he or she was sent automated feedback describing the reasons why they were ineligible to participate (see Figure 1 for the CONSORT flow diagram).
Randomisation methods and allocation concealment.
Those who met the inclusion criteria were randomised using a simple randomisation procedure developed by SPSS. The study was single blinded in that participants were not informed about the study group to which they were allocated. All participants were told that they would be participating in testing a new web portal resource but that they would receive initial access to the portal at different times. All participants were given the same information and treated equally through the use of automated online systems and standardised emails.
All information was delivered online, and data were collected using an online questionnaire system. Participants were sent tasks by email (Figure 2) at three different times. The intervention group was allowed access to the web portal immediately after randomisation through an email containing the URL, and given three days to explore its content and tools before receiving the first task. Each of the tasks corresponded to the web portal's three main content sections, namely: the improved use of research based information, improved critical appraisal of health information, and improved beliefs about participation (search and activation).
The purpose of the first task, the searching task, was to evaluate the degree to which participants used research based sources to obtain healthcare information. Participants were asked to formulate a question and to answer it by searching for online information. The intervention group was not asked specifically to use the web portal for this task and were thus free to search for information using any resource they felt would be useful, including the web portal.
The purpose of the second task, the critical appraisal task, was to evaluate participant critical appraisal skills related to health information. All participants were asked to rate the trustworthiness of an online article describing how to prevent swine flu, and that included information about vaccinations and alternative therapies. The intervention group was asked explicitly to use the full DISCERN critical appraisal tool which was provided on the web portal . DISCERN is an instrument for patients and other lay-users of health information and is designed to allow them to evaluate the reliability of written information about treatment choices . Swine flu was chosen as the subject for this task for a number of reasons. All vaccinations are voluntary in Norway, and related healthcare decisions were therefore seen as a topic that would be of concern to all parents. Moreover, there has been considerable discussion about vaccinations in the media over the last few years, characterised by strong and often conflicting viewpoints , , , , including debate about swine flu vaccinations , . We therefore viewed this topic as having considerable potential interest to participants. The specific material we chose for evaluation was taken from a health information site identified using a Google search and was typical of the kind of information available on sites used by lay-people searching for health information.
The third task, reporting of beliefs about search for health information and activation, was designed to explore potential differences between the intervention and control groups in terms of beliefs about health information searches (attitudes, social expectations, perceived behavioural control and intention to search). It also served to explore differences in overall levels of activation. In this final task, the intervention group was also given four additional questions in order to evaluate their satisfaction with the web portal.
Each of the tasks was sent to participants at weekly intervals. This timing and the overall length were decided on for two reasons: firstly, participants were provided with sufficient time to complete the tasks at their own pace. Secondly, longer periods or shorter intervals between tasks could possibly have had potentially negative consequences on the response rate and result in participation fatigue. At the end of the study, after all data had been collected, the control group was also given access to the web portal. The time-frame for data collection was pre-specified, beginning March and lasting throughout June 2011.
Basic non-sensitive background information and email addresses were kept on file to allow for a descriptive analysis of losses to follow-up. One automatic reminder per task was sent to participants if they failed to respond within six days.
No attempt was made to impute missing data due to no valid assumption to base this imputation on. All analyses were performed using available data, with all participants being analysed in the group to which they were randomly assigned.
Outcome assessment and analysis.
Few tools are available for evaluating improvements in health literacy skills, and most map only general reading and numeracy skills (in other words, only functional literacy) . Our intervention was intended mostly to target and evaluate critical and social skills and we identified no single available instrument suitable to achieving these goals. The outcomes of this study were therefore evaluated using a selection of instruments which, when combined, were considered adequate to evaluating most of the important health literacy skills targeted by our intervention. The main outcomes and corresponding instruments are described in further detail below (see also table 1) and include both actual behaviour as well as behavioural beliefs. All data were automatically exported into SPSS via the online data collection program.
For the searching task the outcome was evaluated by categorising the accessed Internet material (identified by hyperlinks) as information that either had or had not been based on research. The information was considered to be ‘research-based’ if it took the form of a report about original research (e.g. a primary study) or summarised research that had been based on explicit and systematic criteria (e.g. a systematic review or decision support). The information was excluded if no references were provided or identified, and if there were no explicit systematic criteria related to how and why any included references had been chosen. The material (identified by hyperlinks) was categorised by two independent and blinded researchers with training in the field of evidence-based practice and evidence synthesis. Their conclusions were then discussed further with the lead researcher. It was expected that research based information would be found by more participants in the intervention group than the control group. This hypothesis was tested by calculating the relative risk and corresponding confidence intervals.
For the critical appraisal task we used DISCERN appraisal tool (item number 16) to compare overall respondent ratings of the swine flu information . This tool measures the respondent's overall rating on a scale of 1–5, where a score of 1–2 indicates ‘low quality’ (serious or extensive shortcomings), 3 indicates ‘moderate’ quality (potentially important but not serious shortcomings) and a rating of 4–5 indicates that respondents felt the material to be of ‘high quality’ with only minimal shortcomings . The study group's mean value information rating was measured against ratings of the same material made by two blinded external experts with professional research and healthcare backgrounds. Based on results from previous studies using DISCERN, the mean overall score of the intervention group was expected to be closer in value to the experts' rating than the control group . We also expected that the overall quality rating made by the intervention group would be lower, given that evaluations based on explicit criteria tend to be more critical compared to personal opinion . The effects of the intervention were measured by calculating the mean differences between the groups and applying one-sided t-tests.
For measuring beliefs about intention to search for health information and activation we used two instruments. A questionnaire was designed specifically for the purpose of evaluating beliefs about searching for health information, based on the TPB . The Patient Activation Measure (PAM) was used, with permission from Insignia Health, to evaluate overall activation and participants' self-management abilities .
The TPB is a social cognition model rigorously tested and widely used to predict behavioural intentions , , . According to this model, there are three cognitive variables that can predict behavioural intentions (Figure 3) , . These are:
- Attitudes towards the behaviour (beliefs about the consequences of the behaviour, and positive or negative judgements about these consequences).
- Subjective norms about the behaviour (a person's own perception of social pressure or expectations and motivation to comply).
- Perceived behavioural control (how much a person has control over the behavior and how confident a person feels about being able to perform or not perform the behavior).
These three variables are measured using direct and indirect measures . While the direct measures describe the overall beliefs associated with behavioural intentions, in this study of intention to search for health information, the indirect measures represented the specific beliefs that underlay the overall assessments and were seen as explanatory factors. To evaluate the influence of the intervention on the direct and indirect TPB components, we calculated the differences in mean overall scores and specific beliefs using a two-sided t-test. The minimum and maximum mean composite score for direct measures were 1 to 7, −21 to +21 for specific indirect measures and −63 to 63 for overall composite indirect score for subjective norm and perceived behavioural control, and −84 to +84 for attitude (where higher values indicate more favourable attitudes, greater social pressure and higher perceived behavioural control). The intervention group was expected to have stronger beliefs than the control group.
To explore the effects of the intervention on predicting intention to search, the dependency between ‘intention’ and the composite direct measures from the TPB questionnaire was investigated using a multiple regression model in which ‘intention’ was the dependent variable. Group assignment (web portal vs. no intervention), and the three composite scores from the TPB model (namely ‘Attitudes towards the behaviour’, ‘Subjective norms’ and ‘Perceived behavioural control’) were entered as independent variables. In addition, we estimated the effect of the intervention on the dependency between the three composite scores and the intention by entering the interaction terms between group assignment and each of the three composite scores as independent variables in the model. Furthermore, the TPB model, and consequently the questionnaire, consists of several operationalisations of theoretical constructs with certain assumptions about interrelationships between these (Figure 3). These relationships were explored by computing simple bivariate correlations using Pearson's r. The direct TPB measures are hypothesised to be positively correlated with intention, but the direct measures may also be interrelated as these are not seen as categories independent of each other . The questionnaire is available upon request by contacting the authors.
Overall activation was measured using the Patient Activation Measure (PAM), a validated instrument applicable to both patients and healthy people , . This instrument includes four key domains (believing the patient role is important, having confidence and knowledge necessary to taking action, actually taking action to improve one's health, and staying the course under stress) which are measured by a total of 13 items , . Overall activation is scored on a rating from 0–100, where 100 indicates high activation and 0 indicates no activation , . We anticipated that activation levels in the intervention group would be higher than activation levels for the control group. To evaluate this outcome, a two-sided t-test was used to measure and test the difference in mean overall score.
For measuring satisfaction with the web portal, we obtained user feedback from the intervention group about the web portal. Our evaluation of this was based on the Honeycomb model , a useful instrument applied to measurements of Internet site user experiences . The model encompasses seven domains to assess whether a website is accessible, usable, credible, valuable, findable, desirable and useful . In our study, questions were used to assess three of these measures of user satisfaction (credibility, usefulness and usability), each measured on a satisfaction scale from 1–7 (higher values indicating greater satisfaction). The following data were then summarised for each domain: mean values, standard deviation, median, and interquartile range.
Safety monitoring and adverse events
A tool to encourage participation could create potentially unnecessary pressure on users. This domain was captured using the TPB questionnaire (subjective norm). Other adverse effects were deemed unlikely.
Description of study participants
Four participants were excluded from the study because they did not meet the inclusion criterion of having children aged <4-years. One respondent provided incomplete contact details. In total, 96 participants were included, of which 47 were randomised to the intervention group and 49 to the control group (Figure 1). The overall response rates for the intervention group and control group were 60% (n = 28) and 80% (n = 39) respectively (Table 2). Those who chose to complete the first task generally continued throughout the whole study. The number of parents per outcome was 28 in the intervention group and 39 in the control group for the searching task (use of research), 27 in the intervention group and 39 in the control group for the critical appraisal task and 28 the intervention group and 39 in the control group for the beliefs about search and activation outcome.
Use of research
Two research-based sources were identified in the intervention group, and one in the control group. The relative risk was 2.8 (CI 0.3–29.2) (p = 0.39).
The mean rating of the information was 2.41 (SD 0.80) by the intervention group and 2.44 (SD 1.02) by the control. The mean difference was –0.03 (p = 0.904).
The difference between the expert rating (rated as 1) and the rating of the intervention group was 1.40, and for the control group was 1.43 (difference = −0.03; p = 0.904).
The distribution of the ratings across the two groups was not significantly different (Figure 4; Pearson Chi-Square = 1.605, p = 0.448).
Beliefs about searches for health information and activation
A statistically significant difference of 0.63 was found for overall attitude towards search: 0.6 (p = 0.03) in favour of the intervention group (see Table 3 for mean scores). The mean differences for the overall subjective norm (−0.2; p = 0.49) and perceived behavioural control (0.41; p = 0.15) as well as specific beliefs (all three p-values >0.25) related to search, were not statistically significant.
The dependency between the direct composite measures from the TPB and intention to search was found not to differ significantly across the groups (p>0.1 for all three composite scores). The TPB constructs' overall prediction of intention to search across the complete sample was approximately 37% of the observed variation in the intention to search. Attitude was the most important positive predictor of intention (b = 0.51; p<0.002), whereas the predictive strength of subjective norm and perceived behaviour control was −0.15 (p = 0.25) and −0.06 (p = 0.72) respectively.
Direct attitude and perceived behavioral control had a statistical significant and positive correlation with intention (Pearson's r = 0.56 and 0.41 respectively), and were also found to be inter-correlated (Pearson's r = 0.59) (all p<0.001). Subjective norm had a non-statistically significant negative correlation with intention (Pearson's r = −0.1; p = 0.92), and non-statistically significant positive correlation with attitude (Pearson's r = 0.18; p = 0.15) and perceived behavioral control (Pearson's r = 0.69; p = 0.58).
The overall activation (PAM) score was 66.5 in the intervention group and 61.9 in the control group, and the mean difference was 4.61 (p = 0.20).
This study had limitations but also appreciable strengths. Participant recruitment can be a significant obstacle in the research process, and this proved to be a challenge in our study. Given the resources available, we were unable to recruit the number of participants suggested by the power-analysis. The response rates were also lower than expected. Considering that the content of the web portal and tasks included in the trial may be considered as challenging or difficult by many parents, particularly by those with lower health literacy, a higher drop-out could have been expected among those with lower education. Although the correlation between health literacy and education is not perfect, it is a relevant predictor . Background data information about those who dropped out of the study revealed no statistically significant differences in sex or education levels, compared to those who did not drop out which is a positive finding. There was, however, a small but borderline statistically significant (p<0.06) difference between the intervention and control group in terms of loss to follow-up: more were seen to drop out of the intervention group (Table 1). The difference in drop-out between groups may indicate that people joined the study primarily in order to receive access to the portal. Once they had access, they may have dropped out before the first task was given. Of those who did participate in one or more of tasks, the background characteristics across the intervention group and the control groups showed that participant distributions by sex and education were very similar. The mean level of education, however, was slightly higher in the control group. The fact that the loss to follow-up mainly happened between the time of the first screening questionnaire and the time of the first task may indicate that the tasks included in the trial were considered too extensive.
Using an online questionnaire system was a time-saving and cost-efficient data collection method. It allowed for the confidential treatment of data and provided an opportunity to use automatic systems for keeping track of response rates, sending out reminders, sending notifications directly to participants if data were missing, and to export the data directly into Excel and SPSS. In this way we reduced the risk of potential errors inherent in manual data entry, ensured that all communications were standardised, and that both groups were treated equally.
Intervention effects on critical skills
The effects of the intervention on the critical skills of participants were evaluated by means of a searching task and a critical appraisal task. Only minor improvements in the intervention group were identified by both tasks and these differences were not statistically significant.
The number of research-based sources reported by participants in the searching task was very low in both study groups. Interestingly, the majority of participants in the intervention group chose to use sources not included in the portal. This may imply that the respondents preferred to rely on sources with which they were already familiar, or that that the three-day interval between being given access and having to perform the first task, may have been too short to allow them to become familiarised with the portal itself.
In the analysis, the sources were categorised based on pragmatic yet strict judgments regarding whether the information was research-based or not. This proved to be difficult, particularly because few of the sites provided references or indicated what their information was based on. This applied even to information provided by government and public health organisations. Consequently, information that was research-based may have been excluded if it lacked appropriate references. One source, for example, which was originally excluded because it lacked references, was then re-included in our study based on the knowledge of our lead researcher (this material was the Norwegian translation of BMJ's Best Practice Patient information). That online health information is often unclear, incomplete or even misleading is supported by reviews of online literature , ,  and underlines the importance of providing users with easily accessible research-based information and the need to improve critical appraisal skills. But it also indicates that those who present such information on the Internet should critically review their own publishing practices and place greater emphasis on transparency and sources of their information.
Beliefs about participation and intervention effects
Beliefs about participation were evaluated using a TPB questionnaire that addressed intention to search and underlying factors, and the PAM questionnaire describing overall activation. The overall score for beliefs about participation (based on both the TPB and PAM) was high in both groups, indicating that participants viewed taking a more active role favourably. When comparing the two groups, we found small statistically non-significant improvements in favour of the intervention group with both the instruments indicating higher intention to search and higher activation.
In the TPB questionnaire, both groups were also found to have strong favourable attitudes towards searching for health information. Whereas both groups reported high overall perceived behavioural control, their responses to specific beliefs highlighted the fact that certain aspects of searching were perceived as being slightly difficult. The overall social pressure to search was moderate. Of particular interest was the fact that parents experienced very little pressure or expectation from health professionals who might have been anticipated to have a central role in this. Our findings are similar to those from other studies showing that independent searches for health information are rarely discussed, facilitated or addressed during healthcare consultations , , , .
The mean differences of direct measures across the groups showed a statistically significant difference in favour of the intervention group in terms of overall attitude 0.6 (p = 0.03). This suggests that the web portal may be a useful tool for improving attitudes towards searching for health information, and is an interesting finding in view of the fact that attitude was also found to be the most important predictor of intention to search in both groups. That we found favourable attitude to be an important factor associated with searching for health information is supported by other studies , , , including another study in which the same questionnaire was used . Perceived behavioural control was only a weak and not statistically significant predictor of intention to search but was found to have a strong correlation with both intention and attitude. Thus, although the user feeling of being in control of searching does not directly predict intention, it may still be an important underlying variable. Subjective norm was also found to have a weak relationship with intention and this finding supports earlier research showing that social expectations or pressure do not appear to be an important or positive predictor of intention to search .
We identified no statistically significant difference between the groups in terms of the effect of the intervention on the prediction of intention to search. The TPB components' overall prediction of intention to search was satisfactory and generally consistent with other studies in which the TPB had been used to predict behavioural intentions across a range of health topics .
Satisfaction with the portal
The satisfaction with the web portal was good. This is encouraging given that the purpose of the web portal, and the concepts introduced, were potentially novel to most respondents. A challenge in this project was that although we wanted to improve user access to evidence-based information we also wanted to improve knowledge and skills. When developing the web portal, a compromise was therefore reached between usability (how easy it is to access reliable health information) and educational intention (reflected for example by how people are routed through the web pages). Consequently, the web portal may potentially demand more of its users relative to traditional sources of health information. Despite this, the results of this study indicated that we may have come a long way towards achieving this balance correctly.
In the next phase of our research, the web portal will be made publicly available and will be search optimised. This will mean that when people search for health information using a general search engine, the web portal will range high on the list of hits and, in this way, will contribute to making evidence-based health information more available to the general public. The web portal is also in continuous development, and more features will be considered for inclusion. Ideas for expansion include a discussion forum about science and research, and a blog addressing current issues related to medical and health related information in the public debate.
This study was a pragmatic trial conducted in a real life setting. The recruitment of participants was challenging, the response rates were somewhat low and the intervention effects smaller than expected. Although this resulted in the study being underpowered, we found improvements on attitudes towards searching for health information, a variable identified as the most important predictor of intention to search. The relevance of the web portal to users was confirmed by the fact that the participants considered the web portal to have good usability, usefulness and credibility.
Implications for practice
It is vital that health practice and decision making should be based on the best available, current, valid and relevant evidence. Recognising that users should play a central role in evidence-based practice, people should be encouraged to take on an active role. Moreover, resources such as web portals should be made available in order to facilitate greater access to, and critical use of, research-based information.
Future efforts should aim at experimenting more with web-based resources in order to encourage user involvement in health care. Large samples are needed to identify more robust results. Furthermore, online resources alone may not be sufficient to improve health literacy skills effectively. More intensive interventions could include the use of the web portal during consultations with health providers, or as part of evidence-based practice courses for users such as patient representatives who have a particular interest in healthcare issues.
We are grateful to Andy Oxman for his advice and support in the planning of this study. We would also like to thank Claire Glenton, Liv Merete Reinar, Olof Birna Kristjansdottir and Kirsti Riiser for providing expertise as external outcome assessors in the analysis of this trial.
Conceived and designed the experiments: AA AB JO SH. Performed the experiments: AA. Analyzed the data: AA JO. Contributed reagents/materials/analysis tools: AA AB JO SH. Wrote the paper: AA AB JO SH. Supervised the project: AB SH.
- 1. Coulter A (2002) The Autonomous patient: ending paternalism in medical care. London: TSO. 121 p.A. Coulter2002The Autonomous patient: ending paternalism in medical care.LondonTSO121
- 2. Nutbeam D (2000) Health literacy as a public health goal: A challenge for contemporary health education and communication strategies into the 21st century. Health Promotion International 15: 259–267.D. Nutbeam2000Health literacy as a public health goal: A challenge for contemporary health education and communication strategies into the 21st century.Health Promotion International15259267
- 3. Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, et al. (2005) Sicily statement on evidence-based practice. BMC Medical Education. 1 p.M. DawesW. SummerskillP. GlasziouA. CartabellottaJ. Martin2005Sicily statement on evidence-based practice.BMC Medical Education1
- 4. Zarcadoolas C, Pleasant A, Greer D (2006) Advancing health literacy: a framework for understanding and action. San Francisco, CA: Jossey-Bass. 368 p.C. ZarcadoolasA. PleasantD. Greer2006Advancing health literacy: a framework for understanding and action.San Francisco, CAJossey-Bass368
- 5. Nutbeam D (1998) Health promotion glossary. Geneva: World Health Organization. 36 p.D. Nutbeam1998Health promotion glossary.GenevaWorld Health Organization36
- 6. Freedman DA, Bess KD, Tucker HA, Boyd DL, Tuchman AM, et al. (2009) Public Health Literacy Defined. American Journal of Preventive Medicine 36: 446–451.DA FreedmanKD BessHA TuckerDL BoydAM Tuchman2009Public Health Literacy Defined.American Journal of Preventive Medicine36446451
- 7. Peerson A, Saunders M (2009) Health literacy revisited: what do we mean and why does it matter? Health Promotion International 24: 285–296.A. PeersonM. Saunders2009Health literacy revisited: what do we mean and why does it matter?Health Promotion International24285296
- 8. Berkman N, Sheridan S, Donahue K, Halpern D, Crotty K (2011) Low Health Literacy and Health Outcomes: An Updated Systematic Review. Annals of Internal Medicine 155: 97–101.N. BerkmanS. SheridanK. DonahueD. HalpernK. Crotty2011Low Health Literacy and Health Outcomes: An Updated Systematic Review.Annals of Internal Medicine15597101
- 9. Glenton C, Paulsen E, Oxman A (2005) Portals to Wonderland? Health portals lead confusing information about the effects of health care. BMC Medical Informatics and Decision Making. 15 March ed. 8 p.C. GlentonE. PaulsenA. Oxman2005Portals to Wonderland?Health portals lead confusing information about the effects of health care. BMC Medical Informatics and Decision Making. 15 March ed8
- 10. Hoye S, Hjortdahl P (2002) [“New wonder pill!”–what do Norwegian newspapers write]. Tidsskrift for Den norske Legeforening. 122(17): 1671–1676 (June 30): S. HoyeP. Hjortdahl2002[“New wonder pill!”–what do Norwegian newspapers write]. Tidsskrift for Den norske Legeforening.122(17): 1671–1676 (June30)
- 11. Johansen L, Bjørndal A, Flottorp S, Grøtting T, Oxman A (1996) [Evaluation of health information in newspapers and brochures. What can we believe?]. Tidsskrift for Den norske Legeforening 116: 260–264.L. JohansenA. BjørndalS. FlottorpT. GrøttingA. Oxman1996[Evaluation of health information in newspapers and brochures. What can we believe?].Tidsskrift for Den norske Legeforening116260264
- 12. Andreassen H, Sandaune A, Gammon D, Hjortdahl P (2002) [Norwegian use of Internet health services]. Tidsskrift for Den norske Legeforening 122: 1640–1644.H. AndreassenA. SandauneD. GammonP. Hjortdahl2002[Norwegian use of Internet health services].Tidsskrift for Den norske Legeforening12216401644
- 13. Fox S (2006) Online health search. PEW Internet and American Life Project: PEW Internet and American Life Project. 22 p.S. Fox2006Online health search.PEW Internet and American Life ProjectPEW Internet and American Life Project22
- 14. Schwartz K, Roe T, Northrup J, Meza J, Seifeldin R, et al. (2006) Family medicine patients' use of the Internet for health information: a MetroNet study. Journal of the American Board of Family Medicine 19: 39–45.K. SchwartzT. RoeJ. NorthrupJ. MezaR. Seifeldin2006Family medicine patients' use of the Internet for health information: a MetroNet study.Journal of the American Board of Family Medicine193945
- 15. Fox S, Jones S (2009) The Social Life of Health Information. Washington, DC: PEW Internet and American Life Project. 88 p.S. FoxS. Jones2009The Social Life of Health Information.Washington, DCPEW Internet and American Life Project88
- 16. Sillence E, Briggs P, Harris PR, Fishwick L (2007) How do patients evaluate and make use of online health information? Social Science & Medicine 64: 1853–1862.E. SillenceP. BriggsPR HarrisL. Fishwick2007How do patients evaluate and make use of online health information?Social Science & Medicine6418531862
- 17. Robinson E, Kerr C, Stevens A, Lilford R, Braunholtz D, et al. (2005) Lay public's understanding of equipoise and randomisation in randomised controlled trials. NHS R&D HTA Programme. 192 p.E. RobinsonC. KerrA. StevensR. LilfordD. Braunholtz2005Lay public's understanding of equipoise and randomisation in randomised controlled trials.NHS R&D HTA Programme192
- 18. Glenton C, Oxman A (1998) The use of evidence by health care user organizations. Health Expectations 1: 14–22.C. GlentonA. Oxman1998The use of evidence by health care user organizations.Health Expectations11422
- 19. Woloshin S, Schwartz L, Byram S, Sox H, Fischhoff B, et al. (2000) Women's understanding of the mammography screening debate. Archives of Internal Medicine 160: 1434–1440.S. WoloshinL. SchwartzS. ByramH. SoxB. Fischhoff2000Women's understanding of the mammography screening debate.Archives of Internal Medicine16014341440
- 20. Weinstein N (1999) What does it mean to understand a risk? Evaluating risk comprehension. Journal of the National Cancer Institute. pp. 15–20.N. Weinstein1999What does it mean to understand a risk?Evaluating risk comprehension. Journal of the National Cancer Institute1520
- 21. Lloyd A (2001) The extent of patients' understanding of the risk of treatments. Quality in Health Care 10: Suppl 1i14–18.A. Lloyd2001The extent of patients' understanding of the risk of treatments.Quality in Health Care10Suppl 1i1418
- 22. Schwartz L, Woloshin S, Black W, Welch H (1997) The role of numeracy in understanding the benefit of screening mammography. Annals of Internal Medicine 127: 966–972.L. SchwartzS. WoloshinW. BlackH. Welch1997The role of numeracy in understanding the benefit of screening mammography.Annals of Internal Medicine127966972
- 23. Tickner S, Leman P, Woodcock A (2006) Factors underlying suboptimal childhood immunisation. Vaccine 24: 7030–7036.S. TicknerP. LemanA. Woodcock2006Factors underlying suboptimal childhood immunisation.Vaccine2470307036
- 24. Austvoll-Dahlgren A, Helseth S (2010) What informs parents' decision-making about childhood vaccinations? Journal of Advanced Nursing 66: 2421–2430.A. Austvoll-DahlgrenS. Helseth2010What informs parents' decision-making about childhood vaccinations?Journal of Advanced Nursing6624212430
- 25. Legare F, Ratte S, Gravel K, Graham I (2008) Barriers and facilitators to implementing shared decision-making in clinical practice: update of a systematic review of health professionals' perceptions. Patient Education and Counseling 73: 526–535.F. LegareS. RatteK. GravelI. Graham2008Barriers and facilitators to implementing shared decision-making in clinical practice: update of a systematic review of health professionals' perceptions.Patient Education and Counseling73526535
- 26. Bugge C, Entwistle V, Watt I (2006) The significance for decision-making of information that is not exchanged by patients and health professionals during consultations. Social Science & Medicine 63: 2065–2078.C. BuggeV. EntwistleI. Watt2006The significance for decision-making of information that is not exchanged by patients and health professionals during consultations.Social Science & Medicine6320652078
- 27. Detmer D, Singleton P, MacLeod S, Wait S, Taylor M, et al. (2003) The informed patient: Study Report. Cambridge: University of Cambridge. 53 p.D. DetmerP. SingletonS. MacLeodS. WaitM. Taylor2003The informed patient: Study Report.CambridgeUniversity of Cambridge53
- 28. Coulter A, Jenkinson C (2005) European patients' views on the responsiveness of health systems and healthcare providers. European Journal of Public Health 15: 355–360.A. CoulterC. Jenkinson2005European patients' views on the responsiveness of health systems and healthcare providers.European Journal of Public Health15355360
- 29. Nutbeam D (2008) The evolving concept of health literacy. Social Science and Medicine 67: 2072–2078.D. Nutbeam2008The evolving concept of health literacy.Social Science and Medicine6720722078
- 30. Coulter A, Ellins J (2006) Patient-focused interventions: a review of the evidence. London: Picker Institute Europe. 277 p.A. CoulterJ. Ellins2006Patient-focused interventions: a review of the evidence.LondonPicker Institute Europe277
- 31. Wantland D, Portillo C, Holzemer W, Slaughter R, McGhee E (2004) The effectiveness of Web-based vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes. Journal of Medical Internet Research 6: 40.D. WantlandC. PortilloW. HolzemerR. SlaughterE. McGhee2004The effectiveness of Web-based vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes.Journal of Medical Internet Research640
- 32. Murray E, Burns J, See T, Lai R, Nazareth I (2005) Interactive Health Communication Applications for people with chronic disease. Cochrane Database of Systematic Reviews. CD004274 p.E. MurrayJ. BurnsT. SeeR. LaiI. Nazareth2005Interactive Health Communication Applications for people with chronic disease.Cochrane Database of Systematic ReviewsCD004274
- 33. Austvoll-Dahlgren A, Helseth S (2011) Public health nurses' barriers and facilitators to the use of research in consultations about childhood vaccinations. Scandinavian Journal of Caring Sciences. 15 Dec. 8 p.A. Austvoll-DahlgrenS. Helseth2011Public health nurses' barriers and facilitators to the use of research in consultations about childhood vaccinations.Scandinavian Journal of Caring Sciences. 15 Dec8
- 34. Austvoll-Dahlgren A, Sørum R, Helseth S (2012) Cognitive factors predicting intentions to search for health information: an application of the Theory of Planned Behaviour. Unpublished. 30 p.A. Austvoll-DahlgrenR. SørumS. Helseth2012Cognitive factors predicting intentions to search for health information: an application of the Theory of Planned Behaviour.Unpublished30
- 35. Austvoll-Dahlgren A, Danielsen S, Opheim E, Bjørndal A, Reinar L, et al. (2012) A complex intervention: tailoring a web portal to improve user health literacy skills. Unpublished. 31 p.A. Austvoll-DahlgrenS. DanielsenE. OpheimA. BjørndalL. Reinar2012A complex intervention: tailoring a web portal to improve user health literacy skills.Unpublished31
- 36. Bjørndal A, Flottorp S, Klovning A (2007) Knowledge management in medicine and health sciences [Kunnskapshåndtering i medisin og helsefag] Oslo: Gyldendal akademisk. 195 p.A. BjørndalS. FlottorpA. Klovning2007Knowledge management in medicine and health sciences [Kunnskapshåndtering i medisin og helsefag] Oslo: Gyldendal akademisk195
- 37. DiCenso A, Bayley L, Haynes R (2009) Accessing preappraised evidence: fine-tuning the 5S model into a 6S model. Annals of Internal Medicine 151: 99–101.A. DiCensoL. BayleyR. Haynes2009Accessing preappraised evidence: fine-tuning the 5S model into a 6S model.Annals of Internal Medicine15199101
- 38. Charnock D, Shepperd S, Needham G, Gann R (1999) DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology and Community Health 53: 105–111.D. CharnockS. ShepperdG. NeedhamR. Gann1999DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.Journal of Epidemiology and Community Health53105111
- 39. Irwig L, Irwig J, Trevena L, Sweet M (2008) Smart health choices: making sense of health advice. London: Hammersmith. xi: 242.L. IrwigJ. IrwigL. TrevenaM. Sweet2008Smart health choices: making sense of health advice. London: Hammersmith.xi242
- 40. Statistics Norway (2009) [Use of health services by sex and age]. Statistics Norway: SSB. Norway Statistics2009[Use of health services by sex and age].Statistics NorwaySSB
- 41. Wangberg S, Andreassen H, Kummervold P, Wynn R, Sorensen T (2009) Use of the internet for health purposes: trends in Norway 2000-2010. Scandinavian Journal of Caring Sciences 23: 691–696.S. WangbergH. AndreassenP. KummervoldR. WynnT. Sorensen2009Use of the internet for health purposes: trends in Norway 2000-2010.Scandinavian Journal of Caring Sciences23691696
- 42. Radosevich D (2006) Designing an outcomes research study. In: Kane R, editor. Understanding health care outcomes research. 2nd ed. Sudbury, MA: Jones and Bartlett. pp. 23–58.D. Radosevich2006Designing an outcomes research study.R. KaneUnderstanding health care outcomes research2nd ed. Sudbury, MAJones and Bartlett2358
- 43. Charnock D (1998) The DISCERN handbook: Quality criteria for consumer health information on treatment choices. Oxford: Radcliffe Medical Press Ltd. D. Charnock1998The DISCERN handbook: Quality criteria for consumer health information on treatment choices.OxfordRadcliffe Medical Press Ltd
- 44. Mills E, Jadad AR, Ross C, Wilson K (2005) Systematic review of qualitative studies exploring parental beliefs and attitudes toward childhood vaccination identifies common barriers to vaccination. Journal of Clinical Epidemiology 58: 1081–1088.E. MillsAR JadadC. RossK. Wilson2005Systematic review of qualitative studies exploring parental beliefs and attitudes toward childhood vaccination identifies common barriers to vaccination.Journal of Clinical Epidemiology5810811088
- 45. Wolfe R, Sharp L, Lipsky M (2002) Content and design attributes of antivaccination web sites. Journal of American Medical Association 287: 3245–3248.R. WolfeL. SharpM. Lipsky2002Content and design attributes of antivaccination web sites.Journal of American Medical Association28732453248
- 46. Goldacre B (2007) Medicine and the media: MMR: the scare stories are back. The British Medical Journal 335: 126–127.B. Goldacre2007Medicine and the media: MMR: the scare stories are back.The British Medical Journal335126127
- 47. Haug C (2009) [Fear and expectations]. Tidsskrift for Den norske Legeforening 129: 1619.C. Haug2009[Fear and expectations].Tidsskrift for Den norske Legeforening1291619
- 48. Teasdale E, Yardley L (2011) Understanding responses to government health recommendations: public perceptions of government advice for managing the H1N1 (swine flu) influenza pandemic. Patient Education and Counseling 85: 413–418.E. TeasdaleL. Yardley2011Understanding responses to government health recommendations: public perceptions of government advice for managing the H1N1 (swine flu) influenza pandemic.Patient Education and Counseling85413418
- 49. Ishikawa H, Yano E (2008) Patient health literacy and participation in the health-care process. Health Expectations 11: 113–122.H. IshikawaE. Yano2008Patient health literacy and participation in the health-care process.Health Expectations11113122
- 50. Eysenbach G, Powell J, Kuss O, Eun-Ryoung S (2002) Empirical studies assessing the quality of health information for consumers on the world wide web. Journal of American Medical Association 287: 2691–2700.G. EysenbachJ. PowellO. KussS. Eun-Ryoung2002Empirical studies assessing the quality of health information for consumers on the world wide web.Journal of American Medical Association28726912700
- 51. Steinsbekk A (2008) [Patient Activation Measure]. Tidsskrift for den Norske Legeforening 128: 2316–2318.A. Steinsbekk2008[Patient Activation Measure].Tidsskrift for den Norske Legeforening12823162318
- 52. Francis J, Eccles M, Johnston M, Walker A, Grimshaw J, et al. (2004) Constructing questionnaires based on the theory of planned behaviour: A manual for health services researchers. UK: Centre for Health Services Research, University of Newcastle. 42 p.J. FrancisM. EcclesM. JohnstonA. WalkerJ. Grimshaw2004Constructing questionnaires based on the theory of planned behaviour: A manual for health services researchers.UKCentre for Health Services Research, University of Newcastle42
- 53. Armitage C, Conner M (2001) Efficacy of the theory of planned behaviour: A meta-analytic review. British Journal of Social Psychology 40: 471–499.C. ArmitageM. Conner2001Efficacy of the theory of planned behaviour: A meta-analytic review.British Journal of Social Psychology40471499
- 54. Eccles M, Hrisos S, Francis J, Kaner E, Dickinson H, et al. (2006) Do self- reported intentions predict clinicians' behaviour: a systematic review. Implementation Science. 28 p.M. EcclesS. HrisosJ. FrancisE. KanerH. Dickinson2006Do self- reported intentions predict clinicians' behaviour: a systematic review.Implementation Science28
- 55. Ajzen I (1991) The Theory of Planned Behavior. Organizational Behavior and Human Decision Processes 50: 179–211.I. Ajzen1991The Theory of Planned Behavior.Organizational Behavior and Human Decision Processes50179211
- 56. Hibbard JH, Mahoney ER, Stockard J, Tusler M (2005) Development and testing of a short form of the patient activation measure. Health Services Research 40: 1918–1930.JH HibbardER MahoneyJ. StockardM. Tusler2005Development and testing of a short form of the patient activation measure.Health Services Research4019181930
- 57. Hibbard JH, Stockard J, Mahoney ER, Tusler M (2004) Development of the patient activation measure (PAM): Conceptualizing and measuring activation in patients and consumers. Health Services Research 39: 1005–1026.JH HibbardJ. StockardER MahoneyM. Tusler2004Development of the patient activation measure (PAM): Conceptualizing and measuring activation in patients and consumers.Health Services Research3910051026
- 58. Morville P (2004) User Experience Design. 2: P. Morville2004User Experience Design.2Available: http://www.semanticstudios.com/publications/semantics/000029.php. Accessed: 2012 May. Available: http://www.semanticstudios.com/publications/semantics/000029.php. Accessed: 2012 May.
- 59. Rosenbaum S, Glenton C, Cracknell J (2008) User experiences of evidence-based online resources for health professionals: User testing of The Cochrane Library. BMC Medical Informatics and Decision Making. 34 p.S. RosenbaumC. GlentonJ. Cracknell2008User experiences of evidence-based online resources for health professionals: User testing of The Cochrane Library.BMC Medical Informatics and Decision Making34
- 60. Wainstein B, Sterling-Levis K, Baker S, Taitz J, Brydon M (2006) Use of the Internet by parents of paediatric patients. Journal of Paediatrics and Child Health 42: 528–532.B. WainsteinK. Sterling-LevisS. BakerJ. TaitzM. Brydon2006Use of the Internet by parents of paediatric patients.Journal of Paediatrics and Child Health42528532
- 61. Bylund C, Gueguen J, Sabee C, Imes R, Li Y, et al. (2007) Provider-patient dialogue about internet health information: An exploration of strategies to improve the provider-patient relationship. Patient Education and Counseling 66: 346–352.C. BylundJ. GueguenC. SabeeR. ImesY. Li2007Provider-patient dialogue about internet health information: An exploration of strategies to improve the provider-patient relationship.Patient Education and Counseling66346352
- 62. Tuffrey C, Finlay F (2002) Use of the internet by parents of paediatric outpatients. Archives of Disease in Childhood 87: 534–536.C. TuffreyF. Finlay2002Use of the internet by parents of paediatric outpatients.Archives of Disease in Childhood87534536
- 63. Gordon MM, Capell HA, Madhok R (2002) The use of the Internet as a resource for health information among patients attending a rheumatology clinic. Rheumatology 41: 1402–1405.MM GordonHA CapellR. Madhok2002The use of the Internet as a resource for health information among patients attending a rheumatology clinic.Rheumatology4114021405
- 64. Sabel M, Strecher V, Schwartz J, Wang T, Karimipour D, et al. (2005) Patterns of Internet use and impact on patients with melanoma. Journal of the American Academy of Dermatology 52: 779–785.M. SabelV. StrecherJ. SchwartzT. WangD. Karimipour2005Patterns of Internet use and impact on patients with melanoma.Journal of the American Academy of Dermatology52779785