Skip to main content
Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Evaluation of a Web Portal for Improving Public Access to Evidence-Based Health Information and Health Literacy Skills: A Pragmatic Trial



Using the conceptual framework of shared decision-making and evidence-based practice, a web portal was developed to serve as a generic (non disease-specific) tailored intervention to improve the lay public's health literacy skills.


To evaluate the effects of the web portal compared to no intervention in a real-life setting.


A pragmatic randomised controlled parallel trial using simple randomisation of 96 parents who had children aged <4 years. Parents were allocated to receive either access to the portal or no intervention, and assigned three tasks to perform over a three-week period. These included a searching task, a critical appraisal task, and reporting on perceptions about participation. Data were collected from March through June 2011.


Use of the web portal was found to improve attitudes towards searching for health information. This variable was identified as the most important predictor of intention to search in both samples. Participants considered the web portal to have good usability, usefulness, and credibility. The intervention group showed slight increases in the use of evidence-based information, critical appraisal skills, and participation compared to the group receiving no intervention, but these differences were not statistically significant.


Despite the fact that the study was underpowered, we found that the web portal may have a positive effect on attitudes towards searching for health information. Furthermore, participants considered the web portal to be a relevant tool. It is important to continue experimenting with web-based resources in order to increase user participation in health care decision-making.

Trial Registration NCT01266798


The active involvement of healthcare users (hereafter referred to as ‘users’) is argued in respect for individual autonomy, as a critical component of sustainable health and healthcare [1], [2], and as central to evidence-based practice [3]. However, effective participation is dependent on access to research-based information, and skills that would enable users to obtain, understand, evaluate, and act upon the information available [2]. Such health literacy skills include basic reading and numeracy (functional literacy), as well as critical and social skills [2], [4]. Health literacy is described by the World Health Organization as the main desired outcome of health education [5], as an asset in itself, and as a public health issue [2], [5]. Through health literacy, it is argued that people are able to take better control of their own lives and health, including the personal, social and environmental determinants of health [2], [6], [7]. In a systematic review of the evidence, low health literacy levels were associated with poorer health, increased health care utilisation, inappropriate drug use, and the low uptake of disease prevention services (such as vaccinations) [8].

Although access to health information has been improved greatly by new information technologies, evidence-based information is not readily available to the lay public [9], [10], [11]. Studies have found that users may be overwhelmed and frustrated by the vast amount of information available and unsure about who or what they should trust [12], [13]. Moreover, people struggle to understand and critically appraise health information, do not effectively check the accuracy of health information they find and overrate the trustworthiness of such information [13], [14], [15], [16], [17], [18]. Specifically, people are unfamiliar with the principles of medical and health related research and concepts such as randomisation, risk, uncertainty, and causality [17], [19], [20], [21], [22], [23]. Research has also shown that many people are sub-optimally involved in decision making, unaware of their rights or of treatment alternatives, and uncertain about what they need to ask their health care provider [24], [25], [26], [27], [28].

Theoretical and empirical studies suggest that the development of initiatives targeting critical and interactive skills among users is needed, and that these efforts should be evaluated in order to inform practice [4], [29], [30]. Essential skills includes basic reading, writing and numeracy skills (functional or fundamental literacy), but also critical and social literacy skills, including scientific literacy, civic literacy and cultural literacy [4].

Coulter and Ellins' comprehensive review of the evidence indicates that interactive online interventions may be effective strategies for health education [30], [31], [32]. This method of learning is associated with high levels of user satisfaction [30] and may also be more cost-effective and flexible compared to traditional methods of health education [32]. The content can be also be easily updated and made available to all.

Inspired by these findings we developed a web portal, with the aim of improving the public's access to evidence-based health information and health literacy skills. What health literacy skills really entail has been conceptualised in many different ways. We used the multi-dimensional model formulated by Zarcadoolas and colleagues which contains four central domains: fundamental literacy (reading, writing, speaking and working with numbers), science literacy (understanding and using science and technology), civic literacy (skills and abilities that enables awareness, participation and involvement), and cultural literacy (skills and abilities to recognise, understand and use beliefs, customs, world-views and social identities) [4]. The web portal was designed from a public health perspective to target both healthy people as well as patients, and to be used either independently or in consultation with health providers. The web portal was intended to be used by those who are interested in searching for health information, who would like to know more about medical and health-related information, or need support in decisions related to health. Its content and key intervention targets were informed by extensive literature searches as well as explorative pre-studies with input from people within our target audiences, including focus groups and a questionnaire study based on the Theory of Planned Behaviour (TPB) [24], [33], [34], [35]. Three key barriers to obtaining information were identified; not knowing where to find reliable and relevant information, the inability to understand and critically appraise health information and the inability to exchange information in consultations [35]. The content of the web portal was tailored to address these barriers by facilitating specific domains of health literacy through a choice of evidence based strategies [35]. The web portal was developed within the conceptual framework of the shared-decision making model and evidence based practice, encouraging participation and emphasising the importance of that decisions should be based on the best available evidence [1], [3]. Using illustrations of typical healthcare topics, the web portal focused on how research is conducted and why this is important rather than just reporting conclusions and expert interpretations. Generic and non-disease specific in focus, the web portal was designed to be applicable to a range of healthcare decisions and settings, and included three facilitators or tool-sets to address each of the main barriers to obtaining information:

  1. Access to medical and health-related research databases, an introduction to research methods, the principles of science (based on the steps of the evidence based practice model) and levels of evidence synthesis [36], [37].
  2. A checklist for critically assessing health information (DISCERN) [38] and information about why critical assessment is important.
  3. A checklist for consultations with health care providers [39] and information about what decision making related to treatment and screening entails.

The development and content of the web portal is described in more detail in another paper [35], or can be viewed online at An overview of the targeted barriers, the content of the intervention, the hypothesised corresponding health literacy domains targeted are presented in table 1.

Table 1. Overview of the intervention components, corresponding hypothesised health literacy domains targeted and measurements to evaluate these.

The objective of this study was to evaluate the effects of this web portal intervention compared to no intervention in a real life setting on:

  1. - Beliefs about searching for health information and overall activation (participation).
  2. - Searching for research-based information and the development of critical appraisal skills.

In addition, we also wanted to get feedback from the participants on their satisfaction with the web portal.


The protocol for this trial and supporting CONSORT checklist are available as supporting information; see Checklist S1 and Protocol S1.


The study was a pragmatic parallel-trial in which one group received access to the portal and its tools (the intervention group) while the other group received no intervention (the control group).

Participants and recruitment.

Our intention was to include typical users for the web portal in the participant sample. In addition, we wished to increase the probability that the portal would be used by participants in association with visits to health professionals during the trial. Parents with children under the age of 4-years of age were therefore targeted. At this life stage, parents are typically having many questions about treating and preventing health problems. They are also healthcare users with the highest number of health visits per year both for themselves (a mean number of visits per year of 4. 6), and for their children (mean number of visits of 3) [40]. Such parents are also statistically more likely to search for health information online [41].

Sample size justification.

Sample size calculations should be based on assumptions about expected underlying population event rate and minimum detectable difference in means (and standard deviation of the response) in previous studied populations [42]. Few studies, however, have targeted the publics' health literacy skills including domains other than functional literacy. Thus, for the outcomes included in this trial we had very little previous experience to rely on. Considering that the intervention was passive in nature, we anticipated the effects to be modest. We assumed a conservative minimum detectable difference of means (amounting to a one point difference on seven point scale with a standard deviation of 2,1) on the outcome ‘beliefs about searching’. These assumptions were supported by the means and standard deviations observed in the piloting phase and in the validation study of the TPB questionnaire [34]. Furthermore, we based our sample size calculations on a power of 0.80, a level of significance of 0.95 and the use of a two-sided t-test for statistical analysis. Hence, the required sample size was estimated as 140 persons.

Sources and methods of recruitment.

Information about the study was distributed at maternity and child health centres, in online advertisements on social media networks, on Internet sites such as Google, and discussion forums for parents. Those who were interested and wished to participate were directed to a recruitment web page.

Informed consent and inclusion of participants.

People who expressed an interest in participating received information about the study, were asked to give written consent to participation, and directed to an online questionnaire for inclusion criteria screening. Participants were excluded if anyone else in their household was already participating in the study (to ensure that participants were blinded and to protect against potential sample contamination) and if they did not have children aged <4-years. If a participant did not meet these inclusion criteria, he or she was sent automated feedback describing the reasons why they were ineligible to participate (see Figure 1 for the CONSORT flow diagram).

Randomisation methods and allocation concealment.

Those who met the inclusion criteria were randomised using a simple randomisation procedure developed by SPSS. The study was single blinded in that participants were not informed about the study group to which they were allocated. All participants were told that they would be participating in testing a new web portal resource but that they would receive initial access to the portal at different times. All participants were given the same information and treated equally through the use of automated online systems and standardised emails.

Intervention delivery.

All information was delivered online, and data were collected using an online questionnaire system. Participants were sent tasks by email (Figure 2) at three different times. The intervention group was allowed access to the web portal immediately after randomisation through an email containing the URL, and given three days to explore its content and tools before receiving the first task. Each of the tasks corresponded to the web portal's three main content sections, namely: the improved use of research based information, improved critical appraisal of health information, and improved beliefs about participation (search and activation).

The purpose of the first task, the searching task, was to evaluate the degree to which participants used research based sources to obtain healthcare information. Participants were asked to formulate a question and to answer it by searching for online information. The intervention group was not asked specifically to use the web portal for this task and were thus free to search for information using any resource they felt would be useful, including the web portal.

The purpose of the second task, the critical appraisal task, was to evaluate participant critical appraisal skills related to health information. All participants were asked to rate the trustworthiness of an online article describing how to prevent swine flu, and that included information about vaccinations and alternative therapies. The intervention group was asked explicitly to use the full DISCERN critical appraisal tool which was provided on the web portal [38]. DISCERN is an instrument for patients and other lay-users of health information and is designed to allow them to evaluate the reliability of written information about treatment choices [43]. Swine flu was chosen as the subject for this task for a number of reasons. All vaccinations are voluntary in Norway, and related healthcare decisions were therefore seen as a topic that would be of concern to all parents. Moreover, there has been considerable discussion about vaccinations in the media over the last few years, characterised by strong and often conflicting viewpoints [23], [44], [45], [46], including debate about swine flu vaccinations [47], [48]. We therefore viewed this topic as having considerable potential interest to participants. The specific material we chose for evaluation was taken from a health information site identified using a Google search and was typical of the kind of information available on sites used by lay-people searching for health information.

The third task, reporting of beliefs about search for health information and activation, was designed to explore potential differences between the intervention and control groups in terms of beliefs about health information searches (attitudes, social expectations, perceived behavioural control and intention to search). It also served to explore differences in overall levels of activation. In this final task, the intervention group was also given four additional questions in order to evaluate their satisfaction with the web portal.

Each of the tasks was sent to participants at weekly intervals. This timing and the overall length were decided on for two reasons: firstly, participants were provided with sufficient time to complete the tasks at their own pace. Secondly, longer periods or shorter intervals between tasks could possibly have had potentially negative consequences on the response rate and result in participation fatigue. At the end of the study, after all data had been collected, the control group was also given access to the web portal. The time-frame for data collection was pre-specified, beginning March and lasting throughout June 2011.

Participant retention.

Basic non-sensitive background information and email addresses were kept on file to allow for a descriptive analysis of losses to follow-up. One automatic reminder per task was sent to participants if they failed to respond within six days.

Missing data.

No attempt was made to impute missing data due to no valid assumption to base this imputation on. All analyses were performed using available data, with all participants being analysed in the group to which they were randomly assigned.

Outcome assessment and analysis.

Few tools are available for evaluating improvements in health literacy skills, and most map only general reading and numeracy skills (in other words, only functional literacy) [49]. Our intervention was intended mostly to target and evaluate critical and social skills and we identified no single available instrument suitable to achieving these goals. The outcomes of this study were therefore evaluated using a selection of instruments which, when combined, were considered adequate to evaluating most of the important health literacy skills targeted by our intervention. The main outcomes and corresponding instruments are described in further detail below (see also table 1) and include both actual behaviour as well as behavioural beliefs. All data were automatically exported into SPSS via the online data collection program.

For the searching task the outcome was evaluated by categorising the accessed Internet material (identified by hyperlinks) as information that either had or had not been based on research. The information was considered to be ‘research-based’ if it took the form of a report about original research (e.g. a primary study) or summarised research that had been based on explicit and systematic criteria (e.g. a systematic review or decision support). The information was excluded if no references were provided or identified, and if there were no explicit systematic criteria related to how and why any included references had been chosen. The material (identified by hyperlinks) was categorised by two independent and blinded researchers with training in the field of evidence-based practice and evidence synthesis. Their conclusions were then discussed further with the lead researcher. It was expected that research based information would be found by more participants in the intervention group than the control group. This hypothesis was tested by calculating the relative risk and corresponding confidence intervals.

For the critical appraisal task we used DISCERN appraisal tool (item number 16) to compare overall respondent ratings of the swine flu information [43]. This tool measures the respondent's overall rating on a scale of 1–5, where a score of 1–2 indicates ‘low quality’ (serious or extensive shortcomings), 3 indicates ‘moderate’ quality (potentially important but not serious shortcomings) and a rating of 4–5 indicates that respondents felt the material to be of ‘high quality’ with only minimal shortcomings [43]. The study group's mean value information rating was measured against ratings of the same material made by two blinded external experts with professional research and healthcare backgrounds. Based on results from previous studies using DISCERN, the mean overall score of the intervention group was expected to be closer in value to the experts' rating than the control group [38]. We also expected that the overall quality rating made by the intervention group would be lower, given that evaluations based on explicit criteria tend to be more critical compared to personal opinion [50]. The effects of the intervention were measured by calculating the mean differences between the groups and applying one-sided t-tests.

For measuring beliefs about intention to search for health information and activation we used two instruments. A questionnaire was designed specifically for the purpose of evaluating beliefs about searching for health information, based on the TPB [34]. The Patient Activation Measure (PAM) was used, with permission from Insignia Health, to evaluate overall activation and participants' self-management abilities [51].

The TPB is a social cognition model rigorously tested and widely used to predict behavioural intentions [52], [53], [54]. According to this model, there are three cognitive variables that can predict behavioural intentions (Figure 3) [52], [55]. These are:

Figure 3. Model of the theory of planned behaviour by Ajzen 1991.

  1. Attitudes towards the behaviour (beliefs about the consequences of the behaviour, and positive or negative judgements about these consequences).
  2. Subjective norms about the behaviour (a person's own perception of social pressure or expectations and motivation to comply).
  3. Perceived behavioural control (how much a person has control over the behavior and how confident a person feels about being able to perform or not perform the behavior).

These three variables are measured using direct and indirect measures [52]. While the direct measures describe the overall beliefs associated with behavioural intentions, in this study of intention to search for health information, the indirect measures represented the specific beliefs that underlay the overall assessments and were seen as explanatory factors. To evaluate the influence of the intervention on the direct and indirect TPB components, we calculated the differences in mean overall scores and specific beliefs using a two-sided t-test. The minimum and maximum mean composite score for direct measures were 1 to 7, −21 to +21 for specific indirect measures and −63 to 63 for overall composite indirect score for subjective norm and perceived behavioural control, and −84 to +84 for attitude (where higher values indicate more favourable attitudes, greater social pressure and higher perceived behavioural control). The intervention group was expected to have stronger beliefs than the control group.

To explore the effects of the intervention on predicting intention to search, the dependency between ‘intention’ and the composite direct measures from the TPB questionnaire was investigated using a multiple regression model in which ‘intention’ was the dependent variable. Group assignment (web portal vs. no intervention), and the three composite scores from the TPB model (namely ‘Attitudes towards the behaviour’, ‘Subjective norms’ and ‘Perceived behavioural control’) were entered as independent variables. In addition, we estimated the effect of the intervention on the dependency between the three composite scores and the intention by entering the interaction terms between group assignment and each of the three composite scores as independent variables in the model. Furthermore, the TPB model, and consequently the questionnaire, consists of several operationalisations of theoretical constructs with certain assumptions about interrelationships between these (Figure 3). These relationships were explored by computing simple bivariate correlations using Pearson's r. The direct TPB measures are hypothesised to be positively correlated with intention, but the direct measures may also be interrelated as these are not seen as categories independent of each other [55]. The questionnaire is available upon request by contacting the authors.

Overall activation was measured using the Patient Activation Measure (PAM), a validated instrument applicable to both patients and healthy people [51], [56]. This instrument includes four key domains (believing the patient role is important, having confidence and knowledge necessary to taking action, actually taking action to improve one's health, and staying the course under stress) which are measured by a total of 13 items [56], [57]. Overall activation is scored on a rating from 0–100, where 100 indicates high activation and 0 indicates no activation [56], [57]. We anticipated that activation levels in the intervention group would be higher than activation levels for the control group. To evaluate this outcome, a two-sided t-test was used to measure and test the difference in mean overall score.

For measuring satisfaction with the web portal, we obtained user feedback from the intervention group about the web portal. Our evaluation of this was based on the Honeycomb model [58], a useful instrument applied to measurements of Internet site user experiences [59]. The model encompasses seven domains to assess whether a website is accessible, usable, credible, valuable, findable, desirable and useful [58]. In our study, questions were used to assess three of these measures of user satisfaction (credibility, usefulness and usability), each measured on a satisfaction scale from 1–7 (higher values indicating greater satisfaction). The following data were then summarised for each domain: mean values, standard deviation, median, and interquartile range.

Safety monitoring and adverse events

A tool to encourage participation could create potentially unnecessary pressure on users. This domain was captured using the TPB questionnaire (subjective norm). Other adverse effects were deemed unlikely.

Ethical aspects

Data were treated anonymously, and ethical approval was granted by the Norwegian Social Science Data Services (NSD), and Regional Committees for Medical and Health Research Ethics (REK). The trial was registered under the identification number NCT01266798.


Description of study participants

Four participants were excluded from the study because they did not meet the inclusion criterion of having children aged <4-years. One respondent provided incomplete contact details. In total, 96 participants were included, of which 47 were randomised to the intervention group and 49 to the control group (Figure 1). The overall response rates for the intervention group and control group were 60% (n = 28) and 80% (n = 39) respectively (Table 2). Those who chose to complete the first task generally continued throughout the whole study. The number of parents per outcome was 28 in the intervention group and 39 in the control group for the searching task (use of research), 27 in the intervention group and 39 in the control group for the critical appraisal task and 28 the intervention group and 39 in the control group for the beliefs about search and activation outcome.

Use of research

Two research-based sources were identified in the intervention group, and one in the control group. The relative risk was 2.8 (CI 0.3–29.2) (p = 0.39).

Critical appraisal

The mean rating of the information was 2.41 (SD 0.80) by the intervention group and 2.44 (SD 1.02) by the control. The mean difference was –0.03 (p = 0.904).

The difference between the expert rating (rated as 1) and the rating of the intervention group was 1.40, and for the control group was 1.43 (difference = −0.03; p = 0.904).

The distribution of the ratings across the two groups was not significantly different (Figure 4; Pearson Chi-Square = 1.605, p = 0.448).

Beliefs about searches for health information and activation

A statistically significant difference of 0.63 was found for overall attitude towards search: 0.6 (p = 0.03) in favour of the intervention group (see Table 3 for mean scores). The mean differences for the overall subjective norm (−0.2; p = 0.49) and perceived behavioural control (0.41; p = 0.15) as well as specific beliefs (all three p-values >0.25) related to search, were not statistically significant.

Table 3. Distribution of means and differences between groups.

The dependency between the direct composite measures from the TPB and intention to search was found not to differ significantly across the groups (p>0.1 for all three composite scores). The TPB constructs' overall prediction of intention to search across the complete sample was approximately 37% of the observed variation in the intention to search. Attitude was the most important positive predictor of intention (b = 0.51; p<0.002), whereas the predictive strength of subjective norm and perceived behaviour control was −0.15 (p = 0.25) and −0.06 (p = 0.72) respectively.

Direct attitude and perceived behavioral control had a statistical significant and positive correlation with intention (Pearson's r = 0.56 and 0.41 respectively), and were also found to be inter-correlated (Pearson's r = 0.59) (all p<0.001). Subjective norm had a non-statistically significant negative correlation with intention (Pearson's r = −0.1; p = 0.92), and non-statistically significant positive correlation with attitude (Pearson's r = 0.18; p = 0.15) and perceived behavioral control (Pearson's r = 0.69; p = 0.58).

The overall activation (PAM) score was 66.5 in the intervention group and 61.9 in the control group, and the mean difference was 4.61 (p = 0.20).

Satisfaction with the web portal

The mean usefulness rating of the web portal was 4.71 (SD 1.11), mean usability 4.14 (SD 0.97) and mean credibility 4.75 (SD 0.93) (see table 4 for respondents' rating).


Study limitations

This study had limitations but also appreciable strengths. Participant recruitment can be a significant obstacle in the research process, and this proved to be a challenge in our study. Given the resources available, we were unable to recruit the number of participants suggested by the power-analysis. The response rates were also lower than expected. Considering that the content of the web portal and tasks included in the trial may be considered as challenging or difficult by many parents, particularly by those with lower health literacy, a higher drop-out could have been expected among those with lower education. Although the correlation between health literacy and education is not perfect, it is a relevant predictor [8]. Background data information about those who dropped out of the study revealed no statistically significant differences in sex or education levels, compared to those who did not drop out which is a positive finding. There was, however, a small but borderline statistically significant (p<0.06) difference between the intervention and control group in terms of loss to follow-up: more were seen to drop out of the intervention group (Table 1). The difference in drop-out between groups may indicate that people joined the study primarily in order to receive access to the portal. Once they had access, they may have dropped out before the first task was given. Of those who did participate in one or more of tasks, the background characteristics across the intervention group and the control groups showed that participant distributions by sex and education were very similar. The mean level of education, however, was slightly higher in the control group. The fact that the loss to follow-up mainly happened between the time of the first screening questionnaire and the time of the first task may indicate that the tasks included in the trial were considered too extensive.

Using an online questionnaire system was a time-saving and cost-efficient data collection method. It allowed for the confidential treatment of data and provided an opportunity to use automatic systems for keeping track of response rates, sending out reminders, sending notifications directly to participants if data were missing, and to export the data directly into Excel and SPSS. In this way we reduced the risk of potential errors inherent in manual data entry, ensured that all communications were standardised, and that both groups were treated equally.

Intervention effects on critical skills

The effects of the intervention on the critical skills of participants were evaluated by means of a searching task and a critical appraisal task. Only minor improvements in the intervention group were identified by both tasks and these differences were not statistically significant.

The number of research-based sources reported by participants in the searching task was very low in both study groups. Interestingly, the majority of participants in the intervention group chose to use sources not included in the portal. This may imply that the respondents preferred to rely on sources with which they were already familiar, or that that the three-day interval between being given access and having to perform the first task, may have been too short to allow them to become familiarised with the portal itself.

In the analysis, the sources were categorised based on pragmatic yet strict judgments regarding whether the information was research-based or not. This proved to be difficult, particularly because few of the sites provided references or indicated what their information was based on. This applied even to information provided by government and public health organisations. Consequently, information that was research-based may have been excluded if it lacked appropriate references. One source, for example, which was originally excluded because it lacked references, was then re-included in our study based on the knowledge of our lead researcher (this material was the Norwegian translation of BMJ's Best Practice Patient information). That online health information is often unclear, incomplete or even misleading is supported by reviews of online literature [9], [10], [45] and underlines the importance of providing users with easily accessible research-based information and the need to improve critical appraisal skills. But it also indicates that those who present such information on the Internet should critically review their own publishing practices and place greater emphasis on transparency and sources of their information.

Beliefs about participation and intervention effects

Beliefs about participation were evaluated using a TPB questionnaire that addressed intention to search and underlying factors, and the PAM questionnaire describing overall activation. The overall score for beliefs about participation (based on both the TPB and PAM) was high in both groups, indicating that participants viewed taking a more active role favourably. When comparing the two groups, we found small statistically non-significant improvements in favour of the intervention group with both the instruments indicating higher intention to search and higher activation.

In the TPB questionnaire, both groups were also found to have strong favourable attitudes towards searching for health information. Whereas both groups reported high overall perceived behavioural control, their responses to specific beliefs highlighted the fact that certain aspects of searching were perceived as being slightly difficult. The overall social pressure to search was moderate. Of particular interest was the fact that parents experienced very little pressure or expectation from health professionals who might have been anticipated to have a central role in this. Our findings are similar to those from other studies showing that independent searches for health information are rarely discussed, facilitated or addressed during healthcare consultations [60], [61], [62], [63].

The mean differences of direct measures across the groups showed a statistically significant difference in favour of the intervention group in terms of overall attitude 0.6 (p = 0.03). This suggests that the web portal may be a useful tool for improving attitudes towards searching for health information, and is an interesting finding in view of the fact that attitude was also found to be the most important predictor of intention to search in both groups. That we found favourable attitude to be an important factor associated with searching for health information is supported by other studies [13], [60], [64], including another study in which the same questionnaire was used [34]. Perceived behavioural control was only a weak and not statistically significant predictor of intention to search but was found to have a strong correlation with both intention and attitude. Thus, although the user feeling of being in control of searching does not directly predict intention, it may still be an important underlying variable. Subjective norm was also found to have a weak relationship with intention and this finding supports earlier research showing that social expectations or pressure do not appear to be an important or positive predictor of intention to search [34].

We identified no statistically significant difference between the groups in terms of the effect of the intervention on the prediction of intention to search. The TPB components' overall prediction of intention to search was satisfactory and generally consistent with other studies in which the TPB had been used to predict behavioural intentions across a range of health topics [53].

Satisfaction with the portal

The satisfaction with the web portal was good. This is encouraging given that the purpose of the web portal, and the concepts introduced, were potentially novel to most respondents. A challenge in this project was that although we wanted to improve user access to evidence-based information we also wanted to improve knowledge and skills. When developing the web portal, a compromise was therefore reached between usability (how easy it is to access reliable health information) and educational intention (reflected for example by how people are routed through the web pages). Consequently, the web portal may potentially demand more of its users relative to traditional sources of health information. Despite this, the results of this study indicated that we may have come a long way towards achieving this balance correctly.

In the next phase of our research, the web portal will be made publicly available and will be search optimised. This will mean that when people search for health information using a general search engine, the web portal will range high on the list of hits and, in this way, will contribute to making evidence-based health information more available to the general public. The web portal is also in continuous development, and more features will be considered for inclusion. Ideas for expansion include a discussion forum about science and research, and a blog addressing current issues related to medical and health related information in the public debate.


This study was a pragmatic trial conducted in a real life setting. The recruitment of participants was challenging, the response rates were somewhat low and the intervention effects smaller than expected. Although this resulted in the study being underpowered, we found improvements on attitudes towards searching for health information, a variable identified as the most important predictor of intention to search. The relevance of the web portal to users was confirmed by the fact that the participants considered the web portal to have good usability, usefulness and credibility.

Implications for practice

It is vital that health practice and decision making should be based on the best available, current, valid and relevant evidence. Recognising that users should play a central role in evidence-based practice, people should be encouraged to take on an active role. Moreover, resources such as web portals should be made available in order to facilitate greater access to, and critical use of, research-based information.

Future efforts should aim at experimenting more with web-based resources in order to encourage user involvement in health care. Large samples are needed to identify more robust results. Furthermore, online resources alone may not be sufficient to improve health literacy skills effectively. More intensive interventions could include the use of the web portal during consultations with health providers, or as part of evidence-based practice courses for users such as patient representatives who have a particular interest in healthcare issues.


We are grateful to Andy Oxman for his advice and support in the planning of this study. We would also like to thank Claire Glenton, Liv Merete Reinar, Olof Birna Kristjansdottir and Kirsti Riiser for providing expertise as external outcome assessors in the analysis of this trial.

Author Contributions

Conceived and designed the experiments: AA AB JO SH. Performed the experiments: AA. Analyzed the data: AA JO. Contributed reagents/materials/analysis tools: AA AB JO SH. Wrote the paper: AA AB JO SH. Supervised the project: AB SH.


  1. 1. Coulter A (2002) The Autonomous patient: ending paternalism in medical care. London: TSO. 121 p.
  2. 2. Nutbeam D (2000) Health literacy as a public health goal: A challenge for contemporary health education and communication strategies into the 21st century. Health Promotion International 15: 259–267.
  3. 3. Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, et al. (2005) Sicily statement on evidence-based practice. BMC Medical Education. 1 p.
  4. 4. Zarcadoolas C, Pleasant A, Greer D (2006) Advancing health literacy: a framework for understanding and action. San Francisco, CA: Jossey-Bass. 368 p.
  5. 5. Nutbeam D (1998) Health promotion glossary. Geneva: World Health Organization. 36 p.
  6. 6. Freedman DA, Bess KD, Tucker HA, Boyd DL, Tuchman AM, et al. (2009) Public Health Literacy Defined. American Journal of Preventive Medicine 36: 446–451.
  7. 7. Peerson A, Saunders M (2009) Health literacy revisited: what do we mean and why does it matter? Health Promotion International 24: 285–296.
  8. 8. Berkman N, Sheridan S, Donahue K, Halpern D, Crotty K (2011) Low Health Literacy and Health Outcomes: An Updated Systematic Review. Annals of Internal Medicine 155: 97–101.
  9. 9. Glenton C, Paulsen E, Oxman A (2005) Portals to Wonderland? Health portals lead confusing information about the effects of health care. BMC Medical Informatics and Decision Making. 15 March ed. 8 p.
  10. 10. Hoye S, Hjortdahl P (2002) [“New wonder pill!”–what do Norwegian newspapers write]. Tidsskrift for Den norske Legeforening. 122(17): 1671–1676 (June 30):
  11. 11. Johansen L, Bjørndal A, Flottorp S, Grøtting T, Oxman A (1996) [Evaluation of health information in newspapers and brochures. What can we believe?]. Tidsskrift for Den norske Legeforening 116: 260–264.
  12. 12. Andreassen H, Sandaune A, Gammon D, Hjortdahl P (2002) [Norwegian use of Internet health services]. Tidsskrift for Den norske Legeforening 122: 1640–1644.
  13. 13. Fox S (2006) Online health search. PEW Internet and American Life Project: PEW Internet and American Life Project. 22 p.
  14. 14. Schwartz K, Roe T, Northrup J, Meza J, Seifeldin R, et al. (2006) Family medicine patients' use of the Internet for health information: a MetroNet study. Journal of the American Board of Family Medicine 19: 39–45.
  15. 15. Fox S, Jones S (2009) The Social Life of Health Information. Washington, DC: PEW Internet and American Life Project. 88 p.
  16. 16. Sillence E, Briggs P, Harris PR, Fishwick L (2007) How do patients evaluate and make use of online health information? Social Science & Medicine 64: 1853–1862.
  17. 17. Robinson E, Kerr C, Stevens A, Lilford R, Braunholtz D, et al. (2005) Lay public's understanding of equipoise and randomisation in randomised controlled trials. NHS R&D HTA Programme. 192 p.
  18. 18. Glenton C, Oxman A (1998) The use of evidence by health care user organizations. Health Expectations 1: 14–22.
  19. 19. Woloshin S, Schwartz L, Byram S, Sox H, Fischhoff B, et al. (2000) Women's understanding of the mammography screening debate. Archives of Internal Medicine 160: 1434–1440.
  20. 20. Weinstein N (1999) What does it mean to understand a risk? Evaluating risk comprehension. Journal of the National Cancer Institute. pp. 15–20.
  21. 21. Lloyd A (2001) The extent of patients' understanding of the risk of treatments. Quality in Health Care 10: Suppl 1i14–18.
  22. 22. Schwartz L, Woloshin S, Black W, Welch H (1997) The role of numeracy in understanding the benefit of screening mammography. Annals of Internal Medicine 127: 966–972.
  23. 23. Tickner S, Leman P, Woodcock A (2006) Factors underlying suboptimal childhood immunisation. Vaccine 24: 7030–7036.
  24. 24. Austvoll-Dahlgren A, Helseth S (2010) What informs parents' decision-making about childhood vaccinations? Journal of Advanced Nursing 66: 2421–2430.
  25. 25. Legare F, Ratte S, Gravel K, Graham I (2008) Barriers and facilitators to implementing shared decision-making in clinical practice: update of a systematic review of health professionals' perceptions. Patient Education and Counseling 73: 526–535.
  26. 26. Bugge C, Entwistle V, Watt I (2006) The significance for decision-making of information that is not exchanged by patients and health professionals during consultations. Social Science & Medicine 63: 2065–2078.
  27. 27. Detmer D, Singleton P, MacLeod S, Wait S, Taylor M, et al. (2003) The informed patient: Study Report. Cambridge: University of Cambridge. 53 p.
  28. 28. Coulter A, Jenkinson C (2005) European patients' views on the responsiveness of health systems and healthcare providers. European Journal of Public Health 15: 355–360.
  29. 29. Nutbeam D (2008) The evolving concept of health literacy. Social Science and Medicine 67: 2072–2078.
  30. 30. Coulter A, Ellins J (2006) Patient-focused interventions: a review of the evidence. London: Picker Institute Europe. 277 p.
  31. 31. Wantland D, Portillo C, Holzemer W, Slaughter R, McGhee E (2004) The effectiveness of Web-based vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes. Journal of Medical Internet Research 6: 40.
  32. 32. Murray E, Burns J, See T, Lai R, Nazareth I (2005) Interactive Health Communication Applications for people with chronic disease. Cochrane Database of Systematic Reviews. CD004274 p.
  33. 33. Austvoll-Dahlgren A, Helseth S (2011) Public health nurses' barriers and facilitators to the use of research in consultations about childhood vaccinations. Scandinavian Journal of Caring Sciences. 15 Dec. 8 p.
  34. 34. Austvoll-Dahlgren A, Sørum R, Helseth S (2012) Cognitive factors predicting intentions to search for health information: an application of the Theory of Planned Behaviour. Unpublished. 30 p.
  35. 35. Austvoll-Dahlgren A, Danielsen S, Opheim E, Bjørndal A, Reinar L, et al. (2012) A complex intervention: tailoring a web portal to improve user health literacy skills. Unpublished. 31 p.
  36. 36. Bjørndal A, Flottorp S, Klovning A (2007) Knowledge management in medicine and health sciences [Kunnskapshåndtering i medisin og helsefag] Oslo: Gyldendal akademisk. 195 p.
  37. 37. DiCenso A, Bayley L, Haynes R (2009) Accessing preappraised evidence: fine-tuning the 5S model into a 6S model. Annals of Internal Medicine 151: 99–101.
  38. 38. Charnock D, Shepperd S, Needham G, Gann R (1999) DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology and Community Health 53: 105–111.
  39. 39. Irwig L, Irwig J, Trevena L, Sweet M (2008) Smart health choices: making sense of health advice. London: Hammersmith. xi: 242.
  40. 40. Statistics Norway (2009) [Use of health services by sex and age]. Statistics Norway: SSB.
  41. 41. Wangberg S, Andreassen H, Kummervold P, Wynn R, Sorensen T (2009) Use of the internet for health purposes: trends in Norway 2000-2010. Scandinavian Journal of Caring Sciences 23: 691–696.
  42. 42. Radosevich D (2006) Designing an outcomes research study. In: Kane R, editor. Understanding health care outcomes research. 2nd ed. Sudbury, MA: Jones and Bartlett. pp. 23–58.
  43. 43. Charnock D (1998) The DISCERN handbook: Quality criteria for consumer health information on treatment choices. Oxford: Radcliffe Medical Press Ltd.
  44. 44. Mills E, Jadad AR, Ross C, Wilson K (2005) Systematic review of qualitative studies exploring parental beliefs and attitudes toward childhood vaccination identifies common barriers to vaccination. Journal of Clinical Epidemiology 58: 1081–1088.
  45. 45. Wolfe R, Sharp L, Lipsky M (2002) Content and design attributes of antivaccination web sites. Journal of American Medical Association 287: 3245–3248.
  46. 46. Goldacre B (2007) Medicine and the media: MMR: the scare stories are back. The British Medical Journal 335: 126–127.
  47. 47. Haug C (2009) [Fear and expectations]. Tidsskrift for Den norske Legeforening 129: 1619.
  48. 48. Teasdale E, Yardley L (2011) Understanding responses to government health recommendations: public perceptions of government advice for managing the H1N1 (swine flu) influenza pandemic. Patient Education and Counseling 85: 413–418.
  49. 49. Ishikawa H, Yano E (2008) Patient health literacy and participation in the health-care process. Health Expectations 11: 113–122.
  50. 50. Eysenbach G, Powell J, Kuss O, Eun-Ryoung S (2002) Empirical studies assessing the quality of health information for consumers on the world wide web. Journal of American Medical Association 287: 2691–2700.
  51. 51. Steinsbekk A (2008) [Patient Activation Measure]. Tidsskrift for den Norske Legeforening 128: 2316–2318.
  52. 52. Francis J, Eccles M, Johnston M, Walker A, Grimshaw J, et al. (2004) Constructing questionnaires based on the theory of planned behaviour: A manual for health services researchers. UK: Centre for Health Services Research, University of Newcastle. 42 p.
  53. 53. Armitage C, Conner M (2001) Efficacy of the theory of planned behaviour: A meta-analytic review. British Journal of Social Psychology 40: 471–499.
  54. 54. Eccles M, Hrisos S, Francis J, Kaner E, Dickinson H, et al. (2006) Do self- reported intentions predict clinicians' behaviour: a systematic review. Implementation Science. 28 p.
  55. 55. Ajzen I (1991) The Theory of Planned Behavior. Organizational Behavior and Human Decision Processes 50: 179–211.
  56. 56. Hibbard JH, Mahoney ER, Stockard J, Tusler M (2005) Development and testing of a short form of the patient activation measure. Health Services Research 40: 1918–1930.
  57. 57. Hibbard JH, Stockard J, Mahoney ER, Tusler M (2004) Development of the patient activation measure (PAM): Conceptualizing and measuring activation in patients and consumers. Health Services Research 39: 1005–1026.
  58. 58. Morville P (2004) User Experience Design. 2: Available: Accessed: 2012 May.
  59. 59. Rosenbaum S, Glenton C, Cracknell J (2008) User experiences of evidence-based online resources for health professionals: User testing of The Cochrane Library. BMC Medical Informatics and Decision Making. 34 p.
  60. 60. Wainstein B, Sterling-Levis K, Baker S, Taitz J, Brydon M (2006) Use of the Internet by parents of paediatric patients. Journal of Paediatrics and Child Health 42: 528–532.
  61. 61. Bylund C, Gueguen J, Sabee C, Imes R, Li Y, et al. (2007) Provider-patient dialogue about internet health information: An exploration of strategies to improve the provider-patient relationship. Patient Education and Counseling 66: 346–352.
  62. 62. Tuffrey C, Finlay F (2002) Use of the internet by parents of paediatric outpatients. Archives of Disease in Childhood 87: 534–536.
  63. 63. Gordon MM, Capell HA, Madhok R (2002) The use of the Internet as a resource for health information among patients attending a rheumatology clinic. Rheumatology 41: 1402–1405.
  64. 64. Sabel M, Strecher V, Schwartz J, Wang T, Karimipour D, et al. (2005) Patterns of Internet use and impact on patients with melanoma. Journal of the American Academy of Dermatology 52: 779–785.