Skip to main content
Advertisement
  • Loading metrics

Developing a youth-friendly internet-enabled HIV risk calculator: A collaborative approach with young key populations, living in Soweto, South Africa

  • Mamakiri Mulaudzi ,

    Roles Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Writing – original draft, Writing – review & editing

    mamakiri.mulaudzi1@wits.ac.za, mamakirim@assureafrica.com

    Affiliations African Social Sciences Unit of Research and Evaluation (ASSURE), Wits Health Consortium, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa, Perinatal HIV Research Unit (PHRU), School of Clinical Medicine, Faculty of Health Sciences, University of Witwatersrand, Johannesburg, South Africa, School of Human and Community Development, Psychology, University of the Witwatersrand, Johannesburg, South Africa

  • Gugulethu Tshabalala,

    Roles Methodology, Writing – review & editing

    Affiliations African Social Sciences Unit of Research and Evaluation (ASSURE), Wits Health Consortium, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa, Perinatal HIV Research Unit (PHRU), School of Clinical Medicine, Faculty of Health Sciences, University of Witwatersrand, Johannesburg, South Africa

  • Stefanie Hornschuh,

    Roles Writing – original draft, Writing – review & editing

    Affiliation Perinatal HIV Research Unit (PHRU), School of Clinical Medicine, Faculty of Health Sciences, University of Witwatersrand, Johannesburg, South Africa

  • Kofi Ebenezer Okyere-dede,

    Roles Software, Writing – review & editing

    Affiliation Dekode (Pty) Ltd, Software development company, Midrand, Johannesburg, South Africa

  • Minjue Wu,

    Roles Writing – review & editing

    Affiliation Harvard Global Health Institute, Harvard University, Cambridge, Massachusetts, United States of America

  • Oluwatobi Ifeloluwa Ariyo,

    Roles Writing – review & editing

    Affiliation Harvard Global Health Institute, Harvard University, Cambridge, Massachusetts, United States of America

  • Janan J. Dietrich

    Roles Conceptualization, Funding acquisition, Supervision, Writing – review & editing

    Affiliations African Social Sciences Unit of Research and Evaluation (ASSURE), Wits Health Consortium, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa, Perinatal HIV Research Unit (PHRU), School of Clinical Medicine, Faculty of Health Sciences, University of Witwatersrand, Johannesburg, South Africa, Health Systems Research Unit, South African Medical Research Council, Bellville, South Africa

Abstract

Although South Africa is the global epicenter of the HIV epidemic, the uptake of HIV testing and treatment among young people remains low. Concerns about confidentiality impede the utilization of HIV prevention services, which signals the need for discrete HIV prevention measures that leverage youth-friendly platforms. This paper describes the process of developing a youth-friendly internet-enabled HIV risk calculator in collaboration with young people, including young key populations aged between 18 and 24 years old. Using qualitative research, we conducted an exploratory study with 40 young people including young key population (lesbian, gay, bisexual, transgender (LGBT) individuals, men who have sex with men (MSM), and female sex workers). Eligible participants were young people aged between 18–24 years old and living in Soweto. Data was collected through two peer group discussions with young people aged 18–24 years, a once-off group discussion with the [Name of clinic removed for confidentiality] adolescent community advisory board members and once off face-to-face in-depth interviews with young key population groups: LGBT individuals, MSM, and female sex workers. LGBT individuals are identified as key populations because they face increased vulnerability to HIV/AIDS and other health risks due to societal stigma, discrimination, and obstacles in accessing healthcare and support services. The measures used to collect data included a socio-demographic questionnaire, a questionnaire on mobile phone usage, an HIV and STI risk assessment questionnaire, and a semi-structured interview guide. Framework analysis was used to analyse qualitative data through a qualitative data analysis software called NVivo. Descriptive statistics were summarized using SPSS for participant socio-demographics and mobile phone usage. Of the 40 enrolled participants, 58% were male, the median age was 20 (interquartile range 19–22.75), and 86% had access to the internet. Participants’ recommendations were considered in developing the HIV risk calculator. They indicated a preference for an easy-to-use, interactive, real-time assessment offering discrete and private means to self-assess HIV risk. In addition to providing feedback on the language and wording of the risk assessment tool, participants recommended creating a colorful, interactive and informational app. A collaborative and user-driven process is crucial for designing and developing HIV prevention tools for targeted groups. Participants emphasized that privacy, confidentiality, and ease of use contribute to the acceptability and willingness to use internet-enabled HIV prevention methods.

Author summary

Despite medical advances in the treatment of HIV, the disease remains highly stigmatized, which is hampering the uptake of HIV prevention and treatment programs among South Africa’s young people aged 15–24 years old. There are facilities available for HIV testing, treatment and management for young people in South Africa, however, barriers to privacy and confidentiality prevail. This study explored, through a collaborative process, how the development of a youth-friendly internet-enabled HIV risk calculator can help young people to overcome the hurdles they face in accessing HIV testing, prevention, and treatment, while affording them confidentiality and privacy, and providing valuable information about available services. While previous studies on mHealth for HIV prevention and risk assessment have focused primarily on MSM, this research addresses a gap by including young men and women of diverse sexual orientations and practices.

Introduction

Over the years, there has been considerable progress made in the fight against the HIV epidemic [1]. However, the enduring disparities in infection rates serve as a clear sign that there is still work to be done and challenges to overcome. Although there have been innovations and treatment plans to curb the infection rates of HIV, South Africa is still the country with the highest infection rates globally, with approximately 7.6 million people living with HIV in 2022 [2]. Furthermore, in that same year, the HIV prevalence in South Africa accounted for approximately 19% of the global HIV infections [2]. Among individuals aged 15–24 years, an estimated 56,000 new HIV infections were reported in 2022 [2,3]. The HIV epidemic in South Africa is not homogeneous; rather, prevalence and incidence vary by age, race, gender, and socio-economic status, sexual orientation and are distinctly characterized at the local, district, and provincial levels [4,5]

Central to the HIV epidemic in South Africa are the key populations, including MSM, female sex workers, and LGBT individuals [6,7,8]. These marginalized groups continue to face a disproportionate burden of HIV transmission due to an intricate interplay of biological susceptibilities, social determinants, and structural barriers [9,10]. Their vulnerability to HIV infections often heightened by societal stigmatization, discrimination, and limited access to proper healthcare resources [9].

Studies collectively illuminate the intricate landscape of HIV risks across diverse populations in South Africa, highlighting the interplay of various contributing factors. Studies highlight a concern about increased HIV prevalence in populations such as MSM and female sex workers [6,8]. These findings show that the underlying reason behind these heightened HIV rates is transactional sex, along with instances of low and inconsistent condom usage and difficulties in negotiating condom use [6,8]. This alarming blend of factors increases the susceptibility of these young key populations, thereby amplifying HIV transmission.

Mobile health (mHealth) has made significant strides in HIV prevention for young people across the world, including South Africa. mHealth involves using mobile phones and other wireless technologies to prevent, treat, manage and provide medical and public health care [11]. As smartphone ownership and use and internet accessibility have risen drastically among young people in South Africa [12], mHealth has enormous potential for HIV prevention. Internet-enabled applications (apps) are becoming increasingly viable platforms for HIV prevention for young South Africans [13]. Therefore, mobile phones and internet use for HIV risk assessment may become a highly acceptable strategy for this population group [14,15,16].

Involving young people in defining and designing interventions that work for them is crucial to providing age and culturally appropriate interventions [17]. Doing so encourages awareness among young people and promote access to these interventions [18].

Limited information is available on the use of mobile phones in HIV risk assessment and behavioral data collection among young people in South Africa [15]. Moreover, while many mobile apps have been developed for HIV prevention, there is limited evidence of the extent to which mHealth apps are being used and are acceptable by the target audience [19]. Despite the growing number of mHealth apps such as the WHO HTS Info, The AspectTM HIVST, HIVSMART; the usefulness of their content remains unknown [20,21,22,23]. A study by Raeesi et al. [24] aimed to validate some of these mHealth apps by rating the content of HIV-related mobile apps on the Google Play Store and Cafe Bazaar and determining the extent to which evidence-based medicine is incorporated into their content using a newly developed tool called the Evidence-Based Content Rating Tool of Mobile Health Applications (EBCRT-mHealth). According to the EBCRT-mHealth tool [24], one such app reviewed in the research is "WHO HTS Info" from the Google Play Store, which obtained a high rating of 5 (Excellent). The app was rated positively because it successfully delivered evidence-based information about HIV testing and services. Although the overall content quality of HIV-related apps was rated “poor” in 2018, it increased to “acceptable” in 2021 for Google Play Store apps. On the contrary, content quality in Cafe Bazaar apps declined in 2021, with content classified as “inappropriate” [24]. Developing an mHealth tool is a challenge but persuading the targeted group to use it presents an entirely new and considerably greater challenge [25,26].

Studies on mHealth for HIV prevention and HIV risk assessment have focused primarily on MSM, while overlooking other young population groups [27]. This research addresses a gap by including young men and women of diverse sexual orientations and practices. Furthermore, it describes the collaborative approach taken to develop a youth-friendly internet-enabled HIV risk calculator with young people aged 18–24 years living in Soweto, South Africa. The aim of this study is to describe the process taken to develop a youth-friendly internet-enabled HIV risk calculator for young people to assess their own risk for HIV acquisition and to identify their HIV risk factors.

Methods

Study population and setting

The study was conducted at the Perinatal HIV Research Unit (PHRU) based at Chris Hani Baragwanath Academic Hospital in Soweto, Johannesburg, South Africa. PHRU is affiliated with the University of the Witwatersrand and has been conducting research in Soweto for over 20 years. The unit has a walk-in HIV counseling and testing service (HTS) facility, accessible to anyone wanting to test for HIV. Participants lived in Soweto, a peri-urban area located in Gauteng, the country’s most densely populated province, with approximately 11.3 million people [28]. In 2017, HIV prevalence in the general population (15–49 years) in Gauteng was estimated at 17.6% [3]. A total of 46 young men and women participated in the study, including 23 in peer group discussions (PGDs), 18 in in-depth interviews (IDIs) and 5 Adolescent Community Advisory Board (ACAB) members in a group discussion. IDIs included young key populations: LGBT individuals, MSM, and female sex workers.

Study design

To develop a youth-friendly internet-enabled HIV risk calculator collaboratively, we applied an exploratory qualitative approach, using peer group discussions, and in-depth interviews.

Recruitment process

Peer group discussion (PGD) participants.

We performed active community recruitment in various locations in Soweto, such as community centers, shopping centers, and taxi ranks. Two trained fieldworkers approached young people, informed them about the study, and obtained contact details from those willing to participate to schedule them for a PGD. Of the 14 young women and 14 young men who were invited to the PGDs, 12 women and 11 men took part. To participate in the PGDs, they had to be aged 18–24 years, currently residing in Soweto, and able to read and write English.

In-Depth interview (IDI) participants.

We recruited IDI participants using purposive sampling and a chain referral strategy [29]—one of the most useful techniques to recruit marginalized individuals to access LGBT individuals, MSM and female sex workers [30]. Purposive sampling involved actively selecting participants who met specific inclusion criteria relevant to the study’s objectives. We focused on individuals aged 18–24 years, identifying as LGBT, MSM, or female sex workers, currently residing in Soweto, being sexually active (i.e., having engaged in sexual intercourse in the past six months at the time of the study), and able to read and write English.

To complement the purposive sampling, we employed a chain referral sampling method. Chain referral sampling entailed asking each participant who completed an IDI to refer other young people who met the study’s inclusion criteria. The initial contact with possible participants were recruited via active community recruitment where study fieldworkers went into the community to hand out pamphlets and where possible take contact details (cell phone number, age, gender and residence area) of individuals who showed interest to participate in the study. Following a completed IDI, participants were asked to refer other young people they knew who might meet the criteria for participation. Additionally, we used clinic-based recruitment through PHRU’s HIV counselling and testing service facility and other ongoing projects to reach young key populations.

Young people who met the study criteria were approached for IDIs. Individuals interested in participating received an invitation flyer with the study’s contact details. They could contact the study team either by visiting in person at the clinic, or via short messaging services (SMS), WhatsApp, or by sending a ‘please call me’ notification.

Group discussion with Adolescent Community Advisory Board (ACAB) participants.

Communication was sent to the Community Liaison Officers (CLOs) at the clinic to request participation from the ACAB in Soweto. At the time of the study, the ACAB comprised 14 members—five males and nine females—aged 16–24 years. Following approval from the CLOs, a fieldworker from our research team attended the ACAB meeting to provide information about the study and the purpose of involving the ACAB in the project. While eight ACAB members volunteered for the study, only five participated in the discussion.

Data collection procedures

Fig 1 provides a representation of the data collection process, and a detailed description of each process is provided below.

Peer group discussions.

Two trained female facilitators who were conversant in IsiZulu and Sesotho conducted two rounds of PGDs with young women and men. The first round of PGDs was conducted with a group of young women (n = 12) in the morning and a group of young men (n = 10) in the afternoon on the same day. All PGDs were conducted using participants’ preferred languages, which were IsiZulu, Sesotho and English. Participants completed a paper-based self-administered, five-minute demographic questionnaire and a questionnaire on mobile phone usage followed by a 10-minute paper-based HIV and STI risk assessment questionnaire.

The participants were informed that the researchers were investigating ways to develop the questionnaire into a youth-friendly internet-enabled HIV risk calculator, which would be presented to them at their second PGD. Participants were asked to critically review the content, focusing on the type of questions presented, identifying any missing questions, discussing questions they found difficult to respond to, and providing suggestions on how to reformulate those questions.The second round of PGDs with the same group of young women and men was conducted about one week later to seek young people’s opinions about how the HIV and STIs risk assessment questionnaire could be developed into a youth-friendly internet-enabled HIV risk calculator. A few participants (two women and three men) did not return for the second PGD. Participants were first reminded of the intention to develop a youth-friendly internet-enabled HIV risk calculator and were shown an early-stage prototype (Fig 2)—comprising five questions relating to age, gender, race, HIV status, and partner’s HIV status—via an online link on a tablet. Finally, participants discussed the first five questions in a group setting to assess the relevance and comprehensibility of the questions on the HIV Risk calculator.

During the second PGD participants were asked to explore their preferences for mobile phone-based Apps—ways in which an internet-enabled HIV risk calculator could assess HIV risk and increase awareness of HIV risk among young people, potential barriers, and facilitators to using it, and what additional information should be included in the HIV risk calculator. See interview guide for PGDs (S1 File), for questions participants were asked in the group discussions.

In-depth interviews.

Following the PGDs, 18 IDIs with young people who did not participate in any of the PGDs were conducted by trained female interviewers using a semi-structured discussion guide. IDIs were conducted in the participants’ preferred languages, which were IsiZulu, Sesotho and English. Before the discussion, IDI participants completed a paper-based self-administered, short five-minute socio-demographic questionnaire, a questionnaire on mobile phone usage, and a 10-minute paper-based HIV and STI risk assessment questionnaire. The IDIs covered topics on HIV risk perceptions, sexuality, and possible challenges specific to young key populations and their experiences when wanting to access HIV testing services. The interview guide included sensitive questions, specifically directed to young key population groups. During the interview, participants were shown the HIV risk calculator prototype (Fig 2) on a tablet and completed the five questions with the interviewer. IDIs concluded with a discussion around the aesthetics and design of the HIV risk calculator. See interview guide for IDIs (S2 File), for details on the type of questions asked.

Group discussion with ACAB.

Following the completion of the IDIs and PGDs, feedback was sent to the software developers to implement the participants’ suggestions arising from the PGDs and IDIs. A group discussion with members of ACAB—comprising a mixed group of five young people aged 18–24 years (two males and three females)—was then conducted to test the prototype and solicit opinions on how the HIV risk calculator can be further improved for HIV risk assessments with young people. Participants were shown the HIV risk calculator prototype, asked to interact with it and then provided feedback.

Socio-demographic questionnaire.

The demographic component of the questionnaire comprised questions about age, gender, marital status, level of education, previous experience testing for HIV and undergoing HIV risk reduction counseling.

Mobile phone usage questionnaire.

The section on mobile phone usage covered questions on mobile phone ownership, personal use, and access to a mobile phone and the internet.

HIV and STI risk assessment questionnaire.

This questionnaire was derived from the National HIV Counselling and Testing guidelines developed by the National Institute for Communicable Diseases [31]. HIV and sexual risk behavior questions on the HIV and STI risk assessment questionnaire related to type, number and age of sexual partners; frequency of vaginal, oral and anal sex; sex-related condom use; sex under the influence of substances (i.e., alcohol and drugs); sharing needles when injecting drugs; tattooing and piercing behavior; and emotional health. The type of sexual partner was defined as: husband/wife (referring to a sexual partner to whom the participant was married); dating (a sexual partner who was the main or regular partner, even if not living together); living together (a sexual partner with whom the participant was living, even if not married); or one-night stand (a sexual partner with whom the participant has had sex with for the first and one time only). To assess the history of STIs other than HIV, the questionnaire asked the following question: “have you ever tested positive for STIs such as gonorrhea, syphilis, or herpes?” The HIV and STI risk assessment questionnaire also included a question about emotional health: “Have you ever suffered from depression/stress?”

Semi-structured interview guides.

Semi-structured interview guides were used to facilitate the PGDs and IDIs and covered topics relating to HIV testing experience, barriers and facilitators, and to evaluate the questions on the HIV and STI risk assessment questionnaire. The guides were also used to explore perceptions about an internet-enabled HIV risk calculator and ways in which the HIV risk calculator could assess HIV risk and increase awareness of HIV risk among young people. The semi-structured interview guide for IDIs included specific questions for young people who identified as LGBT individuals, MSM, and female sex workers. The following additional questions were included in the IDI guide: “Given the nature of your work (as a sex worker), what challenges do you think you still experience in accessing HIV testing services?” (only asked of sex workers); “Given the nature of your sexual orientation, what challenges do you think you still experience in accessing HIV testing services?” (asked of LGBT individuals and MSM only); “How can the HIV risk calculator be most effectively used through mobile phones and internet access?” (asked of LGBT individuals, MSM and sex workers only).

HIV risk calculator.

The youth-friendly internet-enabled HIV risk calculator is an internet-enabled webpage with 30 self-reported item questions about HIV risk behavior. The webpage was developed in partnership with Dekode, a software development company in South Africa led by a young medical doctor and entrepreneur in digital technology and mHealth app development. The process of developing the HIV risk calculator began with building in the HIV risk assessment questions which were presented to participants for review during the PGDs and IDIs. We then conducted a thematic analysis of data based on PGDs and IDIs to identify preferences and suggestions for building a youth-friendly internet-enabled HIV risk calculator. The aim of the HIV risk calculator is to provide an opportunity for young people to assess their own risk for HIV infections by anonymously answering a series of questions about sexual practices, preferences and possible exposure to HIV infection. Based on the responses given by the user, the HIV risk calculator will generate a report with information about the possible type of risk identified and what the user can do to reduce their risk for HIV infection. The HIV risk calculator is self-administered and anonymous.

Data analysis

Qualitative data were analyzed manually using a framework analysis approach [32]. First, a list of codes was extracted from the PGD and IDI interview guides to obtain the initial framework of codes on a Microsoft Office Excel spreadsheet. The first two PGD and two IDI transcripts were read and coded by the lead researcher (first author) and an MA student (second author) based on the initial framework, while allowing identification of emerging codes. The list of codes was entered into NVIVO software for qualitative analysis and allowed for coding the remaining transcripts. The remaining transcripts were coded by the lead researcher following a line-by-line technique to capture codes that would otherwise emerge from data that was not initially part of the pre-existing codes. This process was followed until a comprehensive codebook was developed. A list of final codes was presented to the research team involved in this study. Final codes were discussed and approved by the supervising researchers. Codes were reviewed to form themes and sub-themes, which involved categorizing similar and diverging codes. Data was analyzed based on two key ideas: evaluating the HIV risk assessment questionnaire and developing a youth-friendly internet-enabled HIV risk calculator. Socio-demographic characteristics and mobile phone usage data were described using SPSS Version 25.0 [33].

Reflexivity

As part of reflexivity, the researcher MM shared her goals and her interest in the research on HIV prevention for young people. As the researcher, MM was a PhD student during data collection and has had extensive research experience in qualitative data collection and analysis. The research assistants involved in data collection were both MA students and received training in qualitative research methods. The researcher and research assistants had no prior knowledge or interaction with the participants. A rapport was built through initial introduction to the participants prior to the PGDs and in-depth interviews. Female participants were interviewed by the female research assistant and the male participants were matched with the male interviewer, whenever possible.

MM’s initial assumptions were that young people are not aware of their risk for HIV and other sexually transmitted infections. Through the discussion with the participants, it became a reality that young people cannot define what HIV risk is and did not understand the importance of completing an HIV risk assessment questionnaire during HIV counseling and testing. This realization may have biased the data analysis; hence, the data analysis involved two researchers and went through the research team for approval of codes and themes generated during the analysis.

Ethical considerations

This study was approved by the University of the Witwatersrand Human Research Ethics Committee (HREC) (non-medical, ethics reference number H14/09/04). All participants provided written informed consent for completing the HIV and STI risk assessment questionnaire and for participating in a PGD or IDI. Prior to commencing any study procedures, participants signed an informed consent form for the study and a separate informed consent form for the audio-recording. Participants were reimbursed R150 ($8.25) to cover the costs of transport and refreshments.

Results

Participant demographics and mobile phone usage

Table 1 provides a detailed description of the demographics, illustrating forty participants (23 cisgender males, 15 cisgender females, and 2 transwomen), participated in either a PGD or IDI. The median age was 20 (interquartile range 19–22.75), and 58% (23/40) were male. Of the 40 participants, 30 self-identified as straight, 9 as lesbian or gay, and 1 as bisexual. Two participants identified as female sex workers. Seventy eight percent (78%) self-reported HIV negative status and 13% were not sure of their HIV status. Mobile phone ownership was 88% (35/40), of which 97% (34/35) used the prepaid method for accessing mobile airtime, data, or SMS bundles. Overall, 72.5% (29/40) had access to the internet via a mobile phone, tablet, laptop, or computer.

Feedback on the risk assessment for HIV and STI questionnaire

PGD and IDI participants were asked to identify HIV and STI risk assessment questions that they found difficult to comprehend and were asked to suggest how the questions could be modified or clarified to a level of language and wording that young people would understand.

Four categories of feedback were identified based on discussions with PGD and IDI participants: (1) confusing questions, (2) difficult words in questions, (3) vague questions, and (4) controversial questions. Table 2 presents the list of questions identified as problematic and indicates how these were revised following the PGDs and IDIs.

thumbnail
Table 2. List of HIV and STI risk assessment questions identified as problematic, representative extracts, and revised version of the questions.

https://doi.org/10.1371/journal.pdig.0000672.t002

“They [the questions] are straight out clear and it’s simple English…but then I think the questions on this App, they need to be optional whether in English or in Zulu or in whatever language that they may prefer.” (IDI-013, transgender, 20 years old)

“[The] English is straight forward. There’s no school that doesnt teach English. Even if I do English at a certain age, this and that. No one must complain about that English. After all, we are not in primary. (PGD, females, 18–24 years old)

However, in some questions, participants identified words they considered unfamiliar to them, which needed to be revised or removed. Other participants suggested that the HIV risk calculator be made available in all South Africa’s 11 official languages, to cater to young people for whom English is not their primary language.

“Like, before you can answer questions, you must choose which language do you wanna use. Maybe if IsiZulu, then questions must be in [Isi]Zulu. Sometimes, isiZulu does make sense.” (PGD, females, 18–24 years old)

“Ja ja, translations as well they are also important because inclusivity is crucial” (IDI-011, female, 24 years old)

Certain questions were identified as confusing and difficult for young people to understand, while others sparked debate and were questioned for their relevance or appropriateness in assessing HIV risk. For example, a few IDI and PGD participants perceived the question regarding race (i.e., What is your race and what is your partner’s race?) to be irrelevant to HIV risk. The question raised insightful debates across the PGDs and among some IDI participants. Most did not like the question, stating that asking about race prejudices Black people as the population that is HIV infected more than other racial groups.

“If you tick every question and then you change the race question, maybe you choose black, I mean Indian and then the percentage changes, yeah that’s offensive. It means the question must not be asked.” (PGD males, 18–24 years old)

“Yes, why? We live in a democratic society. Do you have to tell me, ask about my partner’s race? I think that’s irrelevant as well. (IDI-010, male 24 years old)

While some participants felt that the question is offensive, other participants misconstrued the relevance of the race question in the questionnaire. For example, a participant in a peer group discussion with males, suggested that the race question is for statistical purposes to prove that HIV came from white people and gay men.

“[the question about] race is very clear. As the gent said earlier, HIV was brought by white people and gays so by asking race you will be able to find the average number even though the painful part is that we will not get the exact number from the U.S, but we will be able to get our own number. We are not going to have sufficient evidence, but it will help to know that as black people how many are affected by HIV, do you get me? We will compare the number of blacks, coloureds, and Indians and if we win it will prove that they came with HIV.” (PGD males, 18–24 years old)

Another participant in the PGD with females, suggested that the race question is to show that most black people from the village are affected by HIV because they love sex and lack of access to condoms when compared to white people.

“I think it [race question]is very important because as people we are different and most people who live in the villages are not exposed to things such as condoms so race can clarify that in this province it is mostly black people, they live in the village and they don’t have facilities to access condoms to protect themselves or maybe to see if black people love sex more than white people you see.” (PGD females, 18–24 years old)

Feedback on developing a youth-friendly internet-enabled HIV risk calculator

Four themes were identified from the discussion with PGD and IDI participants on developing a youth-friendly internet-enabled HIV risk calculator: (1) Preference for an app vs. a webpage, (2) privacy and confidentiality, (3) colorful, fun, and discrete interface, (4) an interactive HIV and STI risk assessment and information providing tool. Table 3 provides an overview of participants’ suggestions and preferences for a youth friendly HIV risk calculator and how these were used to guide the design process and adaptations of the HIV risk calculator.

Preference for an App vs. a webpage

Participants in this study preferred a mobile App HIV risk calculator to a website because the mobile App version will be quick and easy to access and cost less data usage.

“…besides the webpage being more expensive than the app. Umm I think the app you can access it very easy and quicker than the webpage cause with the webpage you probably have to go to your browser, after your browser you click www dot blah blah dot co dot ja whatever, unlike the app it’s one click away” (PGD ACAB, mixed gender, 18–24 years old)

Discrete interface

Participants advised that the HIV risk calculator should not be obvious that it is related to HIV. They suggested a discrete interface especially if it is a mobile App version. Young people preferred discrete and private means of assessing their own risk for HIV. While participants were eager to receive notifications or reminders from the online HIV risk assessment to test for HIV every three months, the majority concurred that the design and logo of the internet-enabled HIV risk assessment should not contain any HIV-related imagery.

"If it’s in a web[page]…a website it should say a HIV Risk Calculator then in [an]App it should be something different then [only]if you have access to it you will see what is happening [what the App is about]” (PGD ACAB, mixed gender, 18–24 years old)

Privacy and confidentiality

Participants emphasized the need for privacy and confidentiality of the HIV risk calculator. Some participants shared their fears about being found having an App for HIV and subsequently being perceived to be sexually active. During an in-depth interview one participant expressed their need for confidentiality of the information about sexual risk behaviours shared on the HIV risk calculator.

“If I’m in a relationship with more than one partner, I wouldn’t want my partner to know about my sexual risk behaviour. I wouldn’t want that person to have access to such that I have 3 partners” (IDI-006, Male, 18–24 years old)

Colorful interface

Participant stated that the App should be eye catching, colorful and have graphics to grab the interest of young people.

“I’m a person who loves color so they should add a little bit of [color]. It should look a bit like a game so, like, people can at least be attracted to it, get to try it.” (IDI-013, transgender, 20 years old)

The same participant suggested that researchers and developers should avoid “being boring” and learn from other social media apps that are appealing.

“Just try not to make something so monotone. The reason why the Instagram(s), the twitter(s) and everything take on so wonderfully is that it’s very graphic, you know”.(IDI-013, transgender, 20 years old)

thumbnail
Table 3. Participant preferences for a youth-friendly HIV risk calculator and representative extracts.

https://doi.org/10.1371/journal.pdig.0000672.t003

Participants reiterated their preference for an interactive and informational online-based HIV risk assessment, which would allow them to ask questions openly about their health and identify nearby youth-friendly clinics for HIV testing.

Throughout the software development phase, participants’ opinions, and recommendations from PGDs and IDIs were considered in developing an HIV risk calculator prototype incorporating a chatbot platform. Fig 3 illustrates the logo and personas, while Fig 4 illustrates the final HIV risk calculator prototype elements, and Fig 5 illustrates the final prototype front page.

thumbnail
Fig 4. HIV risk calculator final prototype with example questions.

https://doi.org/10.1371/journal.pdig.0000672.g004

Discussion

Our study is one of only a few South African studies to take a participatory approach to developing an HIV risk calculator for young people. Using peer group discussions and in-depth interviews, young people in collaboration with researchers were able to develop a youth-friendly internet-enabled HIV risk calculator for self-assessment of HIV risk. A youth-friendly HIV risk calculator suggests an age and context appropriate, accessible, discrete, and confidential HIV risk tool. Our study applied the Principles for Digital Development framework [34], focusing on four of the nine principles. The nine principles being: design with the user; understand the existing ecosystem; design for scale; build for sustainability; be data driven; use open standards, open data, open source and open innovation; reuse and improve; address privacy and security and be collaborative [34]. By applying the four principles of digital development, being: design with the user; reuse and improve; address privacy and security and be collaborative, our study was able to collaboratively develop an HIV risk calculator that is youth-friendly, acceptable, confidential, discreet and user interface design.

Our data revealed the value of obtaining young people’s input when developing a mHealth tool. This is similar to other studies, some of which were in South Africa, that used participatory research to co-develop mobile health applications for HIV prevention among different populations including young people, adolescent girls and young women as well as MSM [18,35,36,37,38]. These studies revealed the importance of including target populations in the development of mHealth for HIV prevention. In their study in Peru, which was aimed to develop an SMS text message intervention on sexual and reproductive health with adolescents and youth, Guerrero et al. [18] explain how they had to rephrase certain messages to the level of understanding of adolescents and youth in Peru. For example, study participants suggested the use of words that youth prefer, such as “boyfriend/girlfriend” instead of “intimate partner” and “v zone” instead of “vagina” to grab the attention of adolescents and youth. Similarly, in this study, we identified certain questions in the HIV risk assessment questionnaire that were not at the desired level of understanding for young people in a peri-urban South African setting. Guerrero et al. [18], revealed the importance of involving adolescents and young people in co-creating age and culturally appropriate mHealth applications. All questions considered problematic were revised in collaboration with participants and discussed further with the research team for corroboration. Questions approved by participants and the research team were then sent to the software developer to incorporate into the HIV risk calculator.

Findings from this study also show that the HIV risk calculator, as a confidential and anonymised self-assessment for HIV risk, can potentially reduce under reporting or over reporting of HIV risk practices. Similar to our findings, Dietrich et al. [35], revealed that participants identified confidentiality and privacy as key factors that can enhance participants’ confidence and comfortability in answering intimate questions. Thus, this could potentially reduce the challenge of providing socially desirable responses inherent in HIV risk assessments conducted in clinics [18,35].

Findings from the PGDs and IDIs for the HIV risk assessment questionnaire showed that the participants understood most of the questions. The main areas of concern were the phrasing and relevance of the question [36].

Additionally, the HIV and STI questionnaire lacked definitions of terms relating to sexual activity, which was a concern, as these are crucial for recognizing potential risks. Therefore, refining some questions was essential to ensuring that young people understood and responded appropriately to the HIV risk questions. Most concerning was the finding that revealed controversy surrounding the question about race in the HIV risk assessment. Some young people did not understand why the question about race is included in the HIV risk assessment questionnaire. Further concern was to learn about the misconstrued perception that young people in the study held about that question on race. In our knowledge, there are no studies that explore young people’s understanding of the questions included in the HIV and STI risk assessment questionnaire, particularly in South Africa. Therefore, we can confirm that allowing young people to scrutinize the questions posed for assessing HIV risk is a critical step in ensuring that questions are age-appropriate and context-relevant. This study emphasizes the need for a youth-engagement approach to developing HIV prevention interventions that are acceptable and effective for young people. While several mobile apps have been developed for HIV prevention [20,21,22,23], there is limited evidence of the extent to which they are being used and are considered acceptable by the target audience. This study adds to the limited existing literature on mobile phones on HIV risk assessment, and behavioral data collection among young people in South Africa [15].

The responses from the PGDs and IDIs provided valuable insights into the acceptability of the HIV risk assessment questions and guidance on the design of the HIV risk calculator.

Designing a reusable, yet simple app that accurately reflects both current HIV risk and change in risk over time entails balancing the app’s efficacy with the responsibility of informing users of how each criterion is used. For instance, using certain sensitive criteria such as race or ill-defined questions can mislead users into reaching erroneous conclusions about the role these factors play in risk outcomes. Comparing different risk measurements yielded by different answer choice combinations can give rise to problematic assumptions that correlative criteria cause HIV exposure, which may reinforce stigma and assumptions regarding the HIV patient demographic. This resembles the findings of a literature review that revealed that strategies employed in the communication of HIV risks had the potential to cause unintended adverse effects, including stigmatization [37]. Certain participants voiced concerns about questions that did not correspond with their sexual identity or beliefs about the range of what they deemed to be acceptable sex actions. Thus, to avoid ambiguity or skewing data with preconceived biases, it is crucial to clarify subjective nuances in criteria with specific technical details such as time, frequency, and physical description. And it is crucial for HIV prevention organizations to ask themselves continually whether young people understand their messaging.

Findings from the PGDs and IDIs also indicated that financial accessibility, privacy, and interface design are vital to creating an effective HIV risk calculator. Data availability and costs are vital factors in marketing mHealth apps, particularly in low-and middle-income countries. Our study participants preferred an app platform over a website because of data constraints. However, some participants voiced concerns that even an app might not be the most equitable platform, given that smartphone access varies across different socio-economic backgrounds. This highlights the importance of creating a data-friendly app that places minimal financial constraints on users. In South Africa, the government has taken the initiative to give young people internet access for educational purposes. While initiatives such as Project Isiswe have made Wi-Fi available in several areas in Soweto [38], there is a need to ensure our app is equitable and accessible to the most vulnerable and disadvantaged populations if we are to succeed in our efforts to provide an effective tool in the prevention of HIV in the youth of South Africa.

Social stigma is one of the biggest barriers to HIV prevention in youth [39,40,41]. As with other studies on HIV prevention using mHealth, [42,43], the participants in this study stated that they preferred an app that prioritised user confidentiality.

When creating the HIV risk calculator app, we discovered young people’s preference for an interactive and colorful interface. Designing a user-friendly interface was critical to maintaining adherence. Considering the information fatigue already reported in several studies on HIV prevention, creating an interactive app presented an excellent opportunity to provide young people with information they want or choose to receive [44,45]. The modifications made to the HIV risk assessment questions and the HIV risk calculator through the collaborative process may present the opportunity for improved accuracy of the HIV risk assessment and acceptability of the App among young people in South Africa.

Limitations

The collaborative nature of the PGDs and IDIs meant that participants may have responded more favorably than if the process had been less participative. Because only young people who were approached and available could take part in the study, the findings are not generalizable to all young people in Soweto, South Africa. The study could be improved by using surveys to rate the HIV risk calculator’s usability, usefulness and acceptability. While participants could identify items that were difficult or vague, some of them did not understand the meaning of HIV risk, and therefore could not suggest how to improve certain question items. Fluency in English was not a criterion for participation, as the study did not want to exclude young people who were not in school. IDIs, PGDs and group discussions were conducted in the participants’ preferred languages. However, the demographic questionnaire, the HIV and STI risk assessment questionnaire, and the HIV risk calculator questions were in English and not translated into other local languages spoken in Soweto (IsiZulu, Sesotho, IsiXhosa), which may have influenced participants’ responses. The study was a small-scale research project and therefore could not apply some of the principles outlined in the Principles for Digital Development which included: understanding the existing ecosystem; design for scale; build for sustainability; be data driven; use open standards, open data, open source and open innovation; reuse and improve. This was mainly due to the limited scope of the research project and therefore further studies can be conducted to improve the uptake, accessibility of the HIV risk calculator and influence policy in HIV prevention efforts in South Africa.

Conclusion

The research team discovered the value of taking a collaborative approach to developing an mHealth app for assessing HIV risk and aiding prevention. We found that privacy, confidentiality and ease of use are crucial to promoting acceptability and a willingness to use novel and internet-enabled HIV prevention tools.

Participants indicated a preference for an easy-to-use, interactive, real-time assessment that not only offers a discrete and private means to self-assess HIV risk, but that is also colorful, interactive and informational. In addition, to ensure effective communication, they suggested creating an app that has accessible language and avoids unnecessary literary barriers.

The young people in this study also proposed providing links to clinics or doctors for more information, which is encouraging and demonstrates their openness to accessing health care online.

Declarative statements

Any opinions, findings, conclusions or recommendation expressed in this material is solely the responsibility of the authors and do not necessarily represent the official views of the funders.

Supporting information

Acknowledgments

The authors wish to thank the young people who took part in this study, and the adolescent community advisory board (ACAB) [name of clinic removed for confidentiality], which was integral to data collection. We also thank Dr Kathryn Hopkins and Prof Kennedy Otwombe for reviewing the first draft of the manuscript.

References

  1. 1. Jones J, Sullivan PS, Curran JW. Progress in the HIV epidemic: Identifying goals and measuring success. PLoS Med. 2019;16(1).
  2. 2. UNAIDS. HIV and AIDS Estimates -South Africa [Internet]. 2022 [cited 2023 Aug 20]. Available from://www.unaids.org/en/regionscountries/countries/southafrica
  3. 3. Simbayi LC, Zuma K, Zungu N, Moyo S, Marinda E, Jooste S, Mabaso M, Ramlagan S, North A, van Zyl J, Mohlabane N, Dietrich C NI and the SVT (2019). South African National HIV Prevalence, Incidence, Behaviour and Communication Survey, 2017 [Internet]. Cape Town, South Africa; 2019. Available from: https://www.hsrcpress.ac.za/books/south-african-national-hiv-prevalence-incidence-behaviour-and-communication-survey-2017
  4. 4. Shisana O, Risher K, Celentano DD, Zungu N, Rehle T, Ngcaweni B, et al. Does marital status matter in an HIV hyperendemic country? Findings from the 2012 South African National HIV Prevalence, Incidence and Behaviour Survey. AIDS Care—Psychological and Socio-Medical Aspects of AIDS/HIV. 2016;28(2).
  5. 5. Mabaso M, Makola L, Naidoo I, Mlangeni LL, Jooste S, Simbayi L. HIV prevalence in South Africa through gender and racial lenses: results from the 2012 population-based national household survey. Int J Equity Health. 2019;18(1). pmid:31666077
  6. 6. Metheny N, Stephenson R, Darbes LA, Chavanduka TMD, Essack Z, van Rooyen H. Correlates of Substance Misuse, Transactional Sex, and Depressive Symptomatology Among Partnered Gay, Bisexual and Other Men Who Have Sex with Men in South Africa and Namibia. AIDS Behav. 2022 Jun 1;26(6):2003–14. pmid:34997385
  7. 7. Sikhosana N, Mokgatle MM. A qualitative exploration on accounts of condom-use negotiation with clients: Challenges and predicaments related to sex work among street-based female sex workers in ekurhuleni district, south africa. Pan African Medical Journal. 2021 Sep 1;40. pmid:35059100
  8. 8. Stone J, Mukandavire C, Boily MC, Fraser H, Mishra S, Schwartz S, et al. Estimating the contribution of key populations towards HIV transmission in South Africa. 2021; Available from: http://onlinelibrary.wiley.com/doi/ pmid:33533115
  9. 9. Chen C, Baral S, Comins CA, Mcingana M, Wang L, Phetlhu DR, et al. HIV- and sex work-related stigmas and quality of life of female sex workers living with HIV in South Africa: a cross-sectional study. BMC Infect Dis. 2022 Dec 1;22(1). pmid:36474210
  10. 10. Mulemfo DM, Moyo I, Mavhandu-Mudzusi AH. LGBTQI+ Experiences of HIV Management Services at Primary Healthcare Facilities in Gauteng Province, South Africa. Adv Public Health. 2023;2023.
  11. 11. Goldenberg T, Mcdougal SJ, Sullivan PS, Stekler JD, Stephenson R, Goldenberg T. Preferences for a Mobile HIV Prevention App for Men Who Have Sex With Men. JMIR mHealth uHealth. 2014;2(4):e47. pmid:25355249
  12. 12. Tshuma S, Bhardwaj S, Alexander H, Benjamin P. mHealth and Young People in South Africa [Internet]. Available from: https://www.unicef.org/southafrica/SAF_resources_saAdolescentSocialMedia.pdf
  13. 13. Dietrich JJ, Lazarus E, Andrasik M, Hornschuh S, Otwombe K, Morgan C, et al. Mobile Phone Questionnaires for Sexual Risk Data Collection Among Young Women in Soweto, South Africa. AIDS Behav [Internet]. 2018;(0123456789). Available from: pmid:29594618
  14. 14. Dietrich JJ, Coetzee J, Otwombe K, Hornschuh S, Mdanda S, Nkala B, et al. Adolescent-friendly technologies as potential adjuncts for health promotion. Health Educ. 2014;114(4):304–18.
  15. 15. Dietrich JJ, Laher F, Hornschuh S, Nkala B, Chimoyi L, Otwombe K, et al. Investigating Sociodemographic Factors and HIV Risk Behaviors Associated With Social Networking Among Adolescents in Soweto, South Africa: A Cross-Sectional Survey. JMIR Public Health Surveill. 2016;2(2):e154. pmid:27683173
  16. 16. UNICEF, Division of Communication, Social and Civic Media Section, Beger G, Sinha A. South African mobile generation Study on South African young people on mobiles [Internet]. New York; 2012. Available from: http://www.unicef.org/southafrica/SAF_resources_mobilegeneration.pdf
  17. 17. Al-Shorbaji N, Geissbuhler A. Establishing an evidence base for e-health: The proof is in the pudding. Vol. 90, Bulletin of the World Health Organization. 2012. pmid:22589560
  18. 18. Guerrero F, Lucar N, Garvich Claux M, Chiappe M, Perez-Lu J, Hindin MJ, et al. Developing an SMS text message intervention on sexual and reproductive health with adolescents and youth in Peru. Reprod Health. 2020 Jul 31;17(1). pmid:32736561
  19. 19. Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC. Scaling Up mHealth: Where Is the Evidence? PLoS Med. 2013;10(2):1–5. pmid:23424286
  20. 20. Sullivan PS, Driggers R, Stekler JD, Siegler A, Goldenberg T, McDougal SJ, et al. Usability and Acceptability of a Mobile Comprehensive HIV Prevention App for Men Who Have Sex With Men: A Pilot Study. JMIR Mhealth Uhealth. 2017;5(3):e26. pmid:28279949
  21. 21. Pai N, Esmail A, Saha Chaudhuri P, Oelofse S, Pretorius M, Marathe G, et al. Impact of a personalised, digital, HIV self-testing app-based program on linkages and new infections in the township populations of South Africa. BMJ Glob Health. 2021 Sep 2;6(9). pmid:34475026
  22. 22. Gous N, Fischer AE, Rhanath N, Phatsoane M, Majam M, Lalla-Edward ST. Evaluation of a mobile application to support HIV self-testing in Johannesburg, South Africa.
  23. 23. Songtaweesin WN, Kawichai S, Phanuphak N, Cressey TR, Wongharn P, Saisaengjan C, et al. Youth-friendly services and a mobile phone application to promote adherence to pre-exposure prophylaxis among adolescent men who have sex with men and transgender women at-risk for HIV in Thailand: a randomized control trial. 2020; Available from: http://onlinelibrary.wiley.com/doi/10.1002/jia2.25564/full
  24. 24. Raeesi A, Khajouei R, Ahmadian L. Evaluation of HIV/AIDS-related mobile health applications content using an evidence-based content rating tool. BMC Med Inform Decis Mak. 2021 Dec 1;21(1). pmid:33892691
  25. 25. Gutierrez MA, Moreno RA, Rebelo MS. Information and Communication Technologies and Global Health Challenges. Global Health Informatics: How Information Technology Can Change Our Lives in a Globalized World. 2017 Jan;50–93.
  26. 26. Schnall R, Bakken S, Rojas M, Travers J, Carballo-Dieguez A. mHealth Technology as a Persuasive Tool for Treatment, Care and Management of Persons Living with HIV. AIDS Behav. 2015 Jun;19 Suppl 2(0 2):81–9. pmid:25572830
  27. 27. Mulawa MI, LeGrand S, Hightow-Weidman LB. eHealth to Enhance Treatment Adherence among Youth Living with HIV. Curr HIV/AIDS Rep. 2018 Aug;15(4):336. pmid:29959649
  28. 28. Stats SA. Mid-year population estimates [Internet]. Pretoria, South Africa; 2019. Available from: https://www.statssa.gov.za/publications/P0302/P03022019.pdf
  29. 29. Creswell JW. Research Design: Qualitative, Quantitative and Mixed Methods Approaches [Internet]. 4th ed. London: Sage Publications Inc.; 2015. Available from: http://repositorio.unan.edu.ni/2986/1/5624.pdf
  30. 30. Preston DB, Starks MT, Cain RE. A discussion of chain referral as a method of sampling hard-to-reach populations. J Transcult Nurs. 2003;14(2):100–7. pmid:12772618
  31. 31. South African National Department of Health. NATIONAL HIV COUNSELLING AND TESTING POLICY GUIDELINES [Internet]. 2015 [cited 2023 Aug 20]. Available from: chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://www.nicd.ac.za/assets/files/HCT-Guidelines-2015.pdf
  32. 32. Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013 Sep;13(1):1–8. pmid:24047204
  33. 33. IBM. No Title [Internet]. [cited 2022 Oct 10]. Available from: https://www.ibm.com/support/pages/downloading-ibm-spss-statistics-25
  34. 34. Waugaman A. Implementing the Principles for Digital Development Perspectives and Recommendations from the Practitioner Community PRINCIPLES for Digital Development [Internet]. [cited 2024 Sep 16]. Available from: chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://www.healthdatacollaborative.org/fileadmin/uploads/hdc/Documents/Working_Groups/RHIS/A._Standards_and_Tools/3._RHIS_management/3b.__Information_and_communication_technology/3-_Digital_Principles_to_Practice.pdf
  35. 35. Dietrich JJ, Benadé GL, Mulaudzi M, Kagee A, Hornschuh S, Makhale LM, et al. “You Are on the Right Track With the App:” Qualitative Analysis of Mobile Phone Use and User Feedback Regarding Mobile Phone Sexual Risk Assessments for HIV Prevention Research. Front Digit Health. 2021 Mar 22;3. pmid:34713080
  36. 36. Warde F, Papadakos J, Papadakos T, Rodin D, Salhia M, Giuliani M. Plain language communication as a priority competency for medical professionals in a globalized world. Can Med Educ J. 2018;9(2).
  37. 37. Ibarra FP, Mehrad M, Di Mauro M, Peraza Godoy MF, Cruz EG, Nilforoushzadeh MA, et al. Impact of the COVID-19 pandemic on the sexual behavior of the population. The vision of the east and the west. International Braz J Urol. 2020;46(Suppl 1):104–12. pmid:32550703
  38. 38. Counted AV, Arawole JO. ‘We are connected, but constrained’: internet inequality and the challenges of millennials in Africa as actors in innovation. J Innov Entrep. 2015;5(1).
  39. 39. Swendeman D, Rotheram-Borus MJ, Comulada S, Weiss R, Ramos ME. Predictors of HIV-related stigma among young people living with HIV. Health Psychology. 2006;25(4). pmid:16846325
  40. 40. Harper GW, Lemos D, Hosek SG. Stigma Reduction in Adolescents and Young Adults Newly Diagnosed with HIV: Findings from the Project ACCEPT Intervention. AIDS Patient Care STDS. 2014;28(10):543. pmid:25216106
  41. 41. Rao D, Kekwaletswe TC, Hosek S, Martinez J, Rodriguez F. Stigma and social barriers to medication adherence with urban youth living with HIV. AIDS Care—Psychological and Socio-Medical Aspects of AIDS/HIV. 2007;19(1). pmid:17129855
  42. 42. Arora S, Yttri J, Nilsen W. Privacy and security in mobile health (mHealth) research. Alcohol Res. 2014;36(1). pmid:26259009
  43. 43. Marent B, Henwood F, Darking M, Leon A, West B, Lockhart D, et al. Development of an mHealth platform for HIV care: Gathering user perspectives through co-design workshops and interviews. JMIR Mhealth Uhealth. 2018;6(10). pmid:30339132
  44. 44. Catalani C, Philbrick W, Fraser H, Mechael P, Israelski DM. mHealth for HIV Treatment & Prevention: A Systematic Review of the Literature. Open AIDS J [Internet]. 2013;7(1):17–41. Available from: http://benthamopen.com/ABSTRACT/TOAIDJ-7-17
  45. 45. Jongbloed K, Parmar S, van der Kop M, Spittal PM, Lester RT. Recent Evidence for Emerging Digital Technologies to Support Global HIV Engagement in Care. Vol. 12, Current HIV/AIDS Reports. 2015. pmid:26454756