Study settings.

This study was conducted in two federally qualified health centers in Philadelphia (PA), serving areas with the highest asthma prevalence in the city. In the study, PCPs (primary care physicians or nurse practitioners) addressed erroneous asthma and medication beliefs in 80 patients (mean age 45; 83% female; 91% black/African American; 9% multiracial) with uncontrolled asthma using a standardized SDM-based intervention. The study was approved by Columbia University and the University of Pennsylvania’s institutional review boards. The full protocol has been reported elsewhere [21].

Inclusion criteria. Patients were included in the study if they met the following criteria: (a) had persistent asthma defined as PCP-diagnosed asthma requiring ICS; (b) had uncontrolled asthma based on the Asthma Control Questionnaire [22]; and (c) had erroneous asthma beliefs and/or ICS beliefs measured by the Conventional and Alternative Management for Asthma instrument [23].

Exclusion criteria. Patients were excluded if they were (a) non-English speakers or (b) had severe mental health conditions.

Study design.

In this group-randomized trial, ten providers were randomized into the active SDM intervention condition (n = 5) or a dose-matched attention control condition (n = 5). Providers then delivered the active or control condition to 80 adult patients meeting inclusion and exclusion criteria (40 patients in each group).

Intervention group. In the intervention group, PCPs were instructed to encourage patient ICS adherence using a 9-minute motivational interviewing approach that advances SDM using the following key principles: (a) raise the subject of non-adherence to the medication; (b) provide feedback; (c) enhance the patient’s motivation using motivational interviewing techniques, (d) and advise and negotiate treatment options with the patient. Motivational interviewing is a patient-centered counseling approach engaging patients in a collaborative partnership with clinicians to change patients’ behaviors [24].

Control group. PCPs delivered a 9-minute healthy lifestyle discussion in the control group. There were no restrictions on what providers could discuss during the visit in the control group.

Study outcomes.

The study used two validated tools to assess outcomes:

1. Patient-reported level of SDM: All patients completed the "SDM Questionnaire-9" (SDM-Q-9) [25]. SDM-Q-9 is a 9‐item, 6‐point Likert scale instrument (range: 1 completely disagree- 6 completely agree) to evaluate patients’ perspectives of the degree to which a PCP uses SDM during healthcare encounters. We selected this questionnaire because a recent systematic review led by our team found SDM-Q-9 to be superior in quality to other SDM instruments, especially when capturing SDM dimensions in chronic care visits [26]. Example questions are, "My doctor made clear that a decision needs to be made" and, "My doctor asked me which treatment option I prefer." Higher scores indicate greater levels of shared decision-making. The patients completed SDM-Q‐9 immediately after the patient participant-PCP encounter.
2. Medication Adherence Report Scale for Asthma (MARS-A) [27]: All patients completed the MARS-A to measure medication adherence. MARS-A is a 10-item, self-report measure of adherence with ICS. Example questions are "I only use it [ICS] when I need it" and "I try to avoid using it [ICS]." Each question is scored on a scale between 1 to 5 (range: 1 always- 5 never, summed and divided by 10 to give a range 1–5); a score of 4.5 or higher indicates greater adherence and correlates with objective measures of adherence as well as prescription refills. The patient completed the MARS-A four times: 1) Immediately before the patient participant-PCP encounter; 2) After one month; 3) After two months; and 4) After three months. The four measurements were implemented to understand longitudinal patient ICS adherence trajectories.

While the original study analysis is under review, see more details about the study protocol [21] and preliminary findings described elsewhere [28].

Phase 1: Processing the audio-recorded verbal communication of PCP-patient encounters.

All verbal communications during patient-PCP encounters were audio-recorded in both control and intervention groups. To process the audio-recorded patient-PCP encounters, we used the following steps:

Step 1: Voice-to-text transcription: We used the Amazon Web Service (AWS) Amazon Transcribe to transcribe all audio-recorded PCP-patient encounters (n = 80) to text. Amazon Transcribe is a speech-to-text Health Insurance Portability and Accountability Act of 1996 (HIPAA) compliant service that automatically creates text transcripts from audio files. Overall, our total recordings (both control and intervention groups) length was about 24 hours, with average encounter recording lasting about 18 minutes.

Step 2: Measuring the quality of automated transcription: To measure the quality of the transcript provided by the Amazon Transcribe, we used the following approach:

Creating a test sample: We evaluated the Quality of automated transcription of AWS by taking a random sample (2%, ~30 minutes out of 24 hours) of all the transcriptions. To create this random sample, we first stratified the transcription based on a) Speaker type, 50% recordings of PCPs, and 50% of patient participants. Amazon Transcribe includes a feature to label each fragment with the identified speaker. b) The average confidence score of transcription for each encounter. The Amazon Transcribe created the confidence score automatically (ranging from 1–100, with higher numbers indicating higher confidence scores) and showed how confident the system was with the transcription. The average Amazon Transcribe confidence score was .88 (minimum = .77, maximum = .98). We extracted 10 minutes of the evaluation sample from the transcribed segments with the lowest Amazon Transcribe confidence scores (range .77-.86); 10 additional minutes of the evaluation sample from the transcribed segment with medium Amazon Transcribe confidence scores (range .87-.9); and ten additional minutes of the evaluation sample from the transcribed segment with highest Amazon Transcribe confidence scores (range .91-.98).

Manual transcription of the test sample: To evaluate the quality of automated transcription by Amazon Transcribe, a member of our team transcribed all the recorded speech and compared them with the Amazon Transcribe’s transcription.

Step 3- Quantifying the transcription quality using the ROUGE technique: each evaluation segment was manually transcribed. Transcription quality was quantified using the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metric [29]. To evaluate the quality of automated transcription by Amazon Transcribe. To calculate the ROUGE score, human-transcribed segments are compared with speech-to-text generated text segments, producing scores reflective of transcription accuracy. We calculated ROUGE recall (how much of the human-transcribed text is captured by the automated speech-to-text transcription), ROUGE precision (how much of the automated speech-to-text transcription was accurate compared to human-transcribed text), and ROUGE F-score (harmonic mean between precision and recall).

Phase 2-Utilizing machine learning algorithms to evaluate the level of SDM and predict ICS adherence.

Study outcomes. For machine learning algorithms, we dichotomized the score of SDM-Q-9 into either a low-moderate level of SDM (average SDM score <5) or a high level of SDM (average SDM score > = 5) [25]. Because MARS-A scores collected at four different time points were consistent within patients (more than 80% internal consistency), we also created an additional average MARS-A score across the four-time points. We dichotomized the average MARS-A score into either low self-reported adherence (average MARS-A score <4.5) or high self-reported adherence (> = 4.5). Previous studies recommended this cut-off as it is correlated with higher prescription refill rates and objective measures (electronic monitoring) of ICS use [30].

Study dataset and data pre-processing. The study dataset included all the audio recorded of verbal communication of patient-PCP encounters that were transcribed by the Amazon Transcribe to text. Each recorded encounter was linked to the score of the SDM-Q-9 and the score of the MARS-A completed by the patients. To prepare the transcribed encounters for machine learning, we implemented several text pre-processing steps, including removing punctuation, converting all letters to lower-case, and removing numbers. We also used a stemming technique that reduces words to their roots, e,g. by removing suffixes such as "ing."

Features used in machine learning models. Overall, after pre-processing, the data included 11,348 features representing unique words and expressions that appeared in transcribed patient-PCP encounters. All data on these features were complete since they were represented as either present or absent in a specific patient-PCP encounter.

Machine learning algorithms. We implemented a series of machine learning algorithms to estimate the level of SDM and predict patients’ adherence to ICS. We applied three algorithms [31], including: 1. Naive Bayes- a probabilistic classifier based on the Bayes’ theorem that (naïvely) assumes independence between the predictor features; 2. Support Vector Machines- a supervised machine learning classifier that performs classification by finding a hyperplane that maximizes the margin between the two classes; and 3. J48 Decision Tree- a supervised classifier that uses a decision tree (as a predictive model) to go from features (represented in the branches) to classification conclusions (represented in the tree leaves).

The algorithms were implemented in Weka with default settings. More information about the algorithms and their operation mechanism for data classification is provided in S1 Appendix [31].

We also conducted feature selection, using information gain criteria [31], to reduce the sets of features to about 1,000 of the most informative features. These features were visualized using a word cloud technique where features with higher entropy scores (indicative of more informative features) were represented as words of larger sizes.

Measuring the predictive performance of the algorithms. Predictive performance of each machine learning algorithm was evaluated using standard metrics [31] of precision (defined as the number of true positives out of the total number of predicted positives), recall (defined as the number of true positives out of the actual number of positives) and F-score (defined as the weighted harmonic mean of the precision and recall). Because our sample was relatively small, we used 10-fold cross-validation to estimate the predictive performance. Cross-validation evaluates predictive algorithms by partitioning the original sample into a training set to train machine learning algorithms and a testing set to evaluate them [31]. In 10-fold cross-validation, this procedure is repeated ten times.

Sensitivity analysis. Finally, to account for the possible effect of the study intervention, we conducted sensitivity analysis by repeating each analysis for both experimental and control study groups. We compared algorithms’ predictive performances to explore whether results remained consistent regardless of the study group.

Further work.

In the next phase of this study, we plan to extract temporal vocal features (time-domain features), such as the energy of signal or maximum amplitude, and spectral features (frequency-based features), such as fundamental frequency and spectral centroid from the speech. These features may help improve the performance of machine learning algorithms in evaluating the level of the SDM during patient-PCP encounters and predicting ICS adherence. Further work should also explore the replicability of our results in larger data samples and additional health domains. Finally, in the near future, such machine learning approaches can be applied to create intelligent health assistants to help clinicians identify at-risk patients. Such intelligent assistants can also be applied to evaluate the quality of ongoing care and uncover which clinician verbal behaviors are associated with better patient-reported SDM.