Improving information retrieval from electronic health records using dynamic and multi-collaborative filtering

Due to the rapid growth of information available about individual patients, most physicians suffer from information overload and inefficiencies when they review patient information in health information technology systems. In this paper, we present a novel hybrid dynamic and multi-collaborative filtering method to improve information retrieval from electronic health records. This method recommends relevant information from electronic health records to physicians during patient visits. It models information search dynamics using a Markov model. It also leverages the key idea of collaborative filtering, originating from Recommender Systems, for prioritizing information based on various similarities among physicians, patients and information items. We tested this new method using electronic health record data from the Indiana Network for Patient Care, a large, inter-organizational clinical data repository maintained by the Indiana Health Information Exchange. Our experimental results demonstrated that, for top-5 recommendations, our method was able to correctly predict the information in which physicians were interested in 46.7% of all test cases. For top-1 recommendations, the corresponding figure was 24.7%. In addition, the new method was 22.3% better than the conventional Markov model for top-1 recommendations.


Introduction
When we consider buying a book on Amazon's Website, we often benefit from items listed in a section called "Recommended for you."These recommendations, generated by a method called Collaborative Filtering (CF ) [1], suggest items of possible interest based on what other customers have viewed and purchased.Often, these suggestions are very useful and lead to additional purchases.However, when physicians search the electronic health records (EHRs) with regard to a particular patient problem, the EHRs do not make suggestions for potentially useful information.Instead, it requires physicians to go through the same manual, cumbersome and laborious process of searching for and retrieving information for similar patients/problems every single time.
In this manuscript, we present DmCF , a novel hybrid D ynamic and multi-C ollaborative F iltering method, for information recommendation when physicians search for information from patient EHRs.DmCF integrates the following two key ideas: • collaborative filtering, which prioritizes information items based on what similar physicians have searched for on similar patients; and • dynamic modeling, which foresees future information items of interest based on how physicians search for information items over time.Here, dynamics refers to the information retrieval patterns over time (e.g., in which order different information items are searched for; which information item will be typically searched for after a certain information item has been retrieved).Multi-collaborative filtering (mCF ) refers to that multiple types of similarities (e.g., physician similarities, patient similarities and information similarities) are integrated to score information items of possible interest.DmCF models information retrieval dynamics by a first-order Markov Chain (MC ), and combines MC transition probabilities (discussed in Section ) with mCF scores to produce final recommendation scores for future interested information items.DmCF recommends the information items with the highest scores to physicians.We tested DmCF on a real dataset from the Indiana Network for Patient Care (INPC).Our experimental results demonstrate 22.3% improvement from DmCF over MC models on top-1 recommendation (i.e., only the top recommended information item is considered), and for 46.7% of all the testing cases, DmCF is able to correctly identify information items that are truly interested by physicians among its top-5 recommendations.

Literature Review
The most relevant research to our work is from Recommender Systems, a research area that originated in computer science.In particular, top-N recommender systems, which recommend the top-N items that are most likely to be preferred or purchased by users, have been used in a variety of applications in e-commerce.The top-N recommendation methods can be broadly classified into two categories [1].The first category is neighborhood-based collaborative filtering methods [2], which leverage information from similar users and/or similar items to generate recommendations.The second category is model-based methods, particularly latent factor models which learn user and item latent factors and determine user preference over items using the factors.Recent recommendation methods also include deep learning based approaches [3], in which user preferences, item characteristics and user-item interactions can be learned in deep architectures.
Dynamic recommender systems have been developed to recommend information of interest over time.Popular techniques include latent factor transition approaches [4], and Markov models [5] that model the transitions among latent factors capturing information preference; state space approaches [6,7] that model the transitions across different states over time; point processes [8] and other statistical models [9] that learn probabilities of future events.
Recommendation methods have been recently used to recommend and prioritize healthcare information, due to the rapid growth of information available about individual patients and the tremendous need for personalized healthcare [10].Current applications of recommender systems in healthcare include recommending physicians to patients on specific diseases [11,12]; recommending drugs [13], medicine [14] and therapies [15]; and recommending nursing care plans [16], etc.In EHR systems, there is no measurement similar to numerical rating values in Amazon that can be used to quantitatively assess how much a physician is interested in a certain information item.In this case, we take a type of implicit feedback as a qualitative measurement.That is, if a physician searches for an information item from a patient's EHR data, the physician is considered as interested in that information item during the diagnostic process of the patient, and that information item is useful for/relevant to the diagnosis of the patient.Thus, to evaluate whether a physician is interested in an information item on a patient, we can check whether the physician searches for the information item from the patient's EHR data.Since search is typically done through submitting a search term, we use the two terms "search term" and "information item" exchangeably, and the problem becomes to recommend the next search term that a physician is interested in on a certain patient.

Terminologies, Definitions and Notations
In this manuscript, a physician is denoted as y , a patient is denoted as p , and a search term is denoted as t.A sequence of search terms that a physician y searches for August 13, 2020 3/18 on a certain patient p during a certain patient visit v is represented as where tv k is the k-th search term during visit v.Note that a physician may have multiple search sequences on a same patient during different visits.The physician who we recommend a next search term to on a patient is referred to as the target physician, and the corresponding patient is referred to as the target patient.A set of physicians/patients similar to the target physician y /target patient p is denoted as Sy( y )/Sp( p ), respectively.A set of search terms similar to a particular search term t is denoted as S t (t).The size of a set S is denoted as |S|.Additional notations will be introduced when they are used (e.g., in Section ).Table 1 presents the important notations that we use in this manuscript.

Overview of the Dynamic and Multi-Collaborative Filtering Method -DmCF
In this manuscript, we tackle the problem of recommending the next search term to a physician while the physician is searching for information about a patient.The key idea is to analyze search patterns in order to make recommendations for potentially useful, other information to the physician.To do so, we score and prioritize possible recommendations based on the following two criteria combinatorially: • which terms the physician has searched for on the patient already and • which terms similar physicians have searched for on similar patients.The first criterion considers the search dynamics under the assumption that the past behavior of physicians is a reasonable approximation for the standard of care [17,18], and their future behavior follows a same standard of care.Thus, future search terms can be inferred from previously searched terms and their orders.The second criterion considers patient similarities and physician similarities.The underlying intuition is that patients share commonalities and similar patients stimulate similar information retrieval patterns by physicians.Likewise, physicians share commonalities which result in similar search patterns on patients.We propose the hybrid method DmCF that considers search dynamics and multiple similarities for the next search term recommendation.DmCF consists two scoring components.The first component is designed to address search dynamics through a first-order Markov Chain [19].The score of a possible search term from this dynamics-based scoring component is denoted as Score DYN .The second component is to score search terms based on similarities via multi-collaborative filtering.The score of a possible search term from this similarity-based scoring component is denoted as Score CF .Thus, DmCF scores a next possible search term t for a physician y on a patient p after a sequence of searches T ( y , p , v) (Equation 1) as a linear combination of Score DYN and Score CF , that is, where α ∈ [0, 1] is a weighting parameter.In this manuscript, if a score is generated from a certain method X, a superscript X will be included on the score notation (e.g., Score X , Score X DYN or Score X CF ).In general, a superscript X indicates an associated method X.All possible terms are first scored using the scoring function in Equation 2. The top-scored terms are recommended as the next possible search terms.The first-order Markov Chain-based scoring and the multi-collaborative filtering-based scoring will be discussed in Section and Section , respectively.[19] represents a very fundamental dynamic modeling scheme based on the Markovian assumption.The Markovian assumption states that in a sequence of events (e 0 , e 1 , e 2 , • • • , e t−1 , e t ), each event is only dependent on a small set of previous consecutive events but independent of any earlier events.An MC models a sequence of events so that each of the events follows the Markovian assumption.The Markovian assumption is statistically represented as P (e t |e 0 , e 1 , e 2 , • • • , e t−1 ) = P (e t |e t−k , • • • , e t−2 , e t−1 ), where P (e t |E) is the probability of observing event e t given the previous event sequence E. The number of previous events that e t depends on (i.e., k in P (e t |e t−k , • • • , e t−2 , e t−1 )) defines the order of the MC .A special MC is first-order MC , in which each event only depends on its immediate precursor.MC has been demonstrated to be very effective in modeling, approximating and analyzing real-life sequence data [19].

First-Order Markov Chain-based Scoring -foMC
We use a first-order MC as the dynamic model to simulate the sequence of terms that a physician y searches for on a patient p during a visit.This method is referred to as f irst-order M arkov C hain, denoted as foMC .For a sequence where P (t|t v k ) is the transition probability from tv k to t in a first-order MC .The transition probability P (t j |t i ) from a term ti to another term tj in a first-order MC is calculated as the ratio of the total frequency of transitions from ti to tj over the total frequency of all transitions from ti to any terms, that is, is the frequency of the transitions from ti to tj in T ( y , p , v).Thus, Score foMC DYN as in Equation 3is not specific to a particular physician or patient, but corresponds to clinical practices that are summarized from all available physicians and patients.

Multi-Collaborative Filtering-based Scoring Background on Collaborative Filtering
Collaborative Filtering (CF ) is a popular technique in Recommender Systems [1] for recommending items to a target user.The fundamental idea of CF is that "similar users like similar items".User-based CF methods first identify similar users to the target user, and then recommend to the target user the items that are preferred by similar users.Item-based CF methods first identify items similar to the target user's preferred items, and then recommend to the target user such similar items.Thus, CF methods heavily depend on the calculation of user similarity and item similarity.A typical way to calculate user similarity is to represent each user using her preference profile over items, and calculate user similarity as the item preference profile similarity.Likewise, a typical way to calculate item similarity is to represent each item using its preference profiles across users, and calculate item similarity as the user preference profile similarity.The user similarity function and item similarity function in CF are often pre-defined, and thus the recommendations based on similarities can be easily interpreted.CF is particularly powerful when user and item data are sparse, which is often the case in real-life applications.CF is also well-known for its scalability on large-scale problems, particularly when the user similarity and item similarity can be calculated in parallel trivially.

Physician-Patient-Similarity-based CF Scoring -ypCF
We developed a CF method that generates search term recommendations from similar physicians and patients.This method first identifies similar physicians and similar patients (discussed in Section ) and then scores terms searched by similar physicians on similar patients (discussed in Section ).This method is referred to as phy sician-patient-similarity-based C ollaborative F iltering, and denoted as ypCF .

Identifying similar physicians and similar patients
We developed two approaches to identifying the set of similar physicians and the set of similar patients, depending on which set is identified first.
Patient-First Similarity Identification -simP2Y In the first approach, a set of patients similar to the target patient p is first identified, and then based on the similar patients, a set of physicians similar to the target physician y is then selected.This approach is denoted as simP2Y (i.e., from Patients to phY sicians).In simP2Y , the set of patients similar to the target patient p is represented as and is composed of the top-k p most similar patients to the target patient p (patient-patient similarity will be discussed later in Section ).Given S P2Y p ( p ), a set of physicians similar to the target physician y is represented as and selected as follows: first, physicians who have ever searched for same terms on p and on one or more patients in S P2Y p ( p ) are identified.From such physicians, the top-k y most similar physicians to y are selected into S P2Y y ( y | p ) (physician-physician similarity will be discussed later in Section ).
Physician-First Similarity Identification -simY2P The second approach is to first identify a set of physicians similar to the target physician y , and then based on August 13, 2020 6/18 the similar physicians, to identify a set of similar patients.This approach is denoted as simY2P (i.e., from phY sicians to Patients).In simY2P , the set of similar physicians is represented as and has the top-k y most similar physicians to y .Based on S Y2P y ( y ), a set of patients similar to the target patient p , denoted as is identified as patient p 's top-k p most similar patients on whom physicians in S Y2P y ( y ) have ever searched for same terms as on p .

Collaborative Filtering in ypCF
From Sy( y ) and Sp( p ) (either S P2Y has all the y i , p j , tk triplets such that physician y i ∈ Sy( y ) has searched for term tk for patient p j ∈ Sp( p ).Thus, for a sequence where f ( y , p , • ) = ) is the frequency of the triplet y ′ , p ′ , t (i.e., how many times y ′ searches for t on p ′ in total); f ( y , p , • ) is the average frequency of all possible terms that y searches for on p ; f ( y , p , • ) is the centered frequency for y , p , • (i.e., shifted by f ( y , p , • )) in order to reduce the bias from searches with different frequencies; and simy( y , y ′ ) and simp( p , p ′ ) are the similarity between y and y ′ , and the similarity between p and p ′ , respectively (discussed in Section ).The intuition behind the scoring scheme in Equation 9 is that the possibility that y searches for t on p after a sequence of searches is the aggregation of 1). the average possibility of y searching for arbitrary search terms (i.e., the first term in Equation 9), and 2). the possibility that similar physicians search for t on similar patients (i.e., the second term in Equation 9).
TptCF aggregates from all similar patients the transitions from the last search term in a sequence T ( y , p , v) (Equation 1) to another search term.Specifically, TptCF identifies a set of patients Sp( p ) similar to the target patient p and a set of terms S t (t v k ) similar to the last search term tv k in T ( y , p , v).The set S t (t v k ) contains the terms with term-term similarity (discussed in Section ) to tv k above a threshold β.Then TptCF looks into what physicians search for on patients in Sp( p ) after they searched for a similar term in S t (t v k ).The underlying assumption is that similar patients stimulate similar patterns of search sequences.Thus, the score Score TptCF CF of a next possible search term t is calculated as follows: where g(t ′ → t| p ′ ) is the frequency of transitions from term t ′ to term t for patient p ′ from all possible searches on p ′ , sim t (t v k , t ′ ) is the term-term similarity between tv k and t ′ (discussed in Section ).

Similarity Calculation
Physician-Physician Similarities -simy We first represent each physician y using a vector of search term frequencies, denoted as v.Each dimension of v corresponds to a term, and the value in each dimension of v is the total frequency that the corresponding term has been searched by y .Note that the frequency is aggregated from all the patients that y searches on.This representation scheme is very similar to the bag-of-word representation in text mining [20].Given the representation, the similarity between two physicians y and y ′ is calculated as the cosine similarity between vy and vy′, that is, simy( y , y ′ ) = cos(vy, vy′).
The intuition is that the search term distribution indicates physician specialties and expertise, and physicians of similar specialties and expertise are considered similar.
Patient-Patient Similarities -simp Similarly as for physicians, each patient is also represented using a vector of term frequencies, denoted as u.Each dimension of u corresponds to a term, and the value in each dimension of u is the total frequency of the corresponding term searched for by all physicians.The term distribution represents the health histories of the patient, and thus a reasonable patient representation.Given the representation, the similarity between two patients p and p ′ is calculated as the cosine similarity between up and up′ , that is, simp( p , p ′ ) = cos(up, up′). ( Term-Term Similarities -sim t Each term t is represented using a vector of patient frequencies, denoted as w.Each dimension in w corresponds to a patient, and the value in each dimension of w is the total frequency that term t is searched for by all physicians.The term-term similarity between terms t and t ′ is calculated as the cosine similarity between w t and w t ′ , that is, The underlying assumption is that if two terms are frequently searched for on a same patient, they are considered as similar in their medical meanings and relatedness.In this table, # p is the number of patients; # y is the number of physicians; #t is the number of terms; # T is the number of sequences; len( T ) is total length of sequences; len( T )/# p is average length of sequences per patient and len( T )/# T is average length of sequences.

Materials Data
The data we use for experiments come from the Indiana Network for Patient Care (INPC) 1 .The INPC is Indiana's major health information exchange, and offers physicians access to the most complete, cross-facility virtual electronic patient records in the nation.Implemented in the 1990s, the INPC collects data from over 140 Indiana hospitals, laboratories, long-term care facilities and imaging centers.We extracted the INPC search logs that were generated between 01/24/2013 to 09/24/2013.Table 3 presents the statistics of the INPC dataset.Figure 1 presents the distribution of sequence length in the dataset.It is notable that search sequences are typically very short (on average 2.89 search terms per each sequence).Figure 2 presents the distribution of the number of unique terms for each patient.On average, each patient has 3.85 unique search terms.The short sequences and small number of unique search terms per patient make the recommendation problem difficult, because the available data are very sparse.

Experimental Protocols and Evaluation Metric
We use the following experimental protocol to evaluate our methods on the INPC dataset: all the search sequences are split by the same cut-off time.Any searches before the cut-off time are in the training set, and any searches after the cut-off time are in the testing set.The models are trained using only training set, for example, the transition probabilities (Equation 4) are constructed only using the search sequences and terms in training set, and the various similarities (Equation 11, 12 and 13) are calculated only from the training set.This protocol is referred to as cut-off cross validation, denoted as CUTOFF.Figure 3 demonstrates the CUTOFF experimental protocol.
1 IRB Protocol # 1612682149 "Supporting information retrieval in the ED through collaborative filtering".We use the cut-off time 08/15/2013 (additional results for other cut-off times are available in the supplementary materials2 ).This cut-off time is selected because sufficient search terms from a majority of the search sequences are retained in training set before the cut-off time and meanwhile sufficient search sequences have testing terms after the cut-off time.After the split, the statistics for the training and testing data is presented in Table 3 (in "CUTOFF" rows).This CUTOFF setting is close to the realistic scenario, that is, all the data before a certain time should be used to predict information after that time.However, a shortcoming of CUTOFF is that many early search sequences may not have testing terms, and many late search sequences will not have anything in the training set.Sequences that do not have testing terms are still used to train models.Sequences that do not have training terms are not used.For those sequences which have terms after the cut-off time, the first one of the terms after the cut-off time will be used for evaluation.
The model performance is measured using Hit-Rate at N (HR@N ).For a sequence, a hit is defined as a recommended term that is truly the next search term.HR@N is the percentage of testing sequences that have a hit and the hit appears among the top-N recommended terms.Higher HR@N values indicate better performance.

Experimental Results and Discussions Overall Performance
We compare foMC , ypCF , TptCF and DmCF , as well as their variations, in our experiments.Table 4 presents the best performance of each method.Overall, DmCF -ypCF with simP2Y is the best method because 4 out of 5 results of DmCF -ypCF with simP2Y are the best among all the methods.With parameters α=0.2, |Sp|=1 (i.e., 1 similar patient) and |Sy|=1 (i.e., 1 similar physician), DmCF -ypCF with simP2Y outperforms the simple foMC at 22.3%, 20.2%, 26.0%, 16.7% and 18.1% on HR@1, HR@2, HR@3, HR@4 and HR@5, respectively.The second best method is ypCF with simP2Y because it has better results overall than the rest methods.With parameters |Sp|=1 and |Sy|=1, ypCF with simP2Y outperforms the simple foMC at 23.3%, 19.5%, 20.1%, 10.3% and 8.9% on HR@1, HR@2, HR@3, HR@4 and HR@5, respectively.It is notable that although ypCF is significantly better than foMC , the best DmCF -ypCF with simP2Y has a weight α=0.2 on the ypCF scoring component, but a weight 1-α=0.8 on the foMC scoring component.This indicates the importance of search dynamics in recommending the next search terms.It is also notable that the optimal DmCF -ypCF with simP2Y corresponds to a very small number of similar patients (Sp=1) and physicians (Sy=1).This demonstrates the effectiveness of DmCF -ypCF in identifying most relevant information and leveraging such information for term recommendation.The DmCF -TptCF method is also slightly better than foMC .With parameters α=0.1, |Sp|=1 and β=0.1, DmCF -TptCF outperforms foMC at -1.0%, 4.4%, 2.4%, 0.8% and 5.1% on HR@1, HR@2, HR@3, HR@4 and HR@5, respectively.However, DmCF -TptCF is significantly worse than DmCF -ypCF with simP2Y .The difference between DmCF -TptCF and DmCF -ypCF is that in DmCF -ypCF , the similarity-based scoring component (i.e., ypCF ) does not consider search dynamics and only looks at the search terms that have ever been searched by similar physicians on similar patients, regardless of how such search terms transit to the search term of interest, while TptCF considers such transitions.The performance difference between DmCF -TptCF and DmCF -ypCF may indicate that the transition information captured in TptCF might overlap with that captured in foMC and thus combining them together will not lead to substantial gains.On the other hand, the information captured by ypCF methods could be complementary to that in foMC and thus integration of ypCF and foMC results in significant performance improvement.In DmCF -ypCF , simP2Y is slightly better than simY2P .The simP2Y method first identifies patients similar to the target patient, and based on the identified similar patients identifies physicians similar to the target physician.The simY2P method identifies similar patients and similar physicians in the reversed order as in simP2Y .The better performance of simP2Y over simY2P in DmCF -ypCF demonstrates that when physician search dynamics has been considered via MC , similar patients should be identified first and then based on identified similar patients, similar physicians should be identified.This may be because that when MC already considers all patients and all physicians (Equation 4), a more focused and more homogeneous group of patients similar to the target patient is more critical in order to complement to the MC information.Since physicians may see many patients with different diseases, high physician similarity may be due to patients who are different from the target patient.If such physicians are first selected (e.g., in simY2P ), similar patients identified from these physicians might be very different from the target patient.However, when no information about all the patients and all the physicians is considered like in ypCF , a diverse set of physicians and patients might be beneficial, and that could explain why in ypCF , simY2P actually outperforms simP2Y slightly.
Comparing ypCF and TptCF , it is notable that ypCF is significantly better than TptCF , even though in TptCF more patients similar to the target patient are used to achieve its optimal performance.In TptCF , only terms from similar physicians and patients that are similar to the term of interest are considered in calculating the scores (Equation 10).However, in ypCF , all the terms from similar physicians and patients are used.The improved performance of ypCF compared to that of TptCF may indicate that using more possible terms could benefit recommendation.On the other hand, both foMC and TptCF consider term transitions, while TptCF considers term transitions only among similar terms on similar patients.The experimental results show that TptCF performs worse than foMC .This may indicate that if term transition is a major factor in determining next search term, transitions from more diverse patients should be integrated.
Figure 4, 5, 6, 7 and 8 present HR@1, HR@2, HR@3, HR@4 and HR@5 of DmCF -ypCF with simP2Y over different α values (Equation 2) when |Sy| = 1 and |Sp| = 1, respectively.As the weight α increases from 0, that is, as the CF takes place in the term scoring (Equation 2), the performance of DmCF in terms of HR@1 and HR@2 generally increases.This demonstrates the effect from CF scoring component in DmCF .As α further increases, the performance in general first gets better and then worse (except that the HR@1 performance reaches its best at α=1).This indicates that the dynamic scoring component and CF scoring component in DmCF play complementary roles for recommending terms, and thus considering their combination enables better recommendation performance than each of the two methods alone.7 and Table 8 present the best performance of all the methods for cut-off time 06/26/2013, 07/18/2013 and 09/03/2013, respectively.Overall, DmCF -ypCF achieves the best performance over the other methods on the different cut-off times.The trends among different methods as identified from cut-off time 08/15/2013 remain very similar for the other cut-off times.Note that as using later cut-off times, training data become more as shown in Table 5, and the performance of each method over different cut-off times tends to become worse.For example, the performance of foMC model decreases in general over August 13, 2020 13/18 different cut-off times.This may be due to the increasing heterogeneity among patients as more patients in the system.are less than or equal to 0.2.For simp, 2.65% of the patient-patient similarities are non-zero, and 77.05% of the non-zero similarities are less than or equal to 0.5.Specially, there are some patients whose similarities with one another are relatively high (i.e., the peaks in Figure 10 on larger simp values).This also explains the advantages of simP2Y over simY2P and their performance in Table 4, because more patients with higher simp to the target patient provide better opportunities for DmCF to identify relevant information from such similar patients.Figure 11 presents the distribution of non-zero term-term similarities (sim t ).For sim t , only 0.28% of term-term similarities are non-zero, and 78.36% of the non-zero similarities are less than or equal to 0.3.

Conclusions
In this manuscript, we presented our new dynamic and multi-collaborative filtering method DmCF to recommend search terms relevant to patients for physicians.DmCF   recommendations that are most relevant to the patients and that are most interested to physicians.
foMC calculates a dynamics-based score Score foMC DYN of a next possible search term t after tv k as the transition probability from tv

p
( p ) and S P2Y y ( y | p ), or S Y2P y ( y ) and S Y2P p ( p | y )), a set of physician-patient-term triplets, denoted as

Fig 2 .
Distribution of INPC # unique terms per patient

α
Fig 8. HR@5 over α values Overall Performance on Other Cut-off Times

Fig 9 .Figure 9
Figure9and 10 present the distribution of non-zero physician-physician similarities (simy) and patient-patient similarities (simp), respectively.For simy, 5.65% of physician-physician similarities are non-zero, and 80.98% of the non-zero similarities combines a dynamic first-order Markov chain model and a multi-collaborative filtering model in order to score and prioritize search terms.The collaborative filtering model leverages the key idea originating from Recommender Systems research, and uses patient similarities, physician similarities and term similarities to score potential search terms.The linear combination of the dynamic-based scoring and the August 13, 2020 15/18

Table 2
lists all the methods in the manuscript.

Table 3 .
Statistics of INPC Dataset

Table 4 .
Overall Performance Comparison with CUTOFF (08/15/2013) In this table, the column "sim" corresponds to similarity identification methods; α is the weight on CF component in DmCF ; |Sp| is the number of similar patients; |Sy | is the number of similar physicians; β is the similarity threshold to identify similar terms.The best performance of each method under each metric is bold.The best overall performance of all methods under each metric is underlined.

Table 5 .
Statistics of INPC DatasetIn this table, # p is the number of patients; # y is the number of physicians; #t is the number of terms; # T is the number of sequences; len( T ) is total length of sequences; len( T )/# p is average length of sequences per patient and len( T )/# T is average length of sequences.

Table 6 .
Overall Performance Comparison with CUTOFF 06/26/2013 In this table, the column "sim" corresponds to similarity identification methods; α is the weight on CF component in DmCF ; |Sp| is the number of similar patients; |Sy | is the number of similar physicians; β is the similarity threshold to identify similar terms.The best performance of each method under each metric is bold.The best overall performance of all methods under each metric is underlined.

Table 7 .
Overall Performance Comparison with CUTOFF 07/18/2013 In this table, the column "sim" corresponds to similarity identification methods; α is the weight on CF component in DmCF ; |Sp| is the number of similar patients; |Sy | is the number of similar physicians; β is the similarity threshold to identify similar terms.The best performance of each method under each metric is bold.The best overall performance of all methods under each metric is underlined.

Table 8 .
Overall Performance Comparison with CUTOFF 09/03/2013In this table, the column "sim" corresponds to similarity identification methods; α is the weight on CF component in DmCF ; |Sp| is the number of similar patients; |Sy | is the number of similar physicians; β is the similarity threshold to identify similar terms.The best performance of each method under each metric is bold.The best overall performance of all methods under each metric is underlined.