Planning

The planning includes: (i) defining the research questions according to the objectives, (ii) using a Problem, Intervention, Comparison and Outcome (PICO) [31] approach to aid in the construction of the search strings and finally, (iii) creating a search string. According to the objective for this SMS, given that this research refers to proposals that aim to associate software development with SO, the research questions for this SMS are:

1. RQ1: What are the existing association strategies? It is important to identify how researchers attempt to solve the association between software development and SO, mapping their strategies and approaches, how they developed their solutions and what are the differences and common aspects of each solution.
2. RQ2: What are the input and output information? Software development has different contexts that can be used as input to search for SO information such as code, exceptions, tags and project tasks. At the same time, SO can retrieve different types of content such as posts, tags, user information, code chunks, comments, and questions or answers.
3. RQ3: What are the evaluation methods? Understanding the methodology used to evaluate the proposals is important, as researchers can use the same source of data to compare approaches, or even position their own dataset in regard to datasets other researchers have used.
4. RQ4: What are the reported metrics and results? Explicitly demonstrating metrics and results of each implemented approach, is besides a summary, a procedure to inform researchers on the current used metrics and methods.

To organize and structure the search string based on the objective and research questions, the PICO approach was used [31]. Table 1 presents a PICO description and goals.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. PICO [31] for SMS.

https://doi.org/10.1371/journal.pone.0243852.t001

According to the PICO definitions, the search string (and synonyms) is:

1. “stack overflow” OR “stackoverflow”
2. AND (issue OR task OR context)
3. AND (recommend* OR suggest* OR associat* OR link OR integrat*)
4. AND (“software“OR “software process“OR “develop*”

The scientific databases chosen were: Scopus, Web of Science, IEEE Xplore and ACM.

For the selection of articles, Inclusion and Exclusion criteria were established. These criteria support the decision about whether articles should be read or ignored. The selection criteria are outlined next.

• Inclusion Criteria:
  • Articles related to SO or communities where developers search for support; OR
  • Articles related to the association or recommendation or suggestion or link of software project artifacts; OR
  • Articles that address the use of software development information to find SO posts;

• Exclusion Criteria:
  • Articles not in the computer science area; OR
  • Articles whose research does not focus on Software Engineering; OR
  • Proposals that are not applied to software development; OR
  • Articles not written in the English language; OR
  • Articles that do not state clearly the association strategy, input, output, evaluation method and results between SO posts and software development; OR
  • Articles published before 2010.

The group of control articles [28] that should be retrieved by the search string is:

[10] PONZANELLI, L. et al. Mining StackOverflow to Turn the IDE into a Self-Confident Programming Prompter. Proceedings of the 11th Working Conference on mining software repositories, p. 102-111, May 31, 2014.

[12] PONZANELLI, L.; BACCHELLI, A.; LANZA, M. Seahawk: Stack Overflow in the IDE. Proceedings of the 2013 International Conference on software engineering, p. 1295-1298, May 18, 2013.

[20] CAMPOS, E.C.; SOUZA, L.B.L.; MAIA, M.D.A. Searching Crowd Knowledge to Recommend Solutions for API Usage Tasks. Journal of Software: Evolution and Process, Chichester, v. 28, n. 10, p. 863-892, Oct 2016.

[14] CORREA, D.; SUREKA, A. Integrating Issue Tracking Systems with Community-Based Question and Answering Websites. 2013 22nd Australian Software Engineering Conference, p. 88-96, 2013.

The papers were selected according to the following three steps:

• Step 1—Preliminary selection of publications: Execute the search string on databases to perform the preliminary retrieval of papers and use a reference manager to store retrievals;
• Step 2—Selection of relevant publications—1st filter: For an initial selection of relevant papers, the title and abstracts of each returned paper are read and assessed according to the inclusion and exclusion criteria. For some papers, reading only abstracts may not be sufficient to determine whether or not they should be included. For this reason, the papers are selected for full reading (step 3). Also, for databases that return many results (+200), we have ordered results by relevancy (relevancy selection) and applied abstract and title reading. After 2 pages with no new findings according to inclusion and exclusion criteria after title and abstract reading, we have stopped the search;
• Step 3—Selection of relevant publications—2nd filter: After removing duplicates, all papers selected in the 1st filter (step 2) will be fully read and analyzed to determine if they match the inclusion and exclusion criteria set. The selection of papers is final.

Execution of the search

According to the procedures for selecting papers defined in the planning of this SMS, the next step is to perform the execution of the search string in each of the selected sources. Table 2 presents the total number of papers retrieved from each of the databases.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Number of papers retrieved from each database.

https://doi.org/10.1371/journal.pone.0243852.t002

We have eliminated articles based on titles and abstracts (step 2), and full-text reading (step 3). Additionally, we have decided to add articles using backward snowball, as the retrieval of articles was conducted by the first author and this may pose a threat to the validity of the study (see further discussion in Section Threats to Validity). We have also decided to include papers for full-text reading when in doubt, also to mitigate the possible threats for this study. The second author has reviewed the planning and execution criteria of the SMS and search string testing.

To complement the automated search query, we have conducted a manual search to find articles that were not retrieved by the search query. Our search string did not locate one important source, namely the paper from ČubraniĆ et al. [22]. Since this paper is well-referenced in the literature, we included this publication in our list of research results through manual search. As a result of the manual search, to the best of our knowledge, we found that only this article should be included because it also addresses the general problem of using historical data to find software development artifacts. This paper was also identified by the snowballing process, but not included, as we were already considering this paper inclusion from manual search (identified as a duplicate study).

All articles retrieved by each of the selected databases (Step 1) were stored in a reference manager tool and for databases that returned many results, relevancy selection was executed (Step 2) before storing the articles. Duplicated articles were removed, resulting in 163 articles that had the title and abstract read (Step 2), and when applying inclusion and exclusion criteria, 50 articles resulted. After fully reading these articles (Step 3), twenty-five articles were finally selected, one article added through the manual technique. From snowballing, three more articles were added. These final 29 articles are presented in Table 4. Fig 1 presents the selection/elimination process with the number of articles of each step of the process in detail. In Table 3, we have created an identifier for the papers (ID) used to reference each paper.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Number of papers in each sub-step of selection process.

https://doi.org/10.1371/journal.pone.0243852.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Attributes extracted from selected papers.

https://doi.org/10.1371/journal.pone.0243852.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Systematic mapping study selected articles after Step 3.

https://doi.org/10.1371/journal.pone.0243852.t004

After reading each of the 29 retrieved papers, the attributes extracted are presented in Table 3. The attributes were created based on the research questions. Each field has an item as well as a description of the attribute. The attributes presented in Table 3 were created based on the SMS research questions. All articles presented in Table 4 after being fully read had their attributes (Table 3) extracted and presented in Table 1 in Section S1 Appendix. Systematic Mapping Study Resulting Catalog. The last row of this table presents the results of this proposal, which were later added to the table for comparison purposes, but the Systematic Mapping Study was conducted before the final implementation of the current study. The analysis and discussion of the extracted data are presented in Section Analysis. Note that some of the papers refer to the same study, therefore, they are listed in Table 4 and the reference study is presented in Table 1 in S1 Appendix. The last column of Table 1 in S1 Appendix. refers to the current proposal presented in this paper. These results were later added to this table for comparison purposes, but the Systematic Mapping Study was conducted before the final implementation of the current study.

During the snowballing, 22 new articles were retrieved from analyzing the chosen papers title’s references. After reading title and abstract, 9 papers were chosen to be fully read, and we ended up adding another 3 papers to the final selection. The papers are identified in Table 4 and Table 1 in S1 Appendix. by the sufix SB.

Analysis

The analysis of the information extracted from the papers gathered during the procedure is presented in this section. For this paper, a thematic rationale [28] to discuss the details of the papers is proposed. The authors of this paper were responsible for extracting the information from each retrieved paper and based on the information extracted from the papers, it was possible to answer the research questions of this SMS. Each research question is referenced in Table 1 in S1 Appendix and the next sub-sections provide further detail on each of these research questions.

Threats to validity

We have identified as internal threats in this systematic mapping study that human performance might affect the selection of papers. To mitigate this threat, we used protocols and systematic mapping study established guidelines [28] to enable search replication and minimize bias. As external threats, we identified the use of few research databases can lead to missing studies; we resolve this threat by using four important databases in the Computer Science field. For conclusion threats, we identified that the interpretation of data (selected papers) is a concern once bias can be introduced and only one researcher chooses the papers. The search was executed three times (January 2018, September 2018 and May 2020) aiming at mitigating this threat. Additionally, we have implemented snowballing and the inclusion of papers for full-reading when in doubt and the second author has reviewed in detail the protocol planning and execution of the SMS, both being guidelines suggested by established literature [28].

Overview

Curation in this situation is the act of searching and selecting useful SO posts for a problem encountered during a software development project. The process of curating SO posts is described by the steps enumerated later in this section. Each of these steps can be executed repeatedly until developers are satisfied with the results listed and have chosen an SO post that meets their needs. Once a solution is chosen, curation is over. If results are not helpful, the developer changes the string, trying to use new terms that might provide useful results. The curation steps are represented as a Business Process Model and Notation (BPMN) model in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. BPMN model representing the curation process.

https://doi.org/10.1371/journal.pone.0243852.g002

1. Create search string: A developer has a problem and creates a search string that may retrieve satisfactory results;
2. Submit search string: The search string is submitted to SO;
3. Search and list results: SO executes the search according to internal algorithms and lists the results;
4. Select results: The developer selects one (or a set of) SO post that might be helpful.

Frequently, after a solution from an SO post is selected, the post containing the solution is not associated with the project task and, therefore, cannot be reused by developers dealing with similar or recurring tasks. A growing body of literature recognizes the importance of associating external knowledge (information developers look for on the web) with the related software development task [10–12, 14, 58]. SO is the source of support considered in this paper, and this support is presented in the form of curated SO posts. Through the identification of similar project tasks, SO posts that were once used to support a project task during its solution could be automatically associated with project tasks with similar contexts. Through the identification of similar project tasks, SO posts that were once used to support a project task during its solution could be automatically associated with project tasks with similar contexts.

As for the mechanism that will allow reuse, we propose to find similar project tasks, in regards to their context, and reuse the same SO post on similar new tasks. Our study aims at using project task information and a text similarity to aid SO Post curation and reuse during software development. SO posts that were once used to support a task during the task solution could be automatically linked to a similar task that is encountered. Fig 3 illustrates this process. The processing functionality receives project task contexts as input, submitting these tasks to a similarity retrieval engine and, by discovering the similarity between pairs of tasks, links the SO Post of a project task to the most similar project task identified by the similarity retrieval process.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Proposed similarity approach overview.

https://doi.org/10.1371/journal.pone.0243852.g003

In Fig 3, there are four tasks in the Project (1). Each project task is composed of information, which we refer to as “Context” (2). Three of these tasks (task1, task2 and task3) have a status of “resolved”, represented by the white color of the “Task Status” caption (3), and have SO Posts (4) associated with them. One task is not resolved (task4), nor does this task have an associated SO Post. When all the tasks are submitted to a similarity retrieval algorithm (5), the similarities between the project tasks are retrieved (6). These similarities are expressed in a range of 0 to 1. In the example, when comparing task4 to task1, the similarity index between them is 0.6, meaning these tasks are 60% similar. When comparing task4 to task2, the retrieved similarity is 0.7, meaning these tasks are 70% similar. Finally, when comparing task4 to task3, the result produced was a 30% similarity between these two tasks. Based on the proposed approach, the most similar tasks should share the same SO Post (7). Thus, task4 should be related to the SO Post associated with task1 and task2, as they are the tasks most similar to task4 based on the similarity index used in the illustration.

Developers perform the work of associating SO posts to project tasks (4), after they have curated an SO post. Therefore, part of the solution is a developer’s responsibility. The suggested associations created automatically by the similarity retrieval execution are represented in Fig 3 by marker 7. This proposed approach does not consider a direct association of the project tasks with SO. The curation effort is the process that associates the project task to the curated SO post, highlighting the need for developer to be involved in the curation effort.

Developers perform the work of associating Stack Overflow posts to project tasks (4), after they have curated this SO post. Therefore, part of the solution is one developer’s responsibility. The association’s suggestions created automatically by the similarity retrieval execution are represented in Fig 3 by marker 7. This proposed approach does not consider a direct association of the project tasks with Stack Overflow. The curation effort is what associates the project task to the curated Stack Overflow post, highlighting the need for developer effort for curation.

After a preliminary investigation [18], this enquiry was developed further. In this preliminary investigation, we have performed a qualitative evaluation, asking software developers if similar tasks found among 4000 project tasks could reuse the same Stack Overflow posts. Results c 30% of the possibility of reuse pointed out by the developers that participated in the study. As of the current research, we investigated the elements of the context of a project task and outstanding results are presented in Section Project Task Context. Then, we implemented a process using this project-task context using a data science platform (RapidMiner—rapidminer.com). The results of this research are in Section Similarity Model Implementation. Section Evaluation evaluates and further discusses the similarities retrieved by this implementation.

Current and foreseen situations

This Section provides details about the current situation related to associating SO posts with software development tasks and the key additions provided by the research described in this paper. Currently, developers search for an SO post, use the information found (if useful), and go on to the next task in the project. The developers do not record the SO post, how they found it, or the work they had to employ to transform the information obtained on SO into useful information for the project.

We argue that the knowledge supplied by the SO post could be valuable in the project in future, but because the developer does not associate the information of the useful SO post with the project, this knowledge is lost. If the same developer or another member of the team is working on a similar task, this person will have to look for the same SO post again, and the curation effort is duplicated. Our proposal aims to reuse the curation required from the first developer. This first search for SO posts (curation) will have to be executed. But once executed, and associated with the project task that required the extra support, the curation effort can be minimized or even avoided.

There are still two aspects of reuse that have to be considered: the association of the SO post with the project task and the mechanism that will allow the reuse of the curation. First, the association is solved by the first developer, who performed the curation, manually providing information to the tool that manages the project tasks and indicating, which SO post supported that specific task. Recording such information would require tool customization. For example, this association can be performed by creating a customized field used by the project management tool to store SO posts used. To illustrate Fig 4 shows an example of a project task of the Redmine open source project. This project task has as context: a title, description and other information regarding the project task. If the developer that solves this task needs SO support, the developer could describe which SO post was used in a customized field, as presented in Fig 5. Note that this field does not exist in Redmine, we are proposing this illustration for clarification purposes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Redmine project task without customized field.

https://doi.org/10.1371/journal.pone.0243852.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Redmine project task with SO customized field—proposal.

https://doi.org/10.1371/journal.pone.0243852.g005

Our proposal does not solve the problem of infrastructure, or tool customization. Our aim is to study if the information of project tasks is enough and effective to reuse the curation of SO posts. The next Section presents the study performed regarding project tasks and their contexts.

Project task context

We performed an investigation of the elements that might be related to project tasks, considering that such elements will be used by the implemented model to retrieve similarities between project tasks. Project Task Context is a set of elements that compose a project task. Project tasks are the project assets used in this study since the suggested association of a curated post to a project task will be based on the similarity of the project task. Investigating possible project task context elements is essential to help clarify what information from project tasks are available for similarity comparisons. In addition, this investigation is vital to avoid research bias. For completeness, we have taken advantage of both academic and industrial perspectives, including project management tools that support project tasks in software development.

We adopted two sources of information to determine the information present in a project task context. The first source was an ad-hoc literature review and the second was the information used to specify tasks in popular project management tools. From the literature, we found that software engineering is knowledge-intensive because of its dynamic nature and the amount of technology often encountered [1, 59]. According to Lindvall and colleagues [59], software engineering has both technical and business domain knowledge associated with the project. Technical knowledge refers to design (design patterns, heuristics, best practices, technical constraints, and estimation models), programming (programming languages and development tools) and software processes (methodology, code testing, and debugging procedures). Business domain knowledge refers to information regarding aspects of a specific application (business processes, business rules, activities, stakeholder needs, and business goals). This work does not consider business domain information; it only considers technical information because it aims to be agnostic to business characteristics. We propose a context element that captures technological information about project tasks such as a tag. A tag is a piece of information related to an element. In this case, elements are any technical information directly related to the task that can characterize the task. Each project has a specific context with respect to a product being developed [60], such as technical characteristics that every task will inherit necessarily, indicating the need for a project tag.

It is important to consider research sources other than just the formal literature in software engineering [61], to ensure that practical insights are brought to the activity. Therefore, we analyzed project management tools, by verifying the default elements each tool has for tasks. In this ad-hoc analysis, we concluded that some of the project task context elements identified in the literature were also identified in software development tools that support project workflows. The analyzed tools were JIRA (atlassian.com/Jira), Trello (trello.com) and Redmine (redmine.org), which are broadly used. After performing an analysis of both literature and project management tools contents, we present a list in Table 5 of the context elements that were identified.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Context elements.

https://doi.org/10.1371/journal.pone.0243852.t005

Similarity model implementation

After project task context elements were investigated in both the literature and project management tools, we proposed a method to find the similarities between project task context elements. The method comprises a process model that receives as input a dataset containing project tasks associated with SO posts, and retrieves similarity indices from pairs of project tasks. These indices evaluate if the SO posts are the same (an indication of potential reuse) between project tasks with a high degree of similarity. RapidMiner (RapidMiner Studio version 8.2) was used to implement a similarity process model. RapidMiner is a data science platform, requiring a small learning curve and widely adopted in the academic field [62].

RapidMiner implements a vast number of algorithms to extract text similarities. Each step of the RapidMiner process model is represented as an operator. In this sub-section, we present the implementation of the process model. The evaluation of the model is performed in Section Evaluation. The implemented RapidMiner process model is illustrated in Fig 6 and some details of the implemented model are described next.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. RapidMiner process model.

https://doi.org/10.1371/journal.pone.0243852.g006

In Fig 6, a straight line connects the sequential operators. Each operator runs a unique procedure, and the result is provided to the next operator. Operators have semicircles that are ports for inputs or outputs. Each operator has specific functionality. The Retrieve Operator imports a dataset. The Select Attributes Operator selects attributes from the dataset that are input to the similarity algorithm, to enable choosing a subset of elements from the dataset. The Process Documents from Data Operator is a sub-process responsible for the text pre-processing transformations. The transformations executed are the transformation of characters to lowercase (Transform Cases), the removal of every character that is not an alphanumeric character (Tokenize), stop-words filtering (Filter Stopwords) and finally, the transformation of inflected words into a base or root form of the word (Stem). This sub-process is presented in Fig 7.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Process document from data operator sub-process.

https://doi.org/10.1371/journal.pone.0243852.g007

This operator’s output is connected to the next operator’s input port, the Set Role Operator which is responsible for identifying the dependent variable. This dependent variable is needed by the Data to Similarity Data Operator, which is responsible for retrieving the text similarities, according to the algorithm chosen. The Data to Similarity Data operator is configured to run the Jaccard algorithm [63] at first, to retrieve text similarities, an algorithm that is widely used in this type of textual problem [64]. But there are other textual similarities that can be implemented in the model. Therefore, we have also used and we demonstrate results with different similarity algorithms. The similarities retrieved as a result of the execution of similarity algorithms have a range from 0 to 1 and can be interpreted in percentages (0 to 100%). The Set Role (2) Operator identifies which values of the dataset will be chosen and which will be the base for comparison. The Filter Examples Operator sets a threshold of similarities. We defined a threshold of 50% similarity (similarity index ≥0.5). The threshold of 50% is chosen as this threshold is the mean between the minimum and maximum possible values, 0% and 100%, respectively. This threshold is not fixed, and can be changed depending on domain or design of application, or characteristics of the dataset used. It is defined as a way to eliminate false positives. The Performance Operator is used for statistical evaluation of classification tasks, and is configured to retrieve precision and accuracy metrics. Regarding the Jaccard similarity algorithm, a statistic metric used to compare finite sets of data is used to compare the similarity and diversity of sample sets and uses the ratio of the intersecting set to the union set as the measure of similarity. Thus, it equals to zero if there are no intersecting elements and equals to one if all elements intersect. An example of the equation of Jaccard similarity is presented below in the following equation.

Using documents as examples, supposing there are the three following documents:

• Doc1: “Word2”, “Word3”, “Word4”, “Word2”.
• Doc2: “Word1”, “Word5”, “Word4”, “Word2”.
• Doc3: “Word1”

Comparing Doc1 and Doc2, as the total number of unique words of both documents is 5, and the number of shared words between the documents is 2, gives a Jaccard similarity of 2/5 = 0.4. Comparing Doc1 and Doc1 gives a Jaccard similarity of 1. Comparing Doc1 and Doc3, the result is 0, because they share no similarities between the documents. As stated by Tan et al. [63], this is the most often used metric for document comparison.

Executing the implemented process model

To execute the process model, a dataset with project tasks has to be loaded into the RapidMiner process model implementation. The selected dataset was gathered from a company in Brazil that has been developing software products for more than 20 years and has 30 employees. The software development projects in this company follow agile guidelines, and the project tasks are managed with the support of a project management tool. We were able to gather 25 project tasks with associated SO posts for each of the 25 tasks. The software developers and managers of the company provided a spreadsheet with all information needed from tasks (context elements and SO posts associated with each task). The tasks required for this study should have two mandatory requirements: (i) all of the tasks should have at least one associated SO post and (ii) different tasks should share the same SO post. Our goal is to evaluate if different tasks having context similarities could share the same SO post. Tags were added to the dataset by the developers at the company. The company did not have tags previously associated with each project task but were able to identify tags for both the project and each project task for this study. With this dataset as input in the process model, we were able to execute the process model in RapidMiner. The dataset was composed by the following information regarding project tasks: Project, Category, Iteraction, Title, Description, ProcessActivity, Project Tags, Task Tags and Stack Overflow Post. For a description of each of these elements, please refer to Table 5. The complete dataset and implementation can be found on github.com/glauciams/task2stackRapidMiner.

Results and discussion

After executing the evaluation of the dataset, precision and accuracy are calculated, firstly using the Jaccard algorithm. Other similarity algorithms are also included and demonstrated in Table 8. A matrix is generated through the execution of the RapidMiner process model, which supports the extracted results.

Answering RQ1, the accuracy for the given dataset with the elements identified in Section Knowledge Reuse in Software Projects is 77.78%, and the precision mean 71.60%, when the threshold is set to 50% similarity. Table 6 presents the results reported in the Related Work section. The results presented are the highest scores reported in each of the related works, and the last row presents the findings from our current work. Depicting what these results represent, we argue that these numbers indicate a broader perspective of our investigation, namely the ability to reuse knowledge through the identification of project task similarities. Curated Stack Overflow posts represent the tacit knowledge captured and reused in this proposal. For information purposes, we also disclose the results setting the algorithm similarity threshold to 0.3 (30%) and 0.7(70%) similarity. If the algorithm is set with a 0.3 threshold, the results are 40.87% for precision and 36.57% for accuracy. When set to 0.7, precision is 91.67% and accuracy is 94.44%.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. RQ1 results and comparison with related works.

https://doi.org/10.1371/journal.pone.0243852.t006

Answering RQ2, Table 7 presents the context attributes’ selected combination, the precision and accuracy extracted for each combination, and changes made in each combination, as it is not easy to perceive from the attribute list which attributes were selected and which were not in the combination. To facilitate the visualization of the information, we also provide a graph with the results, presented in Fig 8.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. RQ2 results: Context combinations in a graph.

https://doi.org/10.1371/journal.pone.0243852.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. RQ2 results: Context combinations.

https://doi.org/10.1371/journal.pone.0243852.t007

When considering the project task context elements initially identified in Section Knowledge Reuse in Software Projects, precision and accuracy are the highest among all combinations. When considering the Iteration context, which appeared in the dataset sample gathered in industry, precision and accuracy are the lowest, indicating that either the Iteration context element hampers SO posts reuse, or in the dataset sample, the information was not properly written. The same is perceived when removing other elements of the context, such as Process, ProjectTags and TaskTags, and Title and Description. These results also indicate that the hypothesis of the evaluation Project task similarity can provide useful suggestions of curated SO posts that are correct. There are indications that project task similarity can provide accurate associations between project tasks and curated SO posts, as the prediction and accuracy are as high as 70% and higher than most of the results reported in the related work. As well, the model built in RapidMiner can be used with other datasets with similar characteristics, which indicates the study is replicable with other data.

We have also run our model with different similarity algorithms, for comparison with Jaccard. The algorithms and results are demonstrated in Table 8. The threshold set for demonstration is 50%. It is not our goal to find the best algorithm or to study different algorithms in this study. Nevertheless, we decided to demonstrate the same model and dataset with different algorithms, to enrich our study and provide the means to compare Jaccard, one of the most used text similarities in our related work, to other available nominal and numeral distances and similarity algorithms. For the numeral distances (Euclidean, Cosine and Manhattan), because there is no encoding, the title and description of the dataset are not considered, and we believe this is the reason why the values are low. Regarding the dataset sample characteristics, a higher number of tasks or different datasets might lead to a more robust general model to predict which SO Post to reuse when there are similar project tasks identified. Moreover, there has also been pointed out in the literature that researchers software development context that “… Although many research tools have been proposed that use historical information” (which is our case in this study, we use historical information) “…, few tools are available to practicing developers.” [66]. Without tools that actually gather context from historical information, it is less likely there are available datasets for this matter. In conclusion, the sample size in the dataset is smaller than typically desired, yet the value of using real data seemed to outweigh the benefits of generating a simulated dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Comparison of dataset with different similarity/distance algorithms.

https://doi.org/10.1371/journal.pone.0243852.t008

With respect to the application of the proposed approach in other domains, we argue knowledge bases can be a useful asset in diverse domains such as language studies, health or mathematics. We believe these and other domains could use similar approaches to discover and reuse knowledge through automated tools and methods. Although the approach presented in our paper focuses on software development, the general principles of the proposed approach could be applied to knowledge bases in many additional domains. Of course, new approaches would require data from the experts using these knowledge bases. For example, in the case of health, physicians could take advantage of specific health information already curated by other experts that have used those knowledge bases. The approach could use clinical guideline tasks instead of software development tasks, and instead of Stack Overflow, the approach in health care could use a medical knowledge base Q&A website such as medhelp.org. The proposed approach could also be applied to knowledge bases in domains other than health such as mathematics (math.stackexchange.com) and the law (avvo.com).

According to the model proposed by Wohlin et al. [67], the internal threats to the validity of this evaluation are the small sample size, which can produce distortions in the expected result and conclusions. A strategy to mitigate this problem can include the use of more extensive samples. The lack of another sample and the small size of the dataset can also be characterized as external threats because it can impact the generalization of the study results. However, the selected dataset does not involve only one project, but considers multiple projects (five) of the company in an extended period of time (around 7 years). These multi-purpose projects are: (1) a legacy project with maintenance tasks, (2) a project that is new and uses modern programming languages for front-end, back-end and database—Ruby on Rails, NodeJS and MongoDB, (3) a main project in the company, that had in 2018 been in production for over 8 years and lastly, (4) a project created to coordinate the migration of the application server, and lastly, (5) a project that the system architects of the company managed their backlog. We consider this variability in the purpose of projects to be valuable to mitigate the reduced dataset threat. Moreover, although the sample size in the dataset is smaller than typically aspired, yet the importance of utilizing real data seemed to excel in the advantages of designing a simulated dataset. In the future, we plan to mitigate this threat of external validity by experimenting using a larger dataset. As construction threats, we can cite the selection of methods and definitions of measures, which were performed based on the related work, but gave no indication of being representative of the situation. Finally, the fact that the same developers can be involved in the study might indicate conclusion threats, given the possibility the text of project tasks can be standardized, in case they were written by the same person.