Ten simple rules for making research software more robust

Software produced for research, published and otherwise, suffers from a number of common problems that make it difficult or impossible to run outside the original institution or even off the primary developer’s computer. We present ten simple rules to make such software robust enough to be run by anyone, anywhere, and thereby delight your users and collaborators.


Introduction
Scientific software is typically developed and used by a single person, usually a graduate student or postdoc [1].It may produce the intended results in their hands, but what happens when someone else wants to run it?Everyone with a few years of experience feels a bit nervous when told to use another person's code to analyze their data: it will often be undocumented, work in unexpected ways (if it works at all), rely on nonexistent paths or resources, be tuned for a single dataset, or simply be an older version than was used in published papers.The potential new user is then faced with two unpalatable options: hack the existing code to make it work, or start over.
Being unable to replicate results is so common that one publication refers to it as "a rite of passage" [2].The root cause of this problem is that most research software is essentially a prototype, and therefore is not robust.The lack of robustness in published, distributed software leads to duplicated efforts with little practical benefit, which slows the pace of research [3,4].Bioinformatics software repositories [5,6] catalogue dozens to hundreds of tools that perform similar tasks: for example, in 2016 the Bioinformatics Links Directory included 84 different multiple sequence aligners, 141 tools to analyze transcript expression, and 182 pathway and interaction resources.Some of these tools are legitimate efforts to improve the state-of-the-art, but often they are difficult to install and run [7,8], and are effectively abandoned after publication [9].This problem is not unique to bioinformatics, or even to computing [2].Best practices in software engineering specifically aim to increase software robustness.However, most bioinformaticians learn what they know about software development on the job or otherwise informally [1,10].Existing training programs and initiatives rarely have the time to cover software engineering in depth, especially since the field is so broad and developing so rapidly [4,10].In addition, making software robust is not directly rewarded in science, and funding is difficult to come by [1].Some proposed solutions to this problem include restructuring educational programs, hiring dedicated software engineers [4,11], partnering with private sector or grassroots organizations [1,5], or using specific technical tools like containerization or cloud computing [12,13].Each of these requires time and, in some cases, institutional change.
The good news is, you don't need to be a professionally-trained programmer to write robust software.In fact, some of the best, most reliable pieces of software in many scientific communities are written by researchers [3,11] who have adopted strong software engineering approaches, have high standards of reproducibility, use good testing practices, and foster strong user bases through constantly evolving, clearly documented, useful, and useable software.In the bioinformatics community, Bioconductor and Galaxy follow this path [12,14].Not all scientific software needs to be robust [15], but if you publish a paper about your software, it should, at minimum, satisfy these rules.
So what is "robust" software?We implied above that it is software that works for people other than the original author, on machines other than its creator's.More specifically, we mean that: • it can be installed on more than one computer with relative ease, • it works consistently as advertised, and • it can be integrated with other tools.
Our rules are generic and can be applied to all languages, libraries, packages, documentation styles, and operating systems for both closed-source and open-source software.They are also necessary steps toward making computational research replicable and reproducible: after all, if your tools and libraries cannot be run by others, they cannot be used to verify your results or as a stepping stone for future work [16].
Version control is essential to sustainable software development [17,18].In particular, developers will struggle to understand what they have actually built, what it actually does, and what they have actually released without some mechanical way to keep track of changes.They should therefore put everything created manually into version control as soon as it is created, including programs, original field observations, and the source files for papers.Files that can be regenerated at need, such as the binaries for compiled programs or intermediate files generated during data analysis, should not be versioned; instead, it is often more sensible to use an archiving system for them, and store the metadata describing their contents in version control instead [19].
If you are new to version control, it is simplest to treat it as "a better Dropbox1 " (or, if you are of a certain age, a better FTP), and to use it simply to synchronize files between multiple developers and machines [20].Once you are comfortable working that way, you should use a feature branch workflow: designate one parallel copy (or "branch") of the repository as the master, and create a new branch from it each time you want to fix a bug or add a new feature.This allows work on independent changes to proceed in isolation; once the work has been completed and tested, it can be merged into the master branch for release.
2 Rule 2: Document your code and usage How to write high quality documentation has been described elsewhere [21] and so here we only cover two minimal types: the README and usage.The README is usually available even before the software is installed, exists to get a new user started, and points them towards more help.Usage is a terse, informative command-line help message that guides the user in the correct use of the software.
Numerous guidelines exist on how to write a good README file [22,23].At a minimum, your README should: 1. Explain what the software does.There's nothing more frustrating than downloading and installing something only to find out that it doesn't do what you thought it did.
2. List required dependencies.We address dependencies in more detail in Rule 5.

Provide compilation or installation instructions.
4. List all input and output files, even those considered self-explanatory.Link to specifications for standard formats and list the required fields and acceptable values in other files.If there is no rigorous format, explain what the sections mean.These files are often full of valuable information that can be mined for the user's specific purpose.
5. List a few example commands to get a user started quickly.
6. State attributions and licensing.Attributions are how you credit your contributors; licenses dictate how others may use and need to credit your work.
The program should also print usage information when launching from the command line.Usage provides the first line of help for both new and experienced users.Terseness is important: usage that extends for multiple screens is difficult to read or refer to on the fly.
Almost all command-line applications use a combination of POSIX [24] and GNU [23] standards for usage.More standard command-line behaviours are detailed in [8].Your software's usage should: 1. Describe the syntax for running the program, including the name of the program, the relative location of optional and required flags, other arguments, and values for execution.
2. Give a short description to remind users of the software's primary function.

3.
List the most commonly used arguments, a description of each, and the default values.

4.
State where to find more information.
Usage should be printed to standard output so that it can be combined with other bash utilities like grep, and it should finish with an appropiate exit code.Documentation beyond the README and usage is up to the developer's discretion.We think it is very important for developers to document their work, but our experience is that people are unlikely do it during normal development However, it is worth noting that software that is widely used and contributed to has and enforces the need for good documentation [14].
3 Rule 3: Make common operations easy to control.
Being able to change parameters on the fly to determine if and how they change the results is important as your software gains more users, as it facilitates exploratory analysis and parameter sweeping.Programs should therefore allow the most commonly changed parameters to be configured from the command line.
Users will want to change some values more often than others.Since parameters are software-specific, the appropriate 'tunable' ones cannot be detailed here, but a short list includes input and reference files and directories, output files and directories, filtering parameters, random number generation seeds, and alternatives such as compressing results, use a variant algorithm, or verbose output.
Check that all input values are in a reasonable range at startup.Few things are as annoying as having a program announce after running for two hours that it isn't going to save its results because the requested directory doesn't exist.
To make programs even easier to use, choose reasonable defaults where they exist and set no defaults at all when there aren't any reasonable ones.You can set reasonable default values as long as any command line arguments override those values.
Changeable values should never be hard-coded: if users have to edit your software in order to run it, you have done something wrong.Changeable but infrequently-changed values should therefore be stored in configuration files.These can be in a standard location, e.g..packagerc in the user's home directory, or provided on the command line as an additional argument.Configuration files are often created during installation to set up such things as server names, network drives, and other defaults for your lab or institution.
Software evolves over time, with developers adding or removing features as need dictates.Making official releases stamps a particular set of features with a project-specific identifier so that version can be retrieved for later use.For example, if a paper is published, the software should be released at the same time so that the results can be reproduced.
Most software has a version number composed of a decimal number that increments as new versions are released.There are many different ways to construct and interpret this number, but most importantly for us, a particular software version run with the same parameters should give identical results no matter when it's run.Results include both correct output as well as any errors.Increment your version number every time you release your software to other people.
Semantic versioning 2 is one of the most common types of versioning for open-source software.Version numbers take the form of MAJOR.MINOR[.PATCH], e.g., 0.2.6.Changes in the major version number herald significant changes in the software that are not backwards compatible, such as changing or removing features or altering the primary functions of the software.Increasing the minor version represents incremental improvements in the software, like adding new features.Following the minor version number can be an arbitrary number of project-specific identifiers, including patches, builds and qualifiers.Common qualifiers include alpha, beta, and SNAPSHOT, for applications that are not yet stable or released, and -RC for release candidates prior to an official release.
The version of your software should be easily available by supplying --version or -v on the command line.This command should print the software name and version number, and it should also be included in all of the program's output, particularly debugging traces.If someone needs help, it's important that they be able to tell whoever's helping them which version of the software they're using.
While new releases may make a program better in general, they can simultaneously create work for someone who integrated the old version into their own workflow a year or two ago, and won't see any benefits from upgrading.A program's authors should therefore ensure that old released versions continue to be available.A number of mechanisms exist for controlled release that range from as simple as adding an appropriate commit message or tag to version control [20], to official releases alongside code on Bitbucket or GitHub, to depositing into a repository like apt, yum, homebrew, CPAN, etc. Choose the method that best suits the number and expertise of users you anticipate.

Rule 5: Reuse software (within reason)
In the spirit of code reuse and interoperability, developers often want to reuse software written by others.With a few lines, a call is made out to another library or program and the results are incorporated into the primary script.Using popular projects reduces the amount of code that needs to be maintained and leverages the work done by the other software.
Unfortunately, reusing software (whether software libraries or separate executables) introduces dependencies, which can bring their own special pain.The interface between two software packages can be a source of considerable frustration: all too often, support requests descend into debugging errors produced by the other project due to incompatible libraries, versions, or operating systems [16].Even introducing libraries in the same programming language can rely on software installed in the environment, and the problem becomes much more difficult when relying on executables, or even on web services.
Despite these problems, software developers in research should re-use existing software provided a few guidelines are adhered to.
First, make sure that you really need the auxiliary program.If you are executing GNU sort instead of figuring out how to sort lists in Python, it may not be worth the pain of integration.Reuse software that offers some measurable improvement to your project.
Second, if launching an executable, ensure the appropriate software and version is available.Either allow the user to configure the exact path to the package, distribute the program with the dependent software, or download it during installation using your package manager.If the executable requires internet access, check for that early in execution.
Third, ensure that reused software is robust.Relying on erratic third party libraries or software is a recipe for tears.Prefer software that follows good software development practices, is open for support questions, and is available from a stable location or repository using your package manager.
Exercise caution especially when transitioning across languages or using separate executables, as they tend to be especially sensitive to operating systems, environments, and locales.
6 Rule 6: Rely on build tools and package managers for installation.
To compile code, deploy applications, and automate other tasks, programmers routinely use build tools like Make, Rake, Maven, Ant or MS Build.These tools can also be used to manage runtime environments, i.e., to check that the right versions of required packages are installed and install or upgrade them if they are not.As mentioned in Rule 5, a package manager can mitigate some of the difficulties in software reuse.
The same tools can and should be used to manage runtime environments on users' machines as well.Accordingly, developers should document all dependencies in a machine-readable form.Package managers like apt and yum are available on most Unix-like systems, and application package managers exist for specific languages like Python (pip), Java (Maven/Gradle), and Ruby (RubyGems).These package managers can be used together with the build utility to ensure that dependencies are available at compile/run time.
For example, it is common for Python projects to include a file called requirements.txtthat lists the names of required libraries, along with version ranges: requests>=2.0pygithub>=1.26,<=1.27python-social-auth>=0.2.19,<0.3 This file can be read by the pip package manager, which can check that the required software is available and install it if it is not.Whatever is used, developers should always install dependencies using their dependency description, especially on their personal machines, so that they're sure it works.
Conversely, developers should avoid depending on scripts and tools which are not available as packages.In many cases, a program's author may not realize that some tool was built locally, and doesn't exist elsewhere.At present, the only sure way to discover such unknown dependencies is to install on a system administered by someone else and see what breaks.As use of virtualization containers becomes more widespread, software installation can also be tested on a virtual machine or container system like Docker.
7 Rule 7: Do not require root or other special privileges to install or run.
Root (also known as "superuser" or "admin") is a special account on a computer that has (among other things) the power to modify or delete system files and user accounts.Conversely, files and directories owned by root usually cannot be modifed by normal users.
Installing or running a program with root privileges is often convenient, since doing so automatically bypasses all those pesky safety checks that might otherwise get in the user's way.However, those checks are there for a reason: scientific software packages may not intentionally be malware, but one small bug or over-eager file-matching expression can certainly make them behave as if they were.Outside of very unusual circumstances, packages should not require root privileges to set up or use.
Another reason for this rule is that users may want to try out a new package before installing it system-wide on a cluster.Requiring root privileges will frustrate such efforts, and thereby reduce uptake of the package.Requiring, as Apache Tomcat does, that software be installed under its own user account-i.e., that packagename be made includes a full set of very small files, and a 'Getting Started with HISAT2' section in its manual that leads you through the entire data lifecycle [25].
Equally, make the test script's output easy to interpret.Screens full of correlation coefficients do not qualify: instead, the script's output should be simple to understand for non-experts, such as one line per test, with the test's name and its pass/fail status, followed by a single summary line saying how many tests were run and how many passed or failed.If many or all tests fail because of missing dependencies, that fact should be displayed once, clearly, rather than once per test, so that users have a clear idea of what they need to fix and how much work it's likely to take.
Research has shown that the ease with which people can start making contributions is a strong predictor of whether they will or not [26].By making it simpler for outsiders to contribute, a test suite of any kind also makes it more likely that they will, and software with collaborators stands a better chance of surviving in the busy field of scientific software.
10 Rule 10: Produce identical results when given identical inputs.Given a set of parameters and a dataset, a particular version of a program should produce the same results every time it is run to aid testing, debugging, and reproducibility.Even minor changes to code can cause minor changes in output because of floating-point issues, which means that getting exactly the same output for the same input and parameters probably won't work during development, but it should still be a goal for people who have deployed a specific version.
Many applications rely on randomized algorithms to improve performance or runtimes.As a consequence, results can change between runs, even when provided with the same data and parameters.By its nature, this randomness renders strict reproducibility and therefore debugging more difficult.If even the small test set (#9) produces different results for each run, new users may not be able to tell whether the software is working properly.When comparing results between versions or after changing parameters, even small differences can confuse or muddy the comparison.And especially when producing results for publications, grants or diagnoses, any analysis should be absolutely reproducible.
Given the size of biological data, it is unreasonable to suggest that random algorithms be removed.However, most programs use a pseudo-random number generator, which uses a starting seed and an equation to approximate random numbers.Setting the seed to a consistent value can remove randomness between runs.Allow the user to optionally provide the random seed as an input parameter, thus rendering the program deterministic for those cases where it matters.If the seed is set internally (e.g., using clock time), echo it to the output for re-use later.If setting the seed is not possible, make sure the acceptable tolerance is known and detailed in documentation and in the tests.

Conclusion
There has been extended discussion over the past few years of the sustainability of research software, but this question is meaningless in isolation: any piece of software can be sustained if its users are willing to put in enough effort.The real equation is the ratio between the skill and effort available, and the ease with which software can be installed, understood, used, maintained, and extended.Following the ten rules we outline here reduce the denominator, and thereby enable researchers to build on each other's work more easily.
That said, not every coding effort needs to be engineered to last.Code that is used once to answer a specific question related to a specific dataset doesn't require comprehensive documentation or flexible configuration, and the only sensible way to test it may well be to run it on the dataset in question.Exploratory analysis is an iterative process that is developed quick and revised often [4,11].However, if a script is dusted off and run three or four times for slightly different purposes, is crucial to a publication or a lab, or being passed on to someone else, it may be time to make your software more robust.
The usage message tells users what the program could do.It is equally important for the program to tell users what it actually did.Accordingly, when the program starts, it should echo all parameters and software versions to standard out or a log file alongside the results.This feature supports greater reproducibility because any result can be replicated with only the previous output files as reference.