MMTF—An efficient file format for the transmission, visualization, and analysis of macromolecular structures

Recent advances in experimental techniques have led to a rapid growth in complexity, size, and number of macromolecular structures that are made available through the Protein Data Bank. This creates a challenge for macromolecular visualization and analysis. Macromolecular structure files, such as PDB or PDBx/mmCIF files can be slow to transfer, parse, and hard to incorporate into third-party software tools. Here, we present a new binary and compressed data representation, the MacroMolecular Transmission Format, MMTF, as well as software implementations in several languages that have been developed around it, which address these issues. We describe the new format and its APIs and demonstrate that it is several times faster to parse, and about a quarter of the file size of the current standard format, PDBx/mmCIF. As a consequence of the new data representation, it is now possible to visualize structures with millions of atoms in a web browser, keep the whole PDB archive in memory or parse it within few minutes on average computers, which opens up a new way of thinking how to design and implement efficient algorithms in structural bioinformatics. The PDB archive is available in MMTF file format through web services and data that are updated on a weekly basis.


Introduction
The Protein Data Bank (PDB) [1] is the global archive of 3D structures of proteins, nucleic acids, and complex assemblies. Recent advances in experimental techniques have led to an explosion in both the number and size of such structures. The entire PDB now exceeds one billion atoms and the largest structure currently contains about 2.4 million atoms [2] (Fig 1A). In addition to a growing number of depositions per year ( Fig 1B) and average number of atoms per structure (Fig 1C), 68 of the 100 largest structures were deposited in the past three years. In Fig 1D, we show the rising importance of Cryo-Electron microscopy as a technique [3]. It is expected that much larger molecular machines and molecular assemblies will be modeled by combining multiple experimental techniques [4]. Significant increases in data sizes have been seen in many fields. Efficient storage and transmission of data using novel file formats and data compression methods are integral to these development, e.g., for the transport of HD-TV, video, and audio. A similar trend has emerged in the handling of whole genome data [5].
Few notable developments have been made in developing such a format for macromolecules. First, WPDB [6] stored the data as binary files with limited precision, allowing efficient access. WPDB is however no longer maintained and was tied to the Windows operating system. The second development is mmJSON [7], which represents all data from the PDBx/ mmCIF format (http://mmcif.wwpdb.org/) in the JSON serialization format that can be efficiently parsed by modern web browsers. After compression with gzip (a commonly used general purpose compression tool) the largest structure (PDB ID 3J3Q) takes up 27 MB in mmJSON. In addition, a "lite" version of mmJSON is available that contains a minimal amount of information to render backbone structures (3.3 MB for PDB ID 3J3Q). Neither WPDB, mmJSON, nor other formats such as PDBx/mmCIF, provide all data necessary to represent a full macromolecular model including bond information. Furthermore, as text based formats, they are slow to parse, and clean Application Programming Interfaces (APIs) are generally not made available.
Commercial software providers have produced their own internal representations of macromolecular structures. No such format, however, is openly available and thus they cannot be incorporated into third party software or developed with community involvement. For this reason, structural analysis is currently a laborious and error-prone process, often involving substantial duplicated effort to reliably process the entire PDB archive into a 3rd party data structure. Structure visualization can be equally challenging for large structures, due to slow data download and high client-side memory requirements to parse large structure files. Some of the largest structures in the PDB require more memory than is typically available within in web browser.
In this paper we describe a new data representation, the MacroMolecular Transmission Format (MMTF) (http://mmtf.rcsb.org/) that aims to resolve these deficiencies. MMTF is a binary machine-readable file format that can be parsed, in some instances at least an order of magnitude faster than existing text-based formats. Custom lossless and lossy compression methods with either full atom level detail and a reduced representation (C-alpha, P atoms) are applied [8] to reduce the file size and thus further improve transmission and parsing speeds. MMTF uses a combination of encoding and compression techniques, such as delta encoding, and reduced precision for lossy compression, which have also been used for MD trajectory compression [9][10][11]. Finally, MMTF is designed for interoperability and use by a broad community. APIs are provided in common programming languages and a full chemical description required to understand a structure is included in the file. The PDB archive is provided in MMTF format through web services and updated weekly. A number of third-party tools already support MMTF.

Design and implementation Design considerations
Above we demonstrated that existing file formats are becoming less suitable for modern macromolecular data. Due to these challenges, the MMTF format was designed with three core aims. First, to minimize data storage requirements and transfer times, the format should represent data in compressed form without loss of accuracy. Second, it should be fast to parse, since I/O is often a bottleneck in structural analysis and visualization. Third, we designed MMTF to be as extensible, self-contained, and interoperable as possible. As a binary, machine-readable format, the preferred access to MMTF data is through the APIs provided in several programming languages. This allows the developers to focus on scientific applications and not on developing custom file parsers.

Data items and encoding
The MMTF format was designed to include the core data commonly used by macromolecular visualization and analysis tools (Table 1), rather than support all metadata available in PDBx/ mmCIF. A comprehensive list of the data items is available in the MMTF specification. Additional metadata present in PDBx/mmCIF files and other annotations, if required, can be accessed through web services [12][13] [14].
MMTF files have been augmented with calculated DSSP [15] secondary structure using the BioJava implementation [16]. This information speeds up both visualization and analysis applications and ensures data consistency across all structures in the PDB archive. MMTF includes the full chemical description of all molecules in a PDB entry. Bonds and bond orders for both standard and non-standard residues, e.g., ligands, are included from the Chemical Component Dictionary [17] and additional covalent bonds (struct_conn category in the PDBx/mmCIF files), such as disulfide bonds or covalent bonds between ligands and polymers are also included in MMTF. Metal coordination and hydrogen bond information is not included in MMTF, since there are no generally agreed upon standards how to define them.

Encoding strategies
In order to reduce the overall file size, we applied specialized encoding techniques to make the data more compressible. These techniques either reduce redundancy in the data or reduce the dynamic range (entropy) of numbers, to make them more compressible using standard entropy encoding techniques.
Fields of the same type are grouped together in MMTF to create a flat data structure. For instance, the coordinates of all atoms are stored together, instead of in atom objects with other atom-related data. This avoids imposing a deeply nested hierarchical structure on consuming programs, while still allowing efficient traversal of models, chains, groups, and atoms. This approach represents a columnar encoding [18] of data, which facilitates data encoding and enhances data compressibility. Columnar encoding is also used in mmJSON [7] to increase compressibility.
Lossless integer encoding is applied to all fixed precision floating point numbers. Integer numbers have a simpler bitwise representation and are therefore more compressible than the equivalent floating-point numbers [8]. Atomic coordinates are typically represented with a precision of 3 decimal places, and temperature factors with 2 decimal places. For lossless encoding, we multiply coordinate and temperature factor values by 1000 and 100, respectively, and round the values to the nearest integer. A further increase in compression can be obtained through lossy encoding by rounding coordinates to 0.1 Å precision and temperature factors and occupancy to 0.1 precision. Lossy compression is particularly important for the visualization of large complexes, for which the reduced precision is not visually perceptible [6,8].
Dictionary encoding is used for data repeated across multiple residues. In standard PDB and PDBx/mmCIF files, atoms within a residue are listed in a standard order. Exploiting this, atom name, element symbol, intra residue bonds and bond orders, etc. can be stored once for each unique residue type and not repeated across the file, as shown for the dictionary entry for serine (Fig 3). MMTF has been designed to handle exceptions to a consistent atom order, if they occur, however, the encoding will be less efficient.
Delta encoding is applied to data of large magnitude that change in small increments. For example, instead of storing absolute atom coordinate values, differences in the x, y, and z coordinates are stored. Due to the covalent bond structure in molecules, these differences typically lie within a small dynamic range bound by their bond distances. Previous work determined this method to be the most effective encoding technique [8]. Temperature factors are also delta encoded, since their variation from residue to residue is typically low.
Run-length encoding compresses a list of repeated values, such as occupancy values in Xray structures, most of which are constant (1.0). Here the value itself and the number of repetitions is stored. For atom serial numbers, delta and run-length encoding are combined to achieve a very compact encoding.
Recursive indexing-Given the small dynamic range of delta encoded coordinates, most, but not all values can be represented as 16-bit signed integers, rather than 32-bit signed integers. We have explored the effect of packing on compression [8] and identified recursive indexing as a simple and effective packing strategy for this data type. Any values that lie outside the 16-bit integer range [-32,768, 32,767] are decomposed into a series of values, such that the individual values fit into the 16-bit range (Fig 4D), and their sum adds up to the original value.
The overall workflow for the encoding of columnar data is shown in Fig 4. Serialization MMTF data are stored in the MessagePack format (version 5, http://msgpack.org) binary container format. MessagePack is an efficient binary serialization format, similar to JSON, but faster to parse and more compact. Encoding and decoding libraries for MessagePack are available in many languages. The top-level of the container holds the field names as keys and field data as values. Non-columnar data are described using standard MessagePack data types. Columnar data, e.g., most data columns in the "ATOM" records, are custom encoded. The MMTF specification defines Codec Types used to custom encode columnar data. These data  records are described by the following data structure (Fig 5), which is represented as a binary array in MessagePack.

MMTF data files
MMTF files for all PDB entries are updated weekly as part of the RCSB PDB weekly update pipeline. Semantic versioning (http://semver.org) is employed to the file specification and the APIs.
Major version changes of the specification may require code updates to decode and parse data. For this reason, after the release of a new major version of the specification, the previous major version will be retained for a number of months to allow time for code updates and testing. Such version changes will be disseminated through a mailing list and updates to the documentation. MMTF files are generated with two types of molecular representation ( Table 2). The reduced representation, which uses lossy compression and less atomic level detail is suitable for 3D visualization, e.g., ribbon diagrams, or calculations that require only a C-alpha representation.

Results
The benefits of the MMTF file format were assessed in three different ways. First, the relative sizes of the files in different formats were measured. Second, the file load time was

Load time benchmarks
The following benchmarks assess the file load time for MMTF compared to PDBx/mmCIF and PDB data formats. The load times reported in the figures below consist of reading the files from a local disk, decompressing and parsing the data, instantiating a hierarchical molecular data structure (model->chain->residue->atom), and storing the metadata. All parsing benchmarks were performed using a single core on a MacMini, 2.6 GHz Intel Core i5, 16 GB RAM 1600 MHz DDR3, with a solid state drive. The first benchmark uses the existing file parsers (PDBx/mmCIF, PDB) in BioJava and compares their performance with the new BioJava MMTF parser, which uses the MMTF-Java API. In Fig 8 we compare the load times for~127,000 PDB entries as individual gzip compressed PDBx/mmCIF, PDB, and MMTF files, and as uncompressed Hadoop Sequence files.
The MMTF format has clear advantages over PDBx/mmCIF and PDB. For BioPython, MMTF is parsed about 4 times faster than PDBx/mmCIF using the FastMMCIFParser (353 seconds), and about 30 times faster compared to the default MMCIFParser (2650 seconds), which creates a more complete data model. For NGL (JavaScript), MMTF loading is about 18 times faster than PDBx/mmCIF. For BioJava, loading MMTF files is about 45 times faster than loading the corresponding PDBx/mmCIF files.
To assess the effect of structure size on load time (Fig 10), we created samples of 100 structures around the 25 percentile (S2 Table), 50 percentile (S3 Table), and 75 percentile (S4 Table) from the atom size distribution of the PDB archive. To create these subsets, we selected 100 structures symmetrically around the quartile values. S1 Appendix contains links to the software repositories to run the BioPython and BioJava benchmarks.
According to this benchmark, most small to medium sized PDB structures can be parsed in milliseconds using the BioJava/MMTF-Java API and NGL/MMTF-JavaScript API. The load time is approximately linear with the number of atoms in a PDB entry. MMTF file loading with BioPython is consistently about a factor of 40-50 slower than with BioJava. This is due in part that Python is an interpreted language. Our profiling points to the creation of the hierarchical molecular data structure as the time limiting factor for BioPython.
MMTF was specifically designed to handle the efficient transfer and visualization of very large structures that could not be parsed and visualized using the PDBx/mmCIF format due to the large memory overhead. For example, the currently largest asymmetric structure (PDB ID 3J3Q) in the PDB with 2,440,800 atoms, shown in Fig 1A, was rendered with NGL viewer using the MMTF-reduced format. Table 3 compares the load times for this entry using BioPython, NGL, and BioJava.

Simple application programming interfaces
The MMTF file format is designed to be easy to use and incorporate into 3rd party applications. While MMTF has a flat, columnar data structure, it can be traversed following the structure hierarchy: Models -> Chains -> Groups -> Atoms. In Fig 11, we demonstrate the simplicity in retrieving data using the Java, JavaScript and Python APIs.

Availability and future directions
In this paper we present a modern macromolecular transmission format. MMTF addresses the growing size and complexity of macromolecular structures in the PDB archive through a new binary, custom compressed file format. Furthermore, MMTF is self-contained and simple APIs are provided in multiple popular programming languages. Software developers do not need to implement their own parsers-often an error-prone process, but rather build on the tools provided by MMTF. Through both these advances MMTF allows rapid user-friendly access to any structure in the PDB archive with a few lines of code. We demonstrate that the format is 75% smaller, an order of magnitude faster to parse, and is provided along with a user friendly API that promotes interoperability.
The MMTF project page (http://mmtf.rcsb.org) is the entry point to all documentation and software, including the MMTF specification, links to GitHub repositories of the MMTF APIs (Java, JavaScript, Python, C, and C++), and API descriptions. The versioned specification and all software libraries are available under either an Apache 2 or MIT license. A description how to download MMTF files is also available. A "Try it" feature demonstrates the transfer and parsing performance of MMTF-JavaScript in a web browser, and a "See it in Action" page demonstrates the fast data transfer, parsing, and rendering in NGL viewer [20].
Due to simple API, user-friendly specification and licensing model, the format has already been incorporated into several protein analysis tools and 3D structure visualization tools (Table 4).
We envisage the above advances will have a major impact in two areas of structural bioinformatics (Fig 12).
The first key area of impact is visualization of macromolecular structures, in particular when used on mobile device or in a web browser. MMTF enables low bandwidth file transfer, low client-side memory consumption, and fast parsing of PDB structures. For example, the 3D visualization on the RCSB PDB website is powered by MMTF [24], using the MMTF-full representation for entries < 10,000 residues and the MMTF-reduced representation for larger entries. Using the NGL viewer [20] and MMTF, the currently largest structure in the PDB, the HIV viral capsid (PDB ID 3J3Q) [2], can now be visualized on a mobile device (Fig 1A).
Second, by greatly increasing file-parsing speed, a rapid analysis of the entire PDB archive can be carried out. As an example, we have used the MMTF format to rapidly mine the PDB for interatomic distance distributions. Coupled with the use of an efficient geometric hashing algorithm in BioJava, the distances between all C-alpha carbons can be calculated in minutes. Parsing of the text-based PDBx/mmCIF format alone would take several hours. Using a Hadoop Sequence file with MMTF records enables the scalable analysis of the PDB using standard distributed parallel processing frameworks. Further work is ongoing to demonstrate the use of MMTF with Big Data frameworks.
MMTF is an open source project and we welcome additions and new applications that use the new technology. As an example, the MMTF-C and MMTF-C++ decoders were developed in collaboration with community members.   11. Traversal of the structure hierarchy using the MMTF API. These code snippets (A) Java, (B) JavaScript, and (C) Python demonstrate how to load and decode an MMTF file (PDB ID 4CUP) from http://mmtf.rcsb.org and then traverse the hierarchical data structure (Models -> Chains -> Groups -> Atoms). The code shown here loops through the Model and Chain hierarchy. For each model, the model index is printed, and for each chain, the chainId, chainName, and number of groups (residues) are printed. In an analogous fashion, the group and atom data can be traversed.
Supporting information S1