Problem statement

The translation of cuneiform texts into modern languages requires three steps: (1) a sign-by-sign transliteration; (2) word segmentation, i.e. joining the individual sign readings into words which are expressed by a hyphen (-) or a dot (.); (3) translation into a modern language. The first stage is vital because of the polyvalent nature of the cuneiform script, i.e. almost every sign has different possible readings.

Generally speaking, cuneiform sign-values can be divided into three categories: (1) logographic, one sign (or a combination of signs) representing one word, and usually written in capital letters in modern transliterations; (2) phonetic, one sign representing one syllable, and usually written in italics in transliterations; and (3) determinatives, which are semantic classifiers whose purpose is to aid the correct reading of the word which either follows or precedes them. In transliterations, they are written in superscript. Thus, while modern scholarly convention differentiates between logograms, phonetic readings, and determinatives, the cuneiform signs are exactly the same for all three. For example, the sign can be read as the logogram DINGIR, meaning “god”; it can be read syllabically an, as part of a word; and it can be a determinative ^d, appearing before divine names. For example, the name of Sîn, the Babylonian moon god, will be written and it will appear in transliteration as ^d30 (the moon god was written logographically with the sign for the number 30; the number of days in a lunar month).

Additionally, each of these categories can contain more than one value for any given period or genre of the Akkadian language. Thus, the same sign can be read logographically DINGIR, “god”, but also AN, “sky”. In addition to the most common phonetic reading an, which is found in texts from all periods, the sign has nine more phonetic readings, some of which are only found in particular periods, regions or genres (for example, ilu in Nuzi, le₄ in El-Amarna, or šubul in scholarly texts).

The process described above, of determining appropriate sign readings by looking at the context from previous and following signs, can be efficiently solved by NLP tagging algorithms, as our results show. Therefore, the results of our project can make a significant step toward rediscovering the lost heritage of the ancient Near East. Our models receive as input a string of standard cuneiform Unicode glyphs and produce a segmented transliteration. In order to train the models, we used significant amounts of existing transliterations for which strings of cuneiform Unicode glyphs have already been generated, see Related Work and Materials and methods. We release all code and data used during this project, see here.

Related work

Automating the process of making cuneiform text editions requires a pipeline which deals with each of the following three problems with different AI models: (1) extraction and visual recognition of 2-dimensional or 3-dimensional representations of cuneiform signs; (2) automatic transliteration and segmentation; (3) translation and/or annotation. In our project, we offer a solution for the second problem. For each of these steps, machine learning models require large amounts of training data. The following section presents a brief overview of the current availability of digitized cuneiform data and the state-of-the-art on each stage in the pipeline.

The largest repository of cuneiform digital textual corpora is the Open Richly Annotated Cuneiform Corpus (ORACC). It contains tens of thousands of texts with varying levels of transliteration, translation, annotation, and metadata in at least 20 different scholarly projects. However, one should bear in mind that ORACC is far from comprehensive for the entire cuneiform corpus. The Neo-Assyrian period is the most prevalent on ORACC; made recently available on a wide variety of formats (TEI/XML, JSON and TXT) thanks to the Munich Open-access Cuneiform Corpus Initiative (MOCCI). The Cuneiform Digital Library Initiative (CDLI) offers more than 300,000 text entries on their website, mostly with metadata only. Many entries contain low-resolution pictures of the tablets or only their hand copies (black and white drawings depicting signs in 2D). Less then half of the entries include transliterations and some include translations. There are many more online resources and databases for cuneiform studies, too many to mention in the scope of this article. For a comprehensive survey up till 2014, see D. Charpin [5].

The first stage of the pipeline for automatically creating cuneiform text editions requires procuring digital representations of cuneiform tablets. Currently, these are 3D scans, 2D or 2D+ photographs, and images of hand-copies: drawings made by Assyriologists which imitate the tablet in a 2-dimensional, black and white presentation. Hand-copies are still a standard in the publishing of traditional cuneiform text editions, and therefore they exist in abundance. They were the most cost-effective method for publishing a visual representation of cuneiform tablets until recent years. The situation, however, is very different for 3D and 2D representations. Several major projects developed effective methods for 3D or 2D+ scans and photographs of cuneiform tablets, such as the pioneering, but now defunct iClay project of J. Cohen et al. [4]. The leading methods for acquiring 2D+ images are currently the Polynomial Texture Mapping (PTM) and Reflection Transformation Imaging (RTI) dome shaped systems developed independently by G. Earl et al. [6], on the one hand, and by H. Hameeuw and G. Willems [7], on the other. Furthermore, there are two leading research groups creating 3-dimensional scans of tablets, that are developing methods of sign extraction and identification. A joint Dortmund-Würzburg team, led by G. Müller, G. Fink and F. Weichert, made 3-dimensional scans of cuneiform fragments and created the CuneiformAnalyser. It displays 3-dimensional scans of tablets, enhances features, extracts and recognizes signs and also reconstructs fragmentary tablets [8–10]. Another group from Heidelberg, led by H. Mara, developed the GigaMesh software, which also displays 3D scans of cuneiform tablets and provides philological tools such as sign extraction [11, 12]. In addition, they are developing methods for sign identification, both from 2D projections of 3D scans and hand-copies [13–17]. The best results are usually attained with 3D scans, but there are not enough scanned tablets. 3D scans are still expensive to produce in terms of the equipment necessary, and are also time-consuming, since the high-quality required for machine learning algorithms is considerable and this lengthens the duration of individual scans. However, there have been developments in the field of photogrammetry: there are now cheaper and quicker methods for creating 3D models, which do not require a specialized scanner, but rather use a turntable and standard cameras to create a 3-dimensional scan from multiple pictures, sometimes achieving a higher quality image than a standard 3D scanner, see T. Collins et al. [18].

After signs are extracted and identified, they need to be transliterated and segmented. The first research team to attempt transliteration of cuneiform texts are B. Bogacz et al. [19]. They attempted a pipeline of sign extraction, identification, and transliteration: they took 30 raster images of hand-copies with their corresponding transliterations from the Cuneiform Commentary Project (CCP) website, visually segmented the signs using Histogram of oriented Gradients (HoG) feature descriptors, and labeled them to their corresponding transliteration for training. Their best-performing algorithm was a Hidden Markov Model (HMM), but nevertheless, they achieved low accuracy scores.

The first to attempt segmentation are T. Homburg and C. Chiarcos [20]. They used rule-based, dictionary-based, and machine learning algorithms in order to segment the input of Unicode cuneiform glyphs, prepared from texts available at the CDLI website. They took texts from the Old Babylonian, Middle Babylonian and Neo Assyrian periods, and trained and tested each time-period individually. Their results for texts of different periods varied, but overall the best performing algorithms were dictionary-based algorithms, originally developed for Chinese and Japanese, with an F-score averaging between 60%-80%. They also attempted transliteration, by developing a baseline formed on the most common transliteration for each sign that reaches an accuracy of 40%. Therefore, in this paper we present our state-of-the art approach for automatically transliterating and segmenting Unicode cuneiform glyphs.

The only other project using Unicode cuneiform characters that we are aware of is T. Jauhiainen et al. [21], a task force for language identification in cuneiform texts which was part of the VarDial 2019 workshop. The Project used as data sets texts from ORACC in the Sumerian language and in six dialects of Akkadian. Out of the groups which participated in the task force, the third, fourth, and fifth places published their results in [22–24], respectively.

To our knowledge, there have been no publications attempting automatic translation of texts written in cuneiform. There have been various studies on the automatic annotation of cuneiform texts written in the Sumerian language, e.g. [25–28]. Currently, a research team is working on the project Machine Translation and Automated Analysis of Cuneiform Languages (MTAAC), led by H. Baker, C. Chiarcos, R. Englund, and É. Pagé-Perron: see [29, 30] and their website.

Formulation as a tagging problem

We assume each Unicode glyph corresponds to one sign in the transliteration, in order to cast the problem as a tagging problem, where each sign is marked, or tagged, by its proper transliteration. However, the cuneiform writing system can use compound signs, creating new meaning with the combination of two or more signs, much like the Chinese writing system. For example, the logogram SIPA (“herdsman”) is made of two signs, PA and LU, and the logogram ENSI₂ (“governor”) is made of three, PA, TE and SI. In Unicode cuneiform, as in actual cuneiform, the compound signs are not shown as one character, but rather they are formed from a group of characters. As a result, these compound signs are read together as one sign in the transliterations, which results in an alignment problem.

Thanks to the efforts of the OIMEA project team at LMU Munich, the RINAP texts are also available in JSON format on their website. In the JSON files we could find each sign or group of signs with their matching transliteration. This resolved the alignment problem. To handle these compound signs, we used the BIO-tagging scheme [31], common in Named Entity Recognition (NER) tagging in NLP, which allows tagging a sequence of contiguous signs by the same tag. We used an index for each such sign reading to mark they are all part of the same transliteration. For example, the two signs (DUMU, “son”) and (GEŠ₃, “male member”) that should be read together as the logogram IBILA (“descendent”) appear as IBILA(0) and IBILA(1). In the output, only IBILA(0) is shown, since there is no need for the transliteration to appear twice.

To handle both transliteration and word segmentation in a single model, we double the number of possible transliteration symbols, where for every transliteration t we add a symbol (representing a sign with a hyphen or a dot after it). This denotes that this tag is not the last tag of a word. Thus, after tagging a sentence, we can decode both the transliteration and segmentation.

A Hidden Markov model

A Hidden Markov Model (HMM) is a classical statistical model [32] that embodies a set of statistical assumptions concerning the generation of sample data (and similar data from a larger population). It is often used in NLP for tagging [33, 34]. It can be applied to speech recognition, part-of-speech tagging and various bioinformatics applications inter alia. Given an input sequence of n signs X = (x₁, …, x_n) and a possible output transliteration Y = (y₁, …, y_n), an HMM provides the probability p(X, Y). Thus, given a sequence of signs X = (x₁, …, x_n), we can choose the transliteration that maximizes .

HMMs factorize the probability p(X, Y) into transition probabilities and emission probabilities. Concretely, we use a trigram HMM: where e(⋅) is the emission probability of observing the sign x_i given the transliteration y_i, and the transition probability q(y_i|y_i−1, y_i−2) is the probability of the transliteration y_i given the previous two transliteration. The probabilities q(⋅) and e(⋅) are estimated using maximum likelihood from a training corpus, and the predicted sequence that maximizes is found using the Viterbi algorithm (A dynamic programming algorithm for finding the most likely sequence according to a model, used as part of the HMM and MEMM algorithms [35]). Thus, an HMM captures the correlations between signs and transliterations, as well as which transliteration trigrams (triplets of adjacent transliterations, hence the name trigram HMM) are likely according to the data.

Choosing the order of an n-gram model (trigram in our case) depends on two factors: (a) computational efficiency: using a higher order model (for example, 4-grams) makes decoding slower. (b) statistical efficiency: given infinite data, higher order models are more accurate. However, with limited amount of data, using high-order models quickly leads to sparsity—many n-grams are never observed in the training data, leading to poor performance. Thus, we use in this work trigram models, which empirically lead to good performance and an efficient model.

Because the number of possible transliterations for every sign is large, there are cases where the probability p(X, Y) is zero for all transliteration sequences Y (for example, when for a pair of signs all possible transliteration bigrams have not been observed in the training set). In this case, we use a backoff mechanism, where we choose the most common transliteration for each sign.

A Maximum-entropy Markov model

A Maximum-entropy Markov model is a graphical model for sequence labeling that combines features of hidden Markov models (HMMs) and maximum entropy (MaxEnt) models. It is a model that outperforms HMMs as it can capture richer information [36] [37]. In MEMM, we directly estimate a conditional distribution where the probability p(y_i|y_i−1, y_u−2, X) is estimated as and the scoring function is linear: . Each feature f_j() is extracted from the input sequence and the transliteration trigram, and the parameters w_j are learned.

Because f_j() is an arbitrary function of its inputs, richer features that are useful for prediction can be used. For example, a feature can capture the correlation between a sign and the transliteration of the following sign, which is difficult to do with HMMs. Learning is done by maximizing the L2-regularized maximum likelihood objective over the training set, and the best sequence is found again using Viterbi which finds that maximizes .

The main advantage in MEMMs is the ability to define arbitrary and flexible features [38]. In this work, we found the following features to be most useful for our model p(y_i|y_i−1, y_u−2, X): the current sign, the last two signs, the next two signs, and the transliterations’ bigram and trigram. We also added some domain-specific features that are based on the error types we observed in the HMM algorithm:

• Is the previous sign the last sign of a word or does the word continue with the current sign?
• Is the transliteration of the previous sign phonetic or is it a logogram? There are many cases in which a transliteration may be both (see below under error analysis).
• Is the transliteration of the previous sign a determinative? This may change the context of the word.
• Is the transliteration of the previous sign a compound sign?

While we used Viterbi as our inference algorithm, we also experimented with a greedy inference algorithm that simply chooses the best transliteration for each sign from left-to-right. This inference algorithm does not guarantee theoretically that the output will maximize , but is much faster than Viterbi. In practice, we found its quality is almost as high as Viterbi.

A neural sequence model

In recent years, deep learning models based on neural networks, which allow to capture complex dependencies between inputs and outputs, have improved performance and become standard in NLP. Tagging is no exception, where a common approach for modelling problems in which the input is a sequence, is to use a Recurrent Neural Network (RNN), using the Long-Short Term Memory (LSTM) cell [39]. The advantages of LSTM are: (a) they can capture long-range dependencies, while HMMs and MEMMs only look at a bounded window around every position; (b) the class of functions that they express is larger.

In an LSTM, a neural network “reads” the input sequence word by word from left-to-right, and creates a representation for every word that is conditioned on all previous words. Thus, each word depends on the entire history and can capture long-range dependencies. We use a Bidirectional LSTM, that is, one LSTM reads the input from left to right, and another reads the input from right to left: see Fig 3. Thus, the representation of every word depends on all of the words that occur both before it and after it, which has been shown to be useful in many tasks, including tagging [40, 41]. Once these representations are created, we predict the transliteration in every position from the learned representations. Training is done again using maximum likelihood over the training data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. An illustration of the bidirectional LSTM.

https://doi.org/10.1371/journal.pone.0240511.g003

We adapt the BiLSTM from the open source deep learning library AllenNLP for our purposes [42]. We performed hyper-parameter search for the word embedding size, hidden representation size and learning rate, by doing grid search and choosing the values that maximize performance on a development set. We use 200-dimensional vectors as the word embedding size as well as the hidden representation size, and a learning rate of 0.3.

We optimized the learning rates, the embedding size, and hidden representation size. Details regarding how accuracy changes for different parameters can be found in the S1 File. We observe that a smaller dimension size dramatically reduces performance, while a larger dimension size only slightly reduces performance, but leads to a longer training time. The learning rate of 0.3 gave best results compared to other options on the development set.

Error analysis

We recognize four main types of errors:

• The most common error is word segmentation, as there is an almost 4% difference between total accuracy and transliteration only accuracy in HMM for example. The algorithms tend to transliterate the signs well but they sometimes do not segment the words properly. The main reason for that can be that these words do not have a typical suffix such as a case marker or number. Many examples of different contexts are needed in order to improve our results.
• Some signs can have the same value in their logographic and phonetic reading. For example, has the logographic reading SAG (‘head’), and also the phonetic reading sag. The algorithms can correctly identify “sag”, but cannot identify correctly whether it should be a logogram or a phonetic reading.
• The same type of error occurs with determinatives. For example, can be a logogram KUR (‘country’), or the determinative ^kur which appears before names of lands (and some cities).
• errors appear more often with compound signs. This is not surprising, considering that HMM takes into consideration the two previous signs and MEMM takes into consideration the two previous signs and the two following signs: if there is a complex sign made from two or three signs, it reduces the observable context.

It is important to note, however, that some of these apparent errors are to be evaluated on a case-by-case basis. We need to remember that the choice between reading a sign as a logogram or a determinative is a matter of scholarly convention based on lexical and contextual analysis. For example, some scholars treat KUR as a logogram, transliterating: KUR Hat-ti and some treat it as a determinative: ^kur Hat-ti. The meaning is not changed (“the land of Hatti” or simply “Hatti”).

The same can also apply to choosing between a logogram or a phonetic reading, particularly in proper names. For example, the name of the temple of Marduk in Babylon, Esagila, may be transliterated E₂.SAG.IL₂ or e₂ -sag-il₂.

Four additional case studies

After evaluating the successful results of the algorithms, we selected four case studies from later and earlier periods. Our purpose was to examine the accuracy of the algorithms on texts from corpora the models had not observed. Therefore, we wanted to examine if our models can be used on unfamiliar texts in order to speed up the process of mass transliterations. The principles of cuneiform writing, as elaborated upon, remained the same throughout its usage, but there are naturally variants in common signs and common sign readings, spelling, vocabulary, and formulae used.

We selected the four following texts from two sub-projects in the OIMEA project on ORACC, The Royal Inscriptions of Assyria online (RIAo) and The Royal Inscriptions of Babylonia online (RIBo). We chose four royal inscriptions from different periods, two of which are written in a different dialect and two are written with Babylonian orthography (as opposed to the Assyrian one of our training corpus):

• A Middle-Assyrian Royal Inscription of Aššur-rem-nišešu I, ca. 1398-1391 BCE, found in Assur (link).
• A Middle-Babylonian kudurru (boundary stone) Royal Inscription of Nebuchadnezzar I, ca. 1125-1104 BCE, found in Sippar (link).
• A Neo-Babylonian cylinder Royal Inscription of Nabonidus, ca. 551-542 BCE, found in Sippar (link).
• A Hellenistic Royal Inscription of Antiochus I, ca. 281-261 BCE, found in Borsippa (link).

The Results of the models are summarised in Table 3. They are very satisfactory, considering the differences between the case studies and the corpora used for learning. The complete output of the algorithms for these texts is published alongside our article, following a full qualitative evaluation, in S2 File. In what follows we present a summary of said qualitative evaluation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Accuracy on case-studies.

https://doi.org/10.1371/journal.pone.0240511.t003

We have noticed, unsurprisingly, that the errors referred to in section Error Analysis are the most common. For example, segmentation errors are around 12%-17% in HMM, 11%-16% in MEMM, and 4%-9% in BiLSTM. Errors which need to be evaluated on a case-by-case basis (i.e. logogram/determinative or logogram/phonetic) are around 1%-3%. Additionally, some errors are predictable and can be manually corrected. One such example is sign readings the algorithms had not seen before: in Nabonidus’ Inscription ii l. 13, the first sign of the word kallatu (“bride”), , is unknown to the algorithm with the reading kal. HMM offers the reading reb, MEMM offers the reading KAL, and BiLSTM the reading a. As can be seen with this example, the models have two fall-back options: HMM and MEMM will use another reading of the sign from the dictionary. BiLSTM, on the other hand, has a “wild card” aspect, where it may offer a sign-reading based on the context and not on the sign-values in the dictionary, since it is not limited to the dictionary. In future, we will flag these readings and give the user the option whether they prefer to be given the best contextual reading or the best contextual reading when limiting the options to within the dictionary.

Furthermore, we have noticed that even when the algorithms output errors, the errors stay localized and they do not create a chain-reaction of errors. Were it otherwise, the algorithms would be of no use for corpora they have not learned. This should not be taken for granted, considering the algorithms are context-based.

To conclude, our models will not replace the scholar’s work. Rather, they will provide considerable assistance in speeding up the scholar’s work on transliteration and segmentation. Since our models provide 70% or more correct readings on unlearned corpora and 97% correct readings on learned corpora, minor errors like those expanded upon in S2 File can be easily corrected, and one can dedicate their time to lines which are more difficult to decipher (both for the algorithms and for scholars).