§ 1. OVERVIEW
We define an encoding scheme as a system of binary codes that unambiguously represent the semantic content of manuscripts for purposes of data storage, data transmission, and reconstruction of an abstracted graphical view of the original manuscript. An encoding scheme differs from an electronically scanned image, which stores a 'diplomatic facsimile' of the manuscript, but does not abstract out the content of the manuscript in terms of neumes, text, and other markings. The principal purpose of semantic abstraction is to enable computer analysis of the content of manuscripts, such as correlation analysis and statistical analysis. Semantic abstraction and encoding may lend themselves to other, as yet unanticipated forms of processing in the digital realm.
Our target is primarily musicological analysis, not paleographic analysis, and so a precise encoding of a medieval scribe's writing style is not a priority. (Paleographers tend to do analysis visually from the original manuscripts or high-quality, diplomatic facsimiles; the usefulness of computer technology for their purposes is seen by us as limited, at least given the current state of computer technology.)
Tauber points out what seems to us a useful categorization of data representations into three cases: (1) logical; (2) graphical; and (3) performance. Regarding performance questions, our work is intended for musocological study, in which performance is not an important consideration. Beyond that, there are grave questions surrounding the whole matter of 'performance' of early- and mid-medieval chant. First, there is virtually no disagreement that ecclesial chant was an oral tradition for a very long time before and during the development of a system of music writing. Many scholars have argued that the earliest extant forms of notation served only as guides to the complex liturgical calendar and prescribed chants for each occasion, or as reminders to a cantor about chants that he and the entire schola knew intimately by heart.
Thus, early notation was not conceived as an exact documentation of melodies from which one who was unfamiliar with the chants could sight-sing. In this sense, it is anachronistic to view early manuscripts as having a close correlation to 'performance'. In any case, it would be several centuriies before music notation would be evolved to the point where it could accurately represent the musical features of a 'performance' in the manner that modern music notation does.
1.3. Basis of comparison
Current encoding schemes for medieval neume notation are 'lossy,' in that they focus on an individual researcher's anticipated uses of the data, and the encoding captures only the intension of that researcher's perspective of the data. Lossy data representations inhibit the sharing of data files between researchers. Because manuscripts are housed in such widely dispersed geographical locations, because each manuscript is a unique specimen, and because so much time is invested in encoding a manuscript, a 'lossless' data encoding would, we believe, significantly advance medieval musicology. For a lossless data encoding to be widely adopted by musicologists it must: (1) be comprehensive of musicologists' needs; and (2) be accompanied by a data-input program that is so easy to use that musicologists will be induced to use it in preference to whatever technique they now employ.
Certainly, if a data encoding scheme already exists that will accommodate our needs for medieval manuscripts, then that scheme should be adopted by us. Doing so would reduce the amount of work required in this project and capitalize on compatibility already present in the digital realm. An existing 'standard' representation for music cannot, however, be adopted by us based solely on its wide acceptance in the music industry; it must be evaluated on its own merits, or considered in terms of incidental motivations of compatibility with a 'standard.'
We summarize here the standardized encoding schemes developed for industry, and those that some individual musicologists have devised. Our anticipated conclusion is that the existing standards (used widely in the music industry) are architected for modern notation of music and fail to meet the needs of medieval musicologists. On the other hand, encoding schemes developed by musicologists (at least those we have seen so far) fail the test of being acceptably 'lossless'. Our goal here is to examine the domain of existing representations and determine whether it would be prudent to adopt (or adapt) one of them, or whether inventing our own representation is indeed warranted.
§ 2. STANDARD REPRESENTATIONS
The most widely used standard for digital encoding of music is the MIDI format (Musical Instrument Digital Interchange). It was designed primarily for controlling electronic instruments from a digital data stream. This model is intended to feed data in real time (that is to say, time-sensitive) streams. The MIDI data stream specifies such things as instrument selection, voicing selection, volume, pitch, note-on and note-off, and related performance-oriented information. It does not directly encode information about the graphical representation of written music. There is rarely an unambiguous mapping from MIDI code to musicographic representation, and many important factors about written notation simply find no accommodation in the MIDI standard.
Medieval music manuscripts record chant melodies, liturgical text, and various information not directly related to sound. The relationship between the text and neumes is of crucial importance, but MIDI provides no method for representing text. The entire perspective of music that MIDI takes is anachronistic to medieval chant: (1) chant melodies typically have a melismatic quality that defies quantization into discrete time-sequences of specific pitches; (2) any move toward assigning a tempo to a melody or even specific durations to notes is unjustified, since the oral tradition that was the basis of the manuscripts has been lost; (3) particularly in the case of older manuscripts, there is considerable debate and uncertainty as to the precise meaning of some of the written symbols (indeed, it is not clear that there was a very precise meaning conceived by their writers). In sum, MIDI makes assumptions about the nature of music that have very broad implications that are totally inappropriate to medieval chant and, in particular, to neume-manuscript representations of chant.
Since our project is mainly concerned with lossless representation of written music, with at most a peripheral interest in playback of sound, the MIDI standard can be dismissed as inappropriate and insufficient to our purposes.
Since MIDI has become somewhat of a lingua franca in computer applications to music, often the only way to convert from one music notation format to another is by intermediate translation to MIDI. One must then repair by hand the things that were lost in translation.
NIFF (Notation Interchange File Format) is a new standard for digital representation of written music. Invented in 1995 by a consortium of electronic music-editing software companies and various concerned individuals, its stated purpose is to provide a "nearly universal" representation for written music. Upon examination of the specifications for NIFF, however, it seems clear to us that the principals who framed its design philosophy have given heavy emphasis to the parameters of common practice (modern) music notation.
Some details: NIFF files are formatted in compliance with Microsoft's Resource Interchange File Format (RIFF) with the addition of a data type, called 'tag,' to the definition of a 'chunk'; chunks are combined into 'lists.' RIFF rules provide separate formats for Intel (PC) and Motorola (Macintosh) target CPUs; NIFF uses the Motorola-targeted format, and translation code is required for using NIFF files on a PC. The distinction impacts particularly the byte order of integers. RIFF is principally a binary format, but ASCII data representation is also accommodated. NIFF files use the latter option and are predominantly ASCII data representations (see example).
Tauber adds that, like TIFF format, "There will probably not be any two programs that make use of NIFF in exactly the same way," implying that significant I/O translation will be required for sharing data between music editing programs despite their agreement on a common file format.
NIFF format is used by the Lime and Neume programs, among others.
2.5. ISO/ANSI STANDARD HYTYME
HyTyme and its associated music standard known as SMDL, not complete at the time of this writing.
ENIGMA Transportable File (ETF). Finale can save and read files in this format. ETF files can be fairly large.
"Standard finale file" are not be usable on types of computers different from the one on which they were created.
This is a popular format for folk music.
XML-based formats are likely to be important in the future, but there are several different XML-based music notation formats around, none of which have actually been implemented in music notation software.
Gerd Castan has a very useful page giving information about various file formats for sheet music.