Generic Preservation Reel Structure Specification
Overview of structure of the piqlFilm
1. Introduction
The purpose of this document is to define a generic preservation reel structure, that organizes digital and visual data on an Optical Storage Medium (OSM) based on photo-sensitive, micrographic film. This document describes both the physical and logical reel structure. The goal of the structure is to ensure that the stored data remains complete and accessible in the long-term; spanning several centuries. The structure is therefore designed to be self-contained, all the information needed to safely store, maintain and read back the information in the future is kept on the reel itself.
The physical reel structure is called Archival Reel Format (ARF) and the logical reel structure is called Archival File System (AFS). Both formats derive from several key properties of the storage medium:
- Micrographic film is a long-term stable medium (several hundred years in ideal conditions); no periodical migration of the stored data is necessary for long-term archiving.
- Micrographic film can store two types of data: Non-digital data (analogue, visual or human-readable content) and digital data (machine-readable content), as well as a combination of both (hybrid content).
- Micrographic film can only be written once, and read many times (true WORM technology); it is not possible to alter the data stored in the film later.
- Micrographic film can be read (retrieval of the stored data) independently of the technology used to record the data.
- Micrographic film is a sequential access medium.
- Both ARF and AFS are simplified formats designed for a non-erasable WORM medium that fulfills the above properties.
1.1 Version
The version number for ARF and AFS is in the form M.m.r, where M, m and r are natural numbers. With this three-numbered versioning it is possible to track the different revisions of the ARF and AFS specifications as well as the nature of the changes. The three components of the version number are updated as follows: • M: It is incremented when a major update has been made to the ARF or AFS specifications. A major update is a change to the specification semantics that is expected to break compatibility with older versions. • m: It is incremented when a minor update has been made to the ARF or AFS specifications. A minor update is a change to the specification semantics that is not expected to break compatibility with older versions (with the same M value). • r: It is incremented when a revision has been made to the ARF or AFS specifications. A revision is a change to the specification to improve the clarity of the format, without modifying its semantics.
1.2 Definitions and Terminology
AFS: Archival File System Visual Data: Data that can be read by the human eye. ARF: Archival Reel Format Digital Data: A stream of numerical values. The values must be interpreted to be understood. Frame: Rectangular area on the storage medium. Logical Frame: Describes the content of a frame in logical coordinates and pixels. Logical Reel Format: Describes how the data is organized on the reel, independent of the physical properties of the storage medium. OSM: Optical Storage Medium. The data on the storage medium can be read back using light. Preservation: The processes and systems needed to make sure digital and analogue data is accessible and trustworthy in the future. Physical Reel Format: Describes how data is organized physically on the optical storage medium. Micrographic Film: Optical Storage Medium based on high resolution photosensitive silver halides coted on a polymer film base. Both the silver halides and the polymer are extremely stable and the expected lifetime is over 500 years with correct storage conditions. ZRF: Zero Reference Frame. First content frame on the reel with frame index 0. This frame is also repeated at the end of the reel. ZRM: Zero Reference Mark. Structure inside the ZRF easily recognized by machine vision applications. Allows for automatic detection of the start of the reel.
2. Archival Reel Format
The physical storage medium is a high-resolution micrographic film wound into reels of any length. The Archival Reel Format (ARF) describes the structure of the reel. A reel is a linear storage unit which includes a Storage Area, used for storing content (visual or digital), with some extra film material at both ends to allow mounting on a film reader. The Storage Area is divided into a linear sequence of frames. The physical dimensions follow 35 mm motion picture film as defined in ANSI/SMPTE 139–1996.
2.1 Reel Structure
The structure of the film reel is shown in Figure 1.
**Figure 1. Film reel structure in ARF v0.0.3**
At the head of the reel there is a leading film strip of at least 130 cm length (leader), and a short blank film strip of exactly 6 perforations. Next, there is the storage area, a linear sequence of frames that stores the content. The first frame in the storage area is the zero reference frame with frame index 0. Finally, there is a blank film strip (tail) at the end of the reel of at least 130 cm.
2.1.1 Frame Structure
A frame is a rectangular area in the film used for storing digital and analogue content. The frame structure is shown in detail in Figure 2. Current version of ARF uses a full aperture 3-perforation frame on 35 mm film as defined by SMPTE-59, where the width of a complete full frame shall be 24.950 mm. The height of a complete frame shall be 13.500 mm. The start of each frame shall be centered between two consecutive perforations. The frames shall be strictly aligned to the perforations, but a variation of 50 microns from frame to frame is acceptable. The printed elements shall be located not closer than 100 microns from the inner perforation edge.
Figure 2. Film frame structure
Table 1. Key measurements of a frame (35 mm film perforated KS – SMPTE-59)
Width Height Perforation 2.8 mm 1.98 mm Perf pitch 4.72 mm Frame 24.92 mm 13.87 mm
2.1.2 Zero Reference Frame
The first frame in the Storage Area is the Zero Reference Frame (ZRF), used to mark the start of the Storage Area. The ZRF is an empty frame with a small black patch, the Zero Reference Mark (ZRM) that covers half of the width of the frame and is located aligned with the first perf of the frame. The height of the black patch is the same as that of a perf. The structure of the ZRF is shown in Figure 3.
Figure 3. Zero reference frame (ZRF) structure
From the key measurements of the ZRF, summarized in Table 1, the following quantities can be calculated:
3. Archival File System
The Archival File System (AFS) defines the logical frame that occupies a physical frame on the storage medium and divides the storage area of the ARF into logical sections, where each section contains 1 or more logical frames. The frames are addressed by their frame index, starting from zero. Some of the sections are optional and the order is not mandatory, apart from the ZRM and the Control Frame that must start on frame index 0 and 1 respectively. The sections are based on the properties of the ARF but the design is intended to be independent of the physical storage medium so the AFS could be used for other storage mediums as well if the physical format can be organized in a linear sequence of frames.
The structure of the AFS is shown in Figure 4:
3.1 Logical Frame Format
Defines a rectangular area of pixels where each pixel can have a bit-depth from 1 to 8. The logical frame format is described in Generic 4K Frame Format Description document. All frames on the AFS volume, apart from the ZRF adheres to this format.
3.2 Control Frame
The control frame describes, in digital format, how to decode the rest of the reel. The Control Frame is stored in a single frame, the first frame after the ZRF. It is divided into two sections: Administrative and Technical metadata. Administrative Metadata is administrative information such as short description of the data contents, author/creator name, creation dates, and so on. Technical Metadata is information required to decode the data contents in the rest of the AFS volume. It points to the frame with the Table of Content and has a description of the coding format used (boxing format). All the other digital frames in the volume shall be decoded with the boxing format defined in the Control Frame.
The ContentType parameter in Metadata Band of this frame shall be of type ControlFrame. The digital format used to store the Control Frame is XML. The XML schema is described in Appendix A.
The boxing format used to decode the Control Frame is one of the predefined boxing formats defined in Appendix B. If the boxing format is not known when decoding the Control Frame, all of the predefined formats shall be tried starting with the newest format and descending. The first boxing format that decodes the frames successfully is the valid format.
3.3 Visual Table of Content
The visual Table of Content (V-TOC) is a human readable index of the reel. It should index the content using the physical frame index. The format is specified by the user, but the recommendation is that it’s localized in the language of the reel content. The ContentType parameter in Metadata Band of this frame shall be of type Visual.
3.4 Representation Information
Metadata sections containing the information needed to decode and reproduce the digital files stored on the reel in the future. The information can be in visual or digital form. The format is specified by the user, but the recommendation is that it’s localized in the language of the reel content. The ContentType parameter in Metadata Band of this frame shall be of type Visual or Digital.
3.5 Digital Table of Content
The digital Table of Content (TOC) is an index for of the files stored in the visual and digital content sections. It also contains metadata for the files. The TOC is stored digital in the XML format described in Table of Contents Format description document. This section is repeated at the end of the reel to add recovery safety if the start of the reel is damaged. The ContentType parameter in Metadata Band of this frame shall be of type TOC.
3.6 Digital Content
The ContentType parameter in Metadata Band of this frame shall be of type Digital.
3.7 Visual Content
The ContentType parameter in Metadata Band of this frame shall be of type Visual.
4. References
- ANSI/SMPTE 139–1996. SMPTE STANDARD for Motion-Picture Film (35 mm) - Perforated KS
- ANSI/SMPTE 59 -1998. SMPTE STANDARD for Motion-Picture Film (35-mm) — Camera Aperture Images and Usage
Appendix A – Control Frame Format
The Control Frame is stored in XML 1.0 Specification. The XML format is described by a XSD schema, also a part of the XML specification.
XSD schema:
<?xml version="1.0"?><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace="http://www.piql.com/afs" xmlns:ns="http://www.piql.com/afs" elementFormDefault="qualified"> <xs:annotation> <xs:appinfo>Control Frame XML validation schema.</xs:appinfo> <xs:documentation> The control frame contains information about who to decode the rest of the reel. It has the boxing format used on this reel and points to the reel table of contents. </xs:documentation> </xs:annotation> <xs:include schemaLocation="boxing_format.xsd"/> <xs:complexType name="Tocs"> <xs:sequence> <xs:any minOccurs="0"/> </xs:sequence> </xs:complexType> <xs:complexType name="BoxerFormat"> <xs:sequence> <xs:element name="work" type="ns:BoxingFormat"/> </xs:sequence> </xs:complexType> <xs:complexType name="AdministrativeMetadata"> <xs:annotation> <xs:appinfo>Administrative Metadata</xs:appinfo> <xs:documentation>Metadata about usage and creation of the reel.</xs:documentation> </xs:annotation> <xs:sequence> <xs:element name="ReelId" type="xs:string"/> <xs:element name="PrintReelId" type="xs:string"/> <xs:element name="JobId" type="xs:string"/> <xs:element name="Title" type="xs:string"/> <xs:element name="Description" type="xs:string"/> <xs:element name="Creator" type="xs:string"/> <xs:element name="CreationDate" type="xs:dateTime"/> <xs:element name="Location" type="xs:string"/> </xs:sequence> <xs:attribute name="version" type="xs:string"/> </xs:complexType> <xs:complexType name="TechnicalMetadata"> <xs:annotation> <xs:appinfo>Technical Metadata</xs:appinfo> <xs:documentation>Metadata needed to decode the rest of the reel.</xs:documentation> </xs:annotation> <xs:sequence> <xs:element name="BoxerFormat" type="ns:BoxerFormat"/> <xs:element name="Tocs" type="ns:Tocs"/> </xs:sequence> <xs:attribute name="version" type="xs:string"/> </xs:complexType> <xs:element name="ControlFrame"> <xs:complexType> <xs:sequence> <xs:element name="AdministrativeMetadata" type="ns:AdministrativeMetadata"/> <xs:element name="TechnicalMetadata" type="ns:TechnicalMetadata"/> </xs:sequence> <xs:attribute name="version" type="xs:string"/> </xs:complexType> </xs:element></xs:schema>Appendix B – Boxing Format
The Boxing Format is stored in XML 1.0 Specification. The XML format is described by a XSD schema, also a part of the XML specification.
XSD schema:
<?xml version="1.0"?><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace="http://www.piql.com/afs" xmlns="http://www.piql.com/afs" elementFormDefault="qualified"> <xs:annotation> <xs:appinfo>Boxing format XML validation schema.</xs:appinfo> <xs:documentation> The boxing format describes the frame layout and the codecs used to code and encode the digital data stored in a frame. The format properties had a name and a value and are grouped into logical classes. The user is responsible for how to interpret a property value. </xs:documentation> </xs:annotation> <xs:complexType name="BoxingFormat"> <xs:sequence> <xs:element name="class" maxOccurs="unbounded"> <xs:complexType> <xs:sequence> <xs:element name="property" maxOccurs="unbounded"> <xs:complexType> <xs:attribute name="key" type="xs:string"/> <xs:attribute name="value" type="xs:string"/> </xs:complexType> </xs:element> </xs:sequence> </xs:complexType> </xs:element> </xs:sequence> <xs:attribute name="version" type="xs:string"/> </xs:complexType></xs:schema>
Appendix C – Table of Content Format
The TOC is a XML file automatically generated during reel preparation and it is stored in text format for easy visualisation.
ToC example:
TABLE OF CONTENT ================ Job ID: 71A2D51D-1C44-22D2-5745-5BF76A4CD6E5 Reel ID: A2402EA4-6937-3784-429B-C9D85AC24E1E Version: 1.0.1 Number of reels: 1 VISUAL LAYOUT DEFINITIONS ======================== <layoutId> <sections> <name> 00000 00010 "10 pages per frame" 00001 00001 "1 page per frame" <layoutId> <sectionId> <x> <y> <width> <height> <rotation> 00000 00001 00000 00000 00805 01046 000 00000 00002 00805 00000 00805 01046 000 00000 00003 01610 00000 00806 01046 000 00000 00004 02416 00000 00806 01046 000 00000 00005 03222 00000 00806 01046 000 00000 00006 00000 01046 00805 01046 000 00000 00007 00805 01046 00805 01046 000 00000 00008 01610 01046 00806 01046 000 00000 00009 02416 01046 00806 01046 000 00000 00010 03222 01046 00806 01046 000 00001 00001 00000 00000 04028 02092 000 FILE FORMAT DEFINITIONS ======================= <formatId> <PronomId> <Name> 00000 fmt/95 "Acrobat PDF/A - Portable Document Format" 00001 fmt/353 "TIFF - Tagged Image File Format" <formatId> <fileId> <type> 00000 00000 METADATA 00000 00001 DOCUMENT 00000 00002 SOURCE-CODE 00001 00003 METADATA 00001 00004 DOCUMENT 00001 00005 SOURCE-CODE REEL ==== Reel ID: A2402EA4-6937-3784-429B-C9D85AC24E1E Number of files: 23 DESCRIPTIVE FILES ================= <id> <parentId> <metadataId> <filename> 00000 0000N 00001 PDF-A-1.4.xml 00001 0000N 0000N PDF-A-1.4.pdf 00002 0000N 0000N PDF-A-1.4.tar 00003 0000N 00004 TIFF.xml 00004 0000N 0000N TIFF 6.0.pdf 00005 0000N 0000N libtiff.tar USER DATA FILES =============== <id> <parentId> <metadataId> <filename> 00006 0000P 0000N WorldWonders.tar 00007 00006 00004 New_7_Wonders_Winners_metadata.xml 00008 00006 0000N New_7_Wonders_Winners.tif 00009 00006 00004 Al_Khazneh_metadata.xml PAGE 12 00010 00006 0000N Al_Khazneh.tif 00011 00006 00004 Chichen_Itza_metadata.xml 00012 00006 0000N Chichen_Itza.tif DIGITAL DATA LOCATIONS ====================== <id> <uniqueid> <startframe> <startbyte> <endframe> <endbyte> <size> <date> <sha1 checksum> 00000 7874E91F-5637-809D-8E2D-914FB8B9E9E5 000135 00000000 000259 01209439 0217262080 2015-02-04T08:52:36 c7e72008458dc3e4cb3c73c0f548955234f49ead 00001 22F0DE4E-FF71-3DB6-531F-FC5313600F02 000135 00000000 000135 00001023 0000001024 2015-01-26T15:05:58 e38992c15182ca3259e06cd1b1084694b9796fed 00002 1F2ED6CD-53E9-0AF5-D05C-CAFED6ED5EF6 000135 00001024 000149 00784047 0025176064 2015-01-28T15:44:12 86bfec8a983cb22d65566416068cbb9889400c0c 00003 D24B808E-3673-068D-6D93-7915FF24D2AA 000149 00784048 000149 00785071 0000001024 2015-01-26T14:29:18 f2ff3d5434426984fddaab364bce471c210745ab 00004 4E5CEDA1-4144-7C2E-AB11-4263197EB18F 000149 00785072 000155 01539615 0011208704 2015-01-29T10:10:06 9cb24251e82165999421f8b1a9ca828a92f40599 00005 FEBDB6EF-724F-F6D9-1EA6-0191A1F52742 000155 01539616 000155 01540639 0000001024 2015-01-27T10:07:43 d00cf22efb71b8d447b74e294caecc5dc5a6e3e8 00006 5FF15E0D-04CF-A98C-7327-4B8EC73FE714 000155 01540640 000165 01487663 0017370624 2015-01-28T15:46:56 c8e1aa39ad58745f518fc2eb4384988f79630bb3 00007 605CE59F-A5E4-E705-A2CE-E4CABF996ADC 000165 01487664 000165 01488687 0000001024 2015-01-26T14:34:32 cfb540526d9f184fd6144a901656a4e920cbf7b9 00008 B2898182-6764-E798-9567-92241BE76B31 000165 01488688 000175 00272447 0016207360 2015-01-28T15:41:44 80e1983ac0679a8945f9f17a30f01428e817ece4 VISUAL DATA LOCATIONS ===================== <id> <layoutid> <startframe> <startsection> <section count> <dimensionW> <dimensionH> <overlap WxH> 00001 00000 000176 000000 000900 00000x00000 00000x00000 00004 00000 000256 000000 000100 00000x00000 00000x00000 00008 00001 000266 000000 000001 00000x00000 00000x00000 00010 00001 000267 000000 000001 00000x00000 00000x00000 00012 00001 000268 000000 000001 00000x00000 00000x000