Digital Preservation and How to Retrieve Data
Overview of digital preservation and how you can retrieve data
Table of Contents
1. Introduction
2. Technology
2.1. Security, integrity and authenticity 2.2. Retrieval process
3. How to retrieve data from this film?
3.1. Read the Representation Information 3.2. Build the decoding software 3.3. Capture frame images 3.4. Decode frames 3.5. File format specifications / file readers
1. Introduction
At the end of the 20th century, users of digital storage technologies faced a difficult challenge due to the exponential growth of data produced, the vulnerability of the storage mediums and the ever-changing application software. This challenge forced users to implement strategies for data preservation based on complicated software and hardware migrations. These strategies proved to be expensive and risky, but there were the only available options at the time. Piql answered to this challenge creating a holistic data preservation solution where data integrity and authenticity was preserved using a solid medium with a well-documented lifetime and implementing features which allow users in the future to recreate the reading technology using the information written on the medium.
The solution foundation was a technology based on storing digital data on a photosensitive polyester film creating an ultra-secure, migration-free, long-term preservation solution for digital data. This digital preservation medium is designed to last over 500 years and allows to interpret the data despite software obsolescence. Piql combined the longevity of the polymers and silver halides with the ability to store the instructions on how to decode the information stored on the film in human readable format. This document describes briefly the data preservation technology, the steps taken to ensure the authenticity of the data, and dives into the process to retrieve the content from the film.
2. Technology
Piql’s technology uses an optical recording medium for digital storage. This medium is a 35mm width polyester film coated on one side with a gelatine emulsion containing microscopically small light-sensitive silver halide crystals. The silver halides are darkening when exposed to light, and Piql used this characteristic to expose binary data in frames along the film. Both components of the film, polyester, and silver are extremely stable, and they are proved to last over 500 years when stored properly (film environmental storage recommendations are 21 Celsius and 50% Relative Humidity).
The software technology used to decode the data is open source, freely available and was tested using several operating systems. The hardware technology to capture data from the film is widely available, and the only requirement is an image capturing device able to sample images as a 2D array of color intensity values.
There are three hardware components at the core of the solution. The piqlWriter, a machine using photons to transfer data onto the film medium. The piqlProcessor, a machine where the written information on the film is chemically fixed. The latent silver halide particles exposed to light are converted into metallic silver and fixed to ensure image permanence. The piqlReader, a machine reading frames from the film and converting them into sampled images which are decoded back to digital data.
For retrieving the data from a film, the only component needed is the piqlReader. However, the piqlWriter has been playing a bit part in the preservation process providing integrity and authenticity to the data.
They are several technology elements in the piqlWriter. The core technology element that provides data reliability on film is the imaging technology. The imaging is based on a Texas Instruments1 Digital Lightning Processing (DLP) imaging sensor with 4K resolution (4096 x 2160 pixels), consisting on more than 8’800,000 micro-mirrors able to write pixels of 6μm size. The piqlWriter uses a monochromatic green LED light which is modulated by the DLP. The DLP micromirrors tilt either toward the light source (ON) or away from it (OFF) creating a light or dark pixel on the projected film surface.
For Piql’s digital application, every pixel represented a binary data container. This exposing technology allows writing pixels with excellent contrast allowing to achieve high data density per frame. In this was, it is possible to reproduce the binary stream of the client digital content. After exposing the film, the pixels’ value cannot be changed, providing a true WORM (Write Once Read Many) storage medium. The Imaging technology, combined with the software and the production processes makes the data content in this film secured, integral and reliable.
2.1. Security
2.1.1 Security, integrity and authenticity
Piql makes sure clients’ valuable digital data is safe and accessible, irrespective of future financial capabilities and technological developments. Piql Preservation Services uses an OAIS2 compliant turnkey solution to provide clients with a secure, accessible and migration-free solution for digital preservation. The OAIS reference model provides the guidelines for creating a Piql digital repository. This repository must fulfill the criteria for which clients entrust their data to Piql technology. These criteria are data security, integrity, authenticity, and reliability. Data security was ensured during the entire preservation process, from ingestion to physical storage. When transferring data, mechanisms for secure data transfer are applied. There are secure online data transfer protocols and secure physical transport services. The online transfer protocols used are HTTP over SSL, SSH FTP, and secure VPN tunnels. The advantage of these protocols is the ability to leverage a secure connection to transfer files. The Piql system was configured to work in a closed network environment. The receiving data server in Piql was in the Demilitarized Zone (DMZ) protected by firewalls with restricted access ports.
Only one trusted person had access to client files and is responsible to ingesting the files into the Piql system. The physical transfer service was provided on a ruggedized and shockproof hard drive with 256-bit encryption support with USB interface. The client entrusted one person to unencrypt the hard drive when arriving at Piql
Independent of the transfer method, the client had the option to encrypt the data content. The content could be preserved on film encrypted or unencrypted. The client always has control over the data.
After the data arrives securely to Piql, the data is encoded, exposed on film and verified in a secured environment. Piql infrastructure is placed behind a firewall without access to the public. There is not unauthorized access to the data, and it is limited to two people. As soon as the data arrives at Piql, an encoding process creates a virtual reel of two-dimensional images representing the binary stream of the files. These images are sent to the piqlWriter to be exposed on film and developed. Then, the integrity of the data content is verified and finally sent to physical storage.
After this last process, the files are preserved in an offline medium which cannot be hacked or tampered. As mentioned previously, the film is a Write Once Read Many medium, meaning it is not possible to modify the content.
The warehouse has security protocols and safe guarding devices, such as employees background checks, security cameras, intrusion alarms, disaster prevention systems and energy backup systems. The films are stored and protected in a customized container (piqlBox), that has been tested for the same longevity properties as the film. The warehouse has tight environmental settings, so the film is stored in optimal conditions (21 Celsius and 50% Relative Humidity). When the film is stored by the client, storage recommendations are provided to the client.
1 Texas Instruments Inc. (TI) is an American technology company that designs and manufactures semiconductors. In 1987, TI invented the digital light processing device (also known as the DLP chip). At the time of writing the film the website is: http://www.ti.com/ 2 OAIS reference Model (Open Archival Information System) described in ISO 14721:2003. At the time of writing the film the link to the standard is http://public.ccsds.org/publications/archive/650x0m2.pdf
2.1.2. Integrity
The Piql system is designed to prevent unintentional changes to information and ensure all data is written and retrieved with the highest possible data integrity. Before writing the data, Forward Error Correction (FEC) techniques and checksums are used to make sure the accuracy and authenticity of the data is kept over the entire preservation process. The solution implements data redundancy applying standard FEC algorithms (Cross-Interleaved Reed–Solomon Coding (CIRC)) used by traditional storage mediums like CD, DVD, Raid disk, etc. The FEC algorithm used is is provides redundancy within a frame, so if a frame is scratch or have areas with mechanical damage, still the information can be recover. The data is also distributed along several frames (data splitting), so it is possible to lose entire frames and still able to decode the data.
The second element to provide integrity is the checksums. A checksum or hash sum is a small-size datum computed from an arbitrary block of digital data for detecting errors that may have been introduced during its transmission or storage. SHA-1 and CRC64 checksums are used to ensure data integrity. SHA-1 (Secure Hash Algorithm 1) checksum was originally developed as a cryptographic hash function, but it is widely used for ensuring that the data has not changed due to accidental corruption. In SHA-1, given 10.000.000.000 single files, the chance of collision is ~3,4-29.
The first integrity step in the preservation process is the client calculating the SHA-1 checksums of their archival package before transferring to Piql. After the file is received, the checksum (SHA-1) is generated for the file, sent back to the client for verification and saved in the Piql database for final integrity check. After data transfer verification approval, the file is decoded and split into small elements of information (binary form) that fit into a frame. For each frame to be written, another checksum (Cyclic Redundancy Check - CRC64) is generated and added to the data frame before sending it to the piqlWriter. CRC64 checksums are well suited for the detection of burst errors during transfer, and it has the probability of one block colliding is ~1,8-19.
The piqlWriter firmware has a built-in checksum generation, and when the piqlWriter receives a frame, it calculates the CRC64 checksum and compares it with the CRC64 checksum it receives. In this way, each frame checksum is verified assuring data integrity during transmission and then written on film.
There is a final step of data verification before physical storage. After the reel is exposed and processed, the reel is scanned in the piqlReader, and the sampled images are decoded back to digital files. The files (SHA-1) checksums are calculated and compared with the ones stored in the database. After successful verification, the original files are deleted from the database, and the reel is sent to the physical storage.
The goal of Piql is to ensure data integrity throughout the entire hardware chain.
2.1.3. Authenticity
There are several layers of authenticity in the Piql film. They range from the physical medium until the data preserved.
The physical medium The manufacturer of the film proofs the authenticity of the stock writing an edge signature to each film stock. This edge signature is written in the margin area outside the perforations and describes certain parameters that provide an accurate picture of where and when the film is manufactured (see figure 1).
The edge signature is exposed on film during the film manufacturing process using specialized equipment in dark conditions. Commercial film recorders cannot expose outside the perforations, and they cannot expose on the format the edge signature is designed. This specialized equipment placed in an intermediate production process makes tough to copy the signature after the film has left the factory.
Figure 1. piqlFim edge signature manufactured by Kodak
On figure 1. the edge signature describes the following parameters: piqlFilm by Manufacturer cccc eee rrrpp ss ffff yyyy
Where: • Manufacturer = Company manufacturing the film • c (4 digits) = product code (manufacturer code) • e (3 digits) = emulsion (manufacturer emulsion code) • r (3 digits) = roll number (master roll) • p (2 digits) = part number (part in the master roll) • s (2 digits) = slit number (slit from the master roll/part, 1 to 38 from left to right) • f (4 digits) = footage (incremental foot numbering along the slit) • y (4 digits) = year (manufacturing year)
These parameters answer the following questions: • When the film is produced, • From what master roll is coming from, • Which slit from the master roll belongs to, • What base and what emulsion was used, their composition and their production date • Where within a reel, a problem is coming from?
Tampering a reel is tough, the tampering production facility does not have the knowledge of the chemical combination of the emulsion and the base and needs the know-how to produce a very high resolution and low noise film. Besides, expensive equipment must be designed and implemented to copy the edge signature.
Age determination methods There are mechanical or non-mechanical methods applied to the film which serves as a clock to proof the age of the film. Several options are provided to the clients, such as Carbon 14 analysis3, radioactive signature4 or rare earth metal5 inserted in the base. These methods are optional to the client and provide an extra layer of security ensuring the authenticity of the medium. This document does not cover these methods. It only highlights the options available to clients to proof medium authenticity.
Data authenticity There are digital signature technologies that provide data authenticity proof. These methods generate a time stamp which is verified by the client and others. After the data is verified, it is written on film with this digital signature. When the data is restored, the authenticity of the data can be proved using this signature.
At the time of writing this document there was an emerging technology called “Blockchain.6” This technology used a decentralized model where multiple public nodes verify the transactions (blocks). Each verification timestamp is linked to a previous block. By design, blockchains are inherently resistant to modification of the data — once recorded, the data in a block cannot be altered retroactively. This document does not cover any of these methods but highlights the options for data authenticity on film.
3 Sheridan Bowman, Radiocarbon Dating, University of California Press, 1990 4 Felix Gradstein, James Ogg , Lethaia, An International Journal of Palaeontology and stratigraphy. Article: Geologic Time Scale 2004 – why, how, and where next!, Volume 37, issue 2, June 2004, pg 175-181 5 Voncken, J.H.L., The Rare Earth Elements. An Introduction, 2016 6 Melanie Swan, Blockchain. Blueprint for a New Economy, 2015
2.2. Retrieval process
The retrieval process could occur at any time; the procedure changes if this is done while the technology is available or when the current technology has faded.
The current retrieval workflow allows the client to search for, and access files using web browser application and the piqlReader. When the file is selected, the piqlFilm is retrieved from physical storage and placed in the piqlReader. The piqlReader reads the control Frame (first frame) and provides all parameters for decoding the frames. Then it read the Table of Content, identifies the frames corresponding to the file requested, and forwards the film until those specific frames. Consequently, the frames are scanned, decoded and the file is restored; the file is verified calculating its checksum and comparing it to the checksum written on the film’s Table of Contents. After a successful verification, the file is made available to the client for download. The following chapter explains the steps to follow to restore the data from the film when the technology has faded out.
3. How to retrieve data from this film?
The following section will guide you on how to decode this preservation medium without having Piql technology available.
3.1. Read the Representation Information
The first step to decoding the digital data on this film is understanding how the information is structured on the reel (reel and film terms are used interchangeable along this document). Representation Information is a section of the reel represented by documents written in human readable format (visual format). The document you are reading now is part of the representation information. To have a detailed overview of what documents are on this film, forward the film until finding a document called Representation Information Format Description.
If you are reading this document, it means you have used a magnifying glass, a film projector, a microfilm monitor, or simply you have taken pictures of the film frames. This ability to write readable information on the medium is the advantage of the film, all metadata needed to decode the data is self-contained and easily readable.
In addition, the Representation Information provides context to the data. At the beginning of the film you might have read information provided by the owner of this film. This information allows you to understand what kind of content you are about to decode.
After you have identified the documents written on this film, proceed to read the Generic Preservation Reel Structure Specification document. The purpose of this document is to define a generic preservation reel structure, which organizes digital and visual data on an Optical Storage Medium (OSM) based on photo-sensitive, micrographic film. This document describes both the physical and logical reel structure. The goal of the structure is to ensure that the stored data remains complete and accessible in the long-term; spanning several centuries. An optional document to read is the Generic 4K Frame Format description document. it describes the structure and properties of a frame which represents a storage unit in the physical archival medium.
After understanding what the film is about and how the information is structured, there are two steps that can be performed in parallel. These steps are, creating the software application to decode the data and the process of capturing the frame images. These steps are described below.
3.2. Build the decoding software
In the Representation Information section, there is an open source decoding software (named Unboxer test application) document written in a code editor in human readable format. The software compiles in any computer operating system supporting C language. The C language is a programming language creating in 1972 and became one of the most widely used programming languages supported by all the widely-used compilers. Together with the open source decoding software code, there are two supporting documents, a Source Code (Unboxer Library) description document explaining the software library, and an install description document.
The first step is writing the code into a code editor, it is possible to type it or OCR (Optical Character recognition.) If OCR is used, double check the results as the OCR technology might not be able to recognize some characters, causing errors while compiling the code. After the code is extracted, proceed to compile it in a C language compatible Operating System. The result is an application that can input an image of a two-dimensional array of color intensity values and output a binary data object. The input images are the captured frame images on the next step.
3.3. Capture frame images
Frames are the rectangular areas along the reel that contain the data. The frames are digitized and ingested into the Unboxer software. To digitized a frame(s) in a reel you need to use a film scanner or any image capture device using a backlight source to illuminate the frame. The image capturing device needs to have a proper sensor resolution and a proper optical system to be able to capture and zoom into an entire frame with the required resolution. Read the document “Data Retrieval Technology” document in the Representation Information to understand the parameters you need to consider when choosing a capturing device. After achieving the quality required per frame, input the images into the Unboxer application created in the previous step.
3.4. Decode frames
Now, you have the decoding software and the frame images. The first step is to check if the software works as expected. Use the first frame image from the film for this verification. The first frame of the film is the Control Frame. This frame describes, in digital format, how to decode the rest of the reel. The Control Frame is stored in a single frame in a lower resolution using only white and black pixels (1-bit writing). The Control Frame is also written in a visual format, so it is possible to compare the outcome of the application with the visual information. After decoding the Control Frame, input all frames of the film, so the result is the binary object representing the archival package. This package contains all data files in the film.
3.5. File format specifications / file readers
After decoding the data package, it is likely that some of the file formats of the data preserved are not available any longer. The software to read those file formats might not exist. For this reason, in the Representation Information, Open Source file readers (if available) and file format specifications (if accessible to the public) have been added. A programmer literate person can recreate a file format reader following the file format specification. There are two open source file readers included in the Representation Information; a POSIXTAR7 application (the data package is contained in this file format) and a PDF reader8 (PDF/A is a preservation file format standard used for displaying documents). Other file format readers or specifications are written on the film based on the digital data preserved and its public accessibility.
Now you have all elements to retrieve the files within this film.
7 The TAR file format is an archiver program which stores files in a single archive without compression. This source code is structured using the POSIX.1-1998 archive format standard. The application has been written by Piql using POSIXTAR libraries. 8 The PDF reader used is Xpdf software which is copyright 1996-2014 Glyph & Cog, LLC. (http://www.foolabs.com/xpdf/). Xpdf is licensed under the GNU General Public License (GPL) version 2 or 3.