MARS EXPLORATION ROVERS
PROJECT
Operations Product
Generation Subsystem (OPGS)
Experiment Data Record (EDR)
and
Camera Reduced Data Record
(RDR)
Archive Volume
Software
Interface Specification (SIS)
VERSION 1.7
JPL
D-27086
July 25,
2005
Jet Propulsion Laboratory
California
Institute of Technology
TABLE OF CONTENTS
1. Introduction................................................................................................................................. 1
1.1. Purpose and Scope............................................................................................................... 1
1.2. Content Overview................................................................................................................ 1
1.3. Applicable Documents and Constraints.............................................................................. 2
1.4. Relationships with Other Interfaces.................................................................................... 2
2. Archive Volume Contents........................................................................................................... 2
2.1. Root Directory Contents..................................................................................................... 2
2.2. Data Directory Contents and Naming................................................................................. 3
2.3. Index Directory Contents.................................................................................................... 3
2.4. Document Directory Contents............................................................................................ 3
2.5. Catalog Directory Contents................................................................................................. 4
2.6. Label Directory Contents (optional)................................................................................... 4
2.7. Software Directory Contents (optional).............................................................................. 4
2.8. Calib Directory Contents (optional).................................................................................... 5
2.9. Geometry Directory Contents (optional)............................................................................ 5
2.10. Browse Directory Contents (optional).............................................................................. 5
2.11. Extras Directory Contents (optional)................................................................................ 5
3. Archive Volume Format.............................................................................................................. 6
3.1. Disk Format......................................................................................................................... 6
3.2. File Formats......................................................................................................................... 6
3.2.1. Document File Format.............................................................................................. 6
3.2.2. Tabular File Format................................................................................................... 6
3.2.3. PDS Label Format..................................................................................................... 7
3.2.4. Software File Format................................................................................................. 7
3.2.5. Catalog File Format................................................................................................... 7
3.2.6. Science Data File
Formats......................................................................................... 7
4. Archive Volume Generation........................................................................................................ 8
4.1. Data Transfer, Validation Methods, and Peer Review........................................................ 8
4.2. Interface Media Characteristics........................................................................................... 8
4.3. Backup and Duplicates........................................................................................................ 8
4.4. Labeling and Identification................................................................................................... 9
5. Support Staff and Cognizant Persons......................................................................................... 9
5.1. Data Providers..................................................................................................................... 9
5.2. PDS Contacts..................................................................................................................... 10
Appendix A – MER OPGS EDR AND CAMERA RDR ARCHIVE
VOLUME CONTENTS
Appendix B – MER OPGS CAMERA RDR MOSAICS VOLUME
CONTENTS
Appendix C – MER OPGS CAMERA RDR MESHES ARCHIVE
VOLUME CONTENTS
DOCUMENT CHANGE LOG
|
Change |
Date |
Affected Portions |
|
Initial Release,
Version 1.0 |
06-13-04 |
All |
|
Version 1.1 |
06-15-04 |
Updated Volume Set
Ids, in Appendices, to contain "OP" |
|
Version 1.2 |
06-23-04 |
Added Anaglyph
datasets, RMC dataset. Removed Hazcam mosaic dataset. |
|
Version 1.3 |
07-02-04 |
Removed references
to Mini-TES throughout the document. |
|
Version 1.31 |
07-11-04 |
Added browse file
.LBL requirement and renamed the dataset catalog files. |
|
Version 1.4 |
11-01-04 |
Added 6 new RDR
datasets. |
|
Version 1.5 |
03-01-05 |
Removed the 6 new
RDR datasets and added the SOLAR RDR dataset. Rearranged the DATA directory
structure for Appendix A. |
|
Version 1.6 |
05-09-05 |
Moved the .svf and
.rvf files to the EXTRAS directory. |
|
Version 1.7 |
07-25-05 |
Removed the .svf
and .rvf files from the EXTRAS directory as well as the RMC_SIS from the documents
directory. Instead, they will become a new volume of ancillary data. |
|
|
|
|
TBD ITEMS
|
Section |
Description |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ACRONYMS AND ABBREVIATIONS
|
APXS |
Alpha Particle X-ray Spectrometer |
|
ASCII |
American Standard Code for Information Interchange |
|
CODMAC |
Committee On Data Management And Computation |
|
CORNELL |
Cornell University |
|
DESCAM, DES |
Descent Camera |
|
DVD |
Digital Video Disc |
|
EDR |
Experiment Data Record |
|
HAZCAM, HAZ |
Hazard Avoidance Camera |
|
HTML |
HyperText Markup Language |
|
ILUT |
Inverse Look-Up Table |
|
IMG |
Image |
|
ISO |
International Standards Organization |
|
JPEG, JPG |
Joint Photographic Experts Group |
|
JPL |
Jet Propulsion Laboratory |
|
MB |
Mssbauer Spectrometer |
|
MER |
Mars Exploration Rover |
|
MI |
Microscopic Imager |
|
MIPL |
Multi-mission Image Processing Laboratory |
|
NASA |
National Aeronautics and Space Administration |
|
NAVCAM, NAV |
Navigation Camera |
|
PANCAM, PAN |
Panoramic Camera |
|
NSSDC |
National Space Science Data Center |
|
OPGS (OPS) |
Operations Product Generation Subsystem (Operations) |
|
|
Adobe¨ Portable Document Format |
|
PDS |
Planetary Data System |
|
RAT |
Rock Abrasion Tool |
|
RDR |
Reduced Data Record |
|
RMC |
Rover Motion Counter |
|
SIS |
Software Interface Specification |
|
TBD |
To Be Determined |
GLOSSARY
Archive – An archive consists of one
or more data sets along with all the documentation and ancillary information
needed to understand and use the data. An archive is a logical construct
independent of the medium on which it is stored.
Archive Volume, Archive
Volume Set
– A volume is a unit of media on which data products are stored; for
example, one CD-ROM or DVD-ROM. An archive
volume is a volume containing all or part of an archive; that is, data
products plus documentation and ancillary files. When an archive spans multiple
volumes, they are called an archive
volume set. Usually the documentation and some ancillary files are repeated
on each volume of the set, so that a single volume can be used alone.
Catalog Information – Descriptive
information about a data set (e.g. mission description, spacecraft description,
instrument description), expressed in Object Description Language (ODL), which
is suitable for loading into a PDS catalog.
Data Product – A labeled grouping
of data resulting from a scientific observation, usually stored in one file. A
product label identifies, describes, and defines the structure of the data. An
example of a data product is a planetary image, a spectrum table, or a time
series table.
Data Set – An accumulation of data
products. A data set together with supporting documentation and ancillary files
is an archive.
This Software Interface Specification is intended to be used by those who wish to understand the format and content of the Mars Exploration Rovers (MER) Archives. Typically, these individuals would be software engineers, data analysts, or planetary scientists.
The specifications in this document apply to the MER Operations Product Generation Subsystem (OPGS) Experiment Data Record (EDR) standard product archive volumes , MER Camera Reduced Data Record (RDR) standard product archive volumes, MER Camera RDR Mosaics archive volumes, and MER Camera RDR Meshes archive volumes that are generated by the MER Project.
The MER Archives are intended to be stored online for electronic distribution. The online version will conform to the structure described in this document. In addition, copies of the archives will be stored on physical media such as CDs or DVDs for long-term preservation. The requirements for these physical copies are described in section 4.
The MER OPGS EDR Archive volume set consists of the MER raw
data products acquired and used during ground operations. The MER Camera RDR
Archive volume sets consist of MER OPGS EDR derived data products. All archives
contain documentation and other ancillary material.
The MER raw data products are OPGS EDRs, produced from
telemetry data from instruments onboard the MER Project rovers. Telemetry data is
processed into data records (CODMAC Level 2), with attached dual PDS/VICAR
labels, by MIPL of the Jet Propulsion Laboratory. The MER Camera RDR products
are produced from OPGS EDRs for the Pancam, Navcam, Hazcam, and MI instruments.
MIPL is the producer of all OPGS EDR and Camera RDR data
records and is responsible for assembling the archives. The PDS is responsible
for validating the archives for compliancy of structure and format against PDS
specifications. The various MER instrument science teams are responsible for
reviewing the archive in terms of science validity and integrity.
This Software Interface Specification (SIS) describes the format, content, and generation of the Mars Exploration Rover (MER) Archives. Section 2, Archive Volume Contents, describes the general structure of archive volumes and the contents of each file. Section 3, Archive Volume Format, describes the file formats used on the archive volumes. Section 4, Archive Volume Generation, describes the procedure for transferring data products to archive media. Section 5, Support Staff and Cognizant Persons, lists the individuals and institutions responsible for generating the archive volumes. Finally, Appendices A-D, describe the specific identifiers, specifications, and structure of the archive volumes produced along with a listing of any relevant documentation such as the Data Product SISs and schedules for release of data products.
This Archive Volume SIS is intended to be consistent with the following documents:
1.
Mars Exploration Program Data Management Plan,
R. E. Arvidson and S. Slavney, Rev. 3, March 20, 2002.
2.
Mars Exploration Rover Project Archive
Generation, Validation and Transfer Plan, R. E. Arvidson and S. Slavney, JPL
D-19658 Rev. B, July 14, 2004.
3.
Data Product SIS
[Please refer to appendices for instrument specific SISs.]
4. Planetary Data System Data Preparation Workbook, February 17, 1995, Version 3.1, JPL D-7669, Part 1.
5. Planetary Data System Standards Reference, August 1, 2003, Version 3.6, JPL D-7669, Part 2.
6. ISO 9660-1988, Information Processing - Volume and File Structure of CD-ROM for Information Exchange, April 15, 1988.
7. Universal Disk Format Specification, Revision 1.02, August 30, 1996, Optical Storage Technology Association (OSTA).
This Archive Volume SIS could be affected by changes to the design of any of the MER standard data products (Applicable Document #3 ).
This section describes the general contents of the MER OPGS EDR and Camera RDR Archive volumes, including directory names, file names, file contents, file types, and organization responsible for providing the files. Volume set specific MER archive contents can be found in the appendices.
The MER Archives are organized with each volume set on a separate logical volume (Several small data sets may be stored together on one physical volume, and a particularly large data set may span more than one physical volume). Each logical volume includes the required directories listed below, and may or may not include some or all of the optional directories.
Files in the Root Directory include an overview of the archive, a description of the volume for the PDS Catalog, and a list of errata or comments about the archive. The following files are contained in the Root Directory.
|
File Name |
File Contents |
File Provided By |
|
AAREADME.TXT |
Volume content and format information |
(e.g., PDS Node or Instrument Team) |
|
AAREADME.HTM |
Hypertext version of AAREADME.TXT (optional) |
PDS Node |
|
AAREADME.LBL |
A PDS detached label that describes both AAREADME.TXT and AAREADME.HTM (optional, could be attached to AAREADME.TXT). |
PDS Node |
|
ERRATA.TXT |
A cumulative listing of comments and updates concerning all archive volumes published to date |
Data provider or |
|
VOLDESC.CAT |
A description of the contents of this volume in a PDS format readable by both humans and computers |
PDS Node |
Contents and naming scheme of the
data sub-directories for specific instruments is described in the appendices.
Data file naming format and nomenclature is described in each instrumentŐs
respective Data Product SIS [Applicable Document #3].
Files in the Index Directory are provided to help the user locate products on this archive volume and on previously released volumes in the archive. The following files are contained in the Index Directory.
|
File Name |
File Contents |
File Provided By |
|
INDXINFO.TXT |
A description of the contents of this directory |
PDS Node |
|
INDEX.TAB |
A table listing all data products on this volume |
PDS Node or Data Provider |
|
INDEX.LBL |
A PDS detached label that describes INDEX.TAB |
PDS Node or Data Provider |
|
CUMINDEX.TAB |
A cumulative listing of all data products on this volume and on previous volumes in this set |
PDS Node or Data Provider |
|
CUMINDEX.LBL |
A PDS detached label that describes CUMINDEX.TAB |
PDS Node or Data Provider |
The Document Directory contains documentation to help the user understand and use the archive data. The following files are contained in the Document Directory.
|
File Name |
File Contents |
File Provided By |
|
DOCINFO.TXT |
A description of the contents of this directory |
PDS Node |
|
DPSIS.ASC or .HTM |
The Data Product SIS as text or hypertext |
Data Provider |
|
DPSIS.PDF |
The Data Product SIS as a PDF file |
Data Provider |
|
DPSIS.LBL |
A PDS detached label that describes both DPSIS.TXT(HTM) and DPSIS.PDF |
PDS Node |
|
ARCHSIS.ASC or .HTM |
The Archive Volume SIS (this document) as text or hypertext |
PDS Node or Data Provider |
|
ARCHSIS.PDF |
The Archive Volume SIS (this document) as a PDF file |
PDS Node or Data Provider |
|
ARCHSIS.LBL |
A PDS detached label that describes both ARCHSIS.TXT(HTM) and ARCHSIS.PDF. |
PDS Node |
|
[*.ASC files] |
Other Documents |
Data Provider |
The files in the Catalog Directory provide a top-level understanding of the mission, spacecraft, instruments, and data sets. The files in this directory are coordinated with the PDS data engineer, who is responsible for loading them into the PDS catalog. The following files are found in the Catalog Directory.
|
File Name |
File Contents |
File Provided By |
|
CATINFO.TXT |
A description of the contents of this directory |
PDS Node |
|
DATASET.CAT |
Data set information for the PDS catalog |
Data Provider |
|
INSTHOST.CAT |
Instrument host (i.e., spacecraft) information for the PDS catalog |
MER Project |
|
INST.CAT |
Instrument information for the PDS catalog |
Data Provider |
|
MISSION.CAT |
Mission information for the PDS catalog |
MER Project |
|
PERSON.CAT |
Personnel information for the PDS catalog (Team and PDS personnel responsible for generating the archive) |
Data Provider |
|
REF.CAT |
References mentioned in other *.CAT files |
Data Provider |
|
SOFTWARE.CAT |
Software information for the PDS catalog |
Data Provider |
The Label Directory contains files that describe data format and organization. These files are referred to in the PDS labels that accompany the data products. They are "include" files that are intended to be parsed as if they were part of the PDS labels that refer to them. The following files are contained in the Label Directory.
|
File Name |
File Contents |
File Provided By |
|
LABINFO.TXT |
A description of the contents of this directory |
PDS Node |
|
[*.FMT files] |
Format files |
Data Provider |
The Software Directory contains utilities or application programs to aid the user in viewing or extracting data from the data product files. The following files are contained in the Software Directory.
|
File Name |
File Contents |
File Provided By |
|
SOFTINFO.TXT |
A description of the contents of this directory |
PDS Node |
|
Software files |
Software files, applications,
source code, etc. |
Data Provider |
The Calib Directory contains calibration files used to process the data products, or calibration data needed to use the data products. The following files are contained in the Calib Directory.
|
File Name |
File Contents |
File Provided By |
|
CALINFO.TXT |
A description of the contents of this directory |
PDS Node |
|
Calibration files |
Image Calibration Files |
Data Provider |
The Geometry Directory contains files needed to understand observation geometry. The following files are contained in the Geometry Directory.
|
File Name |
File Contents |
File Provided By |
|
GEOMINFO.TXT |
A description of the contents of this directory |
PDS Node |
|
Geometry files |
Spacecraft Geometry files |
Data Provider |
The Browse Directory contains reduced-size, easily viewed versions of data products to be used to help identify products of interest. The following files are contained in the Browse Directory.
|
File Name |
File Contents |
File Provided By |
|
BROWINFO.TXT |
A description of the contents of this directory |
PDS Node |
|
browse_image.JPG |
Thumbnail size versions of the full resolution image files |
Data Provider |
|
browse_image.LBL |
A PDS detached label that describes browse_image.JPG |
Data Provider |
The Extras Directory contains documentation, utility programs, or other materials that the user may find helpful, but that are beyond the scope of the required elements of the archive. The contents of this directory are exempt from PDS requirements for labeling, etc. The Extras Directory is intended for "value-added" material, handy to have but not crucial for understanding the data. An example would be a set of web pages for displaying the browse data. Since the directory is nonstandard, a thorough explanation of its purpose should be included. The following files are contained in the Extras Directory.
|
File Name |
File Contents |
File Provided By |
|
EXTRINFO.TXT |
A description of the contents of this directory |
PDS Node |
|
[other files] |
|
Data Provider |
This section describes the format of the MER OPGS EDR and Camera RDR Archive Volumes. Data that comprise the Archive will be formatted in accordance with Planetary Data System specifications [Applicable Documents #4 and #5].
Archive Volumes have a digital video disk (DVD) format that is compatible with the computer operating systems MS‑DOS, Macintosh, and SunOS. The volume format is in accordance with ISO 9660 level 1 Interchange Standard or level 2, if any file names are longer than 8.3. [Applicable Documents #6 and #7].
This section describes file formats for the kinds of files contained on Archive Volumes.
Document files with the .TXT suffix exist in the Root, Index, Software, Catalog, Document, and Label directories. They are ASCII files which may have embedded PDS labels. Lines in a .TXT file end with a carriage return character, <CR> (ASCII 13) and a line feed character, <LF> (ASCII 10). PDS recommends plain text files have line length restricted to 80 characters or fewer, including the <CR><LF>. This allows the files to be readable under various operating systems.
Documents in the Document directory may contain formatting and figures that cannot be rendered as ASCII text. Therefore each document is given in two formats, hypertext and PDF. The hypertext file contains ASCII text plus hypertext markup language (HTML) commands that enable it to be viewed in a Web browser such as Netscape Navigator or Microsoft Internet Explorer. The hypertext file may be accompanied by ancillary files such as images and style sheets that are incorporated into the document by the Web browser. The second format, PDF (Portable Document Format) is a proprietary format of Adobe Systems Incorporated that is frequently used for distributing documents. Adobe offers free software, Acrobat Reader, for viewing PDF files.
Tabular files (.TAB suffix) exist in the Index directory and in any data directory where the data consists of ascii tables. Tabular files are ASCII files formatted for direct reading into many database management systems on various computers. All fields are separated by commas and character fields are enclosed in double quotation marks ("). (Character fields are padded with spaces to keep quotation marks in the same columns of successive records.) Character fields are left justified, and numeric fields are right justified. The "start byte" and "bytes" values listed in the labels do not include the commas between fields or the quotation marks surrounding character fields. The records are of fixed length, and the last two bytes of each record contain the ASCII carriage return and line feed characters. This allows a table to be treated as a fixed length record file on computers that support this file type and as a text file with embedded line delimiters on those that don't.
All tabular files are described by PDS labels that are either embedded at the beginning of the file or detached. If detached, the PDS label file has the same name as the data file it describes, with the extension .LBL; for example, the file INDEX.TAB is accompanied by the detached label file INDEX.LBL in the same directory.
All data files in the archive have PDS labels as detached files or embedded at the beginning of the file. For examples of PDS labels for each type of data product, see the Data Product SISs [Applicable Document #3].
A PDS label, provides descriptive information about the associated file. The PDS label is an object-oriented structure consisting of sets of 'keyword=value' declarations. The object to which the label refers to (e.g. IMAGE, TABLE, etc.) is denoted by a statement of the form:
^object = location
in which the carat character (^, also called a pointer in this context) indicates where to find the object. The location is an integer representing the starting record number of the object (the first record in the file is record 1). Below is the format for the ^object definition.
^object
= n
where n is the starting record or byte number of the object, counting from the beginning of the file (record 1, byte 1).
Software
is provided in a Zip-compressed file with a detached PDS label as specified in
the PDS Standards Reference, chapter 20, Zip Compression. The Zip file includes
all files required to use the software, including user manuals.
Catalog files (suffix .CAT) exist in the Root and Catalog directories. They are text files formatted in an object-oriented structure consisting of sets of 'keyword=value' declarations.
Each line in a catalog file must be terminated by the two-character carriage-return/linefeed, <CR><LF>, sequence (ASCII decimal character codes 13 and 10, respectively). PDS requires catalog files have line length restricted to 72 characters or fewer including the <CR><LF>, to accommodate PDSŐ internal database requirements.
See the Data Product SIS for each OPGS EDR product for descriptions of the data file formats.
Data provided to
the MER science teams will meet the specifications detailed in the Data Product
SISs [Applicable Document #3].
The OPGS EDRs and Camera RDRs will be generated by MIPL of the Jet Propulsion Laboratory. MIPL is responsible for the assembly and production not only of PDS formatted data, but of complete, PDS-compliant archive volumes, e.g. DVDs. The MER science teams, with the assistance of their respective PDS Nodes, are responsible for the documentation and ancillary files pertaining to their data.
Before final delivery of the archive to PDS Central Node, the PDS Nodes will conduct both peer review and validation. Peer review may be performed on the volume set as a whole, but validation must be performed on every individual volume. The Central Node may perform additional validation once the volume has been received.
The purpose of the peer review is to confirm that the archive will be useable by members of the science community, both present and future, who are not familiar with the mission and/or instrument. Reviewers include members of the PDS, a distributed representation of the project science teams, and members of the science community not associated with the mission.
The purpose of validation is to verify that
each volume adheres to PDS standards and to this Archive Volume SIS.
All volumes in the MER Standard Product Archive conform to
ISO 9660 standards [ISO 9660, 1988] and UDF standards [OSTA UDF Specification,
Rev. 1.02, 1996].
At an absolute minimum, three physical media copies of each volume will be produced on write once physical media and delivered to the PDS Central Node, the respective PDS Node, and the NSSDC.
Until validated masters have been produced,
or three validated write-once copies have been received by the PDS, volume
contents shall be stored by the archive producer either on magnetic disc or
write-once physical media.
The two MER rovers
are identified as shown in the table below.
|
Rover Name |
Rover ID |
Launch
Date |
Arrival
Date |
|
Spirit |
MER 2 (MER
A) |
June 10,
2003 |
January 4,
2004 |
|
Opportunity |
MER 1 (MER
B) |
July 7,
2003 |
January 25,
2004 |
Please refer to appendices for instrument specific labeling scheme of archive volumes.
Ray Arvidson
MER Data and Archive Working Group Chair
Washington University
1 Brookings Drive, Campus Box 1169
St. Louis, MO 63130
USA
arvidson@wunder.wustl.edu
Jim Bell
MER Pancam Payload Element Lead
Dept. of Astronomy
402 Space Science
Cornell University
Ithaca, NY 14853
USA
jfb8@cornell.edu
Ken Herkenhoff
MER Microscopic Imager Payload Element Lead
U.S. Geological Survey
2255 N. Gemini Drive
Flagstaff, AZ 86001
USA
kherkenhoff@usgs.gov
Rudolf Rieder
MER APXS Payload Element Lead
Max Planck Institute
rieder@mpch-mainz.mpg.de
Gstar Klingelhfer
MER Mssbauer Payload Element Lead
University of Mainz
klingel@mail.uni-mainz.de
Steve Gorevan
MER Rock Abrasion Tool Payload Element Lead
Honeybee Robotics
gorevan@honeybeerobotics.com
Rafael Alanis
PDS Imaging Node
Jet Propulsion Laboratory
MS 168-414
4800 Oak Grove Drive
Pasadena, CA 91109
rafael.alanis@jpl.nasa.gov
Ed Guinness
PDS Geosciences Node
Washington University
1 Brookings Drive, Campus Box 1169
St. Louis, MO 63130
guinness@wunder.wustl.edu
Myche McAuley
PDS Imaging Node
Jet Propulsion Laboratory
MS 168-414
4800 Oak Grove Drive
Pasadena, CA 91109
myche.mcauley@jpl.nasa.gov
Susan Slavney
PDS Geosciences Node
Washington University
1 Brookings Drive, Campus Box 1169
St. Louis, MO 63130
slavney@wunder.wustl.edu
Betty Sword
PDS Central Node MER Data Engineer
Jet Propulsion Laboratory
MS 171-264
4800 Oak Grove Drive
Pasadena, CA 91109
betty.sword@jpl.nasa.gov