MSAM Imaging Data Archive Peer Review

Jan 11, 2023 — Mar 2, 2023

MSAM (Mastcam Stereo Analysis and Mosaics) is a PDART from 2016 that processed MSL Mastcam data from landing through Sol 2579.  It took longer than anticipated to prepare the data, but it is finally ready for review.

The MSAM project conducted a preliminary, informal Peer Review in Feb. 2020.  This provided an opportunity for users (including some of you in this review) to comment on the actual data contents before production processing commenced (it did not include PDS4 labels, just the actual data products).

Production processing is now complete, and there is not enough funding for reprocessing, so we unfortunately cannot entertain any change requests for the data itself during this second, formal Peer Review. However, the documentation is fair game, as are the PDS4 labels (as converted from the VICAR labels) and bundle/collection structure.  That said, if you find deficiencies in the data, please report them, as we can add appropriate notes to the documentation.

The PDS4 label structure closely matches the Mars 2020 image labels, so the labels should look familiar to some of you.

The goals of this Peer Review, as with any PR, are to ensure that:

• The data are scientifically useful
• The metadata correctly and comprehensively describes the data
• The supporting documentation is complete and accurate
• The archive wholly complies with PDS4 standards

Not all review members are expected to comment with respect to all of the above objectives.  The group, as a whole, will accomplish this.

Ultimately, your review of this bundle will consist of a written set of comments, clarifications, and discrepancies that, if implemented and/or corrected, would improve the archive.  These written statements need to be provided to the Review Coordinator(s) by the "Comments Due" date in the Schedule.

GETTING STARTED

The MSAM archive bundle can be viewed in a hierarchical fashion here:

https://pds-imaging.jpl.nasa.gov/data/reviews/pdart/msam/deen_pdart16_msl_msam

Bundle organization

Within this bundle, there are five collections — effectively subdirectories with specific metadata and focus upon specific types of data.

To navigate the science data, go to the judiciously named data/ collection, and drill down through the sol subdirectories to find the data of interest.  The XML labels can be viewed directly in most Web browsers.

Note: The data created from the MSAM processing is quite large.  For the purposes of this review we are only exposing a handful of Sols — and even then, the volume of this review bundle is over 200GB.  The Sols under review were simply chosen to ensure that all data types were present.  The following Sols are contained within this review bundle:

• 00100
• 00300
• 00900
• 01400
• 02100
• 02579

The above Sols contain the full set of MSAM data, including stereo products and mosaics.  See the SIS for a description of all the data. The Peer Review bundle weighs in at about 220 GB;  the full MSAM data set is 38 TB.

The browse/ collection contains viewable image representations of some of the data products.  Generally, web browsers can directly display these images.  Marsviewer (described below) is recommended to access the bulk of the image data.

The calibration_camera/ collection will contain the full set of calibration, lookup tables and parameter files used to process the MSAM data.

The document/ collection will contain several artifacts that, together, document both the MSAM data and its pedigree with the MSL project data.

Lastly, the miscellaneous/ collection contains the Velocity templates used to create the detached PDS4-compliant metadata labels from the original attached VICAR/ODL labels found within each of the data artifacts.

Orienting yourself

Though this peer review covers a lot of data, there is a methodical path through it.  The first thing to look at is probably the SIS, found here:

https://pds-imaging.jpl.nasa.gov/data/reviews/pdart/msam/deen_pdart16_msl_msam/document/MSAM_SIS.pdf

The SIS itself is huge, you are probably better off skimming it and honing in on sections of interest rather than trying to read the whole thing.  Of particular note are the label tables described in Appendix B, they describe in detail all of the label items.  Start with this one:

https://pds-imaging.jpl.nasa.gov/data/reviews/pdart/msam/deen_pdart16_msl_msam/document/MSAM_SIS_Labels_sort_pds.html

(There’s also a PDF version, but the hyperlinks likely work better in the above HTML.)

Then it’s a matter of looking for the data you’re most interested in.

Providing comments

Reviewers are welcome to provide comments in any preferred format via email to the Review Coordinators. The Google Sheet linked below is available for your use, but is not required (each reviewer has a named tab within this Sheet for their own use).

https://docs.google.com/spreadsheets/d/1weqyOvwyVnf81hRNSPpSnG548HbqLcekJT-YaaD_VWA/edit?usp=sharing 

This sheet will ultimately be used to adjudicate reviewer comments and convey the Data Provider’s response to each of these comments.  So, however review comments are delivered to the Review Coordinators, they will be distilled and tracked here.

DATA ACCESS

Overview of access/interaction methods:

1. Possible interactive methods for perusing the imaging archive:

1. Directly interacting with this web site: you can simply go spelunking through the archive structures by clicking on the links.  This method is useful for deciphering the archive bundle structure.  It may also be the simplest way to examine the PDS4 labels.  
2. Browse image collections: the archive bundle includes a browse collection that contains representations of the data in easily rendered formats.  This may be more convenient than interacting with the delivered VICAR/ODL formatted data.  Note: These browse data exist for actual images; many derived products like XYZ images are not represented in the browse collection.  You can navigate through the browse imagery using these directories:

data/sol/nnnnn/opgs/rdr/browse/mosaic/

data/sol/nnnnn/opgs/edr/browse/mcam/

for each Sol.

As these are PNG formatted images, any standard viewer should work (assuming the very large mosaics can be managed by these viewers)

3. Use the Marsviewer application (as described below): Marsviewer allows the interactive perusal and investigation of the data in more detail and with more context.

For mosaics, it is NOT recommended to use Marsviewer. It may work with local mosaics, but some of them are too large for the Web server to efficiently handle.  We recommend the use of xvd and/or direct interaction with the browse images instead.  Marsviewer will show you what mosaics are available, however, and might work on the smaller ones such as finders.

Also, Marsviewer will not view PDS4 labels; simply download them directly or view them in a web browser.

4. Use the xvd image viewer client: xvd is the VICAR image display program.  It can easily view the largest mosaics, if you first download them to your computer.  xvd is available as part of the VICAR Open Source release, which can be found here:

https://www-mipl.jpl.nasa.gov/vicar_open.html

There is a Docker version available that should let you run xvd on any computer (particularly Windows, as it does not run natively on Windows).

For Linux or Mac, you can run xvd natively.

5. Viewing Meshes: Meshes are best viewed in the Meshlab program:

http://www.meshlab.net

although any 3D viewer that reads OBJ format should work.

Meshes are found in the directories:

data/sol/nnnnn/opgs/rdr/mesh/

Download the .obj, .mtl, and .png files for the desired mesh, put them all in the same directory, and import the .obj into Meshlab (all three files are required).

The Meshlab project files (.mlp) are a convenient way to load a whole set of meshes at once.  All of the relevant files must be downloaded to your local directory.  Note that some such sets of meshes are too large for meshlab and may crash the program.

Interactive perusal via Marsviewer

Marsviewer is an interactive, Java-based tool that understands the relationships between the various data products, their organization within the archive, and how to present the data in a visual manner.  Most importantly, it provides visualization for the non-image data types, such as XYZs and surface normals, that are stored in image format.  To get started, simply perform the following:

Windows:

Download the executable from the Peer Review site here:

https://pds-imaging.jpl.nasa.gov/data/reviews/pdart/msam/support/marsviewer-3.7.3pre1-install.exe

Run the installation wizard, specifying the location to which the application suite will be installed on your computer.  The default installation location is %PROGFILES%\JPL\Marsviewer\. (where %PROGFILES% should be replaced with the suitable ‘Program Files’ directory of your system).

After installation, you should see a ‘Marsviewer’ icon on your Windows Desktop.  Double-click the application icon to launch the tool.

Mac OS:

Download the DMG image from the Peer Review site here:

https://pds-imaging.jpl.nasa.gov/data/reviews/pdart/msam/support/Marsviewer_3.7.3_pre1.dmg

Once downloaded, the DMG file should be expanded. If not, then double-click on the file to expand it

Select and drop the Marsviewer (for search and display) and Jadeviewer (image viewer only) apps into your Applications folder. Marsviewer is used for browsing Mars images that reside on PDS servers. Jadeviewer is used for looking at a specific image data product on your computer.

Double-click the application to launch it.  If you get a security violation when doing that, try ctrl-click on the icon and then select Open.  It should present you with a dialog asking if you really want to open it.  Say yes.  This should only be needed once; a simple double-click should be sufficient for future invocations.

Completing Setting Up Of Marsviewer (both OS’s):

When Marsviewer starts, it presents you with a dialog box from which to select a “File Finder”.  You will need to use the following File Finder settings:

File/New File Finder will allow you to change the file finder preset to a different type.  The server hosting this data for the PR can be slow, so be patient during loading - especially with very large images.

LIENS/ERRATA/NOTES/COMMENTS

There are a handful of issues with the bundle as you see here for this review:

1. Some extraneous browse images are present that would not otherwise be present.  These browse images are found under the /rdr/ directories of various sols.
2. Some PDF documents are known to not validate as PDF/A.
3. Tau data (atmospheric opacity) was only available up to sol 2482. This cannot be fixed at this point, but the SIS will be updated to indicate this.  A default of 0.6 was used for sols past that point.
4. The Velocity templates (in the miscellaneous collection) are missing some last-minute changes to fix validation errors.  They will be updated in the final bundle.

SCHEDULE

REVIEWERS