NSSDC Interface – PDS Engineering

The National Space Science Data Center (NSSDC) supports Planetary Data System (PDS) long-term archive. This working group is to ensure that the interface between PDS and NSSDC is well defined to allow PDS to meet its long term perservation requirements.

Working Group

The NSSDC Interface effort initially did not an official working group. It consisted mostly of the Engineering Node staff working directly with the NSSDC staff. Eventually, various node staff were involved to support verification of the Multi-file Package Generator and Analyzer (MPGA) software provided by NSSDC to support electronic delivery. The MPGA was approved to be distributed to the Nodes in Decemeber 2007. The MPGA capability is restricted to 300 GBs in size. Both the packaging and transfer mechanism present a problem for data sets that are several terabytes in size. It was decided at the December 2007 meeting of the PDS Management Council that it is necessary to develop a new process for sending large data sets to the NSSDC. As a result, a formal NSSDC Delievery Working Group (NDWG) was formed to address this need. The WG members include: D. Crichton, E. Grayzeck, S. Hughes, C. Isbell, J. Kodis, E. Law, P. McCaslin, and T. Stein.

Milestones

Milestones include:

Finalize NSSDC large data set delivery process and software package

Current Status

Current working documents associated with this effort can be found on the NSSDC Delivery Working Group page of the PDS Collaboration Wiki. The same user name and password utilized for this site will allow access to that site.

Historical Documents

The NDWG has developed processes to submit, retrieve and replace archive volumes. Initially, data will be sent via ‘data brick’ (see spec below) to the NSSDC. Eventually, NSSDC is planning to extend this to include electronic delivery. NSSDC will be responsible for transferring the data from the data bricks to their archive system. The group also has defined delivery and submission manifests that include checksums and description of the volumes and submission. NSSDC has provided software to generate submission manifests. Initial test was performed by Geosciences (MRO/CRISM) and Imaging Nodes (ODY/THEMIS). Beta Test is currently being performed by Geosciences, Imaging, Rings and Atmospheres Nodes. Specification of the data brick used by Imaging Node: Western Digital My Book Pro II WDG2TP20000N 2TB 7200 RPM USB 2.0 / IEEE 1394a / 1394b External Hard Drive For more details, please go to Product Page. Submission Package Manifest Generator (Xman) software, configuration files, and Operations Guide are accessible at:

Submission Package Manifest Generator (Xman) Operations Guide – Sept 29, 2008
Submission Process – August 11, 2008
NSSDC presentation to MC – July 09, 2008
Sample manifest generated (.xml) – May, 2008
NSSDC presentation to MC – April 03, 2008
Draft Large data set delivery to NSSDC White Paper – March 14, 2008

The PDS-NSSDC MPGA software for Linux was released to the Atmospheres, Imaging and Small Bodies Nodes on March 19, 2007 for Beta Testing. The release included the Installation Guide and the Beta Test Plan/Procedure documents. The participated Nodes executed their tests by selecting Node specific volumes. The volume size ranges from 176KB to ~3.8GB. This test phase was concluded on April 16, 2007. The findings are documented in the Beta Test Report. The production version of the software for Linux and documentation were checked out by EN and SBN and was released it to the Nodes in December 2007. The NDWG was formed in Decemeber 2007 to address large data sets delivery process.

The historical milestones and meetings regarding the PDS-NSSDC Interface, along with their associated documents, are provided below in reverse chronological order:

MPGA Production Release Use and Installation Guide – August 10, 2007
Beta Test Report – April 16, 2007
Beta Test Kickoff Presentation – March 12, 2007
Beta Test Plan/Procedure – March 13, 2007
NSSDC MPGA Installation Guide – March 16, 2007
Testing (Beta Test at EN) – March 19, 2006 This is the test report detailing the results from the execution of the Beta Test plan at the Engineering Node. These tests were completed prior to making the MPGA software available to the Nodes for their own beta test.
Beta Test Kickoff – March 12, 2007 Presentation given to the participating Nodes (Atmospheres, Imaging and Small Bodies) via teleconference, regarding the beta test of the MPGA software.
Status Report – August 3, 2006 Presentation given to the PDS Management Council at the face-to-face meeting at the Imaging Node (USGS).
Testing (Second Round) – April 27, 2006 This is the test report from the second round of testing which verified the ability of the MPGA software to package volumes larger than 2 gigabytes. This round of testing utilized the same test plan/procedure used in the first round. We just used a different volume of data for the test.
A Classification and Evaluation of Data Movement Technologies for the Delivery of Highly Voluminous Scientific Data Products – May 2006 This paper was authored and presented at the 2006 IEEE Mass Storage Systems and Technologies (MSST2006) conference. It focuses on the evaluation of data transfer mechanisms that can be utilized when sending data to the NSSDC.
Status Report – March 2, 2006 Presentation given to the PDS Management Council at the face-to-face meeting at the Rings Node (SETI Institute).
Testing (First Round) – Ended January 9, 2006 The initial round of testing focused on the interface between PDS and NSSDC utilizing the MPGA software. This testing was scoped according to the time frame and milestones specified by the related FY 2006 NASA Award Fee criteria item. The test plan and the test reports from this round of testing follows:
Status Report – October 5, 2005 Presentation given to the PDS Management Council at the face-to-face meeting at the Small Bodies Node (University of Maryland).