Data Integrity/Tracking

PDS has identified maintaining the integrity of the data and archive holdings as well as tracking their existence and locations as a priority. This activity is focused on defining the requirements, developing the associated processes, and implementing needed tools and infrastructure to support the tracking and integrity of the data and archive holdings from the data producer all the way to the NSSDC.

Working Group

The Working Group reports directly to the Management Council and includes :

• Dan Crichton, Mitch Gordon, Ed Guinness, Bill Harris, Steve Hughes, Chris Isbell, Todd King, Al Schultz, Mark Showalter and Tom Stein

Milestones

Near-term milestones include:

• Finalize compiled Integrity/Tracking/Availability/Deep Archive Requirements

Long-term milestones include:

• PDS starts implementation of archive integrity

Current Status

The working group is reviewing the level 4 requirements for archive tracking and assessing using MD5Deep as a PDS checksum. A Data Integrity Report template was created for Nodes to generate annual report.

• Draft PMWG Data Integrity Recommendations “Oct 2008”
• Draft Archive Integrity requirements “June 2008” (PDF)
• Draft MD5 checksum manifest (TXT)
• GEO Data Integrity Process (PDF)
• PPI Data Integrity Process (PDF)
• PPI Inventory Integrity Check Specification (PDF)
• Node Data Integrity Report Template (XLS)

Policies

Policies that are relevant to Archive Integrity are provided below:

• PDS Data Integrity “Checksum” Policy (PDF) – adopted by MC on April 2008
• PDS Archive Integrity Policy (PDF) – adopted by MC on November 2006
• PDS Policy on Data Delivery & Backup (PDF) – adopted by MC on October 2005

Historical Documents

The historical milestones and meetings regarding data integrity and tracking, along with their associated documents, are provided below:

• Draft Integrity / Tracking / Availability compiled requirements “September 2007” (PDF)
• Draft Availability Backup Recovery Use Cases “August 2007” (PDF)
• Draft Availability Backup Recovery requirements “August 2007” (PDF)
• Delivery and Archive Tracking Level 4,5,6 requirements “March 2007” (PDF)
• Delivery and Archive Tracking Level 4 requirements “March 2007” (PDF)
• Delivery and Archive Tracking Use Cases “January 2007” (PDF)
• Data Integrity Use Cases “November 2006” (PDF)
• Data Integrity Level 4 Requirements “November 2006” (PDF)
• Data Integrity Presentation to the Management Council – November 2006 (PDF) Presentation on Data Integrity given the Management Council at the November Face-to-Face meeting.
• PDS Policy on Data Integrity – October 2006 (PDF) Policy defined at the November Face-to-Face meeting on Data Integrity
• Data Integrity Draft Level 4 Requirements – October 2006 (PDF) Draft level 4 requirements for data integrity defined by the working group
• Data Integrity Draft Level 4 Use Cases – October 2006 (PDF) Level 4 use cases for data integrity defined by the working group
• PDS Technical Session Presentation on Data Integrity – October 2006 (PDF) Presentation given at the PDS Technical Session on Data Integrity