Built by Maven

Release Description - Build 10.1.0

This release of the PDS4 System is intended as an operational release of the system components to date. The following sections can be found in this document:

Software Changes

This section details the new, modified and corrected capabilities that comprise this release . They are summarized here for a system-level view. A more detailed list of capabilities can be found in the change logs for each component.

PDS4 JParser

Documentation: https://nasa-pds.github.io/pds4-jparser/

Improvements

Defects

  • None

Pds4 Information Model

Documentation: https://nasa-pds.github.io/pds4-information-model/

Improvements

Defects

PDS App Registry

Documentation: https://nasa-pds.github.io/pds-registry-app/

Improvements

  • pds-registry-app#1update package and documentation after sub-packages updates (harvest, registry)

Defects

Registry

Documentation: https://nasa-pds.github.io/pds-registry-app

Improvements

  • registry#38 - Update Registry documentation per new registry-mgr upgrades
  • registry#20 - Update Registry documentation per collection updates and report-mgr tool
  • registry#23 - Add new search_post utility for posting Solr Docs to the search index
  • registry#25 - Upgrade to Solr 8.2.0
  • registry#27 - Update search-post tool to accomplish all functionality that Solr Post Tool does
  • registry#28 - Add schema for XPath to prevent failure on ingestions with differing data types
  • registry#31 - Expand Registry collection to include additional overall data system metrics on the archive
  • registry#37 - Improve Registry Manager to work in Windows and support more commands

Defects

  • registry#18 - Fix bug in registry installer occurring from open sourcing or project
  • registry#42 - Registry doesn't build on Windows
  • registry#22 - Fix docker and standalone installers to work with new collections

PDS API Server

Documentation:

Improvements

Defects

  • None

PDS Deep Archive

Documentation:

Improvements

Defects

Harvest

Documentation: https://nasa-pds.github.io/pds-registry-app

Improvements

  • harvest#4 - Update harvest to ingest context metadata from context products and ignore information in labels
  • harvest#9 - Include global policy as a default output product
  • harvest#11 - Upgrade to Solr 8.2.0
  • harvest#13 - Update registry-mgr tool to accomplish all functionality that Solr Post Tool does
  • harvest#24 - Review CIS Node Artifact metrics requirements for updates to Harvest config
  • harvest#26 - Prep for Open Source
  • harvest#8 - Update Harvest to use online namespace for easier configuration and validation
  • harvest#15 - Create new Harvest / Registry package to help streamline the documentation and deployment
  • harvest#17 - Update Harvest documentation per new scripts and upgrades
  • harvest#16 - Improve and simplify Harvest execution and configuration to only manage Registry collection

Defects

  • harvest#20 - Bash script does not launch on macos, likely not on linux
  • harvest#22 - Lid & vid validation, logging enhancement, fix skipped file counter
  • harvest#13 - Update Harvest to work with upgraded Solr package

Validate

Documentation: https://nasa-pds.github.io/validate/

Improvements

  • validate#56 - Validate that Table_Character/Table_Binary fields match the field length definitions in the label
  • validate#128 - Develop validate-bundle script for Validation performance improvements using parallelization approach
  • validate#133 - Deprecate --no-data-check to --skip-content-validation to fall in line with other argument naming
  • validate#182 - Upgrade to Saxon 9.9.1-7 to fix Schematron bug
  • validate#187 - Update documentation to include more details about how/where to file bug reports
  • validate#194 - Update validate-bundle to include report filepath flag
  • validate#195 - Improve installation procedure
  • validate#135 - Update config to include telescopes and facilities telescopes and facilities
  • validate#158 - Update POM to use PDS Parent POM
  • validate#197 - Change behaviour for printing of "dots" to show progress
  • validate#215 - Throw warning when data exists after number of records
  • validate#222 - Reduce error messages for overlapping fields
  • validate#63 - Validate Table_Character groups and their specified lengths match the specified group_length
  • validate#132 - Add validated product counter to pass/fail reporting and end summary
  • validate#175 - Fix schematron parsing bug introduced during performance improvements
  • validate#209 - Packed_Data_Fields and bit fields do not validate as expected
  • validate#29 - Improve pds4.bundle validation performance
  • validate#123 - Enhance validate performance to average <1 second with content validation enabled
  • validate#124 - L5.PRP.VA.42 The tool shall average less than 1 second execution time when content validation is disabled.
  • validate#192 - CCB-282: Revert validation checks on strict file naming requirements

Defects

  • validate#137 - Update delimited table handling to allow for empty fields
  • validate#146 - Update parent LID integrity checking to fix bug introduced for collection integrity checking
  • validate#149 - Update software to check number of records specified in label matches the records in the table
  • validate#160 - Remove erroneous timing messages from log output
  • validate#167 - Content validation multi-threading issue
  • validate#170 - Update validate to allow for empty fields in delimited tables
  • validate#173 - Bug with #149 fix: Product counter is invalid and does not reset after each collection
  • validate#178 - Improve performance when content validation is disabled
  • validate#183 - Memory leak issue has returned after Saxon downgrade
  • validate#206 - Validate incorrectly fails ASCII_Integer field that is space-padded (empty) in Table_Character
  • validate#219 - validate v1.22.3 has large performance degradation on products with many tables
  • validate#220 - Incorrect validation of number of records
  • validate#139 - Revisit PDS-605: L5.PRP.VA.36: Check that the LID for a product has the LID of its parent collection product as its base
  • validate#165 - Spot check bug throws internal_error when record number is multiple of spot check number
  • validate#201 - Schematron doesn't fire if <?xml-model ...> has an extra space at the end
  • validate#203 - Assembly plugin non-fatal errors on Windows
  • validate#214 - validate outputs an extra '.'
  • validate#221 - validate unexpectedly throws error for subdir named data/data/
  • validate#177 - Software raises a field_value_overlap error on Table_Binary Packed data fields
  • validate#180 - Fix multi-threading bug where reporting finishes prior to threads completing execution
  • validate#118 - Update Validate error/warning counts per ATM regression testing
  • validate#145 - Incorrect help information for --spot-check-data flag

Ops

Documentation: None

Improvements

  • ops#77 - Update InSight Cameras Resource Products

Defects

  • None

PDS Site

Documentation: https://pds.nasa.gov

Improvements

  • pds-site#2 - Document baseline process for validating schema/schematrons prior to release
  • pds-site#3 - Update PDS Policies/Formats with centralized location for CDF, FITS, and GeoTIFF documentation

Defects

  • None

Search UI

Documentation: None

Improvements

Defects

  • None

MI Label

Documentation: https://nasa-pds.github.io/mi-label/

Improvements

  • mi-label#4 - Complete Generate rename and open source

Defects

  • None

GEO Data Services Pilot

Documentation: None

Improvements

Defects

  • None

Liens

This section details the liens or deviations from the plan for Build 10b along with rationale for the deviation.

Issue Title Rationale
pds4-information-model#159 CCB-209: Correct Definitions of Tagged_*_Object SCR design was not completed and delivered by DDWG.
pds4-information-model#115 CCB-212: Add Modification_History to Ingest_LDD SCR design was not completed and delivered by DDWG.
pds4-information-model#117 CCB-222: Add citation_text to Citation_Information SCR design was not completed and delivered by DDWG.
pds4-information-model#118 CCB-255: Namespace version dependencies are not documented in the Information Model. SCR design was not completed and delivered by DDWG.
pds4-information-model#119 CCB-260: PDS4 label files do not have a unique file extension SCR design was not completed and delivered by DDWG.
validate#51 Provide the capability to specify multiple locations for pds4.bundle validation De-scoped from build for other higher-priority improvements.
validate#10 Check that the LID for a collection product has the LID of its parent bundle as its base De-scoped from build for other higher-priority improvements.
validate#7 Update to support ComplexLSB8 data types and investigate Floating point exception De-scoped from build for other higher-priority improvements.
validate#69 Verify all context objects specified in observational products are referenced in the parent bundle/collection Reference_List De-scoped from build for other higher-priority improvements.
registry#5 Revise Next Gen Registry and Search requirements De-scoped from build in order to develop new prototype PDS API Server to support centralized search.
registry#9 Enhance weighting and expected search results per Search UI upgrades De-scoped from build in order to develop new prototype PDS API Server to support centralized search.

Installation and Operation

This section provides details and links to the installation and operation instructions for setting up Standalone PDS Tools, Discipline Node Environment, and internal Engineering Node Environment and Services.

Discipline Node Environment

This section details the deployment to a Discipline Node environment. The following tools and services are used to ensure the validity and overall management of products in the archive.

The tools and services are organized into two sections:

  • Required - These tools and services should be installed by all discipline nodes. Please notify PDS Operator if you are unable to install this software.
  • Optional - These tools and servers are optional installations at this time. Please read the software descriptions = in the table below to determine if the software is applicable for your node. Some of these components may become required in the future.

NOTE: The next generation Registry is still under construction. Engineering Node will work with Nodes individually to deploy the software as we continue to test it out. Please bare with us during this transition.


Required Software

Tool Installation/Operation Guides Description
Validate On Github Pages Software for validating PDS4 product labels and product data
Feedback Widget On Github Website widget for prompting users in need of assistance or have feedback.

Optional Software

Tool Installation/Operation Guides Description
PDS Registry Application On Github Pages PDS Registry provides functionality for tracking, searching, auditing, locating, and maintaining artifacts within the system. These artifacts can range from data files and label files, schemas, dictionary definitions for objects and elements, services, etc.

Standalone PDS Tools

Tool Installation/Operation Guides Description
MILabel On Github Provides a command-line interface for generating PDS4 Labels using a user provided PDS4 XML template and input (source) data products.
Local Data Dictionary Tool (LDDTool) On Github Parses a local data dictionary definition file and generates PDS4 data standard files for archival Local Data Dictionaries.
PDS Deep Archive On Github Software for validating PDS4 product labels and product data
PDSView On Github Provides support for visualizing PDS3 and PDS4 data products. This application will eventually deprecate NASAView.
Transform On Github Transforms PDS3 and PDS4 product labels and data into various formats.
Validate On Github Software for validating PDS4 product labels and product data

Engineering Node Environment

This section provides an overview of the deployment to the Engineering Node operations environment. The following diagram depicts the EN system flow, starting with central catalog migration, data product registration and finishing up with search index generation.

EN System Flow
Click the image for a larger version.

For this particular release, here are the components that require a new/upgraded operational deployment:

Tool Installation/Operation Guides
Validate On Github Pages
Feedback Widget On Github

Engineering Node Software Catalog

The following list links to all of the software components currently being maintained and operated by the PDS Engineering Node as of the deployment of this build.

Reference Documents

This section details the controlling and applicable documents referenced for this release. The controlling documents are as follows:

  • PDS Level 1, 2 and 3 Requirements, April 20, 2017.
  • PDS4 Project Plan, July 17, 2013.
  • PDS4 System Architecture Specification, Version 1.3, September 1, 2013.
  • PDS4 Operations Concept, Version 1.0, September 1, 2013.
  • PDS General System Software Requirements Document (SRD), Version 1.1, September 1, 2013.
  • PDS Harvest Tool Software Requirements and Design Document (SRD/SDD), Version 1.2, September 1, 2013.
  • PDS Preparation Tools Software Requirements and Design Document (SRD/SDD), Version 0.3, September 1, 2013.
  • PDS Registry Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
  • PDS Report Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
  • PDS Search Service Software Requirements and Design Document (SRD/SDD), Version 1.0, September 1, 2013.
    • PDS Search Scenarios, Version 1.0, September 1, 2013.
    • PDS Search Protocol, Version 1.2, March 21, 2014.
    • PDAP Search Protocol, Version 1.0, March 21, 2014.
  • PDS Security Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
  • PDS Deep Archive Sotware Requirements and Design Document (SRD/SDD)
  • PDS DOI Service Requirements and Design Document (SRD/SDD)