Built by Maven

Release Description - Build 10.0.0

This release of the PDS4 System is intended as an operational release of the system components to date. The following sections can be found in this document:

Reference Documents

This section details the controlling and applicable documents referenced for this release. The controlling documents are as follows:

  • PDS Level 1, 2 and 3 Requirements, April 20, 2017.
  • PDS4 Project Plan, July 17, 2013.
  • PDS4 System Architecture Specification, Version 1.3, September 1, 2013.
  • PDS4 Operations Concept, Version 1.0, September 1, 2013.
  • PDS General System Software Requirements Document (SRD), Version 1.1, September 1, 2013.
  • PDS Harvest Tool Software Requirements and Design Document (SRD/SDD), Version 1.2, September 1, 2013.
  • PDS Preparation Tools Software Requirements and Design Document (SRD/SDD), Version 0.3, September 1, 2013.
  • PDS Registry Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
  • PDS Report Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
  • PDS Search Service Software Requirements and Design Document (SRD/SDD), Version 1.0, September 1, 2013.
    • PDS Search Scenarios, Version 1.0, September 1, 2013.
    • PDS Search Protocol, Version 1.2, March 21, 2014.
    • PDAP Search Protocol, Version 1.0, March 21, 2014.
  • PDS Security Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.

System Requirements, Dependencies, and Assumptions

This section details the high-level system requirements for installing and operating a majority of the software released by PDS Engineering Node. Due to the varying requirements and technologies used by the tools, specific system requirements for each component can be found in their respective Installation documents. Additionally, explicit software dependencies and libraries can also be found in each individual software component.

Preferred Environment Overview

Product Version
Java Runtime Environment 1.8.x
Apache Tomcat 7.x or 8.x
Apache Web Server 2.x

Java Runtime Environment

Many of the components developed by PDS Engineering Node require Java Runtime Environment (JRE). The software was specifically compiled for and tested with Java version 1.8, however, other versions may still work, dependent upon the component.. The following commands test the local Java installation in a UNIX-based environment: 

          % which java
          /usr/bin/java
          
          % java -version
          java version "1.8.0_101"
          Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
          Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

The first command above checks whether the java executable is in the environment's path and the second command reports the version. If Java is not installed or the version is not at least 1.8, Java will need to be downloaded and installed in the current environment. Consult the local system administrator for installation of this software. For the do-it-yourself crowd, the Java software can be downloaded from the Oracle Java Download page. The suggested software package is the Java Standard Edition (SE) 8, either the JDK or the JRE package. The JDK package is not necessary to run the software but could be useful if development and compilation of Java software will also occur in the current environment.

Java Application Server

The other main system requirement pertains to providing a Java application server for hosting certain components (e.g., Registry, etc.). The suggested application server for this release is Apache Tomcat with a minimal version of 7.0.X. Version 8.5.X is also supported in this release. Consult the local system administrator for installation of this software. For the do-it-yourself crowd, see the Tomcat Deployment document for installation and configuration details.

Web Server

The installation of a web server to act as a front-end interface to the Java application server is desired. The suggested web server for this release is the Apache HTTP Server. Consult the local system administrator for installation of this software. For the do-it-yourself crowd, the Apache HTTP Server software can be downloaded from the Apache HTTP Server page.

Software Changes

This section details the new, modified and corrected capabilities that comprise this release . They are summarized here for a system-level view. A more detailed list of capabilities can be found in the change logs for each component.

PDS4 JParser

Documentation: https://nasa-pds-incubator.github.io/pds4-jparser/

Implemented Requirements

  • None

Improvements

Defects

  • None

PDS4 Information Model

Documentation: https://nasa-pds-incubator.github.io/pds4-information-model/

Note: Some issues in this component (PDS-*, CCB-*) are tracked via internal password-protected JIRA systems. For more details, please contact PDS Operator.

Implemented Requirements

  • None

Improvements

  • PDS-906 Open Source PDS4 Model
  • pds4-information-model#74 - LDDTool: Update version flag (-v) to show IM version as well as software version
  • CCB-253 - Add Units_of_Force as a unit of measure
  • CCB-252 - Add "Not Applicable" to wavelength_range in Science_​Facets
  • CCB-250 - Correct schematron rule "pds:Information_Package_Component_Deep_Archive/pds:checksum_type" to test for 'MD5' not 'MD5Deep 4.n'
  • CCB-262 - Add an enumerated value for Primary_Result_Summary.purpose to describe support observations.
  • CCB-244 - Deprecate 'earth-based' from Context Product Instrument_Host types.
  • CCB-251 - Add Units_of_Gmass as a unit of measure.
  • CCB-258 - Use of nilReason with xsi:nil should be required and validated.
  • CCB-266 - Change type of External_Reference/reference_text
  • CCB-254 - Add Enumerated Value to Target_Identification.type for Gravitational Waves

Defects

PDS View

Documentation: https://github.com/NASA-PDS-Incubator/pds-view/#pdsview

Implemented Requirements

  • None

Improvements

  • pds-view#6 - Update docs and distribution to match new PDSView naming
  • pds-view#10 - Replace intro Inspect Tool image with 'Open' dialog

Defects

  • pds-view#14 - Allow multiple periods ('.') in paths to pds4 xml files.
  • pds-view#21 - Fix PDSView build and execution error related to libc6 dependency
  • pds-view#22 - PDSView fails on PDS3 attached product (fails to find InspectTemplate.vm)

Registry

Documentation: https://nasa-pds-incubator.github.io/registry

Implemented Requirements

  • None

Improvements

  • registry#2 - Enhance registry schema and config for refined search capabilities
  • registry#3 - Enhance usability of search endpoints by renaming and simplifying their definitions
  • registry#4 - Review and cleanup applicable search fields for display and index output
  • registry#10 - Create registry blob collection
  • registry#17 - Add capability to handle tracking of Harvest ingestions
  • registry#20 - Update Registry documentation per collection updates and report-mgr tool
  • registry#23 - Add new search_post utility for posting Solr Docs to the search index
  • registry#25 - Upgrade to Solr 8.2.0
  • registry#27 - Update search-post tool to accomplish all functionality that Solr Post Tool does
  • registry#28 - Add schema for XPath to prevent failure on ingestions with differing data types

Defects

  • registry#18 - Fix bug in registry installer occurring from open sourcing or project
  • registry#22 - Fix docker and standalone installers to work with new collections

Feedback Widget

Documentation: https://github.com/NASA-PDS-Incubator/feedback-widget#pds-feedback-widget

Implemented Requirements

  • None

Improvements

  • None

Defects

Validate

Documentation: https://nasa-pds-incubator.github.io/validate/

Implemented Requirements

  • validate#15 - Verify that all name/type attribute values correspond to names denoted context products
  • validate#24 - Update context check to retrieve and use latest context products from EN Registry
  • validate#45 - Check that referenced context products are valid
  • validate#50 - Add new target-manifest flag for user to provide a manifest (file) of file/directory paths to validate

Improvements

  • validate#30 - Enhance content validation to decrease validation time to <=50% of current benchmark
  • validate#87 - Update validate to apply catalog file and local schemas to discipline schema resolution
  • validate#133 - Deprecate --no-data-check to --skip-content-validation to fall in line with other argument naming
  • validate#20 - Improve documentation about usage of flags in config file with latest flags
  • validate#23 - Deprecate -f and -m flags
  • validate#28 - Add capability for user to input JSON file with additional context products
  • validate#29 - Improve pds4.bundle validation performance
  • validate#39 - Investigate and fix memory leak causing validate crash on large bundles
  • validate#41 - Perform benchmarking on validate to enable documenting of system requirements
  • validate#42 - Add capability to pds4.bundle validation to ignore product-level validation
  • validate#43 - Document multi-threaded architecture for maximized performance
  • validate#47 - Add flag to temporarily disable context validation
  • validate#64 - Complete engineering point build 1.15.1
  • validate#65 - Update Maven docs to refer to Github release assets
  • validate#90 - Update context checks to include data_to_investigation
  • validate#135 - Update config to include telescopes and facilities telescopes and facilities

Defects

  • validate#9 - Update content validation to fail when a space character is used when an ASCII_Integer type is specified
  • validate#34 - Add missing core.properties dependency
  • validate#62 - Context validation needs to be updated to include all possible product type use cases
  • validate#71 - Unable to use catalog file after update to always force remote schemas
  • validate#72 - Catalog file flag expects a List of files requiring it not be the last argument
  • validate#74 - Validate execution script throws arguments error with v1.15.1
  • validate#84 - Update Validate to allow specification of schema/schematrons in config file
  • validate#99 - Fix bug introduce by --skip-context-validation feature
  • validate#107 - Registered context product resource lists only one of multiple instrument or target multiple types
  • validate#137 - Update delimited table handling to allow for empty fields
  • validate#33 - Validation succeeds despite throwing exception
  • validate#37 - JAVA Path does not handle whitespaces in MAC as it used to
  • validate#52 - Incorrect package declarations under gov.nasa.tools.web.ui
  • validate#55 - Content validation does not work properly
  • validate#139 - Revisit PDS-605: L5.PRP.VA.36: Check that the LID for a product has the LID of its parent collection product as its base
  • validate#3 - JAXB warning message with Java 9+
  • validate#8 - Large Data File Uncaught Exception Error
  • validate#118 - Update Validate error/warning counts per ATM regression testing

Harvest

Documentation: https://pds-engineering.jpl.nasa.gov/development/pds4/current/harvest/

Note: Some issues in this component (PDS-*, CCB-*) are tracked via internal password-protected JIRA systems. For more details, please contact PDS Operator.

Implemented Requirements

  • None

Improvements

  • PDS-921 - Perform trade study to determine if a config generator or config refactor
  • PDS-922 - Implement configuration updates per trade study
  • PDS-952 - Store document blobs in new registry collection instead of .system
  • PDS-953 - Allow search config lookup by data_class
  • harvest#9 - Include global policy as a default output product
  • harvest#11 - Upgrade to Solr 8.2.0
  • harvest#13 - Update registry-mgr tool to accomplish all functionality that Solr Post Tool does
  • harvest#8 - Update Harvest to use online namespace for easier configuration and validation

Defects

  • None

Tracking Service

Documentation: https://pds-engineering.jpl.nasa.gov/development/pds4/current/tracking-service/

Note: Some issues in this component (PDS-*, CCB-*) are tracked via internal password-protected JIRA systems. For more details, please contact PDS Operator.

Implemented Requirements

  • PDS-687 - Tracking UI: Delivery Tracking Interface
  • PDS-593 - Tracking: Develop a service-based interface for tracking product status

Improvements

  • PDS-914 - Add columns to table to match current data release calendar
  • PDS-913 - Update page to include front matter and other information from current release calendar

Defects

  • None

Liens

This section details the liens or deviations from the plan for Build 10b along with rationale for the deviation.

Issue Title Rationale
validate#44 Notify user when new version of the software is available Re-prioritized for performance improvements
validate#1 Add capability to do data spot checking every nth file Re-prioritized for performance improvements

Installation and Operation

This section provides details and links to the installation and operation instructions for setting up Standalone PDS Tools, Discipline Node Environment, and internal Engineering Node Environment and Services.

Standalone PDS Tools

Tool Installation/Operation Guides Description
Generate On PDS Engineering Node Website Provides a command-line interface for generating PDS4 Labels using a user provided PDS4 XML template and input (source) data products.
Local Data Dictionary Tool (LDDTool) On Github Pages Parses a local data dictionary definition file and generates PDS4 data standard files for archival Local Data Dictionaries.
PDS View (beta release) On Github Provides support for visualizing PDS3 and PDS4 data products.
Transform On Github Pages Transforms PDS3 and PDS4 product labels and data into various formats.
Validate On Github Pages Software for validating PDS4 product labels and product data

Discipline Node Environment

This section details the deployment to a Discipline Node environment. The following tools and services are used to ensure the validity and overall management of products in the archive.

The tools and services are organized into two sections:

  • Required - These tools and services should be installed by all discipline nodes. Please notify PDS Operator if you are unable to install this software.
  • Optional - These tools and servers are optional installations at this time. Please read the software descriptions = in the table below to determine if the software is applicable for your node. Some of these components may become required in the future.

NOTE: The next generation Registry is still under construction. Engineering Node will work with Nodes individually to deploy the software as we continue to test it out. Please bare with us during this transition.


Required Software

Tool Installation/Operation Guides Description
Validate On Github Pages Software for validating PDS4 product labels and product data
Feedback Widget On Github Website widget for prompting users in need of assistance or have feedback.

Optional Software

Tool Installation/Operation Guides Description
Registry On Github Pages Provides functionality for tracking, searching, auditing, locating, and maintaining artifacts within the system. (This software is currently in development as it receives a technology upgrade. EN will work with nodes individually to help install this once the installation and operation have been more thoroughly tested.)
Harvest On PDS Engineering Node website Provides functionality for capturing and indexing product metadata into the Registry. (This is only required if installing the Registry.)
Transport Service On Github Pages Provides functionality for transporting and transforming PDS products using the Registry. This software is optional for the time being until a more integrated architecture is deployed at nodes. This software will be required once the integrated system is ready for deployment.

Testing Discipline Node Environment

Follow along with the tutorial on the PDS Engineering Node website: PDS4 Data Release at a Discipline Node

Engineering Node Environment

This section provides an overview of the deployment to the Engineering Node operations environment. The following diagram depicts the EN system flow, starting with central catalog migration, data product registration and finishing up with search index generation.

EN System Flow
Click the image for a larger version.

For this particular release, here are the components that require a new/upgraded operational deployment:

Tool Installation/Operation Guides
Validate On Github Pages
Feedback Widget On Github

Engineering Node Software Catalog

The following list links to all of the software components currently being maintained and operated by the PDS Engineering Node as of the deployment of this build.