New Capabilities

The following are new capabilities for this release:

• [PDS-591] - Tracking UI: DOI Interface
• [PDS-594] - Tracking: Develop the REST-based service interfaces for insert/update of Product,Archive/Certification/NSSDCA Status
• [PDS-602] - PDS4-Tools/Transform: Add Array_3D_Image to FITS
• [PDS-603] - Core/Validate: Add capability to do spot checking of data content
• [PDS-605] - Core: Check Logical Identifiers of primary members
• [PDS-606] - Core: Check for existence of referenced local identifiers within a label
• [PDS-619] - Tracking: Add additional interfaces for Insert/Update of Delivery, Role, User
• [PDS-636] - Tracking: Add additional interfaces for Insert/Update of References
• [PDS-692] - Tracking UI: Status Interface
• [PDS-711] - Search: Update Data Search results for SPICE
• [PDS-745] - Harvest-Search: Update output log information for better usability
• [PDS-758] - Inspect: Move to Open Source
• [PDS-761] - Search: Containerize Search Service deployment
• [PDS-765] - Harvest-PDAP: Update to ingest into the Search Service
• [PDS-777] - Core: Add check of member status field values for Collection Inventory Tables
• [PDS-787] - Tracking: Develop the REST-based service interface to get the list of all Products with the latest three types of status for each product
• [PDS-793] - Core/PDS4-Tools: Update to support ASCII_BibCode data types

Modified Capabilities

The following capabilities have been modified for this release:

• [PDS-519] - Generate: The tool outputs an unexpected warning
• [PDS-604] - Core: Check for un-labeled files in a bundle
• [PDS-623] - Verify the DOM conversion is correct for CSV format
• [PDS-631] - Generate: Update ProductToolsReader to match changes for VICAR Reader
• [PDS-633] - Search: Upgrade to latest Solr
• [PDS-639] - Search: Provide list of searchable attributes for Keyword Search interface on the EN Site
• [PDS-646] - Portal: Create prototype Feedback widget with website overlay and email service
• [PDS-651] - Verify DOM conversion for rdf format
• [PDS-658] - LDDTool: Verify DOM conversion for .sch format
• [PDS-669] - Search: Update installation and operation document for upgrading to latest Solr
• [PDS-678] - Core/Validate: Check that Context LIDVIDs referenced in product labels exist in the registered list
• [PDS-681] - LDDTool: Update JSON output with new memberTitle attribute
• [PDS-694] - LDDTool: Update output filenames to include LDD Version
• [PDS-695] - LDDTool: Missing Units_of_Pixel_Resolution_Map and Units_of_Pixel_Scale_Map in unit_of_measure_type applicable values
• [PDS-701] - PDS4-Tools/Core/Validate: Update field format validation behavior
• [PDS-738] - DSView: Update viewBundle.jsp to better handle detection of collection_ref types
• [PDS-750] - Deprecate Harvest for Harvest-Search
• [PDS-759] - Deprecate Registry
• [PDS-782] - Build: Update build to include email service and feedback widget
• [PDS-784] - Portal: Update Feedback with better error handling

Corrected Capabilities

The following capabilities have been corrected for this release:

• [PDS-597] - Report: Grok Parse Failures from pdsimg1 and pdsimg2
• [PDS-657] - LDDTool: Multiple choice blocks were generated for every property in the DOM conversion
• [PDS-674] - Ingest: Harvest PDAP fails with TLS error
• [PDS-702] - Search: Instrument Host products not getting returned
• [PDS-714] - Inspect: Array_2D_Map does not display table or image correctly
• [PDS-723] - Core: Support schematron rules using the document-uri() function
• [PDS-726] - Registry: Fix bad Maven repos in POM

Java Runtime Environment

The custom software contained in this release was developed using Java and will run on any platform with a supported Java Runtime Environment (JRE). The software was specifically compiled for and tested with Java version 1.8. The following commands test the local Java installation in a UNIX-based environment:

% which java
/usr/bin/java

% java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
        

The first command above checks whether the java executable is in the environment's path and the second command reports the version. If Java is not installed or the version is not at least 1.8, Java will need to be downloaded and installed in the current environment. Consult the local system administrator for installation of this software. For the do-it-yourself crowd, the Java software can be downloaded from the Oracle Java Download page. The suggested software package is the Java Standard Edition (SE) 8, either the JDK or the JRE package. The JDK package is not necessary to run the software but could be useful if development and compilation of Java software will also occur in the current environment.

Java Application Server

The other main system requirement pertains to providing a Java application server for hosting certain components (e.g., Registry, etc.). The suggested application server for this release is Apache Tomcat with a minimal version of 7.0.X. Version 8.5.X is also supported in this release. Consult the local system administrator for installation of this software. For the do-it-yourself crowd, see the Tomcat Deployment document for installation and configuration details.

Web Server

The installation of a web server to act as a front-end interface to the Java application server is desired. The suggested web server for this release is the Apache HTTP Server. Consult the local system administrator for installation of this software. For the do-it-yourself crowd, the Apache HTTP Server software can be downloaded from the Apache HTTP Server page.

Database Server

The installation of a database server is suggested to support the deployment of the Registry Service. The Registry Service comes prepackaged with the Apache Derby database but isn't necessarily the preferred choice for an operational deployment. Other options supported by the Registry Service include MySQL and PostgreSQL. Consult the local system administrator for installation of these database servers. For the do-it-yourself crowd, the MySQL software can be downloaded from the MySQL page while the PostgreSQL software can be downloaded from the PostgresSQL page.

If another database server is preferred, contact the EN so we can help prepare a database-specific configuration for your environment.

Discipline Node Environment

This section details the deployment to a Discipline Node environment. The Nodes are asked to download, install and exercise the software that makes up this release. The following diagram details an example deployment of selected components:

Click the image for a larger version.

The components depicted above represent a typical deployment of the PDS4 software on a single server. The following diagram depicts a normalized system flow, starting with schema design, continuing onto data product label generation and validation and finishing up with data product registration and search index generation:

Click the image for a larger version.

This diagram identifies where the system components come into play within the system flow. These components and the ones depicted in the deployment diagram, are described in more detail with respect to how the components can be utilized in operations:

• Generate Tool
  

  The Generate Tool is intended to aid users in the generation of their data product labels. This tool was developed by the Imaging Node at JPL and provides a command-line interface for generating PDS4 Labels from either a PDS3 Label or a PDS-specific DOM object.

• Validate Tool
  

  The Validate Tool comes pre-packaged with the latest version of the core XML Schema and Schematron files generated from the data model. The tool allows the user to validate collections of products or single products against the associated core schema. Discipline and mission schemas can be passed into the tool to enable a more detailed validation check. Although the Design Tools listed above provide a file-by-file validation capability, this tool allows the user to validate a batch of products and generates a summary report.

• Transform Tool / PDS4 Tools
  

  The Transform Tool and the PDS4 Tools library provide product transformation capability. The Transform Tool provides a command-line interface to a subset of the functions offered by the PDS4 Tools library. In addition, the PDS4 Tools library provides functionality for accessing PDS4 data objects. The functionality of both of these components will be expanded in future releases.

• Harvest Tool
  

  The Harvest Tool provides the means for registering products with the Registry Service. This tool reads a PDS4 product label, extracts specified metadata and registers the product with the registry. This tool allows the user to register a batch of products and generates a summary report. This tool also supports registration of PDS3 data sets for the purpose of tracking and reporting.

• Search


• Registry
  

  The Registry component provides functionality for tracking, searching, auditing, locating, and maintaining artifacts within the system.

• Transport


• Transport Service (OFSN)
  

  The Online File Specification Name (OFSN) Transport Service provides functionality for transporting and transforming PDS products. This service offers a similar interface to the PDS-D Product Server and allows a user to interface with an archive directory structure for discovering and retrieving product files.

• Transport Service (Registry)
  

  The Registry Transport Service provides functionality for transporting and transforming PDS products. This service interfaces with the Registry Service or Search Service to discover and retrieve products files given a logical identifier.

Installation Procedures for Core Components

Although the Design, Generate and Transform Tools are included with this release, we have not included them in the installation procedure because they may not pertain to every Discipline Node. This procedure will focus on installing the Registry components, Harvest Tool and Validate Tool. The example commands below assume the software is installed in the user's home directory, indicated by the $HOME environment variable. If this is not the case or if this variable is not defined, the absolute path should be used instead of the variable. Some of the commands below have been broken into multiple lines for readability. The commands should be reassembled into a single line prior to execution. Perform the following steps to download, install, configure and test the software in the Node's local environment:

1. Verify System Requirements
   

   As specified in the System Requirements section above, the software requires Java, a Java Application Server (Apache Tomcat), a Web Server (Apache HTTP) and an optional database server to be installed and accessible. These installations should be verified before proceeding to the next step.

   With respect to the Web Server installation, this server may reside on another server and in most cases that would be preferable. The preferred configuration is to install the PDS4 software components on a server that resides behind a firewall at your institution. This offers an initial level of security for the components. Configuring the Web Server for reverse proxies to the query interfaces of the components provides an additional level of security by restricting access to the interfaces that allow updates to the data.

2. Install Harvest Tool
   

   The Installation document for the Harvest Tool provides the details for downloading and installing the software.

3. Install Validate Tool
   

   The Installation document for the Validate Tool provides the details for downloading and installing the software.

Installation Procedures for Registry

1. Registry
   

   The Installation document for the Registry provides the details for downloading, installing and configuring the software. The reverse proxy configuration that follows provides security for the Search Service instance. The instance of the service is accessed at /services/registry on the web server which is facilitated by the following reverse proxy configuration in the Apache HTTP Server httpd.conf file:

Installation Verification for Core Components

To make sure your entire pipeline is ready for operations, there is a tutorial that follows along with the PDS4 Data Release at Discipline Node procedure to test out your installation, as well as get a feel for the tools.

Engineering Node Environment

This section provides an overview of the deployment to the Engineering Node operations environment. The following diagram depicts the EN system flow, starting with central catalog migration, data product registration and finishing up with search index generation:

Click the image for a larger version.

The components depicted above serve as the operational deployment of this release for the Engineering Node. The main purpose of this deployment is to support catalog-level search for the PDS.

• Catalog Tool
  

  The Catalog Tool provides the means for registering PDS3 catalog files with the Registry Service. This tool provides functionality for comparing, validating and ingesting a catalog submission.

• Harvest Tool
  

  The Harvest Tool provides the means for registering products with the Registry/Search Service. This tool reads a PDS4 product label, extracts specified metadata and registers the product with the Search Service at the registry index. This tool allows the user to register a batch of products and generates a summary report.

• Registry
  

  The Registry and Search provides functionality for accepting queries from data consumers for registered products and includes functionality for retrieving search results. The end point for this service is https://pds.nasa.gov/services/registry/.

• Keyword Search User Interface
  

  The Keyword Search User Interface software serves as the user interface for the Search Service on the PDS website. The end point for this application is https://pds.nasa.gov/tools/keyword-search/.

• Report Service
  

  The Report Service provides the capability for capturing and reporting metrics. With this release, the service supports generation of monthly metrics for each of the Nodes.