New Capabilities

The following are new capabilities for this release:

• [PDS-243] - Generate: Add support to use Product Tools library
• [PDS-495] - PDS4-Tools: Add URL support
• [PDS-506] - Transform: Add support for transforming PDS3 tables to CSV
• [PDS-507] - Transform: Add support for transforming a PDS4 label to a PDS3 label
• [PDS-508] - Validate: Add support for validating data content of table products
• [PDS-517] - Tracking: Develop a service-based interface for tracking product status

Modified Capabilities

The following capabilities have been modified for this release:

• [PDS-504] - Validate: Improve Operation guide examples
• [PDS-505] - Generate/Transform: Update PDS4 label transformation for the latest model
• [PDS-509] - Search: Separate the protocol implementation into a separate component
• [PDS-511] - Transform: Add support for reading targets from URLs
• [PDS-512] - Transform: Integrate the tool with the Tools Service
• [PDS-519] - Generate: The tool outputs an unexpected warning
• [PDS-521] - Validate: The -R pds4.bundle option cancels -e "*.xml"

Corrected Capabilities

The following capabilities have been corrected for this release:

• [PDS-467] - Core: Support passing in Schema jar URLs
• [PDS-478] - Search: XSLT variable resolution is not working on the Windows platform
• [PDS-501] - Core: Support bundle.xml and collection.xml file names to trigger their respective rule types
• [PDS-515] - Validate: Handshake failure occurs when trying to validate against schemas located on the PDS machines
• [PDS-523] - Transform: transformation of character table product to PDS3 label fails

Java Runtime Environment

The custom software contained in this release was developed using Java and will run on any platform with a supported Java Runtime Environment (JRE). The software was specifically compiled for Java version 1.7 and has been tested with this version and version 1.8. The following commands test the local Java installation in a UNIX-based environment:

% which java
/usr/bin/java

% java -version
java version "1.7.0_72"
Java(TM) SE Runtime Environment (build 1.7.0_72-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode)
        

The first command above checks whether the java executable is in the environment's path and the second command reports the version. If Java is not installed or the version is not at least 1.7, Java will need to be downloaded and installed in the current environment. Consult the local system administrator for installation of this software. For the do-it-yourself crowd, the Java software can be downloaded from the Oracle Java Download page. The suggested software package is the Java Standard Edition (SE) 7, either the JDK or the JRE package. The JDK package is not necessary to run the software but could be useful if development and compilation of Java software will also occur in the current environment.

Java Application Server

The other main system requirement pertains to providing a Java application server for hosting certain components (e.g., Registry Service, Search Service, etc.). The suggested application server for this release is Apache Tomcat with a minimal version of 7.0.X. Version 8.0.X is also supported in this release and is required if you are running Java version 1.8 in your environment. Consult the local system administrator for installation of this software. For the do-it-yourself crowd, see the Tomcat Deployment document for installation and configuration details.

Web Server

The installation of a web server to act as a front-end interface to the Java application server is desired. The suggested web server for this release is the Apache HTTP Server. Consult the local system administrator for installation of this software. For the do-it-yourself crowd, the Apache HTTP Server software can be downloaded from the Apache HTTP Server page.

Database Server

The installation of a database server is suggested to support the deployment of the Registry Service. The Registry Service comes prepackaged with the Apache Derby database but isn't necessarily the preferred choice for an operational deployment. Other options supported by the Registry Service include MySQL and PostgreSQL. Consult the local system administrator for installation of these database servers. For the do-it-yourself crowd, the MySQL software can be downloaded from the MySQL page while the PostgreSQL software can be downloaded from the PostgresSQL page.

If another database server is preferred, contact the EN so we can help prepare a database-specific configuration for your environment.

Engineering Node Environment

This section provides an overview of the deployment to the Engineering Node operations environment. The following diagram depicts the EN system flow, starting with central catalog migration, data product registration and finishing up with search index generation:

Click the image for a larger version.

The components depicted above serve as the operational deployment of this release for the Engineering Node. The main purpose of this deployment is to support catalog-level search for the PDS.

• Catalog Tool
  

  The Catalog Tool provides the means for registering PDS3 catalog files with the Registry Service. This tool provides functionality for comparing, validating and ingesting a catalog submission.

• Harvest Tool
  

  The Harvest Tool provides the means for registering products with the Registry Service. This tool reads a PDS4 product label, extracts specified metadata and registers the product with the registry. This tool allows the user to register a batch of products and generates a summary report.

• Registry Service
  

  The Registry Service provides functionality for tracking, auditing, locating and maintaining artifacts within the system. Two instances of the service have been deployed, one containing context products migrated from the PDS3 central catalog database and one containing the initial set of PDS4 context products. The end points for these services are:

  • https://pds.nasa.gov/services/registry-pds3/
  • https://pds.nasa.gov/services/registry-pds4/
  
  
• Search Service
  

  The Search Service provides functionality for accepting queries from data consumers for registered products and includes functionality for retrieving search results. This component acts as the interface to the Registry Service for the data consumer. The end point for this service is https://pds.nasa.gov/services/search/.

• Search User Interface
  

  The Search User Interface software serves as the user interface for the Search Service on the PDS website. The end point for this application is https://pds.nasa.gov/tools/data-search/.

• Report Service
  

  The Report Service provides the capability for capturing and reporting metrics. With this release, the service supports generation of monthly metrics for each of the Nodes.

Discipline Node Environment

This section details the deployment to a Discipline Node environment. The Nodes are asked to download, install and exercise the software that makes up this release. The following diagram details an example deployment of selected components:

Click the image for a larger version.

The components depicted above represent a typical deployment of the PDS4 software on a single server. The following diagram depicts a normalized system flow, starting with schema design, continuing onto data product label generation and validation and finishing up with data product registration and search index generation:

Click the image for a larger version.

This diagram identifies where the system components come into play within the system flow. These components and the ones depicted in the deployment diagram, are described in more detail with respect to how the components can be utilized in operations:

• Design Tool
  

  The Design Tool is intended to aid users in the development of their data product label schemas. This release identifies two off-the-shelf products (Oxygen and Eclipse) that are available for this purpose. The provided documentation guides the user with respect to the download, installation and operation of each of these products.

• Generate Tool
  

  The Generate Tool is intended to aid users in the generation of their data product labels. This tool was developed by the Imaging Node at JPL and provides a command-line interface for generating PDS4 Labels from either a PDS3 Label or a PDS-specific DOM object.

• Validate Tool
  

  The Validate Tool comes pre-packaged with the latest version of the core XML Schema and Schematron files generated from the data model. The tool allows the user to validate collections of products or single products against the associated core schema. Discipline and mission schemas can be passed into the tool to enable a more detailed validation check. Although the Design Tools listed above provide a file-by-file validation capability, this tool allows the user to validate a batch of products and generates a summary report.

• Transform Tool / PDS4 Tools
  

  The Transform Tool and the PDS4 Tools library provide product transformation capability. The Transform Tool provides a command-line interface to a subset of the functions offered by the PDS4 Tools library. In addition, the PDS4 Tools library provides functionality for accessing PDS4 data objects. The functionality of both of these components will be expanded in future releases.

• Harvest Tool
  

  The Harvest Tool provides the means for registering products with the Registry Service. This tool reads a PDS4 product label, extracts specified metadata and registers the product with the registry. This tool allows the user to register a batch of products and generates a summary report. This tool also supports registration of PDS3 data sets for the purpose of tracking and reporting.

• Registry


• Registry Service
  

  The Registry Service provides functionality for tracking, auditing, locating and maintaining artifacts within the system. Ultimately, all products (including PDS3 products) will be registered and tracked with the Registry Service.

• Registry User Interface
  

  The Registry User Interface provides a simple viewing capability of the contents of the associated Registry Service. It is intended to offer visual verification of successful registration of products as well as the ability to update the status for registered products.

• Search


• Search Core
  

  The Search Core component provides functionality for generating the search index utilized by the Search Service for retrieving search results.

• Search Service
  

  The Search Service provides functionality for accepting queries from data consumers for registered products and includes functionality for retrieving search results. This component acts as the interface to the Registry Service for the data consumer.

• Product Search User Interface
  

  The Product Search User Interface software serves as the user interface for the Search Service for querying and displaying search results.

• Transport


• Transport Service (OFSN)
  

  The Online File Specification Name (OFSN) Transport Service provides functionality for transporting and transforming PDS products. This service offers a similar interface to the PDS-D Product Server and allows a user to interface with an archive directory structure for discovering and retrieving product files.

• Transport Service (Registry)
  

  The Registry Transport Service provides functionality for transporting and transforming PDS products. This service interfaces with the Registry Service or Search Service to discover and retrieve products files given a logical identifier.

Installation Procedures for Core Components

Although the Design, Generate and Transform Tools are included with this release, we have not included them in the installation procedure because they may not pertain to every Discipline Node. This procedure will focus on installing the Registry components, Harvest Tool and Validate Tool. The example commands below assume the software is installed in the user's home directory, indicated by the $HOME environment variable. If this is not the case or if this variable is not defined, the absolute path should be used instead of the variable. Some of the commands below have been broken into multiple lines for readability. The commands should be reassembled into a single line prior to execution. Perform the following steps to download, install, configure and test the software in the Node's local environment:

1. Verify System Requirements
   

   As specified in the System Requirements section above, the software requires Java, a Java Application Server (Apache Tomcat), a Web Server (Apache HTTP) and an optional database server to be installed and accessible. These installations should be verified before proceeding to the next step.

   With respect to the Web Server installation, this server may reside on another server and in most cases that would be preferable. The preferred configuration is to install the PDS4 software components on a server that resides behind a firewall at your institution. This offers an initial level of security for the components. Configuring the Web Server for reverse proxies to the query interfaces of the components provides an additional level of security by restricting access to the interfaces that allow updates to the data.

2. Install Registry Components
   

   The Registry components consist of the Registry Service and Registry User Interface. The Registry Client and Registry Core, which are also Registry components, are not required to be installed at the Discipline Node.

• Registry Service
  

  The Installation document for the Registry Service provides the details for downloading, installing and configuring the software. The Secure Configuration is skipped for the DN deployment. The reverse proxy configuration that follows provides security for the Registry Service instances. The PDS3 and PDS4 instances of the service are accessed at /services/registry-pds3 and /services/registry-pds4, respectively, on the web server which is facilitated by the following reverse proxy configuration in the Apache HTTP Server httpd.conf file:

  ProxyPass /services/registry-pds3/ http://localhost:8080/registry-pds3/
  ProxyPassReverse /services/registry-pds3/ http://localhost:8080/registry-pds3/
  ProxyPass /services/registry-pds3 http://localhost:8080/registry-pds3
  ProxyPassReverse /services/registry-pds3 http://localhost:8080/registry-pds3
  <Location /services/registry-pds3>
    <Limit PUT POST DELETE>
      Order Deny,Allow
      Deny from All
      Allow from 127.0.0.1
    </Limit>
  </Location>
  
  ProxyPass /services/registry-pds4/ http://localhost:8080/registry-pds4/
  ProxyPassReverse /services/registry-pds4/ http://localhost:8080/registry-pds4/
  ProxyPass /services/registry-pds4 http://localhost:8080/registry-pds4
  ProxyPassReverse /services/registry-pds4 http://localhost:8080/registry-pds4
  <Location /services/registry-pds4>
    <Limit PUT POST DELETE>
      Order Deny,Allow
      Deny from All
      Allow from 127.0.0.1
    </Limit>
  </Location>
                

  The above configuration will likely need to be performed by the System Administrator. This configuration assumes the Apache HTTP Server is on the same server as the Apache Tomcat Server. If that is not the case, the localhost designation above should be replaced with the DNS name of the server where the Apache Tomcat Server is installed. This configuration allows GET requests through the public interface but restricts PUT, POST and DELETE requests to processes running on the local server.

• Registry User Interface
  

  The Installation document for the Registry User Interface provides the details for downloading, installing and configuring the software. The Secure Configuration may be skipped for the DN deployment.

3. Install Harvest Tool
   

   The Installation document for the Harvest Tool provides the details for downloading and installing the software. In order to support multiple instances of the Registry Service, it is suggested to do the following:

   • Make a copy of the harvest script provided with the software for each Registry Service instance:
     

     % cd $HOME/harvest-1.13.0/bin
     % cp harvest harvest-pds3
     % cp harvest harvest-pds4
                   

   • Edit the newly created harvest-pds3 script and change the http://localhost:8080/registry specification to http://localhost:8080/registry-pds3.
   • Edit the newly created harvest-pds4 script and change the http://localhost:8080/registry specification to http://localhost:8080/registry-pds4.
4. Install Validate Tool
   

   The Installation document for the Validate Tool provides the details for downloading and installing the software.

Installation Verification for Core Components

Now that the core software is installed, it can be exercised with a local bundle prepared by the Node or with the Example Bundle available from the EN. The following steps detail exercising of the software with the Example Bundle:

1. Download Example Bundle
   

   The bundle can be downloaded from the PDS4 Repository. Unzipping the package results in a directory named dph_example_archive_VG2PLS.

2. Validate Example Bundle
   

   The next step exercises the Validate Tool by validating the Example Bundle. The Operation document for the Validate Tool provides the details for executing the software. Execute the Validate Tool against the Example Bundle with the following commands:

   % cd $HOME/validate-1.12.0/bin
   % ./validate -t $HOME/dph_example_archive_VG2PLS \
     -x $HOME/dph_example_archive_VG2PLS/xml_schema/PDS4_PDS_1700.xsd \
        $HOME/dph_example_archive_VG2PLS/xml_schema/PDS4_DPH_1700.xsd \
     -S $HOME/dph_example_archive_VG2PLS/xml_schema/PDS4_PDS_1700.sch
               

   The above run results in a successful validation:

   ...
   Summary:
   
     18 of 18 file(s) processed, 0 skipped
     18 of 18 file(s) passed validation
   
   End of Report
               

   Note that the core schema (PDS4_PDS_1700.xsd) was also passed on the command-line because passing any schema on the command-line overrides the schemas provided with the tool. Also note that the -e "*.xml" was not included in the command above. The tool now defaults using *.xml and *.XML as the regular expression for file selection.

3. Register Example Bundle
   

   The next step exercises the Harvest Tool and Registry Service by registering the contents of the Example Bundle. The Operation document for the Harvest Tool provides the details for executing the software. A configuration file (harvest-policy-example.xml) for registering the Example Bundle can be found in the examples directory of the Harvest Tool distribution package. Depending on the location of the Example Bundle in the local environment, the file and path specifications in the configuration file will likely need to be updated for the Example Bundle location in the local environment. Execute the Harvest Tool against the Example Bundle with the following commands:

   % cd $HOME/harvest-1.13.0/bin
   % ./harvest-pds4 -c ../examples/harvest-policy-example.xml
               

   The above run will result in no errors and a few warnings pertaining to references for products that were not found in the local registry. The rest of the products should get registered successfully. The summary should look something like the following:

   ...
   Summary:
   
   13 of 13 file(s) processed, 5 other file(s) skipped
   0 error(s), 2 warning(s)
   
   13 of 13 products registered.
   81 of 81 ancillary products registered.
   
   Product Types Registered:
   5 Product_Collection
   2 Product_Document
   1 Product_Observational
   1 Product_Browse
   3 Product_File_Text
   81 Product_File_Repository
   1 Product_Bundle
   
   79 of 79 generated checksums matched their supplied value in 
   the manifest, 2 value(s) not checked.
   
   67 of 67 generated checksums matched the supplied value in 
   their product label, 14 value(s) not checked.
   
   81 of 81 associations registered.
   
   End of Log
               

4. View Registry Contents
   

   The next step verifies that the registered content from the previous step can be viewed via the Registry Service REST-based API and the Registry User Interface. The Operation document for the Registry Service provides the details for interfacing with the service via the REST-based API. Assuming the registration above was performed against the PDS4 instance of the service, the Registry Service should be accessible via the following URL: http://localhost:8080/registry-pds4/. Accessing the report end point (append report to the URL) from the desired web browser (some browsers display XML better than others), should produce the following output:

   <ns2:report xmlns:ns2="http://registry.pds.nasa.gov" status="OK"
   serverStarted="2016-10-01T11:15:31.291-07:00" associations="337"
   extrinsics="94" services="0" classificationSchemes="2"
   classificationNodes="69" packages="5" events="170"
   registryVersion="1.12.0"/>
               

   The above report shows 94 extrinsics (PDS products) registered which corresponds with the Harvest Tool summary. The number of associations registered is a little harder to verify because there were already some associations in the registry from the configuration step. Feel free to try out some of the other end points (e.g. extrinsics) detailed in the Query Artifacts section of the Operations document.

   Assuming the registration above was performed against the PDS4 instance of the service, the Registry User Interface should be accessible via the following end point: http://localhost:8080/registry-ui/. Select the PDS4 Registry Service instance (http://localhost:8080/registry-pds4/) from the Registry Service pull-down menu and review the resulting product listing. If an "RPC Failure" message appears near the bottom of the screen, this is most likely the result of the Registry Service end point not being configured correctly. The Installation document for the Registry User Interface specifies how to modify the appropriate configuration file to correct this situation.

5. Approve Registry Contents
   

   When content is ingested with the Registry Service, it is given an initial status of Submitted. In order for the Search Core component to index this content, it must be given a status of Approved. Using the Registry User Interface, perform the following steps to approve the Example Bundle:

   1. Select the Packages tab
   2. Identify the package associated with the Example Bundle
   3. Select the check box next to that package
   4. Select the Approved value from the Status pulldown menu
   5. Select the Update Status button

The above steps exercise the core components of the system but are not exhaustive by any measure with respect to testing the software, but they will verify that the software can be installed and executed in the various Node environments. Some Nodes are actively working on generating PDS4 bundles and collections. If a Node would like to attempt to register those products with their local registry the EN would be glad to assist in this effort. In addition, the Harvest Tool supports registering a PDS3 Data Set with the Registry Service. An example configuration file (harvest-policy-pds3.xml) for registering PDS3 Data Sets can be found in the examples directory of the Harvest Tool distribution package. When registering a PDS3 Data Set, the Harvest Tool creates an in-memory proxy label for each product and registers it with the Registry Service. Although not sufficient for PDS4 migration, it does provide a tracking mechanism for PDS3 Data Sets until they can be migrated to PDS4.

Installation Procedures for Search Components

Not all Discipline Nodes will need to install the Search Components and depending on the situation, they may only need to install the Search Service and Search Core components. The installation approach will be determined on a Node-by-Node basis. That said, the following procedure details the installation of all four components. The Search components consist of the Search Service, Search Core and Product Search User Interface. The Search User Interface, which is a Search component, is not required to be installed at the Discipline Node.

1. Search Service
   

   The Installation document for the Search Service provides the details for downloading, installing and configuring the software. The reverse proxy configuration that follows provides security for the Search Service instance. The instance of the service is accessed at /services/search on the web server which is facilitated by the following reverse proxy configuration in the Apache HTTP Server httpd.conf file:

   ProxyPass /services/search/ http://localhost:8080/search-service/
   ProxyPassReverse /services/search/ http://localhost:8080/search-service/
   ProxyPass /services/search http://localhost:8080/search-service
   ProxyPassReverse /services/search http://localhost:8080/search-service
   
   <Location /services/search/admin>
   Order Deny,Allow
   Deny from all
   Allow from 137.78.0.0/255.255.0.0 137.79.0.0/255.255.0.0 .jpl.nasa.gov
   </Location>
               

   The last portion of the configuration above limits access to the Apache Solr Admin interface to folks outside of the institution. The values in the Allow from... line need to be modified for your local network.

2. Search Core
   

   The Installation document for the Search Core software provides the details for downloading, installing and configuring the software.

3. Product Search User Interface
   

   The Installation document for the Product Search User Interface provides the details for downloading, installing and configuring the software. The application can be accessed at /search (the actual end point is up to the user) on the web server which is facilitated by the following reverse proxy configuration in the Apache HTTP Server httpd.conf file:

   ProxyPass /search/ http://localhost:8080/product-search-ui/
   ProxyPassReverse /search/ http://localhost:8080/product-search-ui/
   ProxyPass /search http://localhost:8080/product-search-ui
   ProxyPassReverse /search http://localhost:8080/product-search-ui
               

   The above reverse proxy configuration is optional since there may be a situation where the Node does not want to expose this interface to external users.

4. Data Set View
   

   The Installation document for the Data Set View software provides the details for downloading, installing and configuring the software. When following the instructions in the Configuring the Application section of the Installation document, the values for the registry.url and proxy.registry.url parameters can be ignored (leave them at their defaults). The application is accessed at /ds-view on the web server which is facilitated by the following reverse proxy configuration in the Apache HTTP Server httpd.conf file:

   ProxyPass /ds-view/ http://localhost:8080/ds-view/
   ProxyPassReverse /ds-view/ http://localhost:8080/ds-view/
   ProxyPass /ds-view http://localhost:8080/ds-view
   ProxyPassReverse /ds-view http://localhost:8080/ds-view
               

   The above configuration is only required if the Product Search User Interface is also accessed via a reverse proxy on the web server.

Installation Verification for Search Components

Now that the Search Components are installed, they can be exercised by generating and deploying the search index to the Search Service with the Search Core component. The following command will extract the contents from the PDS4 instance of the Registry Service (populated with the Example Bundle), generate the index and post it to the Search Service:

% search-core -H /usr/local/search-service/pds \
  -p /usr/local/search-core/conf/defaults/pds/pds4/core.properties \
  -l index.log
        

Assuming the above command completes successfully, the Search Service should now contain the products from the Example Bundle. This can be verified via the Search Service interface (http://localhost:8080/search-service). Accessing the service from the desired web browser (some browsers display XML better than others), should produce a default HTML-formatted output listing the products. Accessing the Product Search User Interface (http://localhost:8080/product-search-ui), should also display a list of the products. If products are not displayed, review the Search Core log file for errors. If it shows that zero products were indexed, make sure the Example Bundle products were set to a status of Approved.

Exercise Wrap Up

In addition to the exercises above, the Nodes are encouraged to exercise the Generate Tool, Transform Tool and PDS4 Tools components. See the corresponding Installation and Operation documents for details. For advanced installations, the Transport components may also be installed (details to come later).

The final step of this exercise is to delete the Example Bundle from the Registry Service. This can be accomplished via the Registry User Interface at the following end point: http://localhost:8080/registry-ui/. In the Packages tab just select the entry for the Example package and then select the Delete button. This action may take a few seconds to complete. Now the Registry Service is ready for your Bundles.