Setup

Included in the Harvest package is an example policy configuration file for PDS3 data product registration. Go to the examples/ folder, make a copy of the harvest-policy-pds3.xml file and modify it as necessary. The following breaks down the policy file example and indicates what modifications are to be performed.

PDS3 Directories

<pds3Directories>
  <path>$HOME/VG2-J-PLS-5-SUMM-ELE-MOM-96.0SEC-V1.0/DATA</path>
  <fileFilter>
    <include>*.LBL</include>
  </fileFilter>
</pds3Directories>
        

Specify the top level directory for Harvest to crawl for products to register.

Access URLs

<accessUrls registerFileUrls="false">
  <accessUrl>
    <baseUrl>http://starbase.jpl.nasa.gov</baseUrl>
    <offset>$HOME</offset>
  </accessUrl>
</accessUrls>
        

Specify the base URL of where the physical data products are located. This allows Harvest to provide links to the physical data products in the slots of each registered product in the Registry.

Checksums

<checksums generate="true">
  <manifest basePath="$HOME/VG2-J-PLS-5-SUMM-ELE-MOM-96.0SEC-V1.0/">
    $HOME/VG2-J-PLS-5-SUMM-ELE-MOM-96.0SEC-V1.0/CHECKSUMS.TXT
  </manifest>
</checksums>
        

Specify a Checksum Manifest file. The basePath attribute is used to resolve relative file references, if found, within a Manifest file. If this attribute is not specified in the policy configuration file, the default behavior is to use the target directory as the base path. If there are multiple target directories specified, then it is required to explicitly specify a base path value in the policy.

With this configuration, Harvest will generate checksums for each file object to be registered and compare against the supplied checksums in the data product label as well as the manifest file.

LID Contents

<lidContents prefix="urn:nasa:pds" appendFilename="true">
  <elementName>DATA_SET_ID</elementName>
</lidContents>
        

Specify any additional elements to use to formulate the LID for each PDS3 data product that will get registered.

Title Contents

<titleContents appendFilename="true">
  <elementName>DATA_SET_ID</elementName>
</titleContents>
        

Specify any additional elements to use to formulate the title for each PDS3 data product that will get registered.

Static Metadata

<staticMetadata>
  <slot name="information_model_version">
    <value>1.2.0.1</value>
  </slot>
  <slot name="product_class">
    <value>Product_Proxy_PDS3</value>
  </slot>
  <slot name="data_set_ref">
    <value>urn:nasa:pds:context_pds3:data_set:data_set.vg2-j-pls-5-summ-ele-mom-96.0sec-v1.0</value>
  </slot>
  <slot name="investigation_ref">
    <value>urn:nasa:pds:context_pds3:investigation:mission.voyager</value>
  </slot>
  <slot name="instrument_host_ref">
    <value>urn:nasa:pds:context_pds3:instrument_host:instrument_host.vg2</value>
  </slot>
  <slot name="instrument_ref">
    <value>urn:nasa:pds:context_pds3:instrument:instrument.pls_vg2</value>
  </slot>
  <slot name="target_ref">
    <value>urn:nasa:pds:context_pds3:target:target.planet.jupiter</value>
  </slot>
</staticMetadata>
        

Specify metadata to go with each PDS3 data product registration.

Ancillary Metadata

<ancillaryMetadata>
  <elementName slotName="observation_start_date_time">
    START_TIME
  </elementName>
  <elementName slotName="observation_stop_date_time">
    STOP_TIME
  </elementName>
  <elementName slotName="product_type">
    PRODUCT_TYPE
  </elementName>
  <elementName slotName="creation_date_time">
    PRODUCT_CREATION_TIME
  </elementName>
  <elementName slotName="encoding_type">
    INTERCHANGE_FORMAT
  </elementName>
</ancillaryMetadata>
        

Specify metadata to extract from each data product label.

Running The Tool

The following command demonstrates the recommended way to run Harvest:

%> harvest -c ../harvest-conf/examples/harvest-policy-pds3.xml -C ../search-conf/defaults -l harvest.log
        

In the example above, the -c flag option specifies the example harvest policy configuration file while the -C flag option specifies location for the default search policy configuration files. The following command is a MAVEN-specific example:

The above command will register the full product label into a Solr Collection index named .system, where it can be looked up using its Logical Identifier, but with period . characters instead of colon : characters due to limitations with Solr. Additionally, Harvest will write out the search index files for the target bundle into a solr-docs directory at the current working directory.

Once the Harvest run is complete, use the Solr Post Tool to ingest the Search documents. Depending on the deployment set up of the Search Service, run the appropriate command below:

For Non-Dockerized Search Service Instances

%> $SOLR_HOME/bin/post -c pds -params tr="add-hierarchy.xsl" $HOME/harvest-2.0.0/bin/solr-docs
        

The Search documents will get ingested into a Solr collection named pds. The last parameter assumes that the solr-docs directory was created in the harvest-2.0.0/bin directory.

For Dockerized Search Service Instances

% docker exec -it search-service post -c pds -params "tr=add-hierarchy.xsl" /data/solr-docs
        

The Search documents will get ingested into a Solr collection named pds. The /data/solr-docs directory references a location within the Search Service Docker Container that is bind-mounted to the solr-docs directory at the Host machine. So this path should always get passed in for each Docker Post command.

Once the Search documents have been indexed, it is recommended to rename the solr-docs directory and move it to a safe location for later retrieval in the event that they will need to be re-indexed.

Tool Execution

The Harvest Tool operates with a policy file to register product metadata. Details on how to create this policy file can be found in the Harvest Policy File section.

This section demonstrates some of the other ways that the tool can be executed:

• Registering PDS3 Data Products From a Single Target
• Registering PDS3 Data Products From Multiple Targets
• Registering PDS3 Data Products From Targets Specified In The Policy File
• Registering PDS3 Data Products To A Secured Registry Instance
• Excluding Sub-Directories To Traverse From a Target

Registering PDS3 Products From a Single Target

The following command demonstrates registering PDS3 data products to a non-secured registry instance from a target directory, $HOME/directory, where only files that end with a .LBL file extension, will be processed:

% ./harvest $HOME/directory -pds3 -e "*.LBL" -c policy.xml -C ../search-conf/defaults
        

Registering PDS3 Products From Multiple Targets

The following command demonstrates how to register PDS3 data products to a non-secured registry instance from two target directories, $HOME/directory1 and $HOME/directory2, using the policy file, policy.xml. Only files that end with a .LBL file extension will be processed. The output will go to a log file, harvest.log:

% ./harvest $HOME/directory1, $HOME/directory2 -pds3 -e "*.LBL" -c policy.xml -C ../search-conf/defaults -l harvest.log
        

Registering PDS3 Products From Targets Specified In The Policy File

Targets can either be specified on the command-line or in in the policy file. Any targets specified on the command-line will overwrite any targets specified in the policy file. The following command demonstrates registering PDS3 data products based on targets specified in the policy file, policy.xml:

% ./harvest -c policy.xml
        

Excluding Sub-Directories To Traverse From a Target

The following command demonstrates registering PDS3 data products from a target directory, $HOME/directory, where the tool will not traverse the sub-directory CONTEXT:

% ./harvest $HOME/directory -pds3 -c policy.xml -C ../search-conf/defaults -D "CONTEXT"
        

Persistence Mode

The Harvest Tool can be run in persistence mode through an XML-RPC accessible web service called a daemon. Under this scenario, the Harvest Tool wakes up periodically, inspects a target directory or directories, and registers the latest products. This section details how to set this up.

In order to run the tool through the daemon, the command-line option flags -P and -w need to be used. This tells the Harvest Tool the port number to use and how long to sleep in between crawls, respectively. When the daemon is running, it can be accessed through the following url: http://localhost:{port number}/xmlrpc. The following command demonstrates launching the Harvest Tool through the daemon on port 9001, where it will wait 120 seconds in between crawls:

% ./harvest -c policy.xml -C ../search-conf/defaults -l log.txt -P 9001 -w 120
        

After running the above command, the daemon will be accessible at http://localhost:9001/xmlrpc.

In order to stop the daemon from running, a daemon controller is needed. The bin/ directory of the Harvest Tool release package contains a shell script, harvest-ctrl, and a batch file, harvest-ctrl.bat, which are used to gracefully shut down the daemon service on a UNIX-like and Windows system, respectively. In addition, they can provide a few additional statistics about the crawling.

The following table describes the command-line options available for harvest-ctrl:

The following table describes the operation names available to pass with the --operation command-line option:

The following examples demonstrate how to run harvest-ctrl using a few of the different operations. For demonstration purposes, assume that the daemon service is located at the following url: http://localhost:9001/xmlrpc.

Determine the Status of the Daemon Service

The following command is used to find out if the daemon service is still running:

% harvest-ctrl --url http://localhost:9001/xmlrpc --operation --isRunning
        

Shutdown the Daemon Service

The following command demonstrates shutting down the daemon service:

% harvest-ctrl --url http://localhost:9001/xmlrpc --operation --stop
        

PDS3 Data Product Registration

By default, the tool registers discovered PDS3 products under the Product_Proxy_PDS3 objectType in the registry. Additionally, the tool has to dynamically create certain metadata in order to support ingestion of PDS3 data products into the registry. The following is an example of a policy file to perform product registration of PDS3 data products:

<policy>
  <!-- Specify a single directory containing the PDS3 data products to register. -->
  <pds3Directory>
    <path>/Users/mcayanan/pds3</path>
    <fileFilter>
      <include>*.LBL</include>
    </fileFilter>
  </pds3Directory>
  <candidates>
    <!--
      Harvest will register PDS3 data products under the objectType
      'Product_Proxy_PDS3'.
    -->
    <pds3ProductMetadata>
      <!-- Tells Harvest what element values to use to create the LID. -->
      <lidContents prefix="urn:nasa:pds">
        <elementName>DATA_SET_ID</elementName>
        <elementName>INSTRUMENT_ID</elementName>
        <elementName>PRODUCT_ID</elementName>
      </lidContents>
      <titleContents>
        <elementName>DATA_SET_ID</elementName>
        <elementName>PRODUCT_ID</elementName>
      </titleContents>
      <staticMetadata>
        <slot name="information_model_version">
          <value>0.8.0.0.k</value>
        </slot>
        <slot name="target_ref">
          <value>urn:nasa:pds:target:MARS::1.0</value>
        </slot>
        <slot name="mission_ref">
          <value>urn:nasa:pds:mission.MER</value>
        </slot>
      </staticMetadata>
      <!-- Register any additional metadata. Default is to
        register metadata defined in the identification area of the
        Product_Proxy_PDS3 schema. -->
      <ancillaryMetadata>
        <elementName slotName="start_time">
          START_TIME
        </elementName>
        <elementName slotName="stop_time">
          STOP_TIME
        </elementName>
      </ancillaryMetadata>
      <includePaths>
        <path>/data/pds3/label</path>
      </includePaths>
    </pds3ProductMetadata>
  </candidates>
</policy>
      

This policy file is made up of the following complex type elements: pds3Directory, pds3ProductMetadata, lidContents, titleContents, staticMetadata, ancillaryMetadata, and includePaths.

pds3Directory

Specify this element to tell the Harvest Tool the directory location to crawl. The following table describes the elements that are allowed:

In the example above, the Harvest Tool will crawl for PDS3 data products starting at the location /data/pds3/dataset, looking for files with a .LBL file extension.

pds3ProductMetadata

Specify this element to tell the Harvest Tool what metadata to ingest into the registry when registering PDS3 data products. This element must be specified within the candidates tag as shown in the example. The following table describes the elements that are allowed:

lidContents

Specify this element to tell the Harvest Tool how to generate the logical identifier. The prefix attribute specifies what to prepend to the logical identifier. This is a required attribute. In the example above, the logical identifiers of every discovered PDS3 data product will be prefixed with urn:nasa:pds.

The following table describes the optional attributes that are allowed:

The following table describes the elements that are allowed:

In the policy example above, the logical identifier will be formed using the following contents: prefix + DATA_SET_ID + ":" + INSTRUMENT_ID + ":" + PRODUCT_ID.

titleContents

Specify this element to tell the Harvest Tool how to generate the title of the registered PDS3 product.

There is an optional attribute that can be specified called appendFilename. When this attribute is specified and set to true, the Harvest Tool will append the product label filename (minus the extension) to the end of the title.

The following table describes the elements that are allowed:

In the policy example above, the title contents will be formed using the following contents: DATA_SET_ID + " " + PRODUCT_ID.

staticMetadata

Specify this element to tell the Harvest Tool to register a set of static metadata with the discovered PDS3 data products. The following table describes the elements that are allowed:

In the policy example above, it specifies that for every PDS3 data product that is registered, the following metadata will be added as slots: information_model_version, target_ref, and mission_ref.

ancillaryMetadata

Specify this element to tell the Harvest tool what additional metadata to register. The following table describes the elements that are allowed:

In the example above, the values from the following elements will be extracted from a PDS3 product label: START_DATE_TIME and STOP_DATE_TIME.

includePaths

Specify this element to tell the Harvest tool the locations of where to find file references specified in a label. By default, the tool will look for the file reference in the location of the label file. The following table describes the elements that are allowed:

In the example above, the tool will look at the /data/pds3/label directory for file references if they cannot be found in the same location as the label file.

Defaults

Default configurations are provided for the following data types (assumes Harvest is installed at /usr/local/harvest, if not, update the file paths as needed):

Specifying Resources

The resources/ folder in the Harvest Tool Release Package contains a JSON-formatted file that represents a snapshot of the Resource Products currently registerd at the PDS Engineering Node. This file is read in at execution time so that the tool can populate the resource.* fields set in the Search Core configuration files. Below is an example of how a resource is formatted in this file:

  "urn:nasa:pds:context_pds3:resource:resource.1076_viola__browsert_asteroids-missions": {
    "resource_name": "Solar System Exploration: Missions to Asteroids", 
    "resource_url": "http://solarsystem.jpl.nasa.gov/missions/profile.cfm?Sort=Target\&Target=Asteroids\&Era=Present"
  }
      

urn:nasa:pds:context_pds3:resource:resource.1076_viola__browsert_asteroids-missions represents the LID of the Resource, while resource_name and resource_url is the name to give to the Resource and the URL, respectively. In the event that your particular resource does not exist, simply edit the resources/registered_resources.json file and add to the existing list in the above format.