UNIX-Based Setup

This section details the environment setup for UNIX-based machines. The preferred method is to specify the shell script, Harvest, on the command-line. Setting the PATH environment variable to the location of the script, enables the shell script to be executed from any location on the user's machine.

The following command demonstrates how to set the PATH environment variable, by appending to its current setting:

% setenv PATH ${PATH}:$HOME/harvest-0.3.0/bin
        

The tool can now be executed via the shell script as demonstrated in the following example:

% Harvest <policy file> <command-line arguments>
        

Additional methods for setting up a UNIX-based environment can be found in the UNIX Setup Options section. If viewing this document in PDF form, see the appendix for details.

Windows Setup

This section details the environment setup for Windows machines. The preferred method is to specify the batch file, Harvest.bat, on the command-line. Setting the PATH environment variable to the location of the file, enables the batch file to be executed from any location on the user's machine.

The following command demonstrates how to set the PATH environment variable, by appending to its current setting:

C:\> set PATH = %PATH%;C:\harvest-0.3.0\bin
        

The tool can now be executed via the batch file as demonstrated in the following example:

C:\> Harvest <policy file> <command-line arguments>
        

Additional methods for setting up a Windows environment can be found in the Windows Setup Options section. If viewing this document in PDF form, see the appendix for details.

UNIX-Based Users

Open the Harvest shell script and go to the last line in the file. It should look like the following:

% java -Dpds.registry="http://localhost:8080/registry-service" -jar ${HARVEST_JAR} "$@"
        

Replace the URL value of pds.registry with the URL to the desired instance of the Registry. For example, making the following change to the script will have Harvest pointing to the secured operational instance of the Registry at the Engineering Node:

% java -Dpds.registry="https://pdsops2.jpl.nasa.gov/registry-service" -jar ${HARVEST_JAR} "$@"
        

Windows-Based Users

Open the Harvest batch and go to the last line in the file. It should look like the following:

% java -Dpds.registry="http://localhost:8080/registry-service" -jar "%HARVEST_JAR%" %*
        

Replace the URL value of pds.registry with the URL to the desired instance of the Registry. For example, making the following change to the batch file will have Harvest pointing to a secured operational instance of the Registry at the Engineering Node:

% java -Dpds.registry="https://pdsops2.jpl.nasa.gov/registry-service" -jar "%HARVEST_JAR%" %*
        

Command-Line Options

The following table describes the command-line options available:

Execute Harvest Tool

This section demonstrates execution of the tool using the command-line options. The examples below execute the tool via the batch/shell script. Alternate methods for executing the tool can be found in the Tool Setup section.

The Harvest Tool operates with a policy file to register product metadata. Details on how to create this policy file can be found in the Harvest Policy File section.

The following command demonstrates how to run the Harvest Tool against a policy file, policy.xml, using a valid username and password, with the output going to standard out:

% Harvest policy.xml -u {username} -p {password}
        

The following command demonstrates how to run the Harvest Tool with the output going to a log file, log.txt instead of standard out:

% Harvest policy.xml -u {username} -p {password} -l log.txt
        

When registering product metadata to a non-secured instance of the Registry (such as one running on your local machine), the -u and -p command-line option flags do not need to be passed into the tool. The following command demonstrates how to run the Harvest Tool to register product metadata to a non-secured instance of the Registry Service, with the output going to a log file:

% Harvest policy.xml -l log.txt
        

Harvest Policy File

The Harvest policy file is an XML file that the tool uses to find products and register their metadata. The schema for the policy file can be found in the Harvest Policy Schema section. If viewing this document in PDF form, see the appendix for details.

The following is an example of a policy file:

<?xml version="1.0" encoding="UTF-8"?>
<policy>
   <bundles>
      <file>/home/pds4/context-bundle/bundle.xml</file>
   </bundles>
   <collections>
      <file>/home/pds4/insthost/collection_instrument_host.xml</file>
   </collections>
   <directories>
      <path>/home/user/pds4/geo/product_files</path>
      <filePattern>*.xml</filePattern>
   </directories>
   <validation>
      <enabled>true</enabled>
   </validation>
   <candidates>
      <namespace prefix="geo" uri="http://pds.nasa.gov/schema/pds4/geo"/>
      <productMetadata objectType="character_table">
         <xPath>//geo:Product_Identification_Area/geo:creation_date_time</xPath>
         <xPath>//geo:Subject_Area/geo:instrument_name</xPath>
         <xPath>//Subject_Area/observing_system_name</xPath>
      </productMetadata>
      <productMetadata objectType="Product_Target">
         <xPath>//alternate_title</xPath>
         <xPath>//creation_date_time</xPath>
         <xPath>//identifier</xPath>
         <xPath>//Subject_Area/target_name</xPath>
      </productMetadata>
   </candidates>
</policy>
        

The policy file is made up of the following complex type elements: bundles, collections, directories, validation, candidates, and productMetadata.

Specify this element to tell the Harvest Tool to register and crawl a bundle file. The following table describes the elements that are allowed:

In the example above, the Harvest Tool will register the bundle file named /home/pds4/context-bundle/bundle.xml. It will then crawl the bundle file, looking for collection files to register and process.

Specify this element to tell the Harvest Tool to register and crawl a collection file. Crawling only occurs when the collection file is a primary collection. This is indicated by a value of true in the is_primary_collection element tag within the collection.

The following table describes the elements that are allowed:

In the example above, the Harvest Tool will register the collection file named /home/pds4/insthost/collection_instrument_host.xml. It will then crawl the file, looking for products to register if it is a primary collection.

Specify this element to tell the Harvest Tool where to crawl for data products. The following table describes the elements that are allowed:

In the example above, the Harvest tool will crawl the directory location, /home/user/pds4/geo/product_files, looking for files that have a .xml file extension. The default is to touch all files in the directory if the filePattern element is omitted from the policy file.

validation

Specify this element to tell the Harvest Tool to validate a data product before registering it. If the data product does not pass the validation step, the data product will not be registered. The following table describes the elements that are allowed:

By default, if the validation element is not specified in the policy file, validation is turned on.

candidates

Specify this element to tell the Harvest Tool what product types to register and what metadata to extract from a data product. This is a required element in the policy file. The following table describes the elements that are allowed:

By default, the Harvest Tool defines the default namespace to be the PDS namespace, http://pds.nasa.gov/schema/pds4/pds. To override this default, specify the default attribute in the namespace element and give it a value of true. The following makes the geo namespace the default namespace:

          <candidates>
          <namespace prefix="geo" uri="http://pds.nasa.gov/schema/pds4/geo" default="true"/>
          ...
        

Namespaces need to be defined in the Harvest policy file only if the metadata to be extracted exists in a namespace other than the PDS namespace. In the example above, a namespace with the prefix geo and uri http://pds.nasa.gov/schema/pds4/geo has been defined. This means that any xPath expressions defined in the policy file will be able to use the geo prefix to be able to extract metadata that are within the geo namespace. xPaths will be explained in greater detail in the productMetadata section.

Specify this element to tell the Harvest Tool what metadata to register. It requires an attribute called objectType that tells the Harvest Tool what product types to register. The following table describes the elements that are allowed:

In the example above, the policy file tells the Harvest Tool to look for and register the character_table and Product_Target object types.

Also in the example is a set of xPath elements found under each productMetadata element. This defines what metadata to extract from the different products. XPath is a query language that uses path expressions to select nodes in an XML document. These path expressions look very much like expressions in a traditional computer file system. In its simplest form, prepending a // before a name will find the element no matter where it is in the XML file.

The following XPath expression will find the creation_date_time element within the default namespace, no matter where this element is located in the file:

//creation_date_time
        

The following XPath expression will find the creation_date_time element within the geo namespace, no matter where this element is located in the file:

//geo:creation_date_time
        

The following XPath expression will find all target_name elements that are children of Subject_Area within the default namespace:

//Subject_Area/target_name
        

The following XPath expression will find all target_name elements that are children of Subject_Area and that have a value of MARS:

//Subject_Area/target_name[text()="MARS"]
        

For a more detailed explanation on XPath, go to your favorite search engine and type XPath tutorial.

Continuous Mode

The Harvest Tool can be run in continuous mode through an XML-RPC accessible web service called a daemon. Under this scenario, the Harvest Tool wakes up periodically, inspects a target directory or directories, and registers the latest products. This section details how to set this up.

In order to run the tool through the daemon, the command-line option flags -P and -w need to be used. This tells the Harvest Tool the port number to use and how long to sleep in between crawls, respectively. When the daemon is running, it can be accessed through the following url: http://localhost:{port number}/xmlrpc. The following command demonstrates launching the Harvest Tool through the daemon on port 9000, where it will wait 120 seconds in between crawls:

% Harvest policy.xml -u {username} -p {password} -l log.txt -P 9000 -w 120
        

After running the above command, the daemon will be accessible at http://localhost:9000/xmlrpc.

In order to stop the daemon from running, a daemon controller is needed. The bin/ directory of the Harvest Tool release package contains a shell script, HarvestController, and a batch file, HarvestController.bat, which are used to gracefully shut down the daemon service on a UNIX-like and Windows system, respectively. In addition, they can provide a few additional statistics about the crawling.

The following table describes the command-line options available for the HarvestController:

The following table describes the operation names available to pass into the --operation command-line flag option:

The following examples demonstrate how to run the HarvestController using a few of the different operations. For demonstration purposes, assume that the daemon service is located at the following url: http://localhost:9000/xmlrpc.

Shutdown the daemon service

The following command demonstrates shutting down the daemon service:

% HarvestController --url http://localhost:9000/xmlrpc --operation --stop
        

Find Out The Status Of The Daemon Service

The following command is used to find out if the daemon service is still running:

% HarvestController --url http://localhost:9000/xmlrpc --operation --isRunning