NASA - National Aeronautics and Space Administration

+ NASA Homepage
+ NASA en Español
+ Contact NASA
Go
Planetary Data System - Engineering Node Banner

Operation

This document describes how to operate the Storage Service. The following topics can be found in this document:

Note: The command-line examples in this document have been broken into multiple lines for readability. The commands should be reassembled into a single line prior to execution.

Manage the Service

This section describes how to start and stop the Storage Service. The examples below manage the service via a batch/shell script. The service is started with the following command:

% storage-service start

Starting the File Manager
OK
...
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManager <init>
INFO: File Manager started by user
      

The following table describes the actions available for the storage-service script:

ActionDescription
startStart the service.
stopStop the service.
restartStop and then start the service.
statusDetermine whether the service is running.

When the Storage Service is started for the first time, it creates the logs and run directories under the installation directory to hold the log and pid files, respectively. When the first product is ingested, it creates the catalog directory to hold the Lucene-based catalog. If the product type is defined to move the product to a local repository, then it creates the archive directory to hold this repository.

Initialize for the Catalog Tool

This section describes how to initialize the service for supporting the Catalog Tool. Due to a quirk with the Storage Service, at least one product needs to be ingested for a given product type in order to query for products from that product type. The Catalog Tool utilizes the CatalogFile product type defined in the pds policy. The following command ingests the test file included in the package distribution for that product type:

% storage-client --operation --ingestProduct --productName CatalogTest \
  --productStructure Flat --productTypeName CatalogFile \
  --metadataFile "file:///usr/local/storage-service-1.0.0/test/test.txt.met" \
  --refs "file:///usr/local/storage-service-1.0.0/test/test.txt"
      

Once a successful ingestion has been completed with the Catalog Tool, the test file above may be deleted from the Storage Service. Execute the following command to query for and delete the test product:

% storage-query --sql -query "SELECT * FROM CatalogFile WHERE CAS.ProductName == 'CatalogTest'" \
-outputFormat '$CAS.ProductId' | storage-delete --read
      

Interact with the Service

This section describes how to ingest products into, query for products from and delete products from the Storage Service. The examples below interact with the service via batch/shell scripts and assume that the Storage Service is configured with the example policy.

Ingest Products

This section describes how to ingest a product with the Storage Service and then verify its ingestion via the storage-client script. The following command is an example of ingesting the test file included in the package distribution:

% storage-client --operation --ingestProduct --productName TransferTest \
  --productStructure Flat --productTypeName TransferProduct \
  --metadataFile "file:///usr/local/storage-service-1.0.0/test/test.txt.met" \
  --refs "file:///usr/local/storage-service-1.0.0/test/test.txt"

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
...
ingestProduct: Result: c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85
        

The test file above is ingested as product type TransferProduct. This product type assumes the product will be transferred to the Storage Service repository. By default, the transfer occurs using the local data transfer mechanism. If the product resides on a different machine, the remote data transfer mechanism should be utilized. The following command is identical to the above example but overrides the default data transfer mechanism with the RemoteDataTransferFactory:

% storage-client --operation --ingestProduct --productName TransferTest \
  --productStructure Flat --productTypeName TransferProduct --clientTransfer \
  --dataTransfer org.apache.oodt.cas.filemgr.datatransfer.RemoteDataTransferFactory \
  --metadataFile "file:///usr/local/storage-service-1.0.0/test/test.txt.met" \
  --refs "file:///usr/local/storage-service-1.0.0/test/test.txt"
        

In either case above, the following command verifies the product ingestion:

% storage-client --operation --hasProduct --productName TransferTest

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
...
hasProduct: Result: true
        

The following command is another example of ingesting the test file included in the package distribution, but in this case does not move the product to the repository:

% storage-client --operation --ingestProduct --productName InPlaceTest \
  --productStructure Flat --productTypeName InPlaceProduct --clientTransfer \
  --dataTransfer org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory \
  --metadataFile "file:///user/local/storage-service-1.0.0/test/test.txt.met" \
  --refs "file:///usr/local/storage-service-1.0.0/test/test.txt"

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
...
ingestProduct: Result: 168442dc-fb8a-11e0-b6e9-4bb4c5e6fa85
        

The test file above is ingested as product type InPlaceProduct. This product type assumes the product will be ingested in place, meaning it will not be moved. Because of this, the default data transfer mechanism was overridden with the InPlaceDataTransferFactory.

The storage-client script has a number of other options available. Execute the script with the --help option to see what is available. See the Apache OODT Wiki for more details and examples. The storage-client script is equivalent to the filemgr_client script described on the wiki except that the --url option is specified in the storage-client script and is not required to be passed on the command line.

Query for Products

This section describes how to query the Storage Service for products via the storage-query script. Due to a quirk with the Storage Service, at least one product needs to be ingested for a given product type in order to query for products from that product type. If no products are ingested, the storage-query script will generate an error. The following command queries the service for all ingested products:

% storage-query --sql -query "SELECT * FROM *"

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 2 results
Flat,Product_Test,/Users/shardman/Desktop/storage-0.1.0-dev/archive/TransferTest,\
TransferTest,2011-10-20T19:09:59.587-07:00,test.txt,text/plain,text,plain,\
c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85,TransferProduct
Flat,Product_Test,/Users/shardman/Desktop/storage-0.1.0-dev/test,\
InPlaceTest,2011-10-20T19:12:12.295-07:00,test.txt,text/plain,text,plain,\
168442dc-fb8a-11e0-b6e9-4bb4c5e6fa85,InPlaceProduct
        

The results from the above command are a little cryptic but serviceable. The following command details specification of the metadata elements to return from the query:

% storage-query --sql --query "SELECT * FROM *" \
-outputFormat 'ProductId = $CAS.ProductId | ProductName = $CAS.ProductName'

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 2 results
ProductId = c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85 | ProductName = TransferTest
ProductId = 168442dc-fb8a-11e0-b6e9-4bb4c5e6fa85 | ProductName = InPlaceTest
        

The storage-query script has a number of other options available. Execute the script with the --help option to see what is available. See the Apache OODT Wiki for more details and examples. The storage-query script is equivalent to the query_tool script described on the wiki except that the --url option is specified in the storage-query script and is not required to be passed on the command line.

Note: The query function will throw an exception until the first product is ingested. There is also a bug in the current version of the Apache OODT File Manager that causes the query function to fail unless there is at least one product ingested for each product type.

Delete Products

This section describes how to delete products from the Storage Service via the storage-delete script. The following command deletes a single product from the service:

% storage-delete --productID c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.tools.DeleteProduct <init>
INFO: Commit enabled.
        

Multiple products can be deleted in one command by using the storage-query script in conjunction with the storage-delete script. This is accomplished by formulating the query to return a list of product identifiers that are then piped into the delete command. The following would delete all ingested products from the service:

% storage-query --sql -query "SELECT * FROM *" -outputFormat '$CAS.ProductId' | \
storage-delete --read

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.tools.DeleteProduct <init>
INFO: Commit enabled.
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 2 results
        

FirstGov Logo
+ Freedom of Information Act
+ NASA 2003 Strategic Plan
+ NASA Privacy Statement, Disclaimer, and
   Accessiblity Certification

+ Copyright/Image Use Policy
NASA Logo
Curator: Emily.S.Law
Webmaster: Maryia Sauchanka-Davis
NASA Official: William Knopf
Last Updated:
+ Comments and Questions