Ingest Products

This section describes how to ingest a product with the Storage Service and then verify its ingestion via the storage-client script. The following command is an example of ingesting the test file included in the package distribution:

% storage-client --operation --ingestProduct --productName TransferTest \
  --productStructure Flat --productTypeName TransferProduct \
  --metadataFile "file:///usr/local/storage-service-0.5.0/test/test.txt.met" \
  --refs "file:///usr/local/storage-service-0.5.0/test/test.txt"

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
...
ingestProduct: Result: c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85
        

The test file above is ingested as product type TransferProduct. This product type assumes the product will be transferred to the Storage Service repository. By default, the transfer occurs using the local data transfer mechanism. If the product resides on a different machine, the remote data transfer mechanism should be utilized. The following command is identical to the above example but overrides the default data transfer mechanism with the RemoteDataTransferFactory:

% storage-client --operation --ingestProduct --productName TransferTest \
  --productStructure Flat --productTypeName TransferProduct --clientTransfer \
  --dataTransfer org.apache.oodt.cas.filemgr.datatransfer.RemoteDataTransferFactory \
  --metadataFile "file:///usr/local/storage-service-0.5.0/test/test.txt.met" \
  --refs "file:///usr/local/storage-service-0.5.0/test/test.txt"
        

In either case above, the following command verifies the product ingestion:

% storage-client --operation --hasProduct --productName TransferTest

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
...
hasProduct: Result: true
        

The following command is another example of ingesting the test file included in the package distribution, but in this case does not move the product to the repository:

% storage-client --operation --ingestProduct --productName InPlaceTest \
  --productStructure Flat --productTypeName InPlaceProduct --clientTransfer \
  --dataTransfer org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory \
  --metadataFile "file:///user/local/storage-service-0.5.0/test/test.txt.met" \
  --refs "file:///usr/local/storage-service-0.5.0/test/test.txt"

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
...
ingestProduct: Result: 168442dc-fb8a-11e0-b6e9-4bb4c5e6fa85
        

The test file above is ingested as product type InPlaceProduct. This product type assumes the product will be ingested in place, meaning it will not be moved. Because of this, the default data transfer mechanism was overridden with the InPlaceDataTransferFactory.

The storage-client script has a number of other options available. Execute the script with the --help option to see what is available. See the Apache OODT Wiki for more details and examples. The storage-client script is equivalent to the filemgr_client script described on the wiki except that the --url option is specified in the storage-client script and is not required to be passed on the command line.

Query for Products

This section describes how to query the Storage Service for products via the storage-query script. The following command queries the service for all ingested products:

% storage-query --sql -query "SELECT * FROM *"

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 2 results
Flat,Product_Test,/Users/shardman/Desktop/storage-0.1.0-dev/archive/TransferTest,\
TransferTest,2011-10-20T19:09:59.587-07:00,test.txt,text/plain,text,plain,\
c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85,TransferProduct
Flat,Product_Test,/Users/shardman/Desktop/storage-0.1.0-dev/test,\
InPlaceTest,2011-10-20T19:12:12.295-07:00,test.txt,text/plain,text,plain,\
168442dc-fb8a-11e0-b6e9-4bb4c5e6fa85,InPlaceProduct
        

The results from the above command are little cryptic but serviceable. The following command details specification of the metadata elements to return from the query:

% storage-query --sql --query "SELECT * FROM *" \
-outputFormat 'ProductId = $CAS.ProductId | ProductName = $CAS.ProductName'

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 2 results
ProductId = c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85 | ProductName = TransferTest
ProductId = 168442dc-fb8a-11e0-b6e9-4bb4c5e6fa85 | ProductName = InPlaceTest
        

The storage-query script has a number of other options available. Execute the script with the --help option to see what is available. See the Apache OODT Wiki for more details and examples. The storage-query script is equivalent to the query_tool script described on the wiki except that the --url option is specified in the storage-query script and is not required to be passed on the command line.

Note: The query function will throw an exception until the first product is ingested. There is also a bug in the current version of the Apache OODT File Manager that causes the query function to fail unless there is at least one product ingested for each product type.

Delete Products

This section describes how to delete products from the Storage Service via the storage-delete script. The following command deletes a single product from the service:

% storage-delete --productID c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.tools.DeleteProduct <init>
INFO: Commit enabled.
        

Multiple products can be deleted in one command by using the storage-query script in conjunction with the storage-delete script. This is accomplished by formulating the query to return a list of product identifiers that are then piped into the delete command. The following would delete all ingested products from the service:

% storage-query --sql -query "SELECT * FROM *" -outputFormat '$CAS.ProductId' | \
storage-delete --read

Oct 20, 2011 ... org.apache.oodt.cas.filemgr.tools.DeleteProduct <init>
INFO: Commit enabled.
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 2 results