Software Documentation
- About Storage Service
- Installation
- Operation
- Development
Download
Project Documentation
This document describes how to operate the Storage Service. The following topics can be found in this document:
Note: The command-line examples in this document have been broken into multiple lines for readability. The commands should be reassembled into a single line prior to execution.
This section describes how to start and stop the Storage Service. The examples below manage the service via a batch/shell script. The service is started with the following command:
% storage-service start
Starting the File Manager
OK
...
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManager <init>
INFO: File Manager started by user
The following table describes the actions available for the storage-service script:
| Action | Description |
|---|---|
| start | Start the service. |
| stop | Stop the service. |
| restart | Stop and then start the service. |
| status | Determine whether the service is running. |
When the Storage Service is started for the first time, it creates the logs and run directories under the installation directory to hold the log and pid files, respectively. When the first product is ingested, it creates the catalog directory to hold the Lucene-based catalog. If the product type is defined to move the product to a local repository, then it creates the archive directory to hold this repository.
This section describes how to initialize the service for supporting the Catalog Tool. Due to a quirk with the Storage Service, at least one product needs to be ingested for a given product type in order to query for products from that product type. The Catalog Tool utilizes the CatalogFile product type defined in the pds policy. The following command ingests the test file included in the package distribution for that product type:
% storage-client --operation --ingestProduct --productName CatalogTest \
--productStructure Flat --productTypeName CatalogFile \
--metadataFile "file:///usr/local/storage-service-1.0.0/test/test.txt.met" \
--refs "file:///usr/local/storage-service-1.0.0/test/test.txt"
Once a successful ingestion has been completed with the Catalog Tool, the test file above may be deleted from the Storage Service. Execute the following command to query for and delete the test product:
% storage-query --sql -query "SELECT * FROM CatalogFile WHERE CAS.ProductName == 'CatalogTest'" \
-outputFormat '$CAS.ProductId' | storage-delete --read
This section describes how to ingest products into, query for products from and delete products from the Storage Service. The examples below interact with the service via batch/shell scripts and assume that the Storage Service is configured with the example policy.
This section describes how to ingest a product with the Storage Service and then verify its ingestion via the storage-client script. The following command is an example of ingesting the test file included in the package distribution:
% storage-client --operation --ingestProduct --productName TransferTest \
--productStructure Flat --productTypeName TransferProduct \
--metadataFile "file:///usr/local/storage-service-1.0.0/test/test.txt.met" \
--refs "file:///usr/local/storage-service-1.0.0/test/test.txt"
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
...
ingestProduct: Result: c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85
The test file above is ingested as product type TransferProduct. This product type assumes the product will be transferred to the Storage Service repository. By default, the transfer occurs using the local data transfer mechanism. If the product resides on a different machine, the remote data transfer mechanism should be utilized. The following command is identical to the above example but overrides the default data transfer mechanism with the RemoteDataTransferFactory:
% storage-client --operation --ingestProduct --productName TransferTest \
--productStructure Flat --productTypeName TransferProduct --clientTransfer \
--dataTransfer org.apache.oodt.cas.filemgr.datatransfer.RemoteDataTransferFactory \
--metadataFile "file:///usr/local/storage-service-1.0.0/test/test.txt.met" \
--refs "file:///usr/local/storage-service-1.0.0/test/test.txt"
In either case above, the following command verifies the product ingestion:
% storage-client --operation --hasProduct --productName TransferTest
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
...
hasProduct: Result: true
The following command is another example of ingesting the test file included in the package distribution, but in this case does not move the product to the repository:
% storage-client --operation --ingestProduct --productName InPlaceTest \
--productStructure Flat --productTypeName InPlaceProduct --clientTransfer \
--dataTransfer org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory \
--metadataFile "file:///user/local/storage-service-1.0.0/test/test.txt.met" \
--refs "file:///usr/local/storage-service-1.0.0/test/test.txt"
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
...
ingestProduct: Result: 168442dc-fb8a-11e0-b6e9-4bb4c5e6fa85
The test file above is ingested as product type InPlaceProduct. This product type assumes the product will be ingested in place, meaning it will not be moved. Because of this, the default data transfer mechanism was overridden with the InPlaceDataTransferFactory.
The storage-client script has a number of other options available. Execute the script with the --help option to see what is available. See the Apache OODT Wiki for more details and examples. The storage-client script is equivalent to the filemgr_client script described on the wiki except that the --url option is specified in the storage-client script and is not required to be passed on the command line.
This section describes how to query the Storage Service for products via the storage-query script. Due to a quirk with the Storage Service, at least one product needs to be ingested for a given product type in order to query for products from that product type. If no products are ingested, the storage-query script will generate an error. The following command queries the service for all ingested products:
% storage-query --sql -query "SELECT * FROM *"
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 2 results
Flat,Product_Test,/Users/shardman/Desktop/storage-0.1.0-dev/archive/TransferTest,\
TransferTest,2011-10-20T19:09:59.587-07:00,test.txt,text/plain,text,plain,\
c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85,TransferProduct
Flat,Product_Test,/Users/shardman/Desktop/storage-0.1.0-dev/test,\
InPlaceTest,2011-10-20T19:12:12.295-07:00,test.txt,text/plain,text,plain,\
168442dc-fb8a-11e0-b6e9-4bb4c5e6fa85,InPlaceProduct
The results from the above command are a little cryptic but serviceable. The following command details specification of the metadata elements to return from the query:
% storage-query --sql --query "SELECT * FROM *" \
-outputFormat 'ProductId = $CAS.ProductId | ProductName = $CAS.ProductName'
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 2 results
ProductId = c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85 | ProductName = TransferTest
ProductId = 168442dc-fb8a-11e0-b6e9-4bb4c5e6fa85 | ProductName = InPlaceTest
The storage-query script has a number of other options available. Execute the script with the --help option to see what is available. See the Apache OODT Wiki for more details and examples. The storage-query script is equivalent to the query_tool script described on the wiki except that the --url option is specified in the storage-query script and is not required to be passed on the command line.
Note: The query function will throw an exception until the first product is ingested. There is also a bug in the current version of the Apache OODT File Manager that causes the query function to fail unless there is at least one product ingested for each product type.
This section describes how to delete products from the Storage Service via the storage-delete script. The following command deletes a single product from the service:
% storage-delete --productID c76a527b-fb89-11e0-b6e9-4bb4c5e6fa85
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.tools.DeleteProduct <init>
INFO: Commit enabled.
Multiple products can be deleted in one command by using the storage-query script in conjunction with the storage-delete script. This is accomplished by formulating the query to return a list of product identifiers that are then piped into the delete command. The following would delete all ingested products from the service:
% storage-query --sql -query "SELECT * FROM *" -outputFormat '$CAS.ProductId' | \
storage-delete --read
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.tools.DeleteProduct <init>
INFO: Commit enabled.
Oct 20, 2011 ... org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 2 results