 |
|
|
 |
Operation
This document describes how to operate the Search Core software for a generic environment, see the Engineering Node Operation Procedures for more detailed operation instruction for the Engineering Node installation. The following topics can be found in this document:
Note: The command-line examples in this section have been broken into multiple lines for readability. The commands should be reassembled into a single line prior to execution.
Tool Execution
Search Core can be executed in various ways. This section describes how to run the tool, as well as its behaviors and caveats.
Command-Line Options
The following table describes the command-line options available:
| Command-Line Option | Description |
|---|---|
| -a,--all | Run all components of the Search Core [default] |
| -c,--config-home <directories> | Specify the product class configuration home directory. Must contain product-classes.txt. Multiple directories can be specified to accompany multiple registries. (Default: $SEARCH_CORE_HOME/conf/pds/) |
| -C,--clean-dirs | Turn off removal of all directories from previous Search Core execution output. These directories will still be backed up in the Search Home directory. (Default: True) |
| -d,--debug | Turn on debugger. |
| -e,--extractor | Execute component to extract data from registry |
| -H,--search-home <directory> | Specify the Search Home directory. The tool will output the index files to this directory. When using the Search Service, this should be the $SEARCH_SERVICE_HOME/pds directory (Default: $SEARCH_SERVICE_HOME/pds directory) |
| -h,--help | Display usage. |
| -i,--solr-indexer | Execute component to generate a Solr Index |
| -m,--query-max <integer> | Specify the maximum number of registry values to be returned from query. (Default: 999999999) |
| -p,--properties-file <files> | Specify properties file containing Search Home, Registry URL, and product class configuration home directory. Multiple files can be specified. |
| -r,--registry <urls> | Specify Registry Service instance to query. Multiple registries can be specified. |
| -V,--version | Display application version. |
Execute Search Core Tool
This section demonstrates execution of the tool using the command-line options. The examples below execute the tool via the batch/shell script.
The Search Core requires, at minimum, a SEARCH_HOME directory and REGISTRY_URL be specified via the command-line. The following is the format for the command:
% search-core -H <SEARCH_HOME> -r <REGISTRY_URL> [options]
Since we are running the Search Core with the Search Service, SEARCH_HOME = $SEARCH_SERVICE_HOME/pds. The 'pds' appended to the SEARCH_SERVICE_HOME represents the specific core of Solr we want to apply this index. The following demonstrates how to run the Search Core with a SEARCH_SERVICE_HOME=/usr/local/search-service and REGISTRY_URL=http://localhost:8080/registry:
% search-core -H /usr/local/search-service/pds -r http://localhost:8080/registry
By default, the command above runs all components of the Search Core software and produces a Solr index from the Registry Service data. The solr_index.xml file(s) will appear in the $SEARCH_SERVICE_HOME/pds/index directory. To run each component separately, they must be completed in the following order:
-
The following command will run the Search Core with a SEARCH_SERVICE_HOME=/usr/local/search-service and only run the Registry Extractor component to generate XML index documents for each Registry context product type in the directory /usr/local/search-service/pds/registry-data/<object-type>/:
% search-core -e -H /usr/local/search-service/pds -r http://localhost:8080/registry
-
The following command will run the Search Core with a SEARCH_SERVICE_HOME=/usr/local/search-service and only run the Solr Indexer component, which parses the XML documents produced by the Registry Extractor, and generates Solr indexes located in /usr/local/search-service/pds/index :
% search-core -i -H /usr/local/search-service/pds -r http://localhost:8080/registry
The following command demonstrates how to test the Search Core with a SEARCH_SERVICE_HOME=/usr/local/search-service and only query 100 products for indexing (useful for testing purposes):
% search-core -H /usr/local/search-service/pds -r http://localhost:8080/registry -m 100
The following command demonstrates how to specify a SEARCH_HOME and REGISTRY_URL via a Search Core properties file:
% search-core -p /usr/local/search-core/conf/pds/pds3/core.properties
An example Search Core properties file looks like this:
search.core.search-home = /usr/local/search-service/pds search.core.registry-url = http://localhost:8080/registry search.core.config-home = /usr/local/search-core/conf/pds/pds3
The following command demonstrates how to specify multiple registries:
% search-core -H /usr/local/search-service/pds -r http://localhost:8080/registry http://localhost:8080/registry-psa
The following command demonstrates how to specify multiple Search Core property files:
% search-core -p /usr/local/search-core/conf/pds/pds3/core.properties /usr/local/search-core/conf/psa/pds3/core.properties
Solr Ingestion
Once the Lucene index has been created using the Search Core command-line tool, we want to ingest the data into the Apache Solr web application so it can be searched. To do so we use the solr-post shell script using curl and HTTP POST to ingest the data in Solr. First, in order to run this script, you must go to the $SEARCH_CORE_HOME/bin directory:
Make sure SEARCH_CORE_HOME is in the PATH prior to completing the following procedure
The following command demonstrates how to run the script:
% solr-post <SEARCH_HOME> <SEARCH_SERVICE_URL>
For example:
% $SEARCH_CORE_HOME/solr-post /usr/local/search-service/pds http://localhost:8080/search-service/pds
Basic Index Generation
The initial-index shell script is included in the package in an effort to streamline the process by running the Search Core software to produce the index, and the solr-post script to post the data to the Solr application. Go to the $SEARCH_CORE_HOME/bin directory to update the environment variables ($SEARCH_CORE_HOME/bin/env-vars) and run the script:
Assumptions. The initial-index script is intended only for those installations that use only 1 registry and the default extractor configuration found in $SEARCH_CORE_HOME/conf/pds/pds3/product-classes.txt.
Using Environment Variables
- Update $SEARCH_CORE_HOME/bin/env-vars with, at minimum, the following information:
- SEARCH_HOME - The path for the home directory of local instance of search service.
- SEARCH_SERVICE_URL - URL of the Search Service instance.
- REGISTRY_URL - URL of the Registry Service instance.
- Run the initial-index script:
% $SEARCH_CORE_HOME/initial-index
Using Command-Line Arguments
The initial-index script can also be run using command-line arguments in lieu of updating env-var:
% $SEARCH_CORE_HOME/initial-index <SEARCH_HOME> <SEARCH_SERVICE_URL> <REGISTRY_URL>
For example:
% $SEARCH_CORE_HOME/initial-index /usr/local/search-service/pds http://localhost:8080/search-service/pds http://localhost:8080/registry
Test Search Service
Once this script completes, the data should be available through the Search Service interface. Go to http://localhost:8080/search-service/pds/search/?q=*:* to verify data is available (modify domain and port as needed). See the Search Service - Operate page for more information on how to query data.
Note: To further streamline the process, the command-line arguments can be removed from the initial-index script, and the search-core line can modified according to your specific needs.
Post To Operations
Some installations require building an index to be replicated between multiple secured machines. Secured machines, meaning POSTing data remotely is forbidden. Instead of generating a new index on each machine, the $SEARCH_CORE_HOME/bin/ops-index script was developed to rsync a previously generated index from a remote machine and POST the data to a local Search Service installation. The following describes the how to use the script:
Using Environment Variables
- Update $SEARCH_CORE_HOME/bin/env-vars with, at minimum, the following information:
- SEARCH_HOME - The path for the home directory of local instance of search service.
- SEARCH_SERVICE_URL - URL of the Search Service instance.
- SOURCE - Source machine name where initial index was generated
- SOURCE_USER - Username to connect source machine
- SOURCE_PATH - Optional variable. Path on source machine to directory containing solr_index.xml.* . Defaults to SEARCH_HOME/index
- Run the ops-index script:
% $SEARCH_CORE_HOME/ops-index
Using Command-Line Arguments
The ops-index script can also be run using command-line arguments in lieu of updating env-vars:
% $SEARCH_CORE_HOME/ops-index <SEARCH_HOME> <SEARCH_SERVICE_URL> <SOURCE> <SOURCE_USER> [<SOURCE_PATH>]
For example:
% ./ops-index /usr/local/search-service/pds http://localhost:8080/search-service/pds pdsbeta.jpl.nasa.gov root
Test Search Service
Once this script completes, the data should be available through the Search Service interface. Go to http://localhost:8080/search-service/pds/search/?q=*:* to verify data is available (modify domain and port as needed). See the Search Service - Operate page for more information on how to query data.
Common Errors
Error running Registry Extractor
This error arises when there is an error connecting with the Registry. The following are potential mitigation strategies:
- Verify the Registry URL specified is correct and accessible.
- Verify the Registry is populated with data by going to $REGISTRY_URL/report. See Registry Documentation for more information.
501 Method Not Implemented
This error occurs when using solr-post script and the HTTP POST method to ingest data into Solr. This usually means that the Search Service URL specified is either incorrect or attempting to access a port that is not open to the HTTP POST method. The following are potential mitigation strategies:
- Verify Search Service URL is correct by navigating to the page. The Solr Welcome screen should appear.
- Try changing the URL to access the localhost or an intranet-only accessible port (i.e. 8080). Often servers do not allow the HTTP POST method through port 80 when accessing Tomcat. For instance, if the Search Service URL used is http://my-host/search-service, try http://localhost:8080/search-service or http://my-host:8080/search-service (ports will differ according to Tomcat installation).
search-core: command not found
This error occurs when the system cannot find the search-core script in the PATH. See Installation Instructions for more information on adding the search-core to the PATH.
 |
 |
|
 |
|