Release Description - Build 11.0.0
This release of the PDS4 System is intended as an operational release of the system components to date. The following sections can be found in this document:
- Software Changes
- Liens
- Software Installation and Operation
- Engineering Node Software Catalog
- Reference Documents
Software Changes
This section details the new, modified and corrected capabilities that comprise this release . They are summarized here for a system-level view. A more detailed list of capabilities can be found in the change logs for each component.
Note on requirements: Requirement are now being provided for some project. The coverage is not complete yet. Requirements highlighted in red are those which are impacted by tickets completed for the current build.
- Pds Registry App
- Pds Doi Service
- Pds Deep Archive
- Validate
- Pds4 Jparser
- Transform
- Pds4 Information Model
- Pds Site App
- Pds Api
- Pds Wds
- Pds Github Util
Pds Registry App
Documentation: https://nasa-pds.github.io/pds-registry-app/
Improvements
- pds-registry-app#20 - Update Schema Generator for handling special cases where ancestor classes are needed
- pds-registry-app#16 - Design and Implement capability to handle archive_status
- pds-registry-app#22 - Enhance documentation for PDS4 Attribute Search deployment and customization
- pds-registry-app#38 - Complete revised trade study on Solr vs ElasticSearch
- pds-registry-app#6 - Develop PDS4 Attribute Search capability
- pds-registry-app#7 - Develop Customization capability for PDS4 Attribute Search
- pds-registry-app#8 - Design and implement Multi-Node Deployment capability
- pds-registry-app#9 - Design and implement Security and Authorization component
- pds-registry-app#10 - Develop capability for handling multiple product versions
- pds-registry-app#15 - Develop cability to deploy Test vs Operations Registry environments
- pds-registry-app#11 - Develop Registry reporting / visualization component
- pds-registry-app#30 - Design and document Registry cluster expansion approach
Defects
- pds-registry-app#5 - registry installation's documentation may be insufficient
- pds-registry-app#3 - pds-app-registry doesn't build on Windows
- pds-registry-app#21 - Unable to build application due to file size limit
Requirements
default
- pds-registry-app#57 - The service shall accept artifact registrations.
- pds-registry-app#58 - The service shall provide a means identifying relationships between artifact registrations
- pds-registry-app#59 - The service shall maintain configuration regarding the classes of artifacts to be registered
- pds-registry-app#56 - The service shall assign a global unique identifier to a registered artifact
- pds-registry-app#55 - The service shall allow for queries for registered artifacts
- pds-registry-app#65 - The service shall provide a staging capability for artifacts staged for release
- pds-registry-app#54 - The service should be scalable to enable publishing and management of up to 100 million registrations
publish-artifacts
- pds-registry-app#75 - The service shall accept metadata for a registered artifact in a defined format
- pds-registry-app#74 - The service shall validate metadata for a registered artifact
- pds-registry-app#73 - The service shall require a logical identifier and version be provided for all registered artifacts
- pds-registry-app#72 - The service shall store metadata for a registered artifact in an underlying metadata store
manage-artifacts
- pds-registry-app#71 - The service shall allow updates to registered artifacts
- pds-registry-app#70 - The service shall allow deletion of registered artifacts
quality
- pds-registry-app#69 - The service shall enable replication of registry contents with another instance of the service
- pds-registry-app#68 - The service shall require checksums as metadata for registry artifact to enable system-wide integrity checking
generate-metrics
- pds-registry-app#67 - The service shall require a subset of file system metadata in order to support data metrics generation
authorization
- pds-registry-app#66 - The service shall require user authorization for updating registry metadata
Pds Registry Harvest
Documentation: https://nasa-pds.github.io/pds-registry-app
Improvements
- harvest#24 - Implement date conversion from PDS4 date/time strings to Solr format
Defects
- harvest#4 - Fix bug where ingested product start_date_time is off by 12 hours
- harvest#29 - Could not parse date in yyyy-MM-ddZ format
- harvest#30 - For multivalued fields only unique values are stored
Requirements
- None
Pds Registry Mgr Elastic
Documentation: https://nasa-pds.github.io/pds-registry-app
Improvements
- pds-registry-mgr-elastic#3 - Implement Authentication
Defects
- pds-registry-mgr-elastic#7 - update-schema command doesn't work
Requirements
- None
Pds Doi Service
Documentation: https://nasa-pds.github.io/pds-doi-service
Improvements
- pds-doi-service#46 - Implement the command line and parser to release a doi
- pds-doi-service#4 - Develop Status / Query API and component for OSTI status of a DOI
- pds-doi-service#18 - Develop DOI metadata automated validation component
- pds-doi-service#29 - Develop ability to maintain a transaction log database
- pds-doi-service#31 - Develop maintain internal database of DOI requests
- pds-doi-service#32 - Develop Status / Query capability for querying database and return JSON
- pds-doi-service#33 - Develop Pending DOI Handler component for iteratively querying OSTI for DOI status until status change
- pds-doi-service#41 - Develop simple regression test suite for deployment
- pds-doi-service#42 - Update documentation for operational installation and usage
- pds-doi-service#44 - Perform benchmark tests between original prototype software and new system
- pds-doi-service#56 - validate the submitted OSTI record against a schema
- pds-doi-service#58 - create full_name when first/last name cannot be parsed
- pds-doi-service#60 - Add configuration documentation
- pds-doi-service#70 - add check to ensure landing page is online prior to DOI release
- pds-doi-service#82 - End to end behave test
- pds-doi-service#19 - Design REST API and JSON response
- pds-doi-service#20 - Develop draft PDS Policy for Assigning DOIs
- pds-doi-service#28 - Re-organize and clean-up code to meet coding standards
- pds-doi-service#1 - Develop initial requirements and design for DOI Service
- pds-doi-service#2 - Create / Draft a DOI object capability
- pds-doi-service#21 - Reserve a DOI capability
- pds-doi-service#22 - Release DOI capability
- pds-doi-service#24 - Update Released DOI capability
- pds-doi-service#25 - DOI Batch Processing capability
- pds-doi-service#26 - Implement initial DOI database and management
- pds-doi-service#27 - Develop DOI Status / Query component
- pds-doi-service#40 - Develop operational documentation and test suite
- pds-doi-service#65 - Revise DOI Service Requirements
- pds-doi-service#71 - initialize production deployment with pre-existing dois
- pds-doi-service#83 - Write procedure documentation
Defects
- pds-doi-service#72 - node is not provided as contributor in the reserve records
- pds-doi-service#55 - Update default DOI metadata according to changes in requirements
Requirements
DOI management
- pds-doi-service#5 - The software shall be capable of accepting a request to create a draft DOI.
- pds-doi-service#6 - The software shall be capable of accepting a request to reserve a DOI.
- pds-doi-service#7 - The software shall be capable of accepting a request to release a DOI.
- pds-doi-service#8 - The software shall be capable of accepting a request to deactivate a DOI.
- pds-doi-service#9 - The software shall be capable of accepting a request to update DOI metadata.
- pds-doi-service#10 - The software shall be capable of batch processing >1 DOI requests.
DOI metadata
- pds-doi-service#11 - The software shall be capable of autonomously generating the minimum set of DOI metadata from PDS4 Collection, Bundle, Document products.
- pds-doi-service#12 - The software shall validate a minimum set of metadata is provided when reserving, releasing, or updating a DOI. This minimum set of metadata will be defined by the PDS DOI Working Group.
- pds-doi-service#13 - The software shall validate the DOI metadata when reserving, releasing, or updating a DOI.
DOI interface support
- pds-doi-service#14 - The software shall maintain a database of PDS DOIs and their current state.
- pds-doi-service#15 - The software shall maintain the ability to manage DOIs through OSTI
- pds-doi-service#16 - The software shall maintain the ability to manage DOIs through DataCite.
DOI-management
- pds-doi-service#30 - The software shall provide a Status capability that will allow a user to query for the current status of a DOI
- pds-doi-service#35 - The software shall provide the capability of producing a DOI Status Report based upon a user-specified query
Pds Deep Archive
Documentation: https://nasa-pds.github.io/pds-deep-archive/
Improvements
- pds-deep-archive#21 - Develop one script to run both AIP and SIP generator
- pds-deep-archive#24 - Update software to only include latest collection in when bundle references LIDs
- pds-deep-archive#28 - Update docs to include release instructions
- pds-deep-archive#55 - Add READMEs to AIPs and SIPs
- pds-deep-archive#63 - Add timestamp to SIP / AIP product LIDs and filenames
- pds-deep-archive#65 - Update software to handle Document_File.directory_path_name
- pds-deep-archive#8 - Document SIP Gen component into Node Delivery procedures
- pds-deep-archive#13 - Improve SIP Gen performance for very large data sets
Defects
- pds-deep-archive#29 - SIP and Transfer Manifest duplicate files and LIDVIDs
- pds-deep-archive#39 - SIP Generator not outputting correctly for spice example bundle
- pds-deep-archive#41 - AIP Generator not outputting correctly for spice example bundle
- pds-deep-archive#44 - SIP label contains invalid manifest URL using file path
- pds-deep-archive#64 - Software does not exit after running against large data set
- pds-deep-archive#69 - Software appears to be excluding all the products associated with the collections
- pds-deep-archive#71 - AIP Generator outputs duplicate records in the checksum / transfer manifest
- pds-deep-archive#82 - Output SIP uses backslash in output URLs on Windows
- pds-deep-archive#75 - Invalid filenames to match SIP / AIP LIDs
- pds-deep-archive#78 - Standalone AIP Generator Fails
- pds-deep-archive#59 - Update Usage instructions to more clearly define command-line arguments
- pds-deep-archive#77 - Deep Archive doesn't do Windows
Requirements
NSSDCA-delivery-requirements
- pds-deep-archive#45 - The tool shall be capable of generating a valid Archive Information Package transfer manifest and PDS4 XML label in accordance with the PDS4 Information Model
- pds-deep-archive#46 - The tool shall be capable of generating a valid AIP and SIP products and PDS4 XML label in accordance with the PDS4 Information Model
- pds-deep-archive#47 - The tool shall be capable of generating product manifests from an online Registry Service
- pds-deep-archive#48 - The tool shall be capable of generating product manifests by crawling a file system using the information contained within the specified bundle and collection products
- pds-deep-archive#49 - The tool shall generate a product manifest based upon the specification of a bundle as input
- pds-deep-archive#50 - The tool shall include products in the manifests based upon the relationships described in the PDS4 bundle and collection metadata
- pds-deep-archive#51 - The tool shall be capable of accepting a checksum manifest as input to use in place of Registry or on-demand product checksum generation
- pds-deep-archive#52 - The tool shall generate a PDS4 label for the SIP and AIP using at least PDS4 Information Model v1.13.0.0
- pds-deep-archive#53 - The tool shall require AIP and SIP products contain only products with valid logical identifiers according to the PDS4 Standards Reference
- pds-deep-archive#54 - The Submission Information Package manifest shall be a tab-delimited table with one record per product and four fields per record containing the following fields: checksum value, checksum type, resolvable URL to data product, unique product lidvid for the associated product
Validate
Documentation: https://nasa-pds.github.io/validate/
Improvements
- validate#215 - Throw warning when data exists after number of records
- validate#222 - Reduce error messages for overlapping fields
- validate#63 - Validate Table_Character groups and their specified lengths match the specified group_length
- validate#209 - Packed_Data_Fields and bit fields do not validate as expected
Defects
- validate#149 - Update software to check number of records specified in label matches the records in the table
- validate#206 - Validate incorrectly fails ASCII_Integer field that is space-padded (empty) in Table_Character
- validate#219 - validate v1.22.3 has large performance degradation on products with many tables
- validate#220 - Incorrect validation of number of records
- validate#233 - Product validation does not detect the number of table records correctly for Table + Array object
- validate#177 - Software raises a field_value_overlap error on Table_Binary Packed data fields
- validate#234 - Validate gives incorrect records mismatch WARNING for interleaved data objects
- validate#243 - validate 1.24.0-SNAPSHOT chokes on a probably good file
- validate#203 - Assembly plugin non-fatal errors on Windows
- validate#214 - validate outputs an extra '.'
Requirements
default
- validate#15 - Verify that all name/type attribute values correspond to names denoted context products
- validate#24 - Update context check to retrieve and use latest context products from EN Registry
- validate#45 - Check that referenced context products are valid
- validate#50 - Add new target-manifest flag for user to provide a manifest (file) of file/directory paths to validate
- validate#63 - Validate Table_Character groups and their specified lengths match the specified group_length
- validate#196 - Add new functionality to validate TIFF products used as observational products are uncompressed TIFF
- validate#239 - The tool shall verify bundle directories adhere directory naming standards
general-validation-requirements
- validate#247 - The tool shall accept a pre-defined subset of inputs for specifying the associated schema file(s) as described in this issue
- validate#248 - The tool shall verify that a digital objects which comprise observational data may be used in one and only one product
Pds4 Jparser
Documentation: https://nasa-pds.github.io/pds4-jparser/
Improvements
- None
Defects
- None
Requirements
- None
Transform
Documentation: https://nasa-pds.github.io/transform/
Improvements
- None
Defects
- None
Requirements
default
- transform#21 - Add support for GeoTIFF (uncompressed) to GeoTIFF (compressed) transformation
Pds4 Information Model
Documentation: https://nasa-pds.github.io/pds4-information-model/
Improvements
- pds4-information-model#172 - LDDTool: Add PDS Namespace Registry information to Help output
- pds4-information-model#176 - LDDTool: Add error counter or flag to enable unsuccessful termination System.exit
- pds4-information-model#178 - LDDTool: Improve error messages to clearly designate between FATAL_ERROR vs ERROR
- pds4-information-model#222 - LDDTool: Revise the handling of supplying the IM version (alternate version) via command-line
- pds4-information-model#88 - LDDTool: Add capability to generate LDDs based on user-specified IM version
- pds4-information-model#158 - LDDTool: Setup and configure CI/CD and Nightly Builds
- pds4-information-model#165 - CCB-284: Streamline process for adding or removing standard values.
- pds4-information-model#166 - CCB-285: GeoTIFF format as operational PDS4 image
- pds4-information-model#167 - LDDTool: Use sch:value-of to display a variable in Schematron validation
- pds4-information-model#193 - CCB-308: IM schematron needs to be updated to handle SI unit prefixes
- pds4-information-model#87 - LDDTool: Enhance software error handling and logging for improved usability
- pds4-information-model#143 - CCB-268 how to test this
- pds4-information-model#203 - CCB-290: Add value 'Field Campaign' to Investigation_Area type enumerated value list
- pds4-information-model#207 - CCB-289: Define new class Encoded_Audio to describe supplementary audio data products
- pds4-information-model#214 - CCB-292: New enumerated values for reference_type.
- pds4-information-model#216 - CCB-296: Duplicated value in enumerated value list of Product_Context /Internal_Reference /reference_type.
- pds4-information-model#218 - CCB-294: Add enumerated value to Units_of_Pixel_Resolution_Angular
- pds4-information-model#220 - CCB-297: Inappropriate values in permissible values list for Product_Observational/Observation_Area/Target_Identification/Internal_Reference/reference_type
- pds4-information-model#130 - CCB-256: Need method for providing permissible value definitions for external namespaces in Ingest_LDD
Defects
- pds4-information-model#188 - LDDTool: requires one class with (element_flag = true), even when no classes defined
- pds4-information-model#192 - LDDtool combining two consecutive choice blocks into one block
- pds4-information-model#227 - LDDTool: erroneously replaces TBD values with null in JSON output
- pds4-information-model#229 - LDDTool: uses wrong version of XSD when referencing IMG dictionary
- pds4-information-model#168 - LDDTool: Version info bug introduced since v11.2.0
- pds4-information-model#230 - LDDTool: update LDD attribute checking to only check for "type" and "_type"
- pds4-information-model#180 - LDDTool does not handle "unbounded" maximum cardinality in Choice blocks correctly
Requirements
- None
Pds Site App
Documentation: https://github.com/NASA-PDS/pds-site-app
Improvements
- pds-site-app#1 - create mockup for training page
Defects
- None
Requirements
- None
Pds Api
Documentation: http://nasa-pds.github.io/pds-api
Improvements
- pds-api#5 - Define collection-level search parameters
- pds-api#6 - Assign and iterate over applicable API spec sections with API WG
- pds-api#8 - Design and elicit requirements for PDS API Spec
- pds-api#10 - Propose an initial server stub based on swagger output
- pds-api#11 - Identify a framework to generate python api client from a swagger file
- pds-api#1 - Design initial collection-level search API
- pds-api#4 - Convert PDS API Spec to Open API
Defects
- None
Requirements
- None
Pds Api Client
Documentation: https://github.com/NASA-PDS/pds-api-client
Improvements
- None
Defects
- None
Requirements
- None
Pds Api Javalib
Documentation: https://github.com/NASA-PDS/pds-api-javalib
Improvements
- None
Defects
- None
Requirements
- None
Pds Api Service
Documentation: https://github.com/NASA-PDS/pds-api-service
Improvements
- pds-api-service#1 - Test the documentation generation from swaggerhub
Defects
- None
Requirements
- None
Pds Wds
Documentation: https://github.com/NASA-PDS/pds-wds
Improvements
- pds-wds#5 - Select tertiary color palettes
- pds-wds#13 - Create color styles for PDS color swatches.
- pds-wds#15 - Add bold heading typography styles
- pds-wds#3 - Complete initial PDS App Bar widget for simple web deployment
Defects
- None
Requirements
- None
Pds Wds React
Documentation: https://github.com/NASA-PDS/pds-wds-react
Improvements
- pds-wds-react#1 - Add PDS Banner App Bar
Defects
- None
Requirements
- None
Pds Wds Web
Documentation: https://github.com/NASA-PDS/pds-wds-web
Improvements
- pds-wds-web#3 - Rename bar-section to pds-app-bar-section
- pds-wds-web#7 - Rotate caret for open dropdown
Defects
- None
Requirements
default
- pds-wds-web#4 - WCAG compliant
Pds Github Util
Documentation: https://github.com/NASA-PDS/pds-github-util
Improvements
- None
Defects
- None
Requirements
- None
Liens
This section details the liens or deviations from the plan for Build 10b along with rationale for the deviation.
To Be Completed
Installation and Operation
This section provides details and links to the installation and operation instructions for setting up Standalone PDS Tools, Discipline Node Environment, and internal Engineering Node Environment and Services.
- Discipline Node Environment - Tools and services intended to be installed at discipline nodes and international partners.
- Standalone PDS Tools - Tools and services intended for the preparation, inspection, transformation, and visualization of PDS data products.
- Engineering Node Environment - Tools and services intended to be deployed in Engineering Node operations.
Discipline Node Environment
This section details the deployment to a Discipline Node environment. The following tools and services are used to ensure the validity and overall management of products in the archive.
The tools and services are organized into two sections:
- Required - These tools and services should be installed by all discipline nodes. Please notify PDS Operator if you are unable to install this software.
- Optional - These tools and servers are optional installations at this time. Please read the software descriptions = in the table below to determine if the software is applicable for your node. Some of these components may become required in the future.
NOTE: The next generation Registry is still under construction. Engineering Node will work with Nodes
individually to deploy the software as we continue to test it out. Please bare with us during this transition.
Required Software
| Tool | Installation/Operation Guides | Description |
|---|---|---|
| Validate | On Github Pages | Software for validating PDS4 product labels and product data |
| Feedback Widget | On Github | Website widget for prompting users in need of assistance or have feedback. |
Optional Software
| Tool | Installation/Operation Guides | Description |
|---|---|---|
| PDS Registry Application | On Github Pages | PDS Registry provides functionality for tracking, searching, auditing, locating, and maintaining artifacts within the system. These artifacts can range from data files and label files, schemas, dictionary definitions for objects and elements, services, etc. |
Standalone PDS Tools
| Tool | Installation/Operation Guides | Description |
|---|---|---|
| MILabel | On Github | Provides a command-line interface for generating PDS4 Labels using a user provided PDS4 XML template and input (source) data products. |
| Local Data Dictionary Tool (LDDTool) | On Github | Parses a local data dictionary definition file and generates PDS4 data standard files for archival Local Data Dictionaries. |
| PDS Deep Archive | On Github | Software for validating PDS4 product labels and product data |
| PDSView | On Github | Provides support for visualizing PDS3 and PDS4 data products. This application will eventually deprecate NASAView. |
| Transform | On Github | Transforms PDS3 and PDS4 product labels and data into various formats. |
| Validate | On Github | Software for validating PDS4 product labels and product data |
Engineering Node Environment
This section provides an overview of the deployment to the Engineering Node operations environment. The following diagram depicts the EN system flow, starting with central catalog migration, data product registration and finishing up with search index generation.
Click the image for a larger version.
For this particular release, here are the components that require a new/upgraded operational deployment:
| Tool | Installation/Operation Guides |
|---|---|
| Validate | On Github Pages |
| Feedback Widget | On Github |
Engineering Node Software Catalog
The following list links to all of the software components currently being maintained and operated by the PDS Engineering Node as of the deployment of this build.
- Ingest
- Catalog Tool(EN Only)
- Harvest Tool
- Harvest-PDAP Tool (EN Only)
- Model
- Portal
- Feedback Widget
- Data Search (EN Only)
- Data Dictionary Search (EN Only)
- Data Set View
- Filter
- Phone Book Search (EN Only)
- Tool Registry (EN Only)
- Preparation Tools
- Report
- Report Manager (EN Only)
- Search
- Registry
- Search User Interface (EN Only)
- Security Service (EN Only)
- Storage
- Storage Service (EN Only)
- Product Service (EN Only)
- Tracking
- Tracking Service (EN Only)
- Transform
- Transport
Reference Documents
This section details the controlling and applicable documents referenced for this release. The controlling documents are as follows:
- PDS Level 1, 2 and 3 Requirements, April 20, 2017.
- PDS4 Project Plan, July 17, 2013.
- PDS4 System Architecture Specification, Version 1.3, September 1, 2013.
- PDS4 Operations Concept, Version 1.0, September 1, 2013.
- PDS General System Software Requirements Document (SRD), Version 1.1, September 1, 2013.
- PDS Harvest Tool Software Requirements and Design Document (SRD/SDD), Version 1.2, September 1, 2013.
- PDS Preparation Tools Software Requirements and Design Document (SRD/SDD), Version 0.3, September 1, 2013.
- PDS Registry Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
- PDS Report Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
- PDS Search Service Software Requirements and Design Document (SRD/SDD), Version 1.0, September 1, 2013.
- PDS Search Scenarios, Version 1.0, September 1, 2013.
- PDS Search Protocol, Version 1.2, March 21, 2014.
- PDAP Search Protocol, Version 1.0, March 21, 2014.
- PDS Security Service Software Requirements and Design Document (SRD/SDD), Version 1.1, September 1, 2013.
- PDS Deep Archive Sotware Requirements and Design Document (SRD/SDD)
- PDS DOI Service Requirements and Design Document (SRD/SDD)