Standards Change Request

Increase DATA_QUALITY_ID to 16 characters                      SCR3-1141.v3

Provenance:
-----------

Date: 2009-06-21
Author(s): Richard Chen and Elizabeth Rye (EN), with edits from R. Simpson (RS)
Working Group: Richard Chen (lead), Elizabeth Rye

Date: 2008-12-05
Author(s): Richard Chen and Elizabeth Rye (EN)
Working Group: Richard Chen (lead), Elizabeth Rye
Title: Increase DATA_QUALITY_ID to 16 characters (SCR3-1141.v2)

Date: 2008-07-09
Author(s): Richard Chen (EN)
Working Group: Richard Chen (lead), Elizabeth Rye
Title: Increase DATA_QUALITY_ID to 16 characters (SCR3-1141.v1)

Problem:
--------

The keyword DATA_QUALITY_ID is currently defined with a maximum length of three characters and a general_data_type of "IDENTIFIER".  The existing standard values are integers ranging from -1 to 4 plus "N/A".  The MDIS (Mercury Dual Imaging System) on Messenger, for the EDR and CDR products, treat data_quality_id as a 16-byte flag:

- byte 0: image source is CCD
- byte 1: non-zero exposure time
- byte 2: excessive number of pixels at or approaching saturation
- byte 3: valid pivot position
- byte 4: filter wheel in position
- byte 5: quality of spacecraft attitude knowledge
- byte 6: CCD temperature range
- byte 7: completeness of data within the commanded selection
- bytes 8-15: spare

(from the MDIS EDR SIS v2G at
http://pdsimg.jpl.nasa.gov/data/messenger/MDIS/DOCUMENT/MDISEDRSIS.PDF
and the "MDIS CDR/RDR Software Interface Specification, v1.2")

Current Urgency:
----------------

Medium.  Some Messenger data has already been ingested, although future releases are still pending.  The MDIS instrument uses this keyword in many if not all labels for their data files.

Proposed Solution:
------------------

Change the general data type of the DATA_QUALITY_ID keyword to "CHARACTER" and increase the length from 3 to 16 characters.

Impact Assessment:
------------------

PDS Standards Reference -- Section 5.5.1b of the Standards Reference uses the DATA_QUALITY_ID keyword as an example of how to change the specific implementation of a keyword for a particular mission using local data dictionaries.  The example uses a case similar to the current one in which the Cassini mission wishes to use the DATA_QUALITY_ID for a character type value up to 50 characters in length.  At minimum, this example will have to be updated to show CHAR(16) rather than CHAR(3) for the PSDD version of the keyword.

Archive Preparation Guide -- no impact

Proposer's Archive Guide -- no impact

Planetary Science Data Dictionary -- The keyword, as currently defined in the Data Dictionary, already has some problems.  The existing integer values are inconsistent with the IDENTIFIER general data type.  Updating the general data type to CHARACTER will resolve this problem.  In addition, the maximum_length will be updated from 3 to 16.

PDS tools -- no impact

Existing archives -- As far as we've been able to determine, the node (PPI) which originally added this keyword to the PSDD has never used it.  The only other data set we've found which employs this keyword is the Clementine EDR set "CLEM-1-L/E/Y-A/B/U/H/L/N-2-EDR-V1.0".  It appears that every product in this data set has the value "N/A" for this keyword, plus the SIS for that data set says:

DATA_QUALITY_ID = "N/A"

Data Quality indicator. This keyword is reserved for future use.         
As processing of Clementine data proceeds, this keyword may                  
be used to describe data quality information. No data quality                
parameters have currently been defined.    

We therefore see no impact by making the proposed changes.

Additional Information:
-----------------------

(none)

Requested Changes:
------------------

In addition to increasing the maximum length for values of this keyword, this SCR proposes to fix a number of other problems with the keyword.  It originally had a general data type of IDENTIFIER, which is invalid for strictly numeric values.  Therefore the following list of changes is proposed:

- change the MAXIMUM_LENGTH from 3 to 16
- change the GENERAL_DATA_TYPE from IDENTIFIER to CHARACTER
- change the STANDARD_VALUE_TYPE from DEFINITION to TEXT
- remove the existing standard values
- change the SQL_FORMAT and BL_SQL_FORMAT from char(3) to char(16)
- change the GENERAL_CLASSIFICATION_TYPE from PLASMA to DATASET
- populate the NOTE field with explanations for the values used
- populate the SOURCE_NAME with "PPI"

PDS_VERSION_ID                    = PDS3
LABEL_REVISION_NOTE               = "2009-06-21: RLC/EN Increased length to 16, changed data type"

OBJECT                            = ELEMENT_DEFINITION
  ELEMENT_NAME                    = "data_quality_id"
  BL_NAME                         = "dataqualid"
  DESCRIPTION                     = "

The data_quality_id element provides a numeric
key which identifies the quality of data available for a
particular time period.
The data_quality_id scheme is unique to a given
instrument and is described by the associated
data_quality_desc element."

  GENERAL_DATA_TYPE               = "CHARACTER"
  MAXIMUM                         = "N/A"
  MINIMUM                         = "N/A"
  MAXIMUM_LENGTH                  = "16"
  MINIMUM_LENGTH                  = "N/A"
  STANDARD_VALUE_TYPE             = "TEXT"
  STANDARD_VALUE_SET_DESC         = ""
  KEYWORD_DEFAULT_VALUE           = ""
  UNIT_ID                         = "none"
  SOURCE_NAME                     = "PPI"
  FORMATION_RULE_DESC             = ""
  NOTE                            = "
As originally proposed for Plasma data sets, the values for this keyword ranged from "-1" to "4", indicating worst to best quality.

Huygens similarly used a range of values from "1" to "5" to indicate data quality.

For the Mercury Dual Imaging System on the Messenger spacecraft, the values have the following meanings:

- byte 0: image source; 0 = CCD, 1 = test pattern
- byte 1: exposure time; 0 = greater than 0 ms, 1 = 0 ms
- byte 2: excessive saturation; 0 = < 5 pixels saturated, 1 = > 5
- byte 3: pivot position; 0 = valid, 1 = not valid
- byte 4: filter wheel position (WAC only); 0 = in position, 1 = 
      not in position (for NAC, value is 0)
- byte 5: spacecraft attitude; 0 = knowledge good, 1 = knowledge 
      bad
- byte 6: CCD temperature; 0 = within calibrated range, 1 = 
      outside of calibrated range
- byte 7: data completeness; 0 = no missing data, 1 = missing data
- bytes 8-15: spare"

  SYSTEM_CLASSIFICATION_ID        = "COMMON"
  GENERAL_CLASSIFICATION_TYPE     = "DATASET"
  CHANGE_DATE                     = "2009-06-21"
  STATUS_TYPE                     = "APPROVED"
  STANDARD_VALUE_OUTPUT_FLAG      = "N"
  TEXT_FLAG                       = "N"
  TERSE_NAME                      = "dataqualid"
  SQL_FORMAT                      = "CHAR(16)"
  BL_SQL_FORMAT                   = "char(16)"
  DISPLAY_FORMAT                  = "JUSTLEFT"
  AVAILABLE_VALUE_TYPE            = ""
END_OBJECT                        = ELEMENT_DEFINITION
END

The example in Section 5.5.1b of the Standards Reference will be updated as shown below:

DATA_QUALITY_ID [PSDD] - CHAR(16)
The data_quality_id element provides a numeric key which identifies the quality of data available for a particular time period. The data_quality_id scheme is unique to a given instrument and is described by the associated data_quality_desc element.