2.2.1   Attributes¶

• name - jointcal_cfht_r
• eups package name of data
• directory within eups data package for dataset
• butler dataId or list of dataIds (optional)
• Metrics (list?)

2.3.1   Prior art¶

• lsst.validate.base.Metric
• Metric.yaml format

2.3.2   Attributes¶

• name
• description
• unit
• tags
• reference.doc
• reference.page
• reference.url

2.3.3   Questions & Notes¶

• Specifications are no longer contained by Metrics.
• In the existing lsst.validate.base.Metric, there is a parameters dictionary that defines constants for the Measurement code. For example, the annulus diameter from AMx Metrics. These parameters will be contained in the Specifications.
• We talked about making the minimum Provenance required for a Measurement/Job being defined in the Metric. Is this still a requirement?

2.4.1   Attributes¶

_ * name — Name of the MetricSet that these Measurements are associated with (e.g. validate_drp) * job_id — Job identifier * provenance — provenance of the executed job (TODO: should this actually live in the job itself?) * measurements — dictionary of name: Measurement.

2.5.1   Prior art¶

• lsst.validate.base.MeasurementBase

2.5.2   Attributes¶

• name — Name of metric that this measured.
• value — scalar Measurement value, required to be persisted in units of Metric.unit as an astropy.Quantity.

2.5.3   Questions & Notes¶

• lsst.validate.base.MeasurementBase originally had a parameters attribute that provided Provenance for how the Measurement was made (e.g., a S/N cut-off for star selection). These will now be part of the Task configuration, and available through the regular Provenance.
• lsst.validate.base.MeasurementBase also had an extras attribute where additional Measurement outputs could be persisted (JSON serializable). Do we still want this? Or do we always want such data to go into a Blob?

2.6.1   Prior art¶

• lsst.validate.base.BlobBase

2.7.1   Prior art¶

• lsst.validate.base.Job

2.7.2   Attributes¶

• Measurements — list of Measurement objects (TODO: or a MeasurementSet?).
• blobs — list of Blob objects. Each blob should be reference by at least one * Measurement.
• Provenance — data structure that fully specifies the Provenance of the pipeline run.

2.7.3   Questions & Notes¶

• What is the schema of Provenance? At minimum, it includes the input dataIds (input dataset) and task configurations.
• Not all Provenance is currently known within the pipeline. We use post-qa to hydrate Job Provenance with package versions and Jenkins environment variables. However, working towards a state where post-qa is no longer used as a shim, it’s not unreasonable to move this into validate_base.

2.8.1   Questions & Notes¶

• How is provenance defined?
• How do we define queries on provenance in a Specification?
• How do we map between this provenance and the one that DAX will maintain?

2.9.1   Attributes¶

• name — Identifier of the Metric that this Specification is attached to.
• provenance_query — only Measurements that have matching Provenance parameters are tested by this Specification.
• parameters - A dict of key:value pairs that must be matched by the Job‘s Provenance regarding particular values used in a calculation (e.g. diameter used for aperture photometry).
• alert_listeners - Slack IDs of people who are alerted if a Measurement fails the Specification.
• alert_channels - Slack Channel IDs that recieve messages when a Measurement fails a Specification.
• threshold and comparison_operator — Measurement passes Specification if Measurement is on the side of the threshold indicated by the comparison operator.
• range — Measurement passes Specification if Measurement is within this range (new).

2.9.2   Questions & Notes¶

• Either threshold or range can be set. Possibly there should be different classes of Specification (i.e., a ThresholdSpecification or a RangeSpecification).
• Note that we’re jettisoning some of the earlier Specification class baggage, like dependencies. This means that the definitions of Metrics are no longer driven by definitions of Specifications, as they currently are for AFx/ADx, for example. Instead, this flexibility is handled by additional Metrics.
• Should the Parameters just be part of the Provenance, or should they be a separate section for maintanence convinience and get ingested into the Provenance?

2.10.1   Attributes¶

• Metric_name
• Provenance_query

3.1.1   Fully qualified metric name¶

Metrics can be referenced universally by their fully qualified name:

{ package name }.{ metric name }

For example, the fully qualified name for the example metric is validate_drp.PA1.

When working inside a package, where context is clear, the validate API can permit metrics to be addressed by name alone, PA1.

3.2.1   Specification tests¶

The binary comparison test is quite common, but its not the only imaginable test structure. Other types of tests that may be supported by the validation framework are:

• tolerance: consisting of a target value, and a symmetric tolerance window.
• window: test if a measurement deviates from the sample of previous measurements in a given window, by a given amount.
• function: specifies an importable Python function that computes a binary True (pass) or False (fails) result.

3.2.2   Specification partials¶

Specifications might repeat information. For example, a provenance_query for a certain test dataset. We apply DRY design principles though partials. A partial has an id field, and can’t be a specification on its own. For example:

---
# specification partial
id: 'base'
metric: 'PA1'
threshold:
  unit: ''
  operator: '<='
provenance_query:
  filter: ['g', 'r', 'I']

---
# design specification instance that mixes in the base partial
# validate_drp.PA1.design
name: 'design'
base: '#base'
threshold:
  value: 5.0

---
# stretch specification instance that mixes in the base partial
# validate_drp.PA1.stretch
name: 'stretch'
base: '#base'
threshold:
  value: 3.0
...

A partial can be referenced from the base field by prefixing the id with #. Partials can also be referenced from across files (but within the same package’s specs directory) by providing a filename:

base: "cfht_gri#base"

3.2.3   Specification inheritance¶

Specifications can also inherit from specifications; generally to add partials. Specifications are referenced through their fully qualified name validate_drp.PA1.design_gri, or the package-relative fully qualified name, PA1.design_gri. For example:

---
# Specification partial
id: 'PA1-base'
metric: 'PA1'
threshold:
  unit: 'mmag'
  operator: "<="

---
# validate_drp.PA1.minimum_gri
name: "minimum_gri"
base: "#PA1-base"
threshold:
  value: 8.0

---
# Partial that queries a cfht_gri dataset
id: 'cfht-base'
provenance_query:
  dataset_repo_url: 'https://github.com/lsst/validation_data_cfht.git'
  filters: ['g', 'r', 'i']
  visits: [849375, 850587]
  ccd: [12, 13, 14, 21, 22, 23]

---
# validate_drp.PA1.cfht_minimum_gri
name: 'cfht_minimum_gri'
base: ['PA1.minimum_gri', '#cfht-base']
...

The fully-hydrated validate_drp.PA1.minimum_gri specification is:

---
name: 'minimum_gri'
metric: 'PA1'
threshold:
  value: 8.0
  unit: 'mmag'
  operator: "<="

And the fully-hydrated validate_drp.PA1.cfht_minimum_gri specification is:

name: 'cfht_minimum_gri'
metric: 'PA1'
provenance_query:
  dataset_repo_url: 'https://github.com/lsst/validation_data_cfht.git'
  filters: ['g', 'r', 'i']
  visits: [849375, 850587]
  ccd: [12, 13, 14, 21, 22, 23]
threshold:
  value: 8.0
  unit: 'mmag'
  operator: "<="