Data-quality segments

In order to successfully search data for gravitational-wave signals, precise records of when each observatory was operating, and in a particular configuration, are kept to enable search teams to pick the best data to analyse.

Time segments are recorded denoting when each observatory was taking science-quality data and when the calibration was nominal, as well as during times of possible problems - electronics glitches or severe weather, for example.

The international collaboration operates using the GPS time standard (seconds since the GPS epoch of midnight on January 6th 1980), and records such times as semi-open GPS [start, stop) segments.

GWpy provides a number of classes for generating and manipulating such segments, inherited most functionality from the glue.segments package. All credits for their usefulness go to the authors of that package.

These basic objects are as follows:

Segment A tuple defining a semi-open interval [start, end)
SegmentList A list of Segments
SegmentListDict A dict of SegmentLists

While these objects are key to representing core data segments, they are usually applied to analyses of data as a DataQualityFlag.

The DataQualityFlag

A DataQualityFlag is an annotated set of segments that indicate something about instrumental operation. Each flag is defined by applying some algorithm on data and generating a SegmentList that indicates some good or bad condition has been met during those times. For example, the times during which the LIGO interferometers are operating under observing conditions are recorded as the ‘analysis-ready’ flag, which are used by data analysis groups to define periods of data over which to run their pipelines. Conversely, high seismic noise around the observatory buildings is recorded in a data-quality flag used by analysis groups to veto periods of analysis as a result of sub-standard data.

Each DataQualityFlag has some key attributes:

name The name associated with this flag.
active The set of segments during which this flag was active.
valid The set of segments during which this flag was known, and its state was well defined.

By convention, the name is typically constructed of three colon-separated components: the ifo, tag, and version, e.g. L1:DMT-ANALYSIS_READY:1.

Combining DataQualityFlags

DataQualityFlags can be combined in a number of ways, using the standard python operators, e.g. & and |.

Intersection (&)

>>> a & b

returns the intersection of both the known and active segment lists, e.g:

>>> a = DataQualityFlag(known=[(0, 5), (10, 15)], active=[(1, 5), (10, 12)])
>>> b = DataQualityFlag(known=[(0, 12)], active=[(3, 7), (10, 12)])
>>> print(a & b)
<DataQualityFlag(No name,
                 known=[[0 ... 5)
                        [10 ... 12)],
                 active=[[3 ... 5)
                         [10 ... 12)],
                 description=None)>

This new flag represents times when both a and b were known and when both were active.

Union (|)

>>> a | b

returns the intersection of both the known and active segment lists, e.g:

>>> print(a | b)
<DataQualityFlag(No name,
              known=[[0 ... 15)],
              active=[[1 ... 7)
                      [10 ... 12)],
              description=None)>

This new flag represents times when either a or b were known and when either was active.

Subtraction (-)

>>> a - b

returns the union of the known segments, and the difference of the active segment lists, e.g.:

>>> print(a - b)
<DataQualityFlag(No name,
                 known=[[0 ... 5)
                        [10 ... 12)],
                 active=[[1 ... 3)],
                 description=None)>

The new flag represents times when both a and b were known, but only a was active.

Addition (+)

>>> a + b

This operation is the same as gwpy-segments-union.

Inversion (~)

>>> ~a

returns the same known segments, and the inverse active segment lists, e.g:

>>> print(~a)
<DataQualityFlag(No name,
                 known=[[0 ... 5)
                        [10 ... 15)],
                 active=[[0 ... 1)
                         [12 ... 15)],
                 description=None)>

The new flag represents times when the state of a was known, but it was not active.

Exclusive OR (^)

>>> a ^ b

returns the intersection of known segments and the exclusive OR of active segment lists, e.g:

>>> print(a ^ b)
<DataQualityFlag(No name,
             known=[[0 ... 5)
                    [10 ... 12)],
             active=[[1 ... 3)
                     [5 ... 7)],
             description=None)>

The new flag represents times when the state of both a and b are known, but exactly one of the flags was active.

The DataQualityDict

Groups of DataQualityFlags can be collected into a DataQualityDict, a simple extension of the OrderedDict with methods for operating on a group of flags in bulk.

The most immediate utility of this group class is a bulk query of the segment database, using the DataQualityDict.query() classmethod. This method is what is actually called by the DataQualityFlag.query() classmethod anyway.

Working with segments

Reading/writing and querying segments:

Generating segments from data:

Reference/API

This reference includes the following class entries:

DataQualityFlag A representation of a named set of segments.
DataQualityDict An OrderedDict of (key, DataQualityFlag) pairs.
Segment A tuple defining a semi-open interval [start, end)
SegmentList A list of Segments
SegmentListDict A dict of SegmentLists