Data Model¶
A primary design goal of bluesky is to enable better research by recording rich metadata alongside measured data for use in later analysis. Documents are how we do this.
A document is our term for a Python dictionary with a schema. The bluesky RunEngine emits documents during plan execution. All of the metadata and data generated by executing the plan is organized into documents. Bluesky’s document-based data model supports complex, asynchronous data collection and enables sophisticated live, prompt, streaming, and post-facto data analysis.
The bluesky documentation describes how outside functions can “subscribe” to a stream of these documents, visualizing, processing, or saving them. This section provides an outline of documents themselves, aiming to give a sense of the structure and familiarity with useful components.
Overview¶
The data model is composed of six types of Documents, which in Python are represented as dictionaries but could be represented as nested mappings (e.g. JSON) in any language. Each document class has a defined, but flexible, schema.
Run Start Document — Everything we know about an experiment or simulation before any data acquisition begins: the who / why / what and metadata such as sample information.
Event — A “row” of measurements with associated timestamps.
Event Descriptor — Metadata about a series of Events. Envision richly-detail column headings in a table, encompassing physical units, hardware configuration information, etc.
Resource — A pointer to an external file (or resource in general).
Datum — A pointer to a specific slice of data within a Resource.
Run Stop Document — Everything that we can only know at the very end, such as the time it ended and the exit status (succeeded, aborted, failed due to error).
Every document contains a unique identifier. The Event documents also have
a descriptor field linking them to the Event Descriptor with their metadata.
And the Event Descriptor and Run Stop documents have a run_start field linking
them to their Run Start. Thus, all the documents in a run are linked back to the
Run Start.
Event documents may contain the literal values or pointers to values that are stored in some external file or networked resource, yet to be loaded. The Resource and Datum document types manage references to externally-stored data.
Example Runs¶
Finally, Event and Datum can be represented in “paged” form, where multiple rows are contained in one structure for efficient transport and vectorized computation. The representations contain equivalent information: an EventPage can always be transformed, without loss, into an Event and vice versa.
The scope of a “run” is left to the instrument team or individual scientist. It is quite generic: a set of Documents generated by following a given sequence of instructions. This might encompass a single short measurement (say, “a count”) or a multi-step procedure such as a raster scan. A run should represent a set of individual measurements that will be collected or processed together.
Document Types in Detail¶
For each type, we will show:
the minimal nontrivial example that satisfies the schema
one or more “typical” examples generated by bluesky’s RunEngine during an experiment
the formal schema, encoded using JSON schema
Run Start Document¶
Again, a ‘run start’ document marks the beginning of the run. It comprises everything we know before we start taking data, including all metadata provided by the user and the plan.
Minimal nontrivial valid example:
Documentation note: It might seem more natural to use json code-blocks here than python ones, but using python allows us to include comments in line.
# 'run start' document
{'time': 1550069716.5092213, # UNIX epoch (seconds since 1 Jan 1970)
'uid': '10bf6945-4afd-43ca-af36-6ad8f3540bcd'} # globally unique ID
A typical example
# 'run start' document
{'data_session': 'vist54321',
'data_groups': ['bl42', 'proposal12345'],
'detectors': ['random_walk:x'],
'hints': {'dimensions': [(['random_walk:dt'], 'primary')]},
'motors': ('random_walk:dt',),
'num_intervals': 2,
'num_points': 3,
'plan_args': {'args': ["EpicsSignal(read_pv='random_walk:dt', " "name='random_walk:dt', " 'value=1.0, ' 'timestamp=1550070001.828528, ' 'auto_monitor=False, ' 'string=False, ' "write_pv='random_walk:dt', " 'limits=False, ' 'put_complete=False)', -1, 1],
'detectors': ["EpicsSignal(read_pv='random_walk:x', " "name='random_walk:x', " 'value=1.61472277847348, ' 'timestamp=1550070000.807677, ' 'auto_monitor=False, ' 'string=False, ' "write_pv='random_walk:x', " 'limits=False, ' 'put_complete=False)'],
'num': 3,
'per_step': 'None'},
'plan_name': 'scan',
'plan_pattern': 'inner_product',
'plan_pattern_args': {'args': ["EpicsSignal(read_pv='random_walk:dt', " "name='random_walk:dt', " 'value=1.0, ' 'timestamp=1550070001.828528, ' 'auto_monitor=False, ' 'string=False, ' "write_pv='random_walk:dt', " 'limits=False, ' 'put_complete=False)', -1, 1],
'num': 3},
'plan_pattern_module': 'bluesky.plan_patterns',
'plan_type': 'generator',
'scan_id': 2,
'time': 1550070004.9850419,
'uid': 'ba1f9076-7925-4af8-916e-0e1eaa1b3c47'}
Note
Time is given in UNIX time (seconds since 1970). Software for looking at the data would, of course, translate that into a more human-readable form.
Projections (Experimental)¶
The Run Start document may include a projections field. It is intended
that a projection is an aid to interacting with external systems using
standardized vocabularies. Projections might be used in a variety of use cases
such as providing run data to analysis tools or suitcases. Each projection
represents multiple ways to represent data from the run. Each field in the
projection dictionary is an unique and externally-identifiable string and each
value is an instruction for accessing data from the run. This feature is
experimetal and subject to backward-incompatible changes in future releases.
The run start document formal schema:
{
"definitions": {
"data_type": {
"title" : "data_type",
"patternProperties": {"^([^./]+)$": {"$ref": "#/definitions/data_type"}},
"additionalProperties": false
},
"projection": {
"description": "Where to get the data from",
"type": "object",
"properties" : {
"type": {"enum": ["linked", "calculated", "static"], "description": "linked: a value linked from the data set, calculated: a value that requires calculation, static: a value defined here in the projection "},
"stream": {"type": "string"},
"location": {"enum" : ["start", "event", "configuration"], "description": "start comes from metadata fields in the start document, event comes from event, configuration comes from configuration fields in the event_descriptor document"},
"field": {"type": "string"},
"config_index": {"type": "integer"},
"config_device": {"type": "string"},
"calculation": {
"title" : "calculation properties",
"description": "required fields if type is calculated",
"properties": {
"callable": {"type": "string", "description": "callable function to perform calculation"},
"args": {"type": "array", "decription": "args for calculation callable"},
"kwargs": {"type": "object", "description": "kwargs for calcalation callable"}
},
"required": ["callable"]
},
"value": {"description": "value explicitely defined in the projection when type==static."}
},
"allOf": [
{
"if": {
"allOf": [
{"properties": {"location": {"enum": "configuration"}}},
{"properties": {"type": {"enum": "linked"}}}
]},
"then": {"required": ["type", "location", "config_index", "config_device", "field", "stream"]}
},
{
"if": {
"allOf": [
{"properties": {"location": {"enum": "event"}}},
{"properties": {"type": {"enum": "linked"}}}
]},
"then": {"required": ["type", "location", "field", "stream"]}
},
{
"if": {
"allOf": [
{"properties": {"location": {"enum": "event"}}},
{"properties": {"type": {"enum": "calculated"}}}
]},
"then": {"required": ["type", "field", "stream", "calculation"]}
},
{
"if": {"properties": {"type": {"enum": "static"}}},
"then": {"required": ["type", "value"]}
}
],
"additionalProperties": false
},
"projections": {
"title" : "Describe how to interperet this run as the given projection",
"properties":{
"name": {"type": "string", "description": "The name of the projection"},
"version": {"type": "string", "description": "The version of the projection spec. Can specify the version of an external specification."},
"configuration" : {"type": "object", "description": "Static information about projection"},
"projection" : {
"type": "object",
"patternProperties": {".": {"$ref": "#/definitions/projection"}},
"additionalProperties": false
}
},
"additionalProperties": false,
"required" : ["projection", "version", "configuration"]
}
},
"properties": {
"data_session": {
"type": "string",
"description": "An optional field for grouping runs. The meaning is not mandated, but this is a data management grouping and not a scientific grouping. It is intended to group runs in a visit or set of trials."
},
"data_groups": {
"type": "array", "items": {"type": "string"},
"description": "An optional list of data access groups that have meaning to some external system. Examples might include facility, beamline, end stations, proposal, safety form."
},
"project": {
"type": "string",
"description": "Name of project that this run is part of"
},
"sample": {
"type": ["object", "string"],
"description": "Information about the sample, may be a UID to another collection"
},
"scan_id": {
"type": "integer",
"description": "Scan ID number, not globally unique"
},
"time": {
"type": "number",
"description": "Time the run started. Unix epoch time"
},
"uid": {
"type": "string",
"description": "Globally unique ID for this run"
},
"group": {
"type": "string",
"description": "Unix group to associate this data with"
},
"owner": {
"type": "string",
"description": "Unix owner to associate this data with"
},
"projections": {
"type": "array",
"items": {"$ref": "#/definitions/projections"}
},
"hints": {
"type": "object",
"description": "Start-level hints",
"properties": {
"dimensions": {
"type": "array",
"description": "The independent axes of the experiment. Ordered slow to fast",
"items": {
"type": "array",
"description": "Each entry in this list is of the from ([<FIELD>, ...], <STREAM>). A 1d scan will have 1 such entry, a scan with 3 independent entries would have 3",
"items": [
{
"type": "array",
"description": "The data key(s) for the given dimension.",
"items":
{
"type": "string"
}
},
{
"type": "string",
"description": "The stream to find the datakeys in."
}
],
"additionalItems": false,
"minItems": 2
}
}
},
"patternProperties": {
"^([^.]+)$": {"$ref": "#/definitions/data_type"}},
"additionalProperties": false
}
},
"patternProperties": {
"^([^./]+)$": {"$ref": "#/definitions/data_type"}},
"additionalProperties": false,
"required": [
"uid",
"time"
],
"type": "object",
"description": "Document created at the start of run. Provides a seach target and later documents link to it"
}
Event Descriptor¶
As stated above, an ‘event descriptor’ document provides a schema for the data in the Event documents. It provides useful information about each key in the data and about the configuration of the hardware. The layout of a descriptor is detailed and takes some time to cover, so we defer it to a another page.
Minimal nontrivial valid example:
# 'event descriptor' document
{'configuration': {},
'data_keys': {'camera_image': {'dtype': 'number',
'shape': [512, 512],
'source': 'PV:...'}},
'hints': {},
'name': 'primary',
'object_keys': {},
'run_start': '10bf6945-4afd-43ca-af36-6ad8f3540bcd', # foreign key
'time': 1550070954.276659,
'uid': 'd08d2ada-5f4e-495b-8e73-ff36186e7183'}
Typical example:
# 'event descriptor' document
{'configuration': {'random_walk:dt': {'data': {'random_walk:dt': -1.0},
'data_keys': {'random_walk:dt': {'dtype': 'number',
'lower_ctrl_limit': 0.0,
'precision': 0,
'shape': [],
'source': 'PV:random_walk:dt',
'units': '',
'upper_ctrl_limit': 0.0}},
'timestamps': {'random_walk:dt': 1550070004.994477}},
'random_walk:x': {'data': {'random_walk:x': 1.9221013521832928},
'data_keys': {'random_walk:x': {'dtype': 'number',
'lower_ctrl_limit': 0.0,
'precision': 0,
'shape': [],
'source': 'PV:random_walk:x',
'units': '',
'upper_ctrl_limit': 0.0}},
'timestamps': {'random_walk:x': 1550070004.812525}}},
'data_keys': {'random_walk:dt': {'dtype': 'number',
'lower_ctrl_limit': 0.0,
'object_name': 'random_walk:dt',
'precision': 0,
'shape': [],
'source': 'PV:random_walk:dt',
'units': '',
'upper_ctrl_limit': 0.0},
'random_walk:x': {'dtype': 'number',
'lower_ctrl_limit': 0.0,
'object_name': 'random_walk:x',
'precision': 0,
'shape': [],
'source': 'PV:random_walk:x',
'units': '',
'upper_ctrl_limit': 0.0}},
'hints': {'random_walk:dt': {'fields': ['random_walk:dt']},
'random_walk:x': {'fields': ['random_walk:x']}},
'name': 'primary',
'object_keys': {'random_walk:dt': ['random_walk:dt'],
'random_walk:x': ['random_walk:x']},
'run_start': 'ba1f9076-7925-4af8-916e-0e1eaa1b3c47',
'time': 1550070005.0109222,
'uid': '0ad55d9e-1b31-4af2-865c-7ab7c8171303'}
Formal schema:
{
"definitions": {
"data_key": {
"title": "data_key",
"description": "Describes the objects in the data property of Event documents",
"properties": {
"dtype": {
"enum": [
"string",
"number",
"array",
"boolean",
"integer"
],
"type": "string",
"description": "The type of the data in the event."
},
"external": {
"pattern": "^[A-Z]+:?",
"type": "string",
"description": "Where the data is stored if it is stored external to the events."
},
"shape": {
"type": "array",
"items": {
"type": "integer"
},
"description": "The shape of the data. Empty list indicates scalar data."
},
"dims": {
"type": "array",
"items": {
"type": "string"
},
"description": "The names for dimensions of the data. Null or empty list if scalar data"
},
"source": {
"type": "string",
"description": "The source (ex piece of hardware) of the data."
}
},
"required": [
"source",
"dtype",
"shape"
],
"type": "object"
},
"data_type": {
"title" : "data_type",
"patternProperties": {"^([^./]+)$": {"$ref": "#/definitions/data_type"}},
"additionalProperties": false
},
"object_hints": {
"title": "Object Hints",
"patternProperties": {"^([^./]+)$": {"$ref": "#/definitions/per_object_hint"}},
"additionalProperties": false
},
"per_object_hint": {
"type": "object",
"properties": {
"fields":
{
"description": "The 'interesting' data keys for this device.",
"type": "array",
"items":{
"type": "string"
}
}
}
},
"configuration": {
"type": "object",
"properties": {
"data": {
"type": "object",
"description": "The actual measurement data"
},
"timestamps": {
"type": "object",
"description": "The timestamps of the individual measurement data"
},
"data_keys": {
"additionalProperties": {
"$ref": "#/definitions/data_key"
},
"type": "object",
"description": "This describes the data stored alongside it in this configuration object."
}
}
}
},
"properties": {
"data_keys": {
"additionalProperties": {
"$ref": "#/definitions/data_key"
},
"type": "object",
"description": "This describes the data in the Event Documents.",
"title": "data_keys"
},
"uid": {
"type": "string",
"description": "Globally unique ID for this event descriptor.",
"title": "uid"
},
"run_start": {
"type": "string",
"description": "Globally unique ID of this run's 'start' document."
},
"time": {
"type": "number",
"description": "Creation time of the document as unix epoch time."
},
"hints": {
"$ref": "#/definitions/object_hints"
},
"object_keys": {
"type": "object",
"description": "Maps a Device/Signal name to the names of the entries it produces in data_keys."
},
"name": {
"type": "string",
"description": "A human-friendly name for this data stream, such as 'primary' or 'baseline'."
},
"configuration": {
"additionalProperties": {
"$ref": "#/definitions/configuration"
},
"type": "object",
"description": "Readings of configurational fields necessary for interpreting data in the Events."
}
},
"patternProperties": {
"^([^./]+)$": {"$ref": "#/definitions/data_type"}},
"additionalProperties": false,
"required": [
"uid",
"data_keys",
"run_start",
"time"
],
"type": "object",
"title": "event_descriptor",
"description": "Document to describe the data captured in the associated event documents"
}
Event Document¶
The Event document may contain data directly:
# 'event' document
{'data': {'camera_image': [[...512x512 array...]]},
'descriptor': 'd08d2ada-5f4e-495b-8e73-ff36186e7183', # foreign key
'filled': {},
'seq_num': 1,
'time': 1550072091.2793343,
'timestamps': {'camera_image': 1550072091.2793014},
'uid': '8eac2f83-2b3e-4d67-ae2c-1d3aaff29ff5'}
or it may reference it via a datum_id from a Datum document.
# 'event' document
{'data': {'camera_image': '272132cf-564f-428f-bf6b-149ee4287024/1'}, # foreign key
'descriptor': 'd08d2ada-5f4e-495b-8e73-ff36186e7183', # foreign key
'filled': {'camera_image': False},
'seq_num': 1,
'time': 1550072091.2793343,
'timestamps': {'camera_image': 1550072091.2793014},
'uid': '8eac2f83-2b3e-4d67-ae2c-1d3aaff29ff5'}
See External Assets for details on how external assets are handled.
Typical example:
# 'event' document
{'data': {'random_walk:dt': -1.0,
'random_walk:x': 1.9221013521832928},
'descriptor': '0ad55d9e-1b31-4af2-865c-7ab7c8171303',
'filled': {},
'seq_num': 1,
'time': 1550070005.0189056,
'timestamps': {'random_walk:dt': 1550070004.994477,
'random_walk:x': 1550070004.812525},
'uid': '7b5343fe-dfd7-4884-bc18-a0b571ff60b7'}
From a data analysis perspective, these readings were simultaneous, but in actuality the occurred at separate times. The separate times of the individual readings are not thrown away (they are recorded in ‘timestamps’) but the overall event ‘time’ is often more useful.
Formal schema:
{
"properties": {
"data": {
"type": "object",
"description": "The actual measurement data"
},
"timestamps": {
"type": "object",
"description": "The timestamps of the individual measurement data"
},
"filled": {
"type": "object",
"additionalProperties": {"type": ["boolean", "string"]},
"description": "Mapping each of the keys of externally-stored data to the boolean False, indicating that the data has not been loaded, or to foreign keys (moved here from 'data' when the data was loaded)"
},
"descriptor": {
"type": "string",
"description": "UID of the EventDescriptor to which this Event belongs"
},
"seq_num": {
"type": "integer",
"description": "Sequence number to identify the location of this Event in the Event stream"
},
"time": {
"type": "number",
"description": "The event time. This maybe different than the timestamps on each of the data entries."
},
"uid": {
"type": "string",
"description": "Globally unique identifier for this Event"
}
},
"required": [
"uid",
"data",
"timestamps",
"time",
"descriptor",
"seq_num"
],
"additionalProperties": false,
"type": "object",
"title": "event",
"description": "Document to record a quanta of collected data"
}
Event contents can also be represented in “paged” form, where multiple rows are contained in one structure for efficient transport and vectorized computation. The representations contain equivalent information: an EventPage can always be transformed, without loss, into an Event and vice versa. Here is the example Event above structured as an Event Page with a single row:
# 'event_page' document
{'data': {'random_walk:dt': [-1.0],
'random_walk:x': [1.9221013521832928]},
'descriptor': '0ad55d9e-1b31-4af2-865c-7ab7c8171303',
'filled': {},
'seq_num': [1],
'time': [1550070005.0189056],
'timestamps': {'random_walk:dt': [1550070004.994477],
'random_walk:x': [1550070004.812525]},
'uid': ['7b5343fe-dfd7-4884-bc18-a0b571ff60b7']}
Formal Event Page schema:
{
"definitions": {
"dataframe": {
"type": "object",
"title": "dataframe",
"description": "A DataFrame-like object",
"additionalProperties": {
"type": "array"
}
},
"dataframe_for_filled": {
"title": "dataframe_for_filled",
"type": "object",
"description": "A DataFrame-like object with boolean or string entries",
"additionalProperties": {
"type": "array",
"items": {
"type": [
"boolean",
"string"
]
}
}
}
},
"properties": {
"descriptor": {
"type": "string",
"description": "The UID of the EventDescriptor to which all of the Events in this page belong"
},
"data": {
"$ref": "#/definitions/dataframe",
"description": "The actual measurement data"
},
"timestamps": {
"$ref": "#/definitions/dataframe",
"description": "The timestamps of the individual measurement data"
},
"filled": {
"$ref": "#/definitions/dataframe_for_filled",
"description": "Mapping each of the keys of externally-stored data to an array containing the boolean False, indicating that the data has not been loaded, or to foreign keys (moved here from 'data' when the data was loaded)"
},
"seq_num": {
"type": "array",
"items": {
"type": "integer"
},
"description": "Array of sequence numbers to identify the location of each Event in the Event stream"
},
"time": {
"type": "array",
"items": {
"type": "number"
},
"description": "Array of Event times. This maybe different than the timestamps on each of the data entries"
},
"uid": {
"type": "array",
"items": {
"type": "string"
},
"description": "Array of globally unique identifiers for each Event"
}
},
"required": [
"descriptor",
"uid",
"data",
"timestamps",
"time",
"seq_num"
],
"additionalProperties": false,
"type": "object",
"title": "event_page",
"description": "Page of documents to record a quanta of collected data"
}
It is intentional that the values in the “data” and “timestamps” dictionaries
do not have structure. The values may be numeric, bool, null (None), or a
homogeneous N-dimensional array of any of these. The values are never objects
or dictionaries (to use the JSON and Python terminology respectively). This
requirement allows document-consumers to make useful simplifying assumptions.
As another justification for this design, consider that if we allowed one level
of nesting in “data”, then it could lead to wanting those values to
allow nesting and so on, which would lead us to accepting arbitrarily nested
structured data. This in turn would makes the Event Descriptors significantly
more complex. Thus, we require that the values in “data” never be structured.
Run Stop Document¶
A ‘run stop’ document marks the end of the run. It contains metadata that is not known until the run completes.
The most commonly useful fields here are ‘time’ and ‘exit_status’.
Minimal nontrivial valid example:
# 'run stop' document
{'uid': '546cc556-5f69-46b5-bf36-587d8cfe67a9',
'time': 1550072737.175858,
'run_start': '61bb1db8-c95c-4144-845b-e248c06d80e1',
'exit_status': 'success',
'reason': '',
'num_events': {}}
Typical example:
# 'stop' document
{'run_start': 'ba1f9076-7925-4af8-916e-0e1eaa1b3c47',
'time': 1580172029.3419003,
'uid': '78c70c2c-2508-479e-9857-05553748022e',
'exit_status': 'success',
'reason': '',
'num_events': {'primary': 10}
Formal schema:
{
"definitions": {
"data_type": {
"title" : "data_type",
"patternProperties": {"^([^./]+)$": {"$ref": "#/definitions/data_type"}},
"additionalProperties": false
}
},
"properties": {
"run_start": {
"type": "string",
"description": "Reference back to the run_start document that this document is paired with."
},
"reason": {
"type": "string",
"description": "Long-form description of why the run ended"
},
"time": {
"type": "number",
"description": "The time the run ended. Unix epoch"
},
"exit_status": {
"type": "string",
"enum": ["success", "abort", "fail"],
"description": "State of the run when it ended"
},
"uid": {
"type": "string",
"description": "Globally unique ID for this document"
},
"num_events": {
"type": "object",
"properties": {},
"additionalProperties": { "type": "integer" },
"description": "Number of Events per named stream"
}
},
"patternProperties": {
"^([^./]+)$": {"$ref": "#/definitions/data_type"}},
"additionalProperties": false,
"required": [
"uid",
"run_start",
"time",
"exit_status"
],
"type": "object",
"description": "Document for the end of a run indicating the success/fail state of the run and the end time"
}
Resource Document¶
See External Assets for details on the role Resource documents play in referencing external assets, such as large array data written by detectors.
Minimal nontrivial valid example:
# 'resource' document
{'path_semantics': 'posix',
'resource_kwargs': {},
'resource_path': '/local/path/subdirectory/data_file',
'root': '/local/path/',
'run_start': '10bf6945-4afd-43ca-af36-6ad8f3540bcd',
'spec': 'SOME_SPEC',
'uid': '272132cf-564f-428f-bf6b-149ee4287024'}
Typical example:
# resource
{'spec': 'AD_HDF5',
'root': '/GPFS/DATA/Andor/',
'resource_path': '2020/01/03/8ff08ff9-a2bf-48c3-8ff3-dcac0f309d7d.h5',
'resource_kwargs': {'frame_per_point': 10},
'path_semantics': 'posix',
'uid': '3b300e6f-b431-4750-a635-5630d15c81a8',
'run_start': '10bf6945-4afd-43ca-af36-6ad8f3540bcd'}
Formal schema:
{
"properties": {
"spec": {
"type": "string",
"description": "String identifying the format/type of this Resource, used to identify a compatible Handler"
},
"resource_path": {
"type": "string",
"description": "Filepath or URI for locating this resource"
},
"resource_kwargs": {
"type": "object",
"description": "Additional argument to pass to the Handler to read a Resource"
},
"root": {
"type": "string",
"description": "Subset of resource_path that is a local detail, not semantic."
},
"path_semantics": {
"type": "string",
"description": "Rules for joining paths",
"enum": ["posix", "windows"]
},
"uid": {
"type": "string",
"description": "Globally unique identifier for this Resource"
},
"run_start": {
"type": "string",
"description": "Globally unique ID to the run_start document this resource is associated with."
}
},
"required": [
"spec",
"resource_path",
"resource_kwargs",
"root",
"uid"
],
"additionalProperties": false,
"type": "object",
"title": "resource",
"description": "Document to reference a collection (e.g. file or group of files) of externally-stored data"
}
Datum Document¶
See External Assets for details on the role Datum documents play in referencing external assets, such as large array data written by detectors.
Minimal nontrivial valid example:
# 'datum' document
{'resource': '272132cf-564f-428f-bf6b-149ee4287024', # foreign key
'datum_kwargs': {}, # format-specific parameters
'datum_id': '272132cf-564f-428f-bf6b-149ee4287024/1'}
Typical example:
# datum
{'resource': '3b300e6f-b431-4750-a635-5630d15c81a8',
'datum_kwargs': {'index': 0},
'datum_id': '3b300e6f-b431-4750-a635-5630d15c81a8/0'}
It is an implementation detail that datum_id is often formatted as
{resource}/{counter} but this should not be considered part of the schema.
Formal schema:
{
"properties": {
"datum_kwargs": {
"type": "object",
"description": "Arguments to pass to the Handler to retrieve one quanta of data"
},
"resource": {
"type": "string",
"description": "The UID of the Resource to which this Datum belongs"
},
"datum_id": {
"type": "string",
"description": "Globally unique identifier for this Datum (akin to 'uid' for other Document types), typically formatted as '<resource>/<integer>'"
}
},
"required": [
"datum_kwargs",
"resource",
"datum_id"
],
"additionalProperties": false,
"type": "object",
"title": "datum",
"description": "Document to reference a quanta of externally-stored data"
}
Like Events, Datum contents can also be represented in “paged” form, and the representations contain equivalent information. This is the Datum example above strucuted as a Datum Page with one row:
# datum
{'resource': '3b300e6f-b431-4750-a635-5630d15c81a8',
'datum_kwargs': {'index': [0]},
'datum_id': ['3b300e6f-b431-4750-a635-5630d15c81a8/0']}
Formal Datum Page schema:
{
"definitions": {
"dataframe": {
"type": "object",
"title": "dataframe",
"description": "A DataFrame-like object",
"additionalProperties": {
"type": "array"
}
}
},
"properties": {
"resource": {
"type": "string",
"description": "The UID of the Resource to which all Datums in the page belong"
},
"datum_kwargs": {
"$ref": "#/definitions/dataframe",
"description": "Array of arguments to pass to the Handler to retrieve one quanta of data"
},
"datum_id": {
"type": "array",
"items": {
"type": "string"
},
"description": "Array unique identifiers for each Datum (akin to 'uid' for other Document types), typically formatted as '<resource>/<integer>'"
}
},
"required": [
"resource",
"datum_kwargs",
"datum_id"
],
"additionalProperties": false,
"type": "object",
"title": "datum",
"description": "Page of documents to reference a quanta of externally-stored data"
}
“Bulk Events” Document (DEPRECATED)¶
This is another representation of Events. This representation is deprecated. Use EventPage instead.
{
"patternProperties": {
"^.*$": {
"type": "array",
"items": {
"type": "object",
"properties": {
"data": {
"type": "object",
"description": "The actual measument data"
},
"timestamps": {
"type": "object",
"description": "The timestamps of the individual measument data"
},
"filled": {
"type": "object",
"description": "Mapping the keys of externally-stored data to a boolean indicating whether that data has yet been loaded"
},
"descriptor": {
"type": "string",
"description": "UID to point back to Descriptor for this event stream"
},
"seq_num": {
"type": "integer",
"description": "Sequence number to identify the location of this Event in the Event stream"
},
"time": {
"type": "number",
"description": "The event time. This maybe different than the timestamps on each of the data entries"
},
"uid": {
"type": "string",
"description": "Globally unique identifier for this Event"
}
},
"required": [
"uid",
"data",
"timestamps",
"time",
"descriptor",
"seq_num"
],
"additionalProperties": false,
"type": "object",
"title": "bulk_events",
"description": "Document to record a quanta of collected data"
}
}
}
}
“Bulk Datum” Document (DEPRECATED)¶
This is another representation of Datum. This representation is deprecated. Use DatumPage instead.
{
"properties": {
"datum_kwargs_list": {
"type": "array",
"items": {"type": "object"},
"description": "Array of arguments to pass to the Handler to retrieve one quanta of data"
},
"resource": {
"type": "string",
"description": "UID of the Resource to which all these Datum documents belong"
},
"datum_ids": {
"type": "array",
"items": {"type": "string"},
"description": "Globally unique identifiers for each Datum (akin to 'uid' for other Document types), typically formatted as '<resource>/<integer>'"
}
},
"required": [
"datum_kwargs",
"resource",
"datum_id"
],
"additionalProperties": false,
"type": "object",
"title": "bulk_datum",
"description": "Document to reference a quanta of externally-stored data"
}