Usage¶
Launch Binder to explore some interactive examples.
This is a quick example to give a feel for how it would work to acquire data using bluesky and access it using intake.
Acquire data. In this case we’ll serialize the stream of “documents” generated by bluesky to simple text files. See the Binder link above for an example with a live MongoDB.
In [1]: from bluesky import RunEngine
In [2]: RE = RunEngine()
In [3]: from bluesky.plans import scan
In [4]: from ophyd.sim import det, motor
In [5]: from suitcase.jsonl import Serializer
In [6]: serializer = Serializer('data')
In [7]: uid, = RE(scan([det], motor, -1, 1, 3), serializer)
In [8]: serializer.close()
We’ll access the data using intake locally. See the Binder link above for an example using a HTTP server, where the user may not have direct access to the file system where the data files are stored.
In [9]: from intake_bluesky.jsonl import BlueskyJSONLCatalog
In [10]: import glob
In [11]: catalog = BlueskyJSONLCatalog(glob.glob('data/*.jsonl'))
Read Data as Xarray¶
In [12]: catalog[uid]
Out[12]: name: 65d84525-ab45-49eb-ab42-c1ef7f06c70a
container: bluesky-run
plugin: ['catalog']
description:
direct_access: forbid
user_parameters: []
metadata:
start:
uid: 65d84525-ab45-49eb-ab42-c1ef7f06c70a
time: 1573521581.3453052
scan_id: 1
hints:
dimensions: [[['motor'], 'primary']]
plan_pattern_args:
num: 3
args: ["SynAxis(prefix='', name='motor', read_attrs=['readback', 'setpoint'], configuration_attrs=['velocity', 'acceleration'])", -1, 1]
plan_pattern: inner_product
plan_name: scan
plan_type: generator
num_intervals: 2
plan_pattern_module: bluesky.plan_patterns
detectors: ['det']
motors: ['motor']
plan_args:
detectors: ["SynGauss(name='det', value=1.0, timestamp=1573521581.3272657)"]
num: 3
args: ["SynAxis(prefix='', name='motor', read_attrs=['readback', 'setpoint'], configuration_attrs=['velocity', 'acceleration'])", -1, 1]
per_step: None
num_points: 3
stop:
run_start: 65d84525-ab45-49eb-ab42-c1ef7f06c70a
time: 1573521581.3531487
uid: d7e6baf1-1b10-47da-8bd6-206e464edc67
exit_status: success
reason:
num_events:
primary: 3
args:
gen_func: <function gen at 0x7f9386e7bd90>
gen_args: ('data/65d84525-ab45-49eb-ab42-c1ef7f06c70a.jsonl',)
gen_kwargs:
filler: <Filler>
In [13]: catalog[uid]()
Out[13]:
Run Catalog
uid='65d84525-ab45-49eb-ab42-c1ef7f06c70a'
exit_status='success'
2019-11-12 01:19:41.345 -- 2019-11-12 01:19:41.353
Streams:
* primary
In [14]: catalog[uid]()['primary']
Out[14]: name: primary
container: xarray
plugin: ['bluesky-event-stream']
description:
direct_access: forbid
user_parameters: []
metadata:
descriptors: [{'run_start': '65d84525-ab45-49eb-ab42-c1ef7f06c70a', 'time': 1573521581.3489022, 'data_keys': {'motor': {'source': 'SIM:motor', 'dtype': 'number', 'shape': [], 'precision': 3, 'object_name': 'motor'}, 'motor_setpoint': {'source': 'SIM:motor_setpoint', 'dtype': 'number', 'shape': [], 'precision': 3, 'object_name': 'motor'}, 'det': {'source': 'SIM:det', 'dtype': 'number', 'shape': [], 'precision': 3, 'object_name': 'det'}}, 'uid': '14c7c113-e869-4139-a45f-113764a5fe7f', 'configuration': {'motor': {'data': {'motor_velocity': 1, 'motor_acceleration': 1}, 'timestamps': {'motor_velocity': 1573521581.324532, 'motor_acceleration': 1573521581.3246017}, 'data_keys': {'motor_velocity': {'source': 'SIM:motor_velocity', 'dtype': 'integer', 'shape': []}, 'motor_acceleration': {'source': 'SIM:motor_acceleration', 'dtype': 'integer', 'shape': []}}}, 'det': {'data': {'det': 0.6065306597126334}, 'timestamps': {'det': 1573521581.3480463}, 'data_keys': {'det': {'source': 'SIM:det', 'dtype': 'number', 'shape': [], 'precision': 3}}}}, 'name': 'primary', 'hints': {'motor': {'fields': ['motor']}, 'det': {'fields': []}}, 'object_keys': {'motor': ['motor', 'motor_setpoint'], 'det': ['det']}}]
args:
get_run_start: <function BlueskyRunFromGenerator.__init__.<locals>.get_run_start at 0x7f9380454c80>
stream_name: primary
get_run_stop: <function BlueskyRunFromGenerator.__init__.<locals>.get_run_stop at 0x7f93803df8c8>
get_event_descriptors: <function BlueskyRunFromGenerator.__init__.<locals>.get_event_descriptors at 0x7f93803df950>
get_event_pages: <function BlueskyRunFromGenerator.__init__.<locals>.get_event_pages at 0x7f93803df9d8>
get_event_count: <function BlueskyRunFromGenerator.__init__.<locals>.get_event_count at 0x7f93803dfa60>
get_resource: <function BlueskyRunFromGenerator.__init__.<locals>.get_resource at 0x7f93803dfae8>
lookup_resource_for_datum: <function BlueskyRunFromGenerator.__init__.<locals>.lookup_resource_for_datum at 0x7f93803dfb70>
get_datum_pages: <function BlueskyRunFromGenerator.__init__.<locals>.get_datum_pages at 0x7f93803dfbf8>
filler: <Filler>
metadata:
descriptors: [{'run_start': '65d84525-ab45-49eb-ab42-c1ef7f06c70a', 'time': 1573521581.3489022, 'data_keys': {'motor': {'source': 'SIM:motor', 'dtype': 'number', 'shape': [], 'precision': 3, 'object_name': 'motor'}, 'motor_setpoint': {'source': 'SIM:motor_setpoint', 'dtype': 'number', 'shape': [], 'precision': 3, 'object_name': 'motor'}, 'det': {'source': 'SIM:det', 'dtype': 'number', 'shape': [], 'precision': 3, 'object_name': 'det'}}, 'uid': '14c7c113-e869-4139-a45f-113764a5fe7f', 'configuration': {'motor': {'data': {'motor_velocity': 1, 'motor_acceleration': 1}, 'timestamps': {'motor_velocity': 1573521581.324532, 'motor_acceleration': 1573521581.3246017}, 'data_keys': {'motor_velocity': {'source': 'SIM:motor_velocity', 'dtype': 'integer', 'shape': []}, 'motor_acceleration': {'source': 'SIM:motor_acceleration', 'dtype': 'integer', 'shape': []}}}, 'det': {'data': {'det': 0.6065306597126334}, 'timestamps': {'det': 1573521581.3480463}, 'data_keys': {'det': {'source': 'SIM:det', 'dtype': 'number', 'shape': [], 'precision': 3}}}}, 'name': 'primary', 'hints': {'motor': {'fields': ['motor']}, 'det': {'fields': []}}, 'object_keys': {'motor': ['motor', 'motor_setpoint'], 'det': ['det']}}]
In [15]: catalog[uid]()['primary']()
Out[15]: <Intake catalog: Stream 'primary' from Run 65d84525...>
In [16]: catalog[uid]()['primary']().read()
Out[16]:
<xarray.Dataset>
Dimensions: (time: 3)
Coordinates:
* time (time) float64 1.574e+09 1.574e+09 1.574e+09
Data variables:
motor (time) float64 -1.0 0.0 1.0
motor_setpoint (time) float64 -1.0 0.0 1.0
det (time) float64 0.6065 1.0 0.6065
motor:motor_velocity (time) int64 1 1 1
motor:motor_acceleration (time) int64 1 1 1
det:det (time) float64 0.6065 0.6065 0.6065
seq_num (time) int64 1 2 3
uid (time) <U36 '7b3b7215-6eeb-4906-8bce-16cabe97bbaa' ... '921cba89-7f2f-4aa9-81e1-de3708e8050b'
The calls accept optional arguments. For example, we can filter by field name to avoid reading data that we do not need.
In [17]: catalog[uid]().primary(include=['det']).read()
Out[17]:
<xarray.Dataset>
Dimensions: (time: 3)
Coordinates:
* time (time) float64 1.574e+09 1.574e+09 1.574e+09
Data variables:
det (time) float64 0.6065 1.0 0.6065
seq_num (time) int64 1 2 3
uid (time) <U36 '7b3b7215-6eeb-4906-8bce-16cabe97bbaa' ... '921cba89-7f2f-4aa9-81e1-de3708e8050b'
If the defaults suffice, intake allows the user to elide the calls, supporting this more succinct expression:
In [18]: catalog[uid].primary.read()
Out[18]:
<xarray.Dataset>
Dimensions: (time: 3)
Coordinates:
* time (time) float64 1.574e+09 1.574e+09 1.574e+09
Data variables:
motor (time) float64 -1.0 0.0 1.0
motor_setpoint (time) float64 -1.0 0.0 1.0
det (time) float64 0.6065 1.0 0.6065
motor:motor_velocity (time) int64 1 1 1
motor:motor_acceleration (time) int64 1 1 1
det:det (time) float64 0.6065 0.6065 0.6065
seq_num (time) int64 1 2 3
uid (time) <U36 '7b3b7215-6eeb-4906-8bce-16cabe97bbaa' ... '921cba89-7f2f-4aa9-81e1-de3708e8050b'
Our simple scan produced one logical table (“event stream”) named ‘primary’. In general experiments can produce multiple tables with different time-bases. They can be merged into one xarray Dataset like so:
In [19]: run = catalog[uid]()
In [20]: run
Out[20]:
Run Catalog
uid='65d84525-ab45-49eb-ab42-c1ef7f06c70a'
exit_status='success'
2019-11-12 01:19:41.345 -- 2019-11-12 01:19:41.353
Streams:
* primary
In [21]: import xarray
In [22]: xarray.merge(run[key].read() for key in run)
Out[22]:
<xarray.Dataset>
Dimensions: (time: 3)
Coordinates:
* time (time) float64 1.574e+09 1.574e+09 1.574e+09
Data variables:
motor (time) float64 -1.0 0.0 1.0
motor_setpoint (time) float64 -1.0 0.0 1.0
det (time) float64 0.6065 1.0 0.6065
motor:motor_velocity (time) int64 1 1 1
motor:motor_acceleration (time) int64 1 1 1
det:det (time) float64 0.6065 0.6065 0.6065
seq_num (time) int64 1 2 3
uid (time) <U36 '7b3b7215-6eeb-4906-8bce-16cabe97bbaa' ... '921cba89-7f2f-4aa9-81e1-de3708e8050b'
Read Data as Bluesky “Documents”¶
In [23]: catalog[uid].canonical()
Out[23]: <generator object BlueskyRun.canonical at 0x7f9380370258>
This generator yields (name, doc) pairs and can be fed into streaming
visualization, processing, and serialization tools that consume this
representation, such as those provided by bluesky. This is the same
representation that was emitted when the data was first acquired, so the user
can apply the same streaming pipelines to data while it is being acquired and
after it is saved.
Search¶
The Catalog.search() method returns another Catalog with a subset of the
original Catalog’s entries. The search results can be searched in turn. Here we
search for data acquired using the ‘scan’ experiment plan, and then narrow that
to results from the last 60 seconds.
In [24]: catalog2 = catalog.search({'plan_name': 'scan'})
In [25]: list(catalog2)
Out[25]: ['65d84525-ab45-49eb-ab42-c1ef7f06c70a']
In [26]: import time
In [27]: catalog3 = catalog2.search({'time': {'$gt': time.time() - 60}})
In [28]: list(catalog3)
Out[28]: ['65d84525-ab45-49eb-ab42-c1ef7f06c70a']
This is accomplished using mongoquery, which provides a MongoDB-like query language for querying Python collections. For Catalogs backed by a real MongoDB instance, as in the Binder example linked above, the full MongoDB query language is supported.
The Catalogs carry the composite search query as internal state.
In [29]: catalog._query
Out[29]: {}
In [30]: catalog2._query
Out[30]: {'plan_name': 'scan'}
In [31]: catalog3._query
Out[31]: {'$and': [{'plan_name': 'scan'}, {'time': {'$gt': 1573521521.7918866}}]}