Technical Overview

DataBroker provides access to saved data from Bluesky. It does this by applying Intake to Bluesky.

Intake takes the pain out of data access. It loads data from a growing variety of formats into familiar Scipy/PyData data structures.

Bluesky is a suite of co-developed Python packages for data acquisition and management designed to drive experiments and capture data and metadata from experiments and simulations in a way that interfaces naturally with open-source software in general and the scientific Python ecosystem in particular.

Transition Plan

DataBroker began development in 2015, before Intake. Intake has goals closely aligned with DataBroker’s and a growing community of users from diverse scientific fields. Starting with DataBroker’s 1.0 release, DataBroker is pivoting to become a distribution of Intake plugins.

This change will proceed in a way that provides a slow and smooth transition for existing DataBroker users.

DataBroker 1.0

The 1.0 release of DataBroker supports multiple usages to enable a smooth transition for users.

  • The module databroker.v0 provides the databroker code exactly as it was in previous pre-1.0 releases. This is available as fallback in case of bugs in v1.

  • The module databroker.v1 provides the same user interface as v0, but built on top of intake instead of databroker’s own implementation. With only narrow exceptions, code that worked on pre-1.0 releases should work on databroker.v1. This is the “default” interface in DataBroker 1.0; importing from databroker ... is an alias for from databroker.v1 ....

  • The module databroker.v2 is a very thin shim around intake.

The Version 1 Interface documentation applies to databroker.v0 and databroker.v1. The Version 2 Interface documentation applies to databroker.v2.

Both databroker.v1.Broker and databroker.v2.Broker have accessor attributes, v1 and v2, that support usage of the other’s interface, making it easy to switch between them.

Here we make a v1-style Broker but the v2 accessor to try v2-style features.

from databroker import Broker
db = Broker.named('MY_CATALOG')  # a databroker.v1.Broker instance
query = dict(plan_name='count')
db(**query)  # v1-style search
db.v1(**query)  # a synonym for the above
db.v2.search(query)  # v2-style search

Here we make a v2-style Broker but the v1 accessor to fall back to v1-style usage.

from databroker import catalog
db = catalog.SOMETHING()  # a databroker.v2.Broker instance
query = dict(plan_name='count')
db.search(query)  # v2-style search
db.v2.search(query)  # a synonym for the above
db.v1(**query)  # v1-style search

Roadmap Beyond 1.0

  • When the v1 implementation is proven reliable, v0 will be removed, greatly reducing the amount of code in databroker. This should not have an effect on users, as v1 provides the same interface as v0. The only differences are internal: v1 uses intake internally instead of its own separate implementation to achieve the same end.

  • If and when the v2 interface is shown to meet users’ needs as well or better than v1, it will become the default. To support existing user code, the v1 interface will continue to be available in databroker.v1.

  • After several years of maintaining both v1 and v2, when all critical user code has been migrated to v2, v1 may be deprecated and eventually removed. At that point, DataBroker will be just a distribution of intake plugins.