Recording Metadata
==================

Usage Example
-------------

Metadata is bundled with the data generated by a scan. It can be used by
callbacks (e.g., to tell plots how to label their axes) or simply used to
identify and analyze the data later.

.. ipython:: python
    :suppress:

    from bluesky.global_state import gs
    from bluesky.examples import det
    from bluesky.plans import Count
    RE = gs.RE
    RE.md['owner'] = 'demo'
    RE.md['group'] = 'demo'

.. ipython:: python

    RE.md['project'] = 'my xray project'
    RE.md['sample'] = {'color': 'red', 'dimensions': [10, 20, 5]}

Or specified when the scan is run.

.. ipython:: python

    plan = Count([det])
    RE(plan, experimenter='Emily', mood='excited')

Special Fields
--------------

Custom metadata keywords can be mapped to strings (``task='calibration'``),
numbers (``attempt=5``), lists (``dimensions=[1, 3]``), or
dictionaries (``dimensions={'width': 1, 'height': 3}``). But certain keywords
are given special significance by bluesky's document model.

String Fields
+++++++++++++

To facilitate searchability, the keywords 'owner', 'group', and 'project' are
given special significance. They are all optional, but if provided they must be
strings like ``owner='Dan'``. A non-string, like ``owner=5`` will produce an
error that will interrupt scan execution immediately after it starts.

Again, these fields are optional.

Sample
++++++

Similarly, the keyword "sample" has special significance. It must be either a
string:

.. code-block:: python

    'red 10 20 5'

or a dictionary:

.. code-block:: python

    {'color': 'red', 'dimensions': [10, 20, 5]}

A dictionary is preferred because it is self-describing and more richly
searchable, but either is allowed.

Scan ID
+++++++

The ``scan_id`` field is expected to be an integer, and it is automatically
incremented between runs. If a ``scan_id`` is not provided by the user or
stashed in the persistent metadata from the previous run, it defaults to 1.

Required Fields
---------------

In current versions of bluesky, **no fields are required**.

In versions v0.4.3 and below, the keys ``owner``, ``group``, and
``beamline_id`` were required.

Persistence Between Runs
------------------------

To set a field of metadata to persist for future runs, add it to ``RE.md``.

.. ipython:: python

    RE.md['color'] = 'blue'

Now it will be included in the metadata of every scan until it is deleted:

.. ipython:: python

    del RE.md['color']

To review the metadata before running ascan, check ``RE.md``, which
behaves like a Python dictionary.

.. ipython:: python

    RE.md['sample']

To start fresh:

.. ipython:: python

    RE.md.clear()

Persistence Between Sessions
----------------------------

The ``RE.md`` attribute shown above may be a Python dictionary or anything
that supports the dictionary interface. To persist metadata between
sessions, we suggest ``historydict`` --- a Python dictionary backed by a
sqlite database.

Example:

.. ipython:: python

    from historydict import HistoryDict
    hist = HistoryDict('metadata-cache.sqlite')
    RE = RunEngine(hist)
    type(RE.md)

Any metadata added to ``RE.md``, including the ``scan_id``, will be saved
and can be re-loaded.

Metadata Validator
------------------

Additional, customized metadata validation can be added to the RunEngine.
For example, to ensure that a run will not be executed unless the parameter
'sample_number' is specified, define a function that accepts a dictionary
argument and raises if 'sample_number' is not found.

.. code-block:: python

    def ensure_sample_number(md):
        if 'sample_number' not in md:
            raise ValueError("You forgot the sample number.")

Apply this function by setting ``RE.md_validator = ensure_sample_number``.
The function will be executed immediately before each new run in opened.