Interruptions

The RunEngine can be cleanly paused and resumed. Plans are resumed at specified checkpoints to ensure that the interruption does not corrupt the data or a miss a step. When an interruption occurs, the RunEngine “rewinds” the plan and carefully re-executes the steps between the interruption and last checkpoint.

Pausing and Suspending

A pause can be initiated interactively:

  • Ctrl+C once: Pause at the next checkpoint (e.g., finish the current step in a scan first).
  • Ctrl+C twice: Pause immediately.
In [1]: In [1]: RE(scan([det], motor, -10, 10, 15), subs)
   ...: +-----------+------------+------------+------------+
   ...: |   seq_num |       time |      motor |        det |
   ...: +-----------+------------+------------+------------+
   ...: |         1 | 07:21:29.2 |    -10.000 |      0.000 |
   ...: |         2 | 07:21:29.3 |     -8.571 |      0.000 |
   ...: |         3 | 07:21:29.4 |     -7.143 |      0.000 |
   ...: ^C^C
   ...: Pausing...
   ...: In [2]:
   ...: 

or planned as part of the logic of an experiment:

# count; pause and wait for the user to resume; count again
In [2]: In [1]: RE(pchain(count([det]), pause(), count([det]))

When the RunEngine is paused, it returns the command prompt to the user. During the pause, the user can do anything: check readings, move motors, etc. Then, from a paused state, the user can choose to resume:

In [3]: In [2]: RE.resume()
   ...: Resuming from last checkpoint...
   ...: |         4 | 07:21:29.5 |     -5.714 |      0.000 |
   ...: |         5 | 07:21:29.5 |     -4.286 |      0.000 |
   ...: |         6 | 07:21:29.6 |     -2.857 |      0.017 |
   ...: |         7 | 07:21:29.7 |     -1.429 |      0.360 |
   ...: (etc.)
   ...: 

or choose to stop/abort. (Read on for the distinction between these two.)

In [4]: In [3]: RE.abort()
   ...: Aborting...
   ...: Out[3]: ['8ef9388c-75d3-498c-a800-3b0bd24b88ed']
   ...: 

It can also be useful to interrupt execution automatically in response some condition (e.g., shutter closed, beam dumped, temperature exceed some limit). We use the word suspension to mean an unplanned pause initialized by some agent running the background. The agent (a “suspender”) monitors some condition and, if it detects a problem, it suspends execution. When it detects that conditions have returned to normal, it gives the RunEngine permission to resume after some interval. This can operate unattended.

In [5]: RE(my_scan)
+------------+-------------------+----------------+----------------+
|   seq_num  |             time  |         theta  |    sclr_chan4  |
+------------+-------------------+----------------+----------------+
|         1  |  16:46:08.953815  |          0.03  |        290.00  |
Suspending....To get prompt hit Ctrl-C to pause the scan
|         2  |  16:46:20.868445  |          0.09  |        279.00  |
|         3  |  16:46:29.077690  |          0.16  |        284.00  |
|         4  |  16:46:33.540643  |          0.23  |        278.00  |
+------------+-------------------+----------------+----------------+

A suspended plan does not return the prompt to the user. Like a paused plan, it stops executing new instructions and rewinds to the most recent checkpoint. But unlike a paused plan, it resumes execution automatically when conditions return to normal.

To take manual control of a suspended plan, pause it using Ctrl+C. This will override its plan to automatically resume.

Read on for an example of installing a suspender.

Checkpoints

Plan are specified as a sequence of granualor instructions like ‘read’ and ‘set’. The instructions can optionally include one or more ‘checkpoint’ messages, indicating a place where it safe to resume after an interruption. For example, checkpoints are placed before each step of a bluesky.plans.scan.

Some experiments are not resumable: for example, the sample may be melting or aging. Incorporating bluesky.plans.clear_checkpoint in a plan makes it un-resuming. If a pause or suspension are requested, the plan will abort instead.

Note

For developers, here some gritty details about checkpoints.

It is not legal to create checkpoint in the middle of a data point (between ‘create’ and ‘save’) Checkpoints are implicitly created after actions that it is not safe to replay: staging a device, adding a monitor, or adding a subscription.

Deferred Pause vs Hard Pause

When a deferred pause is requested (Ctrl+C once), the RunEngine continues processing messages until the next checkpoint or the end of the plan, whichever happens first. When (if) it reaches a checkpoint, it pauses. Then it can be resumed from that checkpoint without repeating any work.

When a hard pause is requested (Ctrl+C twice), the RunEngine pauses as soon as possible — normally within less than second.

Stopping vs Aborting

To stop a paused plan, use RE.stop() or RE.abort(). In both cases, any data that has been generated will be saved. The only difference is that aborted runs are marked with exit_status: 'abort' instead of exit_status: 'success', which may be a useful distinction during analysis.

Suspenders

Bluesky includes several “suspenders” that work with ophyd Signals to monitor conditions and suspend execution. It’s also possible to write suspenders from scratch to monitor anything at all.

We’ll start with an example.

Example: Suspend a plan if a shutter closes; resume when it opens

We will use a built-in utility that watches an EPICS PV. It tells the RunEngine to suspend when the PV’s value goes high. When it goes low again, the RunEngine resumes.

from ophyd import EpicsSignal
from bluesky.suspenders SuspendBoolHigh

shutter = EpicsSignal('XF:23ID1-PPS{PSh}Pos-Sts')  # main shutter PV

sus = SuspendBoolHigh(signal)
RE.install_suspender(sus)

The above is all that is required. It will watch the PV indefinitely. In the following example, the shuttle was closed in the middle of the second data point.

In [6]: RE(my_scan)
+------------+-------------------+----------------+----------------+
|   seq_num  |             time  |         theta  |    sclr_chan4  |
+------------+-------------------+----------------+----------------+
|         1  |  16:46:08.953815  |          0.03  |        290.00  |
Suspending....To get prompt hit Ctrl-C to pause the scan
|         2  |  16:46:20.868445  |          0.09  |        279.00  |
|         3  |  16:46:29.077690  |          0.16  |        284.00  |
|         4  |  16:46:33.540643  |          0.23  |        278.00  |
+------------+-------------------+----------------+----------------+

Notice that the plan was suspended and then resumed. When it resumed, it went back to the last checkpoint and re-took the second data point cleanly.

Built-in Suspenders

The example above demonstrates SuspendBoolHigh. Several other variants are built in, and it is straightforward to write customized ones.

bluesky.suspenders.SuspendBoolHigh Suspend when a boolean signal goes high; resume when it goes low.
bluesky.suspenders.SuspendBoolLow Suspend when a boolean signal goes low; resume when it goes high.
bluesky.suspenders.SuspendFloor Suspend when a scalar falls below a threshold.
bluesky.suspenders.SuspendCeil Suspend when a scalar rises above a threshold.
bluesky.suspenders.SuspendInBand Suspend when a scalar signal leaves a given band of values.
bluesky.suspenders.SuspendOutBand Suspend when a scalar signal enters a given band of values.

Deferred Pause

When a deferred pause is requested, the RunEngine continues processing messages until the next checkpoint or the end of the plan, whichever happens first. When (if) it reaches a checkpoint, it pauses. Then it can be resumed from that checkpoint without repeating any work.

Associated RunEngine Interface

State

The RunEngine has a state machine defining its phases of operation and the allowed transitions between them. As illustrated above, it can be inspected via the state property.

The states are:

  • 'idle': RunEngine is waiting for instructions.
  • 'running': RunEngine is executing instructions.
  • 'paused': RunEngine is waiting for user input. It can be

Request Methods

This method is called when Ctrl+C is pressed or when a ‘pause’ Message is processed. It can also be called by user-defined agents. See the next example.

RunEngine.request_pause(defer=False)

Command the Run Engine to pause.

This function is called by ‘pause’ Messages. It can also be called by other threads. It cannot be called on the main thread during a run, but it is called by SIGINT (i.e., Ctrl+C).

If there current run has no checkpoint (via the ‘clear_checkpoint’ message), this will cause the run to abort.

Parameters:defer (bool, optional) – If False, pause immediately before processing any new messages. If True, pause at the next checkpoint. False by default.

This method is used by the PVSuspend* classes above. It can also be called by user-defined agents.

RunEngine.request_suspend(fut, *, pre_plan=None, post_plan=None, justification=None)

Request that the run suspend itself until the future is finished.

The two plans will be run before and after waiting for the future. This enable doing things like opening and closing shutters and resetting cameras around a suspend.

Parameters:
  • fut (asyncio.Future) –
  • pre_plan (iterable, optional) – Plan to execute just before suspending
  • post_plan (iterable, optional) – Plan to execute just before resuming
  • justification (str, optional) – explanation of why the suspension has been requested

Example: Requesting a pause from the asyncio event loop

Since the user does not control of the prompt, calls to RE.request_pause must be planned in advance. Here is a example that pauses the plan after 5 seconds.

from bluesky.plans import null

def loop_forever():
    "a silly plan"
    while True:
        yield from null()

import asyncio
loop = asyncio.get_event_loop()
# Request a pause 5 seconds from now.
loop.call_later(5, RE.request_pause)

# Execute the plan.
RE(loop_forever())

# Five seconds after ``call_later`` was run, the plan is paused.
# Observe that the RunEngine is in a 'paused' state.
RE.state

Above, we passed True to RE.request_pause to request a deferred pause.