Interruptions

The RunEngine can be safely interrupted and resumed. All plans get this feature “for free.”

Pausing Interactively

Note

Looking for a quick refresher on pausing, resuming, or aborting interactively? Skip to the Summary.

While the RunEngine is executing a plan, it captures SIGINT (Ctrl+C).

Pause Now: Ctrl+C twice

In [14]: RE(scan([det], motor, 1, 10, 10))
Transient Scan ID: 2     Time: 2018/02/12 12:43:12
Persistent Unique Scan ID: '33a16823-e214-4952-abdd-032a78b8478f'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time |      motor |        det |
+-----------+------------+------------+------------+
|         1 | 12:43:13.3 |      1.000 |      0.607 |
|         2 | 12:43:14.3 |      2.000 |      0.135 |
|         3 | 12:43:15.3 |      3.000 |      0.011 |
^C
A 'deferred pause' has been requested. The RunEngine will pause at the next checkpoint. To pause immediately, hit Ctrl+C again in the next 10 seconds.
Deferred pause acknowledged. Continuing to checkpoint.
^C
Pausing...
---------------------------------------------------------------------------
RunEngineInterrupted                      Traceback (most recent call last)
<ipython-input-14-826ee9dfb918> in <module>()
----> 1 RE(scan([det], motor, 1, 10, 10))

~/Documents/Repos/bluesky/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    670
    671             if self._interrupted:
--> 672                 raise RunEngineInterrupted(self.pause_msg) from None
    673
    674         return tuple(self._run_start_uids)

RunEngineInterrupted:
Your RunEngine is entering a paused state. These are your options for changing
the state of the RunEngine:

RE.resume()    Resume the plan.
RE.abort()     Perform cleanup, then kill plan. Mark exit_stats='aborted'.
RE.stop()      Perform cleanup, then kill plan. Mark exit_status='success'.
RE.halt()      Emergency Stop: Do not perform cleanup --- just stop.

Before returning the prompt to the user, the RunEngine ensures that all motors that it has touched are stopped. It also performs any device-specific cleanup defined in the device’s (optional) pause() method.

If execution is later resumed, the RunEngine will “rewind” through the plan to the most recent checkpoint, the last safe place to restart.

Pause Soon: Ctrl+C once

Pause at the next checkpoint: typically, the next step in a step scan. We call this “deferred pause.” It avoids having to repeat any work when the plan is resumed.

Notice that this time when Ctrl+C (^C) is hit, the current step (4) is allowed to complete before execution is paused.

In [12]: RE(scan([det], motor, 1, 10, 10))
Transient Scan ID: 1     Time: 2018/02/12 12:40:36
Persistent Unique Scan ID: 'c5db9bb4-fb7f-49f4-948b-72fb716d1f67'
New stream: 'primary'
+-----------+------------+------------+------------+
|   seq_num |       time |      motor |        det |
+-----------+------------+------------+------------+
|         1 | 12:40:37.6 |      1.000 |      0.607 |
|         2 | 12:40:38.7 |      2.000 |      0.135 |
|         3 | 12:40:39.7 |      3.000 |      0.011 |
^CA 'deferred pause' has been requested. The RunEngine will pause at the next checkpoint. To pause immediately, hit Ctrl+C again in the next 10 seconds.
Deferred pause acknowledged. Continuing to checkpoint.
|         4 | 12:40:40.7 |      4.000 |      0.000 |
Pausing...
---------------------------------------------------------------------------
RunEngineInterrupted                      Traceback (most recent call last)
<ipython-input-12-826ee9dfb918> in <module>()
----> 1 RE(scan([det], motor, 1, 10, 10))

~/Documents/Repos/bluesky/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    670
    671             if self._interrupted:
--> 672                 raise RunEngineInterrupted(self.pause_msg) from None
    673
    674         return tuple(self._run_start_uids)

RunEngineInterrupted:
Your RunEngine is entering a paused state. These are your options for changing
the state of the RunEngine:

RE.resume()    Resume the plan.
RE.abort()     Perform cleanup, then kill plan. Mark exit_stats='aborted'.
RE.stop()      Perform cleanup, then kill plan. Mark exit_status='success'.
RE.halt()      Emergency Stop: Do not perform cleanup --- just stop.

What to do after pausing

After being paused, the RunEngine holds on to information that it might need in order to resume later. It “knows” that it is in a paused state, and you can check that at any time:

In [2]: RE.state
Out[2]: 'paused'

During the pause, we can do anything: check readings, move motors, etc. It will not allow you to execute a new plan until the current one is either resumed or terminated. Your options are:

Resume

In [3]: RE.resume()
|         4 | 07:21:29.5 |     -5.714 |      0.000 |
|         5 | 07:21:29.5 |     -4.286 |      0.000 |
|         6 | 07:21:29.6 |     -2.857 |      0.017 |
|         7 | 07:21:29.7 |     -1.429 |      0.360 |
(etc.)

Depending on the plan, it may “rewind” to safely continue on and ensure all data is collected correctly.

Abort

Allow the plan to perform any final cleanup. For example, some plans move motors back to their starting positions. Mark the data as having been aborted, so that this fact can be noted (if desired) in later analysis. All of the data collected up this point will be saved regardless.

From a paused state:

In [3]: RE.abort()
Aborting...
Out[3]: ['8ef9388c-75d3-498c-a800-3b0bd24b88ed']

Stop

RE.stop() is functionally identical to RE.abort(). The only difference is that aborted runs are marked with exit_status: 'abort' instead of exit_status: 'success'. This may be a useful distinction during analysis.

Halt

Aborting or stopping allows the plan to perform cleanup. We already mentioned the example of a plan moving motors back to their starting positions at the end.

In some situations, you may wish to prevent the plan from doing anything — you want to halt immediately, skipping cleanup. For this, use RE.halt().

Summary

Interactively Interrupt Execution

Command	Outcome
Ctrl+C	Pause soon.
Ctrl+C twice	Pause now.

From a paused state

Command	Outcome
RE.resume()	Safely resume plan.
RE.abort()	Perform cleanup. Mark as aborted.
RE.stop()	Perform cleanup. Mark as success.
RE.halt()	Do not perform cleanup — just stop.
RE.state	Check if ‘paused’ or ‘idle’.

Automated Suspension

It can also be useful to interrupt execution automatically in response to some condition (e.g., shutter closed, beam dumped, temperature exceeded some limit). We use the word suspension to mean an unplanned pause initialized by some agent running the background. The agent (a “suspender”) monitors some condition and, if it detects a problem, it suspends execution. When it detects that conditions have returned to normal, it gives the RunEngine permission to resume after some interval. This can operate unattended.

In [1]: RE(scan([det], motor, -10, 10, 15), LiveTable([motor, det]))
+------------+-------------------+----------------+----------------+
|   seq_num  |             time  |         motor  |           det  |
+------------+-------------------+----------------+----------------+
|         1  |  16:46:08.953815  |          0.03  |        290.00  |
Suspending....To get prompt hit Ctrl-C to pause the scan
|         2  |  16:46:20.868445  |          0.09  |        279.00  |
|         3  |  16:46:29.077690  |          0.16  |        284.00  |
|         4  |  16:46:33.540643  |          0.23  |        278.00  |
+------------+-------------------+----------------+----------------+

A suspended plan does not return the prompt to the user. Like a paused plan, it stops executing new instructions and rewinds to the most recent checkpoint. But unlike a paused plan, it resumes execution automatically when conditions return to normal.

To take manual control of a suspended plan, pause it by hitting Ctrl+C twice. You will be given the prompt. When conditions are good again, you may manually resume using RE.resume().

Installing Suspenders

Bluesky includes several “suspenders” that work with ophyd Signals to monitor conditions and suspend execution. It’s also possible to write suspenders from scratch to monitor anything at all.

We’ll start with an example.

Example: Suspend a plan if the beam current dips low

This defines a suspender and installs it on the RunEngine. With this, plans will be automatically suspended when the beam_current signal goes below 2 and resumed once it exceeds 3.

from ophyd import EpicsSignal
from bluesky.suspenders import SuspendFloor

beam_current = EpicsSignal('...PV string...')
sus = SuspendFloor(beam_current, 2, resume_thresh=3)
RE.install_suspender(sus)

In the following example, the beam current dipped below 2 in the middle of taking the second data point. It later recovered.

In [2]: RE(my_scan)
+------------+-------------------+----------------+----------------+
|   seq_num  |             time  |         theta  |    sclr_chan4  |
+------------+-------------------+----------------+----------------+
|         1  |  16:46:08.953815  |          0.03  |        290.00  |
Suspending....To get prompt hit Ctrl-C to pause the scan
|         2  |  16:46:20.868445  |          0.09  |        279.00  |
|         3  |  16:46:29.077690  |          0.16  |        284.00  |
|         4  |  16:46:33.540643  |          0.23  |        278.00  |
+------------+-------------------+----------------+----------------+

Notice that the plan was suspended and then resumed. When it resumed, it went back to the last checkpoint and re-took the second data point cleanly.

See the API documentation (follow the links in the table below) for other suspender types and options, including a waiting period and cleanup procedures to run pre-suspend and pre-resume.

Built-in Suspenders

bluesky.suspenders.SuspendBoolHigh	Suspend when a boolean signal goes high; resume when it goes low.
bluesky.suspenders.SuspendBoolLow	Suspend when a boolean signal goes low; resume when it goes high.
bluesky.suspenders.SuspendFloor	Suspend when a scalar falls below a threshold.
bluesky.suspenders.SuspendCeil	Suspend when a scalar rises above a threshold.
bluesky.suspenders.SuspendWhenOutsideBand	Suspend when a scalar signal leaves a given band of values.
bluesky.suspenders.SuspendWhenChanged	Suspend when the monitored value deviates from the expected.

Checkpoints

Plans are specified as a sequence of messages, granular instructions like ‘read’ and ‘set’. The messages can optionally include one or more ‘checkpoint’ messages, indicating a place where it is safe to resume after an interruption. For example, checkpoints are placed before each step of a bluesky.plans.scan().

Some experiments are not resumable: for example, the sample may be melting or aging. Incorporating bluesky.plan_stubs.clear_checkpoint() in a plan makes it un-resuming. If a pause or suspension are requested, the plan will abort instead.

Note

Some details about checkpoints and when they are allowed:

It is not legal to create a checkpoint in the middle of a data point (between ‘create’ and ‘save’). Checkpoints are implicitly created after actions that it is not safe to replay: staging a device, adding a monitor, or adding a subscription.

Planned Pauses

Pausing is typically done interactively (Ctrl+C) but it can also be incorporated into a plan. The plan can pause the RunEngine, requiring the user to type RE.resume() to continue or RE.stop() (or similar) to clean up and stop.

import bluesky.plan_stubs as bps

def pausing_plan():
    while True:
        yield from some_plan(...)
        print("Type RE.resume() to go again or RE.stop() to stop.")
        yield from bps.checkpoint()  # marking where to resume from
        yield from bps.pause()

Associated RunEngine Interface

State

The RunEngine has a state machine defining its phases of operation and the allowed transitions between them. As illustrated above, it can be inspected via the state property.

The states are:

• 'idle': RunEngine is waiting for instructions.

• 'running': RunEngine is executing instructions.

• 'paused': RunEngine is waiting for user input.

Suspender-related Methods

RunEngine.install_suspender(suspender)

Install a ‘suspender’, which can suspend and resume execution.

Parameters

suspenderbluesky.suspenders.SuspenderBase

See also

RunEngine.remove_suspender()

RunEngine.clear_suspenders()

RunEngine.remove_suspender(suspender)

Uninstall a suspender.

Parameters

suspenderbluesky.suspenders.SuspenderBase

See also

RunEngine.install_suspender()

RunEngine.clear_suspenders()

RunEngine.clear_suspenders()

Uninstall all suspenders.

See also

RunEngine.install_suspender()

RunEngine.remove_suspender()

The RunEngine also has a suspenders property, a collection of the currently-installed suspenders.

Request Methods

This method is called when Ctrl+C is pressed or when a ‘pause’ Message is processed. It can also be called by user-defined agents. See the next example.

RunEngine.request_pause(defer=False)

Command the Run Engine to pause.

This function is called by ‘pause’ Messages. It can also be called by other threads. It cannot be called on the main thread during a run, but it is called by SIGINT (i.e., Ctrl+C).

If there current run has no checkpoint (via the ‘clear_checkpoint’ message), this will cause the run to abort.

Parameters

deferbool, optional

If False, pause immediately before processing any new messages. If True, pause at the next checkpoint. False by default.

This method is used by the PVSuspend* classes above. It can also be called by user-defined agents.

RunEngine.request_suspend(fut, *, pre_plan=None, post_plan=None, justification=None)

Request that the run suspend itself until the future is finished.

The two plans will be run before and after waiting for the future. This enable doing things like opening and closing shutters and resetting cameras around a suspend.

Parameters

futasyncio.Future

pre_planiterable or callable, optional

Plan to execute just before suspending. If callable, must take no arguments.

post_planiterable or callable, optional

Plan to execute just before resuming. If callable, must take no arguments.

justificationstr, optional

explanation of why the suspension has been requested

Example: Requesting a pause from the asyncio event loop

Since the user does not have control of the prompt, calls to RE.request_pause must be planned in advance. Here is a example that pauses the plan after 5 seconds.

from bluesky.plan_stubs import null

def loop_forever():
    "a silly plan"
    while True:
        yield from null()

import asyncio
loop = asyncio.get_event_loop()
# Request a pause 5 seconds from now.
loop.call_later(5, RE.request_pause)

# Execute the plan.
RE(loop_forever())

# Five seconds after ``call_later`` was run, the plan is paused.
# Observe that the RunEngine is in a 'paused' state.
RE.state

Above, we passed True to RE.request_pause to request a deferred pause.

Experimental: Record Interruptions

In the analysis stage, it can be useful to know if and when a run was interrupted. This experimental feature creates a special event stream recording the time and nature of any interruptions.

Warning

This is an experimental feature. It is tested but not yet widely used. It might be changed or removed in the future.

Activate this feature by setting

RE.record_interruptions = True

In this mode, the RunEngine emits a special event descriptor after opening a new run. This name field in the descriptor is ‘interruptions’. It has a single data key:

{'interruptions': {'dtype': 'string',
                   'shape': None,
                   'source': 'RunEngine'}}

Each time the RunEngine is paused, suspended, or resumed during the run, an Event document for that descriptor is created. The data payload event['data']['interruptions'] is 'pause', 'suspend', or 'resume'. The associated time notes when the interruptions/resume was processed.

To see this in action, try this example:

from bluesky.plans import count
from bluesky.preprocessors import pchain
from bluesky.plan_stubs import pause
from ophyd.sim import det

RE.record_interruptions = True

RE(pchain(count([det]), pause(), count([det])), print)
# ... RunEngine pauses
RE.resume()

In the text that print dumps to the screen, look for the special ‘interruptions’ event descriptor and associated events.