Release History

v1.16.1 (2020-10-15)

Added

  • The “projections” schema in the Run Start document has been enhanced to accept “configuration” locaitons intended to link to fields that are in Event stream Configuration See Projections (Experimental). Additionally, validation of projections was enhanced.

  • The method start was enhanced to check for repeated runs with colliding uids, raising an ValueError when this occurs.

Changed

v1.16.0 (2020-09-03)

Added

Changed

  • The function register_coercion() replaces the misspelled register_coersion(), which is retained as an alias for backward-compatibility.

v1.16.0 (2020-09-03)

Added

Changed

  • The function register_coercion() replaces the misspelled register_coersion(), which is retained as an alias for backward-compatibility.

v1.15.2 (2020-06-12)

Added

  • Various documentation additions.

  • jsonschema 2.x compatibility.

  • Better naming for handler subclasses.

v1.15.1 (2020-05-01)

Fixed

  • A bug was fixed in RunRouter which caused descriptor documents to be sent to subfactory callback start methods.

v1.15.0 (2020-04-27)

Fixed

  • In the data model documentation an erroneous link to the RunStart schema was corrected to a link to the EventDescriptor schema.

Changed

  • SingleRunDocumentRouter was added with convenience methods for getting the start document, the descriptor document for an event document, and the stream name for an event document.

  • In v1.14.0, RunRouter was changed to pass the RunStart document directly to its callbacks. To smooth the transition, any Exception raised by the callbacks was squashed and a warning printed. With v1.15.0 these Exceptions are allowed to propagate. The warning is still printed.

v1.14.1 (2020-04-06)

Fixed

  • In v1.13.0, the Filler object was unintentionally made un-pickleable. It can now be pickled.

  • For validation purposes, we accept numpy arrays as “array”-like.

v1.14.0 (2020-03-11)

Fixed

  • Let register_coersion() tolerate duplicate registration of the same coersion as the long the duplicate is identical with the original (i.e. func is original_func). This is now consistent with how handler registration works.

  • Fix a critical typo in an error message codepath in register_coersion().

Changed

  • The RunRouter hands RunStart documents to its factory functions so they can decide which if any callbacks to subscribe for that run. Formerly, the RunRouter left it up to the factory functions to pass the RunStart document through to any callbacks the factory function returned. Now, the RunRouter passes the RunStart document to the callbacks directly, removing that responsibility from the factory. To smooth this transition, it does so inside a try...except block and warns if any Exception is raised. This is a best effort at backward-compatibility with factories that are currently passing the RunStart document in, though it may not work in every case depending on the details of the callback. Likewise for subfactories: the callbacks that they return will be given the RunStart document and the relevant EventDescriptor document inside a try...except block.

v1.13.3 (2020-03-05)

Fixed

Changed

  • Raise a more specific error when Filler encounters an error due to a malformed document.

See the GitHub v1.13.3 milestone for a complete list of changes in this release.

v1.13.2 (2020-01-31)

Fixed

A bug in the new dispatch logic in DocumentRouter introduced in v1.13.1 caused the dispatcher to sometimes return NotImplemented. Now it always falls back to returning the original document if the subclass returns None or NotImplemented.

v1.13.1 (2020-01-28)

Changed

  • The DocumentRouter converts and routes Event and EventPage documents correctly if either one or both of the methods event or event_page is overridden in the subclass. Likewise for Datum and DatumPage and the methods datum and datum_page. The base class implementations all document-type methods now return the Python built-in sentinel NotImplemented (not to be confused with the exception NotImplementedError).

  • This retry-with-backoff loop in Filler is now applied to handler instantiation as well as handler calls. Either can involve I/O with a filesystem that may lag slightly behind the availability of the documents.

v1.13.0 (2020-01-21)

Added

  • The Filler accepts an optional parameter coerce that can be used to change the behavior of the handlers. This is useful for forcing the filled data to be an in-memory numpy array or a dask array, for example. The options accepted by coerce can be configured at runtime using the new function register_coersion(). The coersions registered by default are as_is() and force_numpy().

  • The NoFiller has been added. It has the same interface as Filler but it merely validates the filling-related documents rather than actually filling in the data. This is useful if the filling may be done later as a delayed computation but we want to know immediately that we have all the information we need to perform that computation.

  • It is sometimes convenient to make an instance of Filler based on an existing instance but perhaps setting some options differently. The new method clone() takes all the same parameters as Filler instantiation. If called with no arguments, it will make a “clone” with all the same options. Pass in arguments to override certain options.

Changed

  • The NumpyEncoder special-cases dask arrays.

  • Several error messages have been made more specific and useful.

Deprecated

  • Field-level filtering in Filler via the parameters include and exclude is deprecated.

Internal Changes

  • The code in DocumentRouter that dispatches based on document type has been factored out of __call__ into a new internal method, _dispatch, which makes it easier for subclasses to modify __call__ but reuse the dispatch logic.

See the GitHub v1.13.0 milestone for a complete list of changes in this release.

v1.12.0 (2019-10-11)

Added

  • The RunRouter can now “fill” documents that reference externally stored data. It accepts an optional handler_registry and root_map which it uses to create instances of Filler internally. The default behavior of RunRouter has not changed because it defaults to handler_registry={} and fill_or_fail=False, meaning that any external reference not found in handler_registry will be passed through unfilled. For advanced customizations—such as custom cache management—use the parameter filler_class to specifiy an API-compatible alternative to Filler.

Changed

Fixed

  • Fix cache management in Filler such that registering a new handler for a given spec clears all cached instances of the previously registered handler.

  • Fix the validation feature in DocumentRouter, which previously raised an error if used.

v1.11.2 (2019-09-03)

Fixed

  • Include requirements.txt in source distribution.

  • When UnresolveableForeignKeyError is raised, it always includes a key attribute with the key in question.

v1.11.1 (2019-08-09)

Fixed

  • Fix some inconsistent behavior in the Filler inplace parameter, and test it better.

v1.11.0 (2019-06-27)

Added

  • Added new optional parameter inplace to Filler.

  • Added new methods fill_event() and fill_event_page().

  • Added rechunk_event_pages().

Fixed

  • Consult the Event Descriptor document to infer which columns need to be filled if there is no explicit 'filled' key in the Event document.

v1.10.0 (2019-05-24)

This release requires jsonschema>3. Previous releases required jsonschema<3.

Added

  • Added schema_validators using the new interface in jsonschema 3.0.

Fixes

  • The counters in num_events in the RunStop document were off by one.

v1.9.0 (2019-05-01)

Added

Fixes

  • unpack_datum_page() errored when datum_kwargs were empty.

  • Fill EventPages in place, as Events are filled in place.

  • Do not assume Events and EventPages have a filled key; it is optional.

v1.8.3 (2019-03-28)

Fixes

  • Add 'configuration' to EventDescriptor schema.

  • Fix path semantics and be robust against empty 'filled'.

  • Fix sequence numbers in compose_descriptor().

  • Fix a typo which made 'num_events' always empty.

v1.8.2 (2019-03-08)

Fix setup.py meta-data to include python_requires. This prevents the wheels from being installed on python < 3.6.

v1.8.0 (2019-03-05)

Added

v1.7.0 (2019-01-03)

Added

  • The DataKey in an EventDescriptors may contain a ‘dims’ key, providing names for each dimension of the data.

  • Convenience functions for composing valid documents have been added. These are experimental and may change in a future release in a non-backward-compatible way.