onsrap.stage module

class onsrap.stage.Stage(name: str, source: ~pathlib.Path | ~typing.Callable[[...], ~typing.Any] | None = None, dependencies: tuple[str, ...] = <factory>, metadata: dict[str, ~typing.Any] = <factory>, entrypoint: str | None = None, backend: str = 'python')

Bases: object

Represents a single unit of work within a pipeline.

Can be defined by a data source process or a Python script/callable item. Stages may be dependant on other stages and can hold metadata for themselves.

Parameters:

  • name (str) – The name of the Stage being run.

  • source (Path, Callable, or None) – Item being implemented in this Stage. E.g. a file path to a Python script or a function being executed directly. The full file path is gathered if a path is used.

  • dependencies (tuple of strings) – Names of stages that must be completed before this stage is attempted. These are cleaned post initialisation to remove leading/trailing whitespace.

  • metadata (dictionary with string:Any key/value pairs) – Location to store any summary information about the stage being run.

  • entrypoint (str, optional) – Name of the starting script to the pipeline.

  • backend (str, default = "python") – The name of the system that the code runs on.

Raises:

StageConfigurationError – If the stage name is empty or if the source is not a supported type.

backend: str = 'python'
dependencies: tuple[str, ...]
entrypoint: str | None = None
classmethod from_callable(callable_object: Callable[[...], Any], *, name: str | None = None, dependencies: Iterable[str] | str | None = None, metadata: Mapping[str, Any] | None = None, backend: str = 'python') → Stage

Class method that retrieves the name of the Stage from a Callable item.

Parameters:

  • callable_object (Callable with any number of arguments of any type) – The name or file path for the script that the Stage will be running.

  • name (str) – The name of the Stage

  • dependencies (Iterable[str], str, or None) – The Stage/s that need to be complete before the Stage currently attempted.

  • metadata (Mapping[str, Any], or None) – Any supporting information for the Stage being run.

  • entrypoint (str or None) – The name of the first script for the Stage.

  • backend (str, default = "python") – The system that the stage is run on.

Returns:

Stage class instance with collected Stage name, normalised dependencies and metadata, and defined the source as the callable_object.

Return type:

Stage

classmethod from_dict(data: Mapping[str, Any]) → Stage

Class method that converts a dictionary stage into a Stage class instance.

Extracts the values from the key/value pairs in the stage and holds them as attributes.

Parameters:

data (any number of key/value pairs of strings) – The information to convert into a Stage class.

Raises:

StageConfigurationError – If the source is not a suitable type (callable or Path).

Returns:

Stage class instance with collected Stage attributes based on the type of source provided.

Return type:

Stage

classmethod from_file(file_path: str | Path, *, name: str | None = None, dependencies: Iterable[str] | str | None = None, metadata: Mapping[str, Any] | None = None, entrypoint: str | None = None, backend: str = 'python') → Stage

Class method that checks and cleans the file path for the Stage.

Expands file path to its full name and checks whether it exists. The method also cleans other parameters in the Stage class in the return line.

Parameters:

  • file_path (str or Path) – The name or file path for the script that the Stage will be running.

  • name (str) – The name of the Stage

  • dependencies (Iterable[str], str, or None) – The Stage/s that need to be complete before the Stage currently attempted.

  • metadata (Mapping[str, Any], or None) – Any supporting information for the Stage being run.

  • entrypoint (str or None) – The name of the first script for the Stage.

  • backend (str, default = "python") – The system that the Stage is run on.

Raises:

StageConfigurationError – If the file path does not exist

Returns:

Stage class instance with cleaned/checked file path, dependencies, and metadata

Return type:

Stage

metadata: dict[str, Any]
name: str
run(context: ExecutionContext, executor: StageExecutor) → StageResult

Checks that the source is valid and then runs the source

Properties

contextset value “ExecutionContext”

Uses ExecutionContext class information to provide required metadata on running source. Any stage-specific configuration resolved by the Pipeline is available through context.stage_config while this stage is running.

executorset value “StageExecutor”

Uses StageExecutor class to extract the .execute method to actually run the source.

rtype:

execute method of the StageExecutor class stored in the StageResult class.

source: Path | Callable[[...], Any] | None = None
property source_label: str | None

Sets a property for the Stage class with a human-readable name for the source.

Return type:

source_label attribute to the Stage class if source is a callable or Path.

property source_path: Path | None

Sets a property for the Stage class if the source is a path.

Return type:

source_path attribute to the Stage class if the source is a path.

validate() → None

Error checking on source attribute.

Raises:

StageConfigurationError – If source attribute does not define a source or does not exist.

with_dependencies(*dependencies: str) → Stage

Method that normalises and adds dependencies to the Stage class attributes.

Parameters:

*dependencies (str) – Information on which scripts need to run before other scripts for this Stage.

Returns:

Stage class instance with normalised dependencies attribute.

Return type:

Stage