executor Package¶

`executor` Package¶

Package with executors of analysis workflows. Executors perform and control the execution of workflows. The setup of workflows is often performed by the user or via dedicated workflow drivers.

`base` Module¶

Module containing base classes for workflow executors.

Workflow executors control the execution of workflows. The setup of workflows is often controlled either by a workflow driver or the user.

class omsi.workflow.executor.base.workflow_executor_base(analysis_objects=None)¶

Bases: omsi.datastructures.analysis_data.parameter_manager

Base class used to execute a workflow of one or many analyses. This is an object-based execution, i.e., the user defines a set of analyses to be executed.

We are given a set of existing analysis objects for which we need to coordinate the execution.

Variables:

analysis_objects – Private set of analysis objects to be executed
DEFAULT_EXECUTOR_CLASS – Define the derived workflow_executor_base class to be used as default executor The default value is None, in which case the greedy_workflow_executor is used. This variable is used by the get_default_executor function to instantiate a default workflow executor on request. Using this variable we can change the default executor to our own preferred executor, e.g., to change the executor used by the omsi.analysis.base.analysis_base functions execute_all(...) and execute_recursive(...)

To implement a derived workflow executor, we need to implement a derived class that implements the main() function.

Initialize the workflow executor

Parameters:	analysis_objects – A list of analysis objects to be executed

DEFAULT_EXECUTOR_CLASS = None¶: The default executor class to be used

add_analysis(analysis_object)¶

Add a given analysis to the set of object to be executed by the workflow

Shorthand for: self.analysis_tasks.add(analysis_object)

add_analysis_all()¶

Add all known analyses to the workflow.

Shorthand for: self.analysis_tasks.add_analysis_all()

add_analysis_dependencies()¶

Add the dependencies of all analyses to the workflow in case they are missing.

This function is recursive, step-by-step adding all dependencies of the workflow to the list of tasks to be executed, until no more dependencies are found.

Usually this function is called by the workflow executor itself before running the analysis and should not need to be called by the user.

Returns:	Integer indicating the number of dependencies added to the list of tasks

add_analysis_from_scripts(script_files)¶

Evaluate the list of scripts and add all (i.e., zero, one, or multiple) analyses to this workflow

NOTE: This function executes scripts using exec(..), i.e., there are NO safeguards against malicious codes.

Parameters:	script_files – List of strings with the paths to the script files. If only a single script is used, then a single string may be used as well.

clear()¶

Remove all analyses from the workflow.

Shorthand for: self.analysis_tasks.clear()

execute()¶: Execute the workflow. This uses the main() function to run the actual workflow.

classmethod from_script_files(script_files)¶

Same as from_script, but the scripts are read from the given list of files.

NOTE: This function executes scripts using exec(..), i.e., there are NO safeguards against malicious codes.

Parameters:	script_files – List of strings with the paths to the script files. If only a single script is used, then a single string may be used as well.
Returns:	Instance of the current workflow executor class for running the given workflow

classmethod from_scripts(scripts)¶

Create and initalize a worklflow executor of the current class type to execute the workflow defined in the given set of scripts.

This function using analysis_task_list.from_scripts to evaluate the workflow scripts to extract all analyses to be created.

NOTE: This function executes scripts using exec(..), i.e., there are NO safeguards against malicious codes.

Parameters:	scripts – The script with the setup of the workflow. This should only include the definition of analyses and their inputs.
Returns:	Instance of the current workflow executor class for running the given workflow

get_analyses()¶

Get the list of analyses to be run.

Shorthand for: self.analysis_tasks

get_analysis(index)¶

Get the analysis with the given index

Shorthand for: self.analysis_tasks[index]

Parameters:	index – Integer index of the analysis
Returns:	omsi.analysis.base.analysis_base object
Raises:	IndexError in case that the index is invalid

classmethod get_default_executor(analysis_objects=None)¶

Create an instance of the default workflow executor to be used.

Parameters:	analysis_objects – A set or unique list of analysis objects to be executed by the workflow
Returns:	Instance of the default workflow executor

classmethod get_default_executor_class()¶

Get the default executor class

Returns:	Derived class of workflow_executor_base

main()¶: Implement the execution of the workflow. We should always call execute(..) or __call__(..) to run the workflow. This function is intended to implementd the executor-specific exeuction behavior and must be implemented in child classes.

`greedy_executor` Module¶

Module used to help with the execution of complex analyses workflows

class omsi.workflow.executor.greedy_executor.greedy_executor(analysis_objects=None)¶

Bases: omsi.workflow.executor.base.workflow_executor_base

Execute a set of analysis objects and their dependencies

Variables:	run_info – The runtime information dictionary for the overall workflow mpi_comm – The MPI communicator to be used when running in parallel mpi_root – The MPI root rank when running in parallel

Additional parameters:

Parameters:	reduce_memory_usage – Boolean indicating whether we should reduce memory usage by pushing analysis data to file after an analysis has been completed. This reduces the amount of data we keep in memory but results in additional overhead for I/O and temporary disk storage.

Initalize the workflow driver

Parameters:	analysis_objects – A list of analysis objects to be executed

main()¶: Execute the analysis workflow

executor Package¶