driver Package

driver Package

Package with drivers for analyses and workflows. Drivers are intented to control the set up —i.e., creation and initialization—of workflows. The execution of workflows itself is typically performed by a workflow executor.

base Module

Module with base classes for workflow drivers.

Workflow drivers are responsible for the creation and initialization of workflows. The execution of workflows is then controlled by the workflow executor.

class omsi.workflow.driver.base.analysis_driver_base(analysis_class)

Bases: omsi.workflow.driver.base.driver_base

Base class defining the minimal interface for drivers of a single analysis based on the type/class of the analysis.

This is a class-based execution, i.e, the user defines only the type of analysis and inputs but does not actually create the analysis.

Derived classes must implement the main(...) function where the analysis is created and executed.

Variables:analysis_class – The analysis class for which we want to execute the analysis. The analysis class must derive from omsi.analysis.analysis_base. May be None in case that we use other means to set the analysis_class, e.g., via the command-line.

Initialize the analysis driver

Variables:analysis_class – The analysis class for which we want to execute the analysis. The analysis class must derive from omsi.analysis.analysis_base. May be None in case that we use other means to set the analysis_class, e.g., via the command-line.
main()

The main function for running the analysis.

class omsi.workflow.driver.base.driver_base

Bases: object

Primitve base class for driving the execution of an object

execute()

Same as main

main()

The main function for running the analysis.

class omsi.workflow.driver.base.workflow_driver_base(workflow_executor)

Bases: object

Base class defining the minimal interface for drivers of complex analysis workflows.

Workflows may be specified via scripts or given via a set of analysis objects.

Derived classes must implement the main(...) function where the analysis is created and executed.

Variables:workflow_executor – The executor of the workflow.

Initalize the workflow driver

Parameters:workflow_executor – The executor of the workflow we want to drive.
main()

The main function for running the analysis.

cl_analysis_driver Module

Module used to help with driving the execution of omsi-based analyses.

class omsi.workflow.driver.cl_analysis_driver.cl_analysis_driver(analysis_class, add_analysis_class_arg=False, add_output_arg=True, add_log_level_arg=True)

Bases: omsi.workflow.driver.base.analysis_driver_base

Command-line analysis driver.

Variables:
  • analysis_class_arg_name – The name for the argument where the positional argument for defining the analysis class should be stored.
  • output_save_arg_name – Name of the optional keyword argument for specifying the name and target for the analysis
  • analysis_class_arg_name – Name of the optional first positional argument to be used to define the analysis class to be used.
  • log_level_arg_name – Name of the keyword cl argument to define the level of logging ot be used.
  • analysis_class – The class (subclass of analysis_base) defining the analysis to be executed
  • analysis_object – Instance of the analysis object to be executed
  • add_analysis_class_arg – Boolean indicating whether an optional positional command line argument should be used to determine the analysis class (or whether the analysis class will be set explicitly)
  • add_output_arg – Boolean indicating whether an optional keyword argument should be added to define the output target for the analysis.
  • add_log_level_arg – Boolean indicating whether the –loglevel argument should be added to the command line
  • parser – The argparse.ArgumentParser instance used for defining command-line arguments
  • required_argument_group – argparse.ArgumentParser argument group used to define required command line arguments
  • custom_argument_groups – Dict of custom argparse.ArgumentParser argument groups specified by the analysis
  • output_target – Specification of the output target where the analysis result should be stored
  • analysis_arguments – Dictionary defining the input arguments to be used for the analysis
  • mpi_root – Integer indicating the root rank when running using MPI
  • mpi_comm – Integer indicating the MPI communicator to be used when running in parallel using MPI
Parameters:
  • analysis_class (omsi.analysis.base.analysis_base) – The analysis class for which we want to execute the analysis. The analysis class must derive from omsi.analysis.analysis_base. May be None in case that we use the command-line to define the analysis class via the optional positional argument for the command class (i.e., set add_analysis_class_arg to True).
  • add_analysis_class_arg – Boolean indicating whether we will use the positional command-line argument to determine the analysis class name
  • add_output_arg – Boolean indicating whether we should add the optional keyword argument for defining the output target for the analysis.
  • add_log_level_arg – Boolean indicating wither we should add the option keyword argument to specify the logging level via the command line.
Raises:

A ValueError is raised in the case of conflicting inputs, i.e., if i) analysis_class==None and add_analysis_class_arg=False, i.e., the analysis class is not determined or ii) analysis_class!=None and add_analysis_class_arg=False, i.e, the analysis class is determined via two separate mechanisms.
add_and_parse_analysis_arguments()

The function assumes that the command line parser has been setup using the initialize_argument_parser(..)

This function is responsible for adding all command line arguments that are specific to the analysis and to then parse those argument and save the relevant data in the self.analysis_arguments dictionary. Command-line arguments that are specific to the command line driver are removed, so that only arguments that can be consumed by the analysis are handed to the analysis.

Side effects: The function sets self.analysis_arguments

analysis_class_arg_name = '__analysis_class'

The name where the positional argument for defining the analysis class will be stored.

create_analysis_object()

Initialize the analysis object, i.e., set self.analysis_object

get_analysis_class_from_cl()

Internal helper function used to get the analysis class object based on the analysis_class_arg_name positional argument from the command line.

Side effects: The function sets ``self.analysis_class`

Raises:ImportError in case that the analysis module cannot be loaded
Raises:AttributeError in case that the analysis class cannot be extracted from the module
initialize_argument_parser()

Internal helper function used to initialize the argument parser. NOTE: self.analysis_class must be set before calling this function.

Side effects: The function sets self.parser and self.required_argument_group

log_level_arg_name = 'loglevel'

Name of the keyword argument used to specify the level of logging to be used

main()

Default main function for running an analysis from the command line. The default implementation exposes all specified analysis parameters as command line options to the user. The default implementation also provides means to print a help text for the function.

Raises:ValueError is raised in case that the analysis class is unknown
output_save_arg_name = 'save'

Name of the key-word argument used to define

parse_cl_arguments()

The function assumes that the command line parser has been setup using the initialize_argument_parser(..)

This function parses all arguments that are specific to the command-line parser itself. Analysis arguments are added and parsed later by the add_and_parse_analysis_arguments(...) function. The reason for this is two-fold: i) to separate the parsing of analysis arguments and arguments of the command-line driver and ii) if the same HDF5 file is used as input and output target, then we need to open it first here in append mode before it gets opened in read mode later by the arguments.

Side effects: The function sets self.output_target and self.profile_analysis

print_settings()

Print the analysis settings.

remove_output_target()

This function is used to delete any output target files created by the command line driver. This is done in case that an error occurred and we do not want to leave garbage files left over.

Side effects The function modifies self.output_target

Returns:Boolean indicating whether we succesfully cleaned up the output
reset_analysis_object()

Clear the analysis object and recreate it, i.e., delete self.analysis_object and set it again.

cl_workflow_driver Module

Module used to help with driving the execution of analysis workflows

class omsi.workflow.driver.cl_workflow_driver.cl_workflow_driver(workflow_executor=None, add_script_arg=False, add_output_arg=True, add_log_level_arg=True, add_profile_arg=False, add_mem_profile_arg=False)

Bases: omsi.workflow.driver.base.workflow_driver_base

Command-line workflow driver.

Variables:
  • script_arg_name – Name of the optional keyword cl argument for defining workflow scripts to be executed
  • output_save_arg_name – Name of the optional keyword argument for specifying the name and target for the workflow. This may be a folder or an HDF5 file ending with .h5
  • profile_arg_name – Name of the keyword argument used to enable profiling of the analysis
  • profile_mem_arg_name – Name of the keyword argument used to enable profiling of memory usage of an analysis
  • log_level_arg_name – Name of the keyword cl argument to define the level of logging ot be used.
  • workflow_executor – The workflow executor object used to execute the analysis workflow. The workflow executor must derive from omsi.workflow.executor.base.workflow_executor_base. May be None in case that we use the command-line to define workflow executor or if the default executor should be used. The default executur class is defined by omsi.workflow.executor.base.workflow_executor_base.
  • script_files – List of strings with the paths to files with workflow scripts to be executed.
  • add_script_arg – Boolean indicating whether the –script keyword argument should be added to the command-line, to define the workflow scripts via the CL (or whether the scripts will be set explicitly)
  • add_output_arg – Boolean indicating whether an optional keyword argument should be added to define the output target for the analysis.
  • add_profile_arg – Add the optional –profile keyword argument for profiling the analysis
  • add_mem_profile_arg – Boolean indicating whether we should add the optional keyword argument for enabling memory profiling of the analysis.
  • add_log_level_arg – Boolean indicating wither we should add the option keyword argument to specify the logging level via the command line.
  • parser – The argparse.ArgumentParser instance used for defining command-line arguments
  • required_argument_group – argparse.ArgumentParser argument group used to define required command line arguments
  • optional_argument_group – argparse.ArgumentParser argument group used to define optional command line arguments
  • custom_argument_groups – Dict of custom argparse.ArgumentParser argument groups specified by the analysis
  • identifier_argname_seperator – String used to seperate the analysis identifier and argument name when creating custom command-line options for the individual analyses of the workflow
  • output_target – Specification of the output target where the analysis result should be stored
  • profile_analyses – Boolean indicating whether we should profile the analysis for time and usage
  • profile_analyses_mem – Boolean indicating whether we should profile the memory usage of the individual analyses
  • analysis_arguments – Dictionary defining the custom input arguments to be used for the analysis
  • workflow_executor_arguments – Dictionary defining the custom input arguments routed to the workflow executor
  • __output_target_self – Private member variable used to store the output files created by this object.
  • user_log_level – The custom logging level specified by the user (or None)
  • mpi_root – The root rank used when running in parallel
  • mpi_comm – The mpit communicator to be used when running in parallel
Parameters:
  • workflow_executor (omsi.analysis.base.analysis_base) – The workflow executor object used to execute the analysis workflow. The workflow executor must derive from omsi.workflow.executor.base.workflow_executor_base. May be None in case that we use the command-line to define workflow executor or if the default executor should be used. The default executur class is defined by omsi.workflow.executor.base.workflow_executor_base.
  • add_script_arg – Boolean indicating whether the –script keyword argument should be added to the command-line, to define the workflow scripts via the CL (or whether the scripts will be set explicitly)
  • add_output_arg – Boolean indicating whether we should add the optional keyword argument for defining the output target for the analysis.
  • add_log_level_arg – Boolean indicating wither we should add the option keyword argument to specify the

logging level via the command line. :param add_profile_arg: Boolean indicating whether we should add the optional keyword

argument for enabling profiling of the analysis.
Parameters:add_mem_profile_arg – Boolean indicating whether we should add the optional keyword argument for enabling memory profiling of the analysis.
Raises:A ValueError is raised in the case of conflicting inputs, i.e., if i) workflow_executor==None and add_script_arg=False, i.e., the analysis class is not determined or ii) workflow_executor!=None and add_script_arg=False, i.e, the analysis class is determined via two separate mechanisms.
add_and_parse_workflow_arguments()

The function assumes that the command line parser has been setup using the initialize_argument_parser(..)

This function is responsible for adding all command line arguments that are specific to the workflow and to then parse those arguments and save the relevant data in the self.analysis_arguments dictionary. Command-line arguments that are specific to the command line driver are removed, so that only arguments that can be consumed by the analysis are handed to the analysis.

Side effects: The function sets self.analysis_arguments and updates the analysis parameters of the
analyses stored in self.workflow_executor.analysis_tasks
create_workflow_executor_object()

Initialize the workflow executor object, i.e., set self.workflow_executor

Side effects This function potentially modifies self.workflow_executor

initialize_argument_parser()

Internal helper function used to initialize the argument parser.

Side effects: The function sets:

  • self.parser
  • self.required_argument_group
  • self.opitonal_argument_group
log_level_arg_name = 'loglevel'

Name of the keyword argument used to specify the level of logging to be used

main()

Default main function for running an analysis from the command line. The default implementation exposes all specified analysis parameters as command line options to the user. The default implementation also provides means to print a help text for the function.

Raises:ValueError is raised in case that the analysis class is unknown
output_save_arg_name = 'save'

Name of the key-word argument used to define

parse_cl_arguments()

The function assumes that the command line parser has been setup using the initialize_argument_parser(..)

This function parses all arguments that are specific to the command-line parser itself. Analysis workflow arguments are added and parsed later by the add_and_parse_workflow_arguments(...) function. The reason for this is two-fold: i) to separate the parsing of analysis arguments and arguments of the command-line driver and ii) if the same HDF5 file is used as input and output target, then we need to open it first here in append mode before it gets opened in read mode later by the arguments.

Side effects: The function sets:

  • self.output_target
  • self.profile_analyses
print_memory_profiles()

Print the memory profiles if available

print_settings()

Print the analysis settings.

print_time_and_usage_profiles()

Print the profiling data for time and usage if available

profile_arg_name = 'profile'

Name of the keyword argument used to enable profiling of the analysis

profile_mem_arg_name = 'memprofile'

Name of the keyword argument used to enable profiling of memory usage of an analysis

remove_output_target()

This function is used to delete any output target files created by the command line driver. This is done in case that an error occurred and we do not want to leave garbage files left over.

Side effects The function modifies self.output_target

Returns:Boolean indicating whether we succesfully cleaned up the output
reset_workflow_executor_object()

Remove and recreate the workflow executor object

script_arg_name = 'script'

The name where the positional argument for defining the analysis class will be stored.