seaquest package

Subpackages

• seaquest.utils package
  • Submodules
  • seaquest.utils.loggus module
    • _init_logging_config()
    • init_logger()
  • seaquest.utils.pod module
    • _delete_pod()
    • _get_list_of_pods()
    • _is_running()
    • _launch_pod()
    • _wait_for_running_state()
    • make_pod_name_unique()
  • seaquest.utils.validate module
    • _parse_args()
    • _parse_config_file()
    • _parse_monitor_args()
    • _parse_runner_args()
    • _update()
    • _validate()
    • parse_and_validate_args()
  • seaquest.utils.vschema module
  • Module contents

Submodules

seaquest.experiment module

seaquest.launcher module

seaquest.launcher._launch_job(api_instance: BatchV1Api, namespace: str, job_name: str, job_config: dict, pvc: str, arguments: dict) → None

launches a kubernetes job based on the provided configuration

Parameters

api_instance: kubernetes client

Kubernetes client

namespace: str

Kubernetes namespace

job_name: str

Name of the job

job_config: dict

Dictionary of job configuration parameters

pvc: str

Name of the PVC

arguments: dict

List of arguments to pass to the runner module that runs inside the job

Returns

None

seaquest.launcher._prepare_afinity(job_config: dict) → V1Affinity

create the affinity object for the k8s job

Parameters

job_config: dict

Dictionary of parsed job configuration arguments

Returns

affinity: client.V1Affinity

kubernetes affinity object

seaquest.launcher._prepare_containers(config: dict, job_name: str, pvc: str, arguments: dict) → list[V1Container]

create the container objects for the k8s job

Parameters

config: dict

Dictionary of parsed job configuration arguments

job_name: str

The name of the job

arguments: dict

A list of arguments to pass to the runner module that will run inside the job

Returns

containers: list[client.V1Container]

list of kubernetes container objects

seaquest.launcher._prepare_job_spec(job_config: dict, job_name: str, pvc: str, arguments: dict) → V1JobSpec

create the specification for the k8s job

Parameters

job_config: dict

Dictionary of job configuration parameters

job_name: str

Name of the job

pvc: str

Name of the PVC

arguments: dict

List of arguments to pass to the runner module that runs inside the job

Returns

status: client.V1JobStatus

Kubernetes job spec object

seaquest.launcher._prepare_volumes(pvc: str) → list[V1Volume]

create the volume objects for the k8s job

Parameters

pvc: str

Name of the PVC

Returns

volumes: list[client.V1Volume]

List of kubernetes volume objects to mount to the job

seaquest.launcher.create_jobs(api_instance: BatchV1Api, num_jobs: int, namespace: str, job_config: dict, prefix: str, suffix: str, model_name: str, model_fun: str, pvc: str, model_dir: str, data_file: str) → list

Launches ‘num_jobs’ kubernetes jobs based on the specified configuration using a unique naming assignment formed of the model name and the data file

Parameters

api_instance: kubernetes client

Kubernetes client

num_jobs: int

Number of jobs to launch

namespace: str

Kubernetes namespace

job_config: dict

Dictionary of job configuration parameters

prefix: str

Prefix for the job name

suffix: str

Suffix for the job name

model_name: str

Name of the model to experiment with

model_fun: str

Name of the function called (either ‘train’ or ‘infer’)

pvc: str

Name of the PVC

model_dir: str

Name of the model directory

data_file: str

Name of the data file

Returns

all_created_jobs: list

A list of names of the succesfully launched jobs

seaquest.model_class module

class seaquest.model_class.NautPipelineModel(output_dir: Path, data_file: Path)

Bases: ABC

Base class to be inherited by all models to ensure minimum required constructor arguments and methods

__abstractmethods__ = frozenset({'infer', 'train'})

__dict__ = mappingproxy({'__module__': 'seaquest.model_class', '__firstlineno__': 24, '__doc__': 'Base class to be inherited by all models to ensure minimum required constructor arguments and methods', '__init__': <function NautPipelineModel.__init__>, 'train': <function NautPipelineModel.train>, 'infer': <function NautPipelineModel.infer>, '__static_attributes__': ('data_file', 'output_dir'), '__dict__': <attribute '__dict__' of 'NautPipelineModel' objects>, '__weakref__': <attribute '__weakref__' of 'NautPipelineModel' objects>, '__abstractmethods__': frozenset({'infer', 'train'}), '_abc_impl': <_abc._abc_data object>, '__annotations__': {}})

__firstlineno__ = 24

__init__(output_dir: Path, data_file: Path)

__module__ = 'seaquest.model_class'

__static_attributes__ = ('data_file', 'output_dir')

__weakref__

list of weak references to the object

_abc_impl = <_abc._abc_data object>

abstractmethod infer()

abstractmethod train()

class seaquest.model_class.OutputContext(output_dir: Path)

Bases: object

Class that handles the context switch for saving output files on the PVC

__dict__ = mappingproxy({'__module__': 'seaquest.model_class', '__firstlineno__': 6, '__doc__': 'Class that handles the context switch for saving output files on the PVC', '__init__': <function OutputContext.__init__>, '__enter__': <function OutputContext.__enter__>, '__exit__': <function OutputContext.__exit__>, '__static_attributes__': ('curr_wd', 'output_dir'), '__dict__': <attribute '__dict__' of 'OutputContext' objects>, '__weakref__': <attribute '__weakref__' of 'OutputContext' objects>, '__annotations__': {}})

__enter__()

__exit__(exc_type, exc_value, traceback)

__firstlineno__ = 6

__init__(output_dir: Path)

__module__ = 'seaquest.model_class'

__static_attributes__ = ('curr_wd', 'output_dir')

__weakref__

list of weak references to the object

seaquest.monitor module

seaquest.runner module

seaquest.runner._load_model_class(namespace, model_name)

Loads the model class that instantiates a model object

Parameters

namespace: str

The namespace that holds the model

model_name: str

The name of the model

Returns

model: class

The model class that instantiates a model object

seaquest.runner._load_model_dir(model_dir)

Load model directory

Parameters

model_dir: str

The name of the model directory

Returns

namespace: str

The name of the namespace where the model can be loaded from

seaquest.runner._load_model_function(model, fun)

seaquest.runner.main(args: dict) → None

Dynamically imports and runs a specified function from a model class.

Parameters:

model_name: str

Name of the model to import.

model_dir: str

Directory where the model is located.

data_dir: str

Directory containing the data for training or inference.

output_dir: str

Directory where the output should be stored.

fun:

str Function to execute from the model class (‘train’ or ‘infer’).

Returns:

None

Raises:

ValueError: If the model or function cannot be found.

seaquest.uploader module

seaquest.uploader._check_pvc_exists(api_instance: CoreV1Api, namespace: str, pvc_name: str)

Create a persistent volume claim (pvc) if it does not exist

Parameters

api_instance: kubernetes client

Kubernetes client

namespace: str

Kubernetes Namespace

pvc_name: str

Name used in the creation of the pvc

Returns

Bool

Wether the PVC exists or not

seaquest.uploader._copy_files_to_pod(api_instance: CoreV1Api, namespace: str, pod_name: str, source_path: Path, dest_path: Path) → None

Copy files to the temporary pod

Parameters

api_instance: kubernetes client

Kubernetes client

namespace: str

Namespace to use for creating pvc

pod_name: str

Name of the pod to copy files to

source_path: pathlib.Path

Path to the files to copy

dest_path: pathlib.Path

Mount path of the pvc in the pod

Returns

None

None

seaquest.uploader._create_pvc(api_instance: CoreV1Api, namespace: str, pvc_name: str) → None

Create a persistent volume claim (pvc) if it does not exist

Parameters

api_instance: kubernetes client

Kubernetes client

namespace: str

Kubernetes Namespace

pvc_name: str

Name used in the creation of the pvc

Returns

None

None

seaquest.uploader._delete_pvc(api_instance: CoreV1Api, namespace: str, pvc_name: str) → None

Delete a persistent volume claim (pvc) if it exists

Parameters

api_instance: kubernetes client

Kubernetes client

namespace: str

Kubernetes Namespace

pvc_name: str

Name used of the pvc to delete

Returns

None

None

seaquest.uploader._update_file_dest_name(api_instance: CoreV1Api, namespace: str, pod_name: str, old_name: Path, new_name: Path) → None

Copy files to the temporary pod

Parameters

api_instance: kubernetes client

Kubernetes client

namespace: str

Namespace to use for creating pvc

pod_name: str

Name of the pod to copy files to

old_name: str

Name of the directory where the files have been copied

new_name: str

Name of the new directory

Returns

None

None

seaquest.uploader.upload_files_to_pvc(namespace: str, prefix: str, pvc: str, files_path: list[Path, Path]) → str

Upload model and data files to the pvc Parameters ———- kube: kubernetes client

Kubernetes client

pvc: str

Name of pvc to use for data and model storage

namespace: str

Namespace to use for creating pvc

Returns

None

None

Module contents