Pipeline#

class pai.pipeline.RegisteredComponent(pipeline_id, manifest=None, workspace_id=None)#

Bases: ComponentBase

RegisteredComponent represent the pipeline schema from pipeline/component.

RegisteredComponent object include the definition of “Workflow” use in PAI pipeline service.

It could be fetched from remote pipeline service or construct from local Pipeline/Component. Saved pipeline component has unique pipeline_id which is generated by pipeline service.

Template constructor.

Parameters

  • manifest – “Workflow” definition of the pipeline.

  • pipeline_id – Unique ID for pipeline in PAI service.

  • workspace_id – ID of the workspace which the pipeline belongs to.

property identifier#
property provider#
property version#
property pipeline_id#

Unique ID of the pipeline in PAI pipeline service.

Returns

Unique pipeline ID of the component instance.

Return type

str

property manifest#

Pipeline manifest schema.

Returns

Pipeline manifest schema in dict.

Return type

dict

property raw_manifest#

Pipeline manifest in YAML format

Returns

Pipeline manifest.

Return type

str

classmethod get_by_identifier(identifier, provider=None, version='v1')#

Get SavedOperator with identifier-provider-version tuple.

Parameters

  • identifier (str) – Pipeline identifier.

  • provider (str) – Provider of the Pipeline, account uid of the current session will be used as default.

  • version (str) – Version of the pipeline.

Returns

SavedOperator instance

Return type

pai.pipeline.SavedOperator

classmethod list(identifier=None, provider=None, version=None, session=None, page_size=10, page_number=1)#

List the SavedOperator in PAI

Search the pipeline component available in remote PAI service. The method return a generator used to traverse the SavedOperator set match the query condition.

Parameters

  • identifier (str) – Pipeline identifier filter.

  • provider (str) – Pipeline provider filter.

  • version (str) – Pipeline version.

  • workspace_id (str) – Workspace id of the pipeline.

  • session – PAI session.

  • page_number (int) –

  • page_size (int) –

Yields

pai.component.SavedOperator – SavedOperator match the query.

update(component)#

Update current registered component/pipeline using the manifest of given component/pipeline.

Parameters

  • component (could be an unregistered) – New pipeline/component spec,

  • component –

  • str. (dict or yaml in) –

delete()#

Delete this registered component/pipeline.

classmethod deserialize(obj_dict)#
classmethod get(pipeline_id, session=None)#

Get SavedOperator with pipeline_id.

Parameters

pipeline_id (str) – Unique pipeline id.

Returns

SavedOperator instance with the

specific pipeline_id

Return type

pai.pipeline.SavedOperator

save(identifier=None, version=None)#
io_spec_to_dict()#
class pai.pipeline.core.Pipeline(steps, inputs=None, outputs=None, **kwargs)#

Bases: UnRegisteredComponent

Represents pipeline instance in PAI Machine Learning pipeliner service.

Pipeline can be constructed from multiple pipeline steps, or single container implementation. It is shareable and reusable workflow, present as YAML format in backend pipeline service.

Pipeline initializer.

property steps#
property ref_name#
validate_step_name(name)#
dot()#
to_dict(identifier=None, version=None)#
to_manifest(identifier, version)#
class pai.pipeline.step.PipelineStep(inputs=None, name=None, depends=None, component=None)#

Bases: object

Represents an execution step in PAI pipeline.

Pipeline steps can be configured together to construct a Pipeline, which is present as workflow in PAI ML pipeline service.

Construct a step which represent component execution in pipeline.

Parameters

  • inputs (dict) – Inputs for the step in dict: key is the component input name, value could be the output artifact/parameter from other step, input of the pipeline, or actual value for the step.

  • name (str) – Name of the step in pipeline, must be unique in the pipeline.

  • depends (list) – A list of PipelineStep which step depends.

  • component (OperatorBase) – The component used by the constructed step.

property is_component_registered#
property component#
gen_name_prefix()#
classmethod from_registered_component(identifier, provider=None, version='v1', inputs=None, name=None, depends=None)#

Build the PipelineStep from the given registered component reference: identifier, version, provider.

Parameters

  • identifier – Identifier of the registered component.

  • provider – Provider of the registered component.

  • version – Version of the registered component.

  • inputs – Inputs for the building step.

  • name – Name for the building step.

  • depends – Depended steps of the building step.

Returns

The built step instantiates from the given registered component and inputs.

Return type

PipelineStep

property repeated_io_config#
set_artifact_count(artifact_name, count)#
property depends#
property name#
classmethod get_component(identifier, provider, version)#
after(*steps)#
property ref_name#
to_dict()#