openfl.pipelines

class openfl.pipelines.KCPipeline(p_sparsity=0.01, n_clusters=6, **kwargs)

A pipeline class to compress data lossly using k-means and GZIP methods.

Class Attributes:

• p (float) – The amount of sparsity for compression.

• n_cluster (int) – The number of K-mean clusters.

class openfl.pipelines.NoCompressionPipeline(**kwargs)

The data pipeline without any compression.

class openfl.pipelines.RandomShiftPipeline(**kwargs)

Random Shift Pipeline.

class openfl.pipelines.SKCPipeline(p_sparsity=0.1, n_clusters=6, **kwargs)

A pipeline class to compress data lossly using sparsity and k-means methods.

Class Attributes:

• p (float) – The sparsity factor.

• n_cluster (int) – The number of K-mean clusters.

class openfl.pipelines.STCPipeline(p_sparsity=0.1, n_clusters=6, **kwargs)

A pipeline class to compress data lossly using sparsity and ternarization methods.

Class Attributes:

p (float) – The sparsity factor.

class openfl.pipelines.TensorCodec(compression_pipeline)

TensorCodec is responsible for the following.

1. Tracking the compression/decompression related dependencies of a given tensor. 2. Acting as a TensorKey aware wrapper for the compression_pipeline functionality.

Class Attributes:

• compression_pipeline – The pipeline used for compression.

• lossless_pipeline – The pipeline used for lossless compression.

static apply_delta(tensor_key, delta, base_model_nparray, creates_model=False)

Add delta to the nparray.

Parameters:

• tensor_key – This is the tensor_key associated with the delta. Should have a tag of ‘trained’ or ‘aggregated’.

• delta – Weight delta between the new model and old model.

• base_model_nparray – The nparray that corresponds to the prior weights.

• creates_model – If flag is set, the tensorkey returned will correspond to the aggregator model.

Returns:

• new_model_tensor_key – Latest model layer tensorkey.

• new_model_nparray – Latest layer weights.

compress(tensor_key, data, require_lossless=False, **kwargs)

Function-wrapper around the tensor_pipeline.forward function.

It also keeps track of the tensorkeys associated with the compressed nparray.

Parameters:

• tensor_key – TensorKey is provided to verify it should be compressed, and new TensorKeys returned will be derivatives of the existing tensor_name.

• data – (uncompressed) numpy array associated with the tensor_key.

• require_lossless – boolean. Does tensor require compression.

Returns:

• compressed_tensor_key – Tensorkey corresponding to the decompressed tensor.

• compressed_nparray – The compressed tensor.

• metadata – metadata associated with compressed tensor.

decompress(tensor_key, data, transformer_metadata, require_lossless=False, **kwargs)

Function-wrapper around the tensor_pipeline.backward function.

It also keeps track of the tensorkeys associated with the decompressed nparray.

Parameters:

• tensor_key – TensorKey is provided to verify it should be decompressed, and new TensorKeys returned will be derivatives of the existing tensor_name.

• data – (compressed) numpy array associated with the tensor_key.

• transformer_metadata – metadata associated with the compressed tensor.

• require_lossless – boolean, does data require lossless decompression.

Returns:

• decompressed_tensor_key – Tensorkey corresponding to the decompressed tensor.

• decompressed_nparray – The decompressed tensor.

find_dependencies(tensor_key, send_model_deltas)

Resolve the tensors required to do the specified operation.

Parameters:

• tensor_key – A tuple containing the tensor name, origin, round number, report, and tags.

• send_model_deltas – A boolean flag indicating whether to send model deltas.

Returns:

tensor_key_dependencies – A list of tensor keys that are dependencies of the given tensor key.

static generate_delta(tensor_key, nparray, base_model_nparray)

Create delta from the updated layer and base layer.

Parameters:

• tensor_key – This is the tensor_key associated with the nparray. Should have a tag of ‘trained’ or ‘aggregated’

• nparray – The nparray that corresponds to the tensorkey.

• base_model_nparray – The base model tensor that will be subtracted from the new weights.

Returns:

• delta_tensor_key – Tensorkey that corresponds to the delta weight array.

• delta – Difference between the provided tensors.

set_lossless_pipeline(lossless_pipeline)

Set lossless pipeline.

Parameters:

lossless_pipeline – The pipeline to be set as the lossless pipeline. It should be a pipeline that is not lossy.

Raises:

AssertionError – If the provided pipeline is not lossless.

kc_pipeline	KCPipeline module.
no_compression_pipeline	NoCompressionPipeline module.
pipeline	Pipeline module.
random_shift_pipeline	RandomShiftPipeline module.
skc_pipeline	SKCPipeline module.
stc_pipeline	STCPipelinemodule.
tensor_codec	TensorCodec module.