Apply Compression Settings

The Open Federated Learning (OpenFL) framework supports lossless and lossy compression pipelines. Federated learning enables a large number of participants to work together on the same model. Without a compression pipeline, this scalability results in increased communication cost. Furthermore, large models exacerbate this problem.

Note

In general, the weights of a model are typically not robust to information loss, so no compression is applied by default to the model weights sent bidirectionally; however, the deltas between the model weights for each round are inherently more sparse and better suited for lossy compression.

The following are the compression pipelines supported in OpenFL:

NoCompressionPipeline

The default option applied to model weights

RandomShiftPipeline

A lossless pipeline that randomly shifts the weights during transport

STCPipeline

A lossy pipeline consisting of three transformations:

  • Sparsity Transform (p_sparsity=0.1), which by default retains only the (p*100)% absolute values of greatest magnitude.

  • Ternary Transform, which discretizes the sparse array into three buckets

  • GZIP Transform

SKCPipeline

A lossy pipeline consisting of three transformations:

  • Sparsity Transform (p=0.1), which by default retains only the(p*100)% absolute values of greatest magnitude.

  • KMeans Transform (k=6), which applies the KMeans algorithm to the sparse array with k centroids

  • GZIP Transform

KCPipeline

A lossy pipeline consisting of two transformations:

  • KMeans Transform (k=6), which applies the KMeans algorithm to the original weight array with k centroids

  • GZIP Transform

Demonstration of a Compression Pipeline

The example template, keras_cnn_with_compression, uses the KCPipeline with six centroids for KMeans. To gain a better understanding of how experiments perform with greater or fewer centroids, you can modify the n_clusters parameter in the template plan.yaml:

compression_pipeline :
  defaults : plan/defaults/compression_pipeline.yaml
  template : openfl.pipelines.KCPipeline
  settings :
    n_clusters : 6