Quickstart#
In this guide, we will train a simple Convolutional Neural Network (CNN) on MNIST handwritten digits dataset. We will simulate a Federated Learning experiment between two collaborators, orchestrated by an aggregator, via few CLI commands.
OpenFL must be installed for this tutorial. Refer to the installation guide.
(Optional) For a simplified setup without TLS, follow these steps as applicable throughout the process:
Set
use_tls: falsein the FL plan before initialization.Skip all certificate-related commands (incl. CA setup).
Manually populate the collaborator names in
cols.yaml. For example:
collaborators:
- bob
- charlie
Create a workspace#
We will reuse an existing workspace. OpenFL provides many reference model templates. Check them out under workspaces.
!fx workspace create --prefix ./mnist_example --template keras/mnist
%cd ./mnist_example
!pip install -r requirements.txt > /dev/null
Show code cell output
Creating Workspace Directories
Creating Workspace Templates from /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl-workspace/keras/mnist in /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example
[06/23/25 17:30:13] INFO Loading DEFAULTS for section aggregator from file ]8;id=40988;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=158285;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl-workspace/workspace/plan/defaults/aggrega
tor.yaml.
INFO Loading DEFAULTS for section assigner from file ]8;id=883913;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=42833;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl-workspace/workspace/plan/defaults/assigne
r.yaml.
INFO Loading DEFAULTS for section collaborator from file ]8;id=150158;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=123985;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl-workspace/workspace/plan/defaults/collabo
rator.yaml.
INFO Loading DEFAULTS for section compression_pipeline from file ]8;id=703998;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=194841;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl-workspace/workspace/plan/defaults/compres
sion_pipeline.yaml.
INFO Loading DEFAULTS for section data_loader from file ]8;id=401008;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=28282;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl-workspace/workspace/plan/defaults/data_lo
ader.yaml.
INFO Loading DEFAULTS for section network from file ]8;id=55691;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=621779;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl-workspace/workspace/plan/defaults/network
.yaml.
INFO Loading DEFAULTS for section task_runner from file ]8;id=619375;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=243322;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl-workspace/workspace/plan/defaults/task_ru
nner.yaml.
INFO Loading DEFAULTS for section tasks from file ]8;id=389270;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=867543;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl-workspace/workspace/plan/defaults/tasks_k
eras.yaml.
INFO FL-Plan hash is 71f93e3c5870ca465935df0180c419181cd57832297069eb737d448dfa00cb6f8474dfab1489dd6621521dbf084e2565 ]8;id=985179;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=318578;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#305\305]8;;\
INFO FL-Plan hash is 0991b8712692d2d83dcb0aa5f2717ff09c72440545c2ae613b96ca2f58b2b26390a03f3ec0a2a8b00be9e437b482b35e ]8;id=880166;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=486819;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#305\305]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=30508;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=932765;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#152\152]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl-workspace/keras/mnist/plan/plan.yaml.
INFO aggregator: ]8;id=562494;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=667095;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#157\157]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: build-28612574-project-638300-openfl
agg_port: 55336
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
transport_protocol: grpc
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
New workspace directory structure:
mnist_example
├── src
│ ├── taskrunner.py
│ ├── __init__.py
│ ├── mnist_utils.py
│ └── dataloader.py
├── cert
├── plan
│ ├── cols.yaml
│ ├── data.yaml
│ ├── plan.yaml
│ └── defaults
├── .workspace
├── data
├── local_state
├── logs
├── requirements.txt
└── save
7 directories, 10 files
✔️ OK
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/IPython/core/magics/osm.py:417: UserWarning: This is now an optional IPython functionality, setting dhist requires you to install the `pickleshare` library.
self.shell.db['dhist'] = compress_dhist(dhist)[-100:]
Initialize a Plan#
This step builds an entire FL experiment plan, along with the initial set of parameters that will be used in the experiment.
We supply localhost as the aggregator address, for simulation purposes. The command below supports models initializations (e.g. when using pre-trained models).
For a complete list of supported options, refer to fx plan initialize.
!fx plan initialize -a localhost
Show code cell output
2025-06-23 17:30:47.620068: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:30:47.620927: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:47.624024: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:47.632058: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699847.644942 630 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699847.648792 630 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:30:47.663457: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
[06/23/25 17:30:49] INFO FL-Plan hash is 9e1ba56efcf0b721c28c1da90aa42f16010b6da507bfc8bade79aca838e705c1180668d1e7fcc30aff039f4664763718 ]8;id=27011;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=527459;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#305\305]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=924837;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=350106;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#152\152]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=438797;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=467843;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#157\157]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: build-28612574-project-638300-openfl
agg_port: 55336
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
transport_protocol: grpc
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
INFO Initialized minimal dataloader for model creation ]8;id=232977;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/utilities/dataloading.py\dataloading.py]8;;\:]8;id=488280;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/utilities/dataloading.py#54\54]8;;\
INFO Building `src.taskrunner.KerasCNN` Module. ]8;id=827445;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=837141;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#243\243]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py:113: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(activity_regularizer=activity_regularizer, **kwargs)
2025-06-23 17:30:49.536905: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
INFO Model: "sequential" ]8;id=788503;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/keras/src/utils/summary_utils.py\summary_utils.py]8;;\:]8;id=877142;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/keras/src/utils/summary_utils.py#389\389]8;;\
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type) ┃ Output Shape ┃ Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d (Conv2D) │ (None, 13, 13, 16) │ 272 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D) │ (None, 5, 5, 32) │ 8,224 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten) │ (None, 800) │ 0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense) │ (None, 100) │ 80,100 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense) │ (None, 10) │ 1,010 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 89,606 (350.02 KB)
Trainable params: 89,606 (350.02 KB)
Non-trainable params: 0 (0.00 B)
INFO Building `openfl.pipelines.NoCompressionPipeline` Module. ]8;id=722557;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=341948;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#243\243]8;;\
WARNING Following parameters omitted from global initial model, local initialization will determine values: [] ]8;id=411189;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/plan.py\plan.py]8;;\:]8;id=802572;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/plan.py#132\132]8;;\
INFO Saving model state to save/init.pbuf ]8;id=912408;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/plan.py\plan.py]8;;\:]8;id=470397;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/plan.py#140\140]8;;\
WARNING Patching Aggregator Addr in Plan 🠆 localhost ]8;id=100149;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/plan.py\plan.py]8;;\:]8;id=711936;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/plan.py#157\157]8;;\
INFO FL-Plan hash is e5455a1aaae22029e1fe92ab5bb878702761ff3c12e2ae7ddc8b037eafd2cc6b18ae2cff34bfc1833276c78073bb5d2b ]8;id=767466;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=898364;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#305\305]8;;\
INFO ['plan_e5455a1a'] ]8;id=239342;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/plan.py\plan.py]8;;\:]8;id=899883;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/plan.py#171\171]8;;\
✔️ OK
Aggregator Setup#
The fx aggregator subcommand options to manage the aggregator-side certificate signing, generation and verification.
Here, we generate the aggregator’s certificate and private key. The aggregator’s certificate is then signed by the CA generated in the previous step.
For all commands that follow, --silent is used to skip manual confirmation. In practice, users are recommended to verify the details before proceeding.
!fx aggregator generate-cert-request --fqdn localhost
!fx aggregator certify --fqdn localhost --silent
Show code cell output
2025-06-23 17:30:54.847728: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:30:54.848480: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:54.851594: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:54.859657: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699854.872709 647 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699854.876530 647 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:30:54.891161: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Creating AGGREGATOR certificate key pair with following settings: CN=localhost, SAN=DNS:localhost
Writing AGGREGATOR certificate key pair to: /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/cert/server
The CSR Hash 5a57cafc03d06d7d49d5a59495fc770fe2c031a9e07a6092efbfddf4e0b51b360ea187363c90ad7fb356b21a5b1788fa
✔️ OK
2025-06-23 17:30:58.279082: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:30:58.279799: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:58.282916: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:58.291058: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699858.304333 653 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699858.308190 653 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:30:58.322985: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
The CSR Hash for file server/agg_localhost.csr = 5a57cafc03d06d7d49d5a59495fc770fe2c031a9e07a6092efbfddf4e0b51b360ea187363c90ad7fb356b21a5b1788fa
Warning: manual check of certificate hashes is bypassed in silent mode.
Signing AGGREGATOR certificate
✔️ OK
Collaborator Setup#
Following aggregator, fx collaborator subcommand manages the collaborator-side certificate signing, generation and verification. In addition, because collaborators bring their own local datasets, we populate dataset information under plan/data.yaml.
Setup first collaborator bob#
This step registers --data_path against collaborator’s name -n under plan/data.yaml.
!fx collaborator create --data_path 0 -n bob --silent
Show code cell output
2025-06-23 17:31:01.753486: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:01.754215: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:01.757284: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:01.765247: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699861.778090 659 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699861.781865 659 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:01.796475: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
✔️ OK
The next step generates bobs certificate and key pair.
!fx collaborator generate-cert-request -n bob --silent
Show code cell output
2025-06-23 17:31:05.116411: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:05.117150: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:05.120291: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:05.128333: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699865.141371 665 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699865.145168 665 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:05.159868: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Creating COLLABORATOR certificate key pair with following settings: CN=bob, SAN=DNS:bob
Moving COLLABORATOR certificate to: /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/cert/col_bob
The CSR Hash b4ceb0fc92d815cbb53217a442ac59b7df4edea2a7158252bfcf7090f4fd72a3064dfa8af7aa76a3f32200f4b1c41aea
Archive col_bob_to_agg_cert_request.zip with certificate signing request created
This file should be sent to the certificate authority (typically hosted by the aggregator) for signing
✔️ OK
Certify bob with the CA.
Note: Here, we assume CA to be the aggregator. Hence the command below runs on the
aggregatorside. In reality,bob’s certificate signing request (CSR) package generated in the last step, is sent to theaggregator, which then signs the certificate with the CA. The signed certificate then has to be sent back tobob.
!fx collaborator certify --request-pkg col_bob_to_agg_cert_request.zip --silent
Show code cell output
2025-06-23 17:31:08.538845: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:08.539561: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:08.542648: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:08.550637: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699868.563405 671 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699868.567124 671 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:08.581599: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
The CSR Hash for file col_bob.csr is b4ceb0fc92d815cbb53217a442ac59b7df4edea2a7158252bfcf7090f4fd72a3064dfa8af7aa76a3f32200f4b1c41aea
Signing COLLABORATOR certificate, Warning: manual check of certificate hashes is bypassed in silent mode.
Registering bob in /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/plan/cols.yaml
✔️ OK
Setup second collaborator charlie#
As before, we create a collaborator by registering --data_path against collaborator’s name -n under plan/data.yaml.
!fx collaborator create --data_path 1 -n charlie --silent
Show code cell output
2025-06-23 17:31:11.950328: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:11.951056: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:11.954132: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:11.962160: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699871.975138 677 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699871.978955 677 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:11.993673: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
✔️ OK
Now, generate charlies certificate and key pair.
!fx collaborator generate-cert-request -n charlie --silent
Show code cell output
2025-06-23 17:31:15.285020: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:15.285767: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:15.288987: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:15.297271: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699875.311803 683 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699875.315681 683 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:15.332069: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Creating COLLABORATOR certificate key pair with following settings: CN=charlie, SAN=DNS:charlie
Moving COLLABORATOR certificate to: /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/cert/col_charlie
The CSR Hash f36286942fbfef54b851f21264c9d6a0546dc89c11dac8f5bfb16f9b5328da7f3cc7dbb0af289963106e54233187479a
Archive col_charlie_to_agg_cert_request.zip with certificate signing request created
This file should be sent to the certificate authority (typically hosted by the aggregator) for signing
✔️ OK
Certify charlie with the CA.
Note: Again, we assume CA to be the aggregator. Hence the command below is expected to be run on the
aggregatorside. In practice,charlie’s certificate signing request (CSR) package generated in the last step, is sent to theaggregator, which then signs the certificate with the CA. The signed certificate then has to be sent back tocharlie.
!fx collaborator certify --request-pkg col_charlie_to_agg_cert_request.zip --silent
Show code cell output
2025-06-23 17:31:18.749679: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:18.750406: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:18.753506: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:18.761526: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699878.774416 689 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699878.778213 689 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:18.792897: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
The CSR Hash for file col_charlie.csr is f36286942fbfef54b851f21264c9d6a0546dc89c11dac8f5bfb16f9b5328da7f3cc7dbb0af289963106e54233187479a
Signing COLLABORATOR certificate, Warning: manual check of certificate hashes is bypassed in silent mode.
Registering charlie in /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/plan/cols.yaml
✔️ OK
Import CA signed certificates#
This step imports the CA-signed certificates of the collaborator and the aggregator into the workspace. We are back on the individual collaborator nodes for this step.
!fx collaborator certify --import agg_to_col_bob_signed_cert.zip
!fx collaborator certify --import agg_to_col_charlie_signed_cert.zip
Show code cell output
2025-06-23 17:31:22.252438: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:22.253180: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:22.256281: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:22.264342: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699882.277882 695 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699882.281964 695 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:22.297170: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Certificate updated in the PKI directory
✔️ OK
2025-06-23 17:31:25.536019: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:25.536740: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:25.539847: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:25.547942: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699885.560899 701 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699885.564671 701 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:25.579351: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Certificate updated in the PKI directory
✔️ OK
Start the simulation#
This step starts the simulation of the FL experiment. The aggregator will orchestrate the training process between the collaborators.
In practice, this command is executed on respective aggregator/collaborator nodes.
!fx aggregator start &\
fx collaborator start -n bob &\
fx collaborator start -n charlie
Show code cell output
2025-06-23 17:31:29.206231: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:29.207187: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:29.210371: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:29.218393: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699889.246772 708 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699889.254787 708 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:29.284734: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-06-23 17:31:29.291727: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:29.292388: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:29.295184: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:29.303038: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699889.320198 709 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699889.328003 709 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:29.355487: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-06-23 17:31:29.363786: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:29.365456: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:29.372335: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:29.386124: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699889.408996 707 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699889.412704 707 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:29.427020: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
[06/23/25 17:31:32] INFO FL-Plan hash is e5455a1aaae22029e1fe92ab5bb878702761ff3c12e2ae7ddc8b037eafd2cc6b18ae2cff34bfc1833276c78073bb5d2b ]8;id=298500;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=821805;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#305\305]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=353336;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=745368;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#152\152]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=216495;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=197853;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#157\157]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: localhost
agg_port: 55336
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
transport_protocol: grpc
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
INFO Data paths: {'one': '1', 'bob': '0', 'charlie': '1'} ]8;id=624750;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/collaborator.py\collaborator.py]8;;\:]8;id=883034;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/collaborator.py#80\80]8;;\
Data = {'one': '1', 'bob': '0', 'charlie': '1'}
INFO 🧿 Starting a Collaborator Service. ]8;id=49892;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/collaborator.py\collaborator.py]8;;\:]8;id=223343;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/collaborator.py#82\82]8;;\
INFO Building `src.dataloader.KerasMNISTInMemory` Module. ]8;id=935516;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=512713;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#243\243]8;;\
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
8192/11490434 [..............................] - ETA: 0s
[06/23/25 17:31:32] INFO FL-Plan hash is e5455a1aaae22029e1fe92ab5bb878702761ff3c12e2ae7ddc8b037eafd2cc6b18ae2cff34bfc1833276c78073bb5d2b ]8;id=734695;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=785566;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#305\305]8;;\
835584/11490434 [=>............................] - ETA: 0s
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=599318;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=141189;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#152\152]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=242098;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=404283;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#157\157]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: localhost
agg_port: 55336
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
transport_protocol: grpc
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
INFO 🧿 Starting the Aggregator Service. ]8;id=603706;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/aggregator.py\aggregator.py]8;;\:]8;id=723304;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/aggregator.py#94\94]8;;\
INFO Building `openfl.component.RandomGroupedAssigner` Module. ]8;id=267692;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=19954;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#243\243]8;;\
[06/23/25 17:31:32] INFO FL-Plan hash is e5455a1aaae22029e1fe92ab5bb878702761ff3c12e2ae7ddc8b037eafd2cc6b18ae2cff34bfc1833276c78073bb5d2b ]8;id=491567;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=528270;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#305\305]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=241004;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=126029;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#152\152]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=891630;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=334081;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#157\157]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: localhost
agg_port: 55336
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
transport_protocol: grpc
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
INFO Data paths: {'one': '1', 'bob': '0', 'charlie': '1'} ]8;id=109680;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/collaborator.py\collaborator.py]8;;\:]8;id=945479;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/collaborator.py#80\80]8;;\
7634944/11490434 [==================>...........] - ETA: 0sData = {'one': '1', 'bob': '0', 'charlie': '1'}
INFO 🧿 Starting a Collaborator Service. ]8;id=719659;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/collaborator.py\collaborator.py]8;;\:]8;id=462838;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/interface/collaborator.py#82\82]8;;\
INFO Building `src.dataloader.KerasMNISTInMemory` Module. ]8;id=381023;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=209470;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#243\243]8;;\
A local file was found, but it seems to be incomplete or outdated because the auto file hash does not match the original value of 731c5ac602752760c8e48fbffcf8c3b850d9dc2a2aedcf2cc48468fc17b673d1 so we will re-download the data.
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
INFO Building `openfl.pipelines.NoCompressionPipeline` Module. ]8;id=688046;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=688182;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#243\243]8;;\
INFO Importing 🡆 Object WaitForAllPolicy from openfl.component.aggregator.straggler_handling Module. ]8;id=475615;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=804386;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#267\267]8;;\
INFO Connector defaults: None ]8;id=379777;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=998627;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#355\355]8;;\
INFO Building `openfl.component.Aggregator` Module. ]8;id=605579;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=66102;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#243\243]8;;\
INFO Persistent checkpoint is enabled, setting persistent db at path local_state/tensor.db ]8;id=108973;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=424113;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py#157\157]8;;\
INFO Initializing persistent db at local_state/tensor.db ]8;id=832833;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=278572;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/databases/persistent_db.py#36\36]8;;\
8192/11490434 [..............................] - ETA: 0sAborted!
606208/11490434 [>.............................] - ETA: 0s
5308416/11490434 [============>.................] - ETA: 0s
11493376/11490434 [==============================] - 0s 0us/step
INFO Starting Aggregator gRPC Server ]8;id=117736;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_server.py\aggregator_server.py]8;;\:]8;id=727065;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_server.py#354\354]8;;\
[06/23/25 17:31:33] INFO MNIST > X_train Shape : (30000, 28, 28, 1) ]8;id=922468;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=797967;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/src/mnist_utils.py#105\105]8;;\
INFO MNIST > y_train Shape : (30000,) ]8;id=984179;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=705351;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/src/mnist_utils.py#106\106]8;;\
INFO MNIST > Train Samples : 30000 ]8;id=364165;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=277491;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/src/mnist_utils.py#107\107]8;;\
INFO MNIST > Valid Samples : 5000 ]8;id=681728;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=469768;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/docs/tutorials/mnist_example/src/mnist_utils.py#108\108]8;;\
INFO Building `src.taskrunner.KerasCNN` Module. ]8;id=777622;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=865349;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#243\243]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py:113: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(activity_regularizer=activity_regularizer, **kwargs)
2025-06-23 17:31:33.018031: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
INFO Model: "sequential" ]8;id=623398;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/keras/src/utils/summary_utils.py\summary_utils.py]8;;\:]8;id=423329;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/keras/src/utils/summary_utils.py#389\389]8;;\
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type) ┃ Output Shape ┃ Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d (Conv2D) │ (None, 13, 13, 16) │ 272 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D) │ (None, 5, 5, 32) │ 8,224 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten) │ (None, 800) │ 0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense) │ (None, 100) │ 80,100 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense) │ (None, 10) │ 1,010 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 89,606 (350.02 KB)
Trainable params: 89,606 (350.02 KB)
Non-trainable params: 0 (0.00 B)
INFO Building `openfl.pipelines.NoCompressionPipeline` Module. ]8;id=934114;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=670059;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#243\243]8;;\
INFO Building `openfl.component.Collaborator` Module. ]8;id=887794;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=86560;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/plan/plan.py#243\243]8;;\
WARNING Argument `device_assignment_policy` is deprecated and will be removed in the future. ]8;id=517821;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=500519;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/collaborator/collaborator.py#131\131]8;;\
INFO Requesting tasks... ]8;id=359966;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=776141;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:31:33] INFO Sending tasks to collaborator charlie for round 0 ]8;id=460622;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=404660;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py#552\552]8;;\
INFO Round: 0 Received Tasks: [name: "aggregated_model_validation" ]8;id=719499;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=439094;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/collaborator/collaborator.py#188\188]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 242ms/step - accuracy: 0.1250 - loss: 2.3036
37/Unknown 0s 1ms/step - accuracy: 0.0755 - loss: 2.3066
75/Unknown 0s 1ms/step - accuracy: 0.0682 - loss: 2.3068
115/Unknown 0s 1ms/step - accuracy: 0.0633 - loss: 2.3073
154/Unknown 0s 1ms/step - accuracy: 0.0613 - loss: 2.30762025-06-23 17:31:33.887122: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.9/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:160: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.0612 - loss: 2.3077
INFO Collaborator charlie is sending task results for aggregated_model_validation, round 0 ]8;id=948051;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=742773;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py#806\806]8;;\
INFO Run 0 epoch of 0 round ]8;id=263320;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=957921;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/federated/task/runner_keras.py#108\108]8;;\
2025-06-23 17:31:36.545857: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
938/938 - 3s - 3ms/step - accuracy: 0.9421 - loss: 0.1965
[06/23/25 17:31:36] INFO Collaborator charlie is sending task results for train, round 0 ]8;id=211714;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=512463;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py#806\806]8;;\
1/Unknown 0s 20ms/step - accuracy: 0.9062 - loss: 0.3803
40/Unknown 0s 1ms/step - accuracy: 0.9580 - loss: 0.1744
79/Unknown 0s 1ms/step - accuracy: 0.9594 - loss: 0.1638
118/Unknown 0s 1ms/step - accuracy: 0.9589 - loss: 0.1584
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - accuracy: 0.9589 - loss: 0.1541
INFO Collaborator charlie is sending task results for locally_tuned_model_validation, round 0 ]8;id=526424;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=19314;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py#806\806]8;;\
INFO Round 0: Collaborators that have completed all tasks: ['charlie'] ]8;id=419164;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=915935;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/component/aggregator/aggregator.py#1333\1333]8;;\
[06/23/25 17:31:36] INFO Requesting tasks... ]8;id=257586;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=154444;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:31:46] INFO Requesting tasks... ]8;id=978008;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=360159;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:31:56] INFO Requesting tasks... ]8;id=141738;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=340042;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:32:06] INFO Requesting tasks... ]8;id=856763;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=151387;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:32:16] INFO Requesting tasks... ]8;id=710376;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=191365;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:32:26] INFO Requesting tasks... ]8;id=779621;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=871832;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:32:36] INFO Requesting tasks... ]8;id=562269;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=335970;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:32:46] INFO Requesting tasks... ]8;id=268027;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=503611;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:32:56] INFO Requesting tasks... ]8;id=714754;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=9833;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:33:06] INFO Requesting tasks... ]8;id=57461;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=454110;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:33:16] INFO Requesting tasks... ]8;id=351794;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=917610;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:33:26] INFO Requesting tasks... ]8;id=928257;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=569639;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:33:36] INFO Requesting tasks... ]8;id=300615;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=377701;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:33:46] INFO Requesting tasks... ]8;id=936675;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=805870;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:33:56] INFO Requesting tasks... ]8;id=454198;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=112270;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:34:07] INFO Requesting tasks... ]8;id=4586;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=133070;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:34:17] INFO Requesting tasks... ]8;id=563924;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=888302;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:34:27] INFO Requesting tasks... ]8;id=372561;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=547857;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:34:37] INFO Requesting tasks... ]8;id=256379;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=908559;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:34:47] INFO Requesting tasks... ]8;id=685061;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=268964;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:34:57] INFO Requesting tasks... ]8;id=782284;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=935621;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:35:07] INFO Requesting tasks... ]8;id=401604;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=596885;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:35:17] INFO Requesting tasks... ]8;id=250660;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=314154;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:35:27] INFO Requesting tasks... ]8;id=708919;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=25120;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:35:37] INFO Requesting tasks... ]8;id=901191;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=87536;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:35:47] INFO Requesting tasks... ]8;id=163004;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=919972;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:35:57] INFO Requesting tasks... ]8;id=870356;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=687919;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:36:07] INFO Requesting tasks... ]8;id=278751;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=695001;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:36:17] INFO Requesting tasks... ]8;id=469032;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=816264;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:36:27] INFO Requesting tasks... ]8;id=718114;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=951734;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:36:37] INFO Requesting tasks... ]8;id=539684;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=820470;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:36:47] INFO Requesting tasks... ]8;id=124898;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=15982;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:36:57] INFO Requesting tasks... ]8;id=991439;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=660991;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:37:07] INFO Requesting tasks... ]8;id=992633;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=463560;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:37:17] INFO Requesting tasks... ]8;id=675857;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=378480;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:37:27] INFO Requesting tasks... ]8;id=89885;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=508134;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:37:37] INFO Requesting tasks... ]8;id=952248;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=399346;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:37:47] INFO Requesting tasks... ]8;id=230761;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=757470;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:37:57] INFO Requesting tasks... ]8;id=189668;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=399575;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:38:07] INFO Requesting tasks... ]8;id=224881;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=171910;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:38:17] INFO Requesting tasks... ]8;id=174354;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=307475;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:38:27] INFO Requesting tasks... ]8;id=211418;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=812702;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:38:37] INFO Requesting tasks... ]8;id=38626;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=245654;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:38:47] INFO Requesting tasks... ]8;id=154278;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=738752;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:38:57] INFO Requesting tasks... ]8;id=542174;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=499967;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:39:07] INFO Requesting tasks... ]8;id=356868;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=992519;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:39:17] INFO Requesting tasks... ]8;id=365846;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=171123;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:39:27] INFO Requesting tasks... ]8;id=963259;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=509771;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:39:37] INFO Requesting tasks... ]8;id=515078;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=9174;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:39:47] INFO Requesting tasks... ]8;id=563581;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=228970;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:39:57] INFO Requesting tasks... ]8;id=670372;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=347578;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:40:07] INFO Requesting tasks... ]8;id=605508;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=23080;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:40:17] INFO Requesting tasks... ]8;id=446900;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=812349;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:40:27] INFO Requesting tasks... ]8;id=545069;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=410696;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:40:37] INFO Requesting tasks... ]8;id=876209;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=298605;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:40:47] INFO Requesting tasks... ]8;id=382416;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=282779;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:40:57] INFO Requesting tasks... ]8;id=77437;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=612072;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:41:07] INFO Requesting tasks... ]8;id=368788;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=92970;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:41:17] INFO Requesting tasks... ]8;id=409463;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=845588;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
[06/23/25 17:41:27] INFO Requesting tasks... ]8;id=228918;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=96739;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.9/openfl/transport/grpc/aggregator_client.py#327\327]8;;\
^C
Aborted!
Next steps#
The simulation will run for a few epochs. Once the simulation is complete, the model will be saved under the save directory. The model can be converted to framework native formats using the fx model save command.
For detailed information on options supported via CLI, refer to the CLI reference.
For an in-depth understanding of the TaskRunner architecture, refer to the TaskRunner API.