Quickstart#
In this guide, we will train a simple Convolutional Neural Network (CNN) on MNIST handwritten digits dataset. We will simulate a Federated Learning experiment between two collaborators, orchestrated by an aggregator, via few CLI commands.
OpenFL must be installed for this tutorial. Refer to the installation guide.
(Optional) For a simplified setup without TLS, follow these steps as applicable throughout the process:
Set
use_tls: falsein the FL plan before initialization.Skip all certificate-related commands (incl. CA setup).
Manually populate the collaborator names in
cols.yaml. For example:
collaborators:
- bob
- charlie
Create a workspace#
We will reuse an existing workspace. OpenFL provides many reference model templates. Check them out under workspaces.
!fx workspace create --prefix ./mnist_example --template keras/mnist
%cd ./mnist_example
!pip install -r requirements.txt > /dev/null
Show code cell output
Creating Workspace Directories
Creating Workspace Templates from /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl-workspace/keras/mnist in /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example
[06/23/25 17:30:12] INFO Loading DEFAULTS for section aggregator from file ]8;id=346636;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=40455;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl-workspace/workspace/plan/defaults/aggre
gator.yaml.
INFO Loading DEFAULTS for section assigner from file ]8;id=427445;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=435372;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl-workspace/workspace/plan/defaults/assig
ner.yaml.
INFO Loading DEFAULTS for section collaborator from file ]8;id=148572;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=813855;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl-workspace/workspace/plan/defaults/colla
borator.yaml.
INFO Loading DEFAULTS for section compression_pipeline from file ]8;id=577235;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=420509;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl-workspace/workspace/plan/defaults/compr
ession_pipeline.yaml.
INFO Loading DEFAULTS for section data_loader from file ]8;id=980303;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=796948;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl-workspace/workspace/plan/defaults/data_
loader.yaml.
INFO Loading DEFAULTS for section network from file ]8;id=883272;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=845802;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl-workspace/workspace/plan/defaults/netwo
rk.yaml.
INFO Loading DEFAULTS for section task_runner from file ]8;id=17459;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=976075;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl-workspace/workspace/plan/defaults/task_
runner.yaml.
INFO Loading DEFAULTS for section tasks from file ]8;id=901395;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=152446;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#186\186]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl-workspace/workspace/plan/defaults/tasks
_keras.yaml.
INFO FL-Plan hash is 71f93e3c5870ca465935df0180c419181cd57832297069eb737d448dfa00cb6f8474dfab1489dd6621521dbf084e2565 ]8;id=188470;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=786010;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#305\305]8;;\
INFO FL-Plan hash is 10b91870638eda742ffd6a224eaacd087b3d89da1b9b31814fe9350c9f67d78301385f59f4985c34361d0e5ebaa454e0 ]8;id=197297;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=869723;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#305\305]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=365012;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=591282;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#152\152]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl-workspace/keras/mnist/plan/plan.yaml.
INFO aggregator: ]8;id=208902;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=288428;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#157\157]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: build-28612573-project-638300-openfl
agg_port: 54370
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
transport_protocol: grpc
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
New workspace directory structure:
mnist_example
├── src
│ ├── taskrunner.py
│ ├── __init__.py
│ ├── mnist_utils.py
│ └── dataloader.py
├── cert
├── plan
│ ├── cols.yaml
│ ├── data.yaml
│ ├── plan.yaml
│ └── defaults
├── .workspace
├── data
├── local_state
├── logs
├── requirements.txt
└── save
7 directories, 10 files
✔️ OK
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/IPython/core/magics/osm.py:417: UserWarning: This is now an optional IPython functionality, setting dhist requires you to install the `pickleshare` library.
self.shell.db['dhist'] = compress_dhist(dhist)[-100:]
Initialize a Plan#
This step builds an entire FL experiment plan, along with the initial set of parameters that will be used in the experiment.
We supply localhost as the aggregator address, for simulation purposes. The command below supports models initializations (e.g. when using pre-trained models).
For a complete list of supported options, refer to fx plan initialize.
!fx plan initialize -a localhost
Show code cell output
2025-06-23 17:30:47.045005: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:30:47.045770: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:47.048834: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:47.056813: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699847.069646 629 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699847.073423 629 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:30:47.087920: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
[06/23/25 17:30:49] INFO FL-Plan hash is 0f7fa25bd4eb74bd36686903898be7cfbc560a7f20741ae982c1ad5ba513f66acbd54947e1a6cdbb18ef35dfcc5bd2f2 ]8;id=15286;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=930960;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#305\305]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=77834;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=887771;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#152\152]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=114904;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=842215;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#157\157]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: build-28612573-project-638300-openfl
agg_port: 54370
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
transport_protocol: grpc
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
INFO Initialized minimal dataloader for model creation ]8;id=304472;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/utilities/dataloading.py\dataloading.py]8;;\:]8;id=478828;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/utilities/dataloading.py#54\54]8;;\
INFO Building `src.taskrunner.KerasCNN` Module. ]8;id=668128;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=445505;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#243\243]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py:113: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(activity_regularizer=activity_regularizer, **kwargs)
2025-06-23 17:30:49.635917: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
INFO Model: "sequential" ]8;id=375714;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/keras/src/utils/summary_utils.py\summary_utils.py]8;;\:]8;id=188590;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/keras/src/utils/summary_utils.py#389\389]8;;\
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type) ┃ Output Shape ┃ Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d (Conv2D) │ (None, 13, 13, 16) │ 272 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D) │ (None, 5, 5, 32) │ 8,224 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten) │ (None, 800) │ 0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense) │ (None, 100) │ 80,100 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense) │ (None, 10) │ 1,010 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 89,606 (350.02 KB)
Trainable params: 89,606 (350.02 KB)
Non-trainable params: 0 (0.00 B)
INFO Building `openfl.pipelines.NoCompressionPipeline` Module. ]8;id=458063;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=461278;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#243\243]8;;\
WARNING Following parameters omitted from global initial model, local initialization will determine values: [] ]8;id=120403;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/plan.py\plan.py]8;;\:]8;id=231214;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/plan.py#132\132]8;;\
INFO Saving model state to save/init.pbuf ]8;id=457106;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/plan.py\plan.py]8;;\:]8;id=248570;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/plan.py#140\140]8;;\
WARNING Patching Aggregator Addr in Plan 🠆 localhost ]8;id=714214;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/plan.py\plan.py]8;;\:]8;id=663134;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/plan.py#157\157]8;;\
INFO FL-Plan hash is b5506672795565d7e9a2278a8f721262cb056ca035e481dbb41cd9b822a936c51254ad05042082374bd357a01ac72a47 ]8;id=13097;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=565588;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#305\305]8;;\
INFO ['plan_b5506672'] ]8;id=451486;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/plan.py\plan.py]8;;\:]8;id=559362;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/plan.py#171\171]8;;\
✔️ OK
Aggregator Setup#
The fx aggregator subcommand options to manage the aggregator-side certificate signing, generation and verification.
Here, we generate the aggregator’s certificate and private key. The aggregator’s certificate is then signed by the CA generated in the previous step.
For all commands that follow, --silent is used to skip manual confirmation. In practice, users are recommended to verify the details before proceeding.
!fx aggregator generate-cert-request --fqdn localhost
!fx aggregator certify --fqdn localhost --silent
Show code cell output
2025-06-23 17:30:54.761811: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:30:54.762534: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:54.765603: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:54.773543: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699854.786269 646 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699854.790048 646 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:30:54.804610: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Creating AGGREGATOR certificate key pair with following settings: CN=localhost, SAN=DNS:localhost
Writing AGGREGATOR certificate key pair to: /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/cert/server
The CSR Hash a2977d682892f6a12fb289572b93c7ac45da1ff6496c82c1ff183ab3f26487ad4f8b19ff24d4c3af084961712e923767
✔️ OK
2025-06-23 17:30:58.165248: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:30:58.165971: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:58.169050: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:30:58.177022: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699858.189875 652 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699858.193637 652 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:30:58.208227: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
The CSR Hash for file server/agg_localhost.csr = a2977d682892f6a12fb289572b93c7ac45da1ff6496c82c1ff183ab3f26487ad4f8b19ff24d4c3af084961712e923767
Warning: manual check of certificate hashes is bypassed in silent mode.
Signing AGGREGATOR certificate
✔️ OK
Collaborator Setup#
Following aggregator, fx collaborator subcommand manages the collaborator-side certificate signing, generation and verification. In addition, because collaborators bring their own local datasets, we populate dataset information under plan/data.yaml.
Setup first collaborator bob#
This step registers --data_path against collaborator’s name -n under plan/data.yaml.
!fx collaborator create --data_path 0 -n bob --silent
Show code cell output
2025-06-23 17:31:01.518136: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:01.518864: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:01.521989: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:01.530020: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699861.542899 658 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699861.546689 658 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:01.561214: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
✔️ OK
The next step generates bobs certificate and key pair.
!fx collaborator generate-cert-request -n bob --silent
Show code cell output
2025-06-23 17:31:04.726940: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:04.727664: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:04.730729: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:04.738691: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699864.751406 664 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699864.755128 664 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:04.769595: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Creating COLLABORATOR certificate key pair with following settings: CN=bob, SAN=DNS:bob
Moving COLLABORATOR certificate to: /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/cert/col_bob
The CSR Hash d6c533a4095829a5bbfa9ceea73de5bbca1ddda4e9b717fff932fad9c142436773ced5ac025825059013532f6bf50718
Archive col_bob_to_agg_cert_request.zip with certificate signing request created
This file should be sent to the certificate authority (typically hosted by the aggregator) for signing
✔️ OK
Certify bob with the CA.
Note: Here, we assume CA to be the aggregator. Hence the command below runs on the
aggregatorside. In reality,bob’s certificate signing request (CSR) package generated in the last step, is sent to theaggregator, which then signs the certificate with the CA. The signed certificate then has to be sent back tobob.
!fx collaborator certify --request-pkg col_bob_to_agg_cert_request.zip --silent
Show code cell output
2025-06-23 17:31:08.038433: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:08.039145: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:08.042217: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:08.050156: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699868.062839 670 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699868.066580 670 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:08.081188: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
The CSR Hash for file col_bob.csr is d6c533a4095829a5bbfa9ceea73de5bbca1ddda4e9b717fff932fad9c142436773ced5ac025825059013532f6bf50718
Signing COLLABORATOR certificate, Warning: manual check of certificate hashes is bypassed in silent mode.
Registering bob in /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/plan/cols.yaml
✔️ OK
Setup second collaborator charlie#
As before, we create a collaborator by registering --data_path against collaborator’s name -n under plan/data.yaml.
!fx collaborator create --data_path 1 -n charlie --silent
Show code cell output
2025-06-23 17:31:11.368506: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:11.369211: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:11.372285: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:11.380234: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699871.393032 676 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699871.396796 676 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:11.411316: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
✔️ OK
Now, generate charlies certificate and key pair.
!fx collaborator generate-cert-request -n charlie --silent
Show code cell output
2025-06-23 17:31:14.594589: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:14.595301: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:14.598370: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:14.606324: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699874.619147 682 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699874.622897 682 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:14.637433: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Creating COLLABORATOR certificate key pair with following settings: CN=charlie, SAN=DNS:charlie
Moving COLLABORATOR certificate to: /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/cert/col_charlie
The CSR Hash f03b9d8600f63bbfdb54fa01ff877b7ab9c272145c075f47a91d7002fbebb7853454c51e70fabe766137295efe164b95
Archive col_charlie_to_agg_cert_request.zip with certificate signing request created
This file should be sent to the certificate authority (typically hosted by the aggregator) for signing
✔️ OK
Certify charlie with the CA.
Note: Again, we assume CA to be the aggregator. Hence the command below is expected to be run on the
aggregatorside. In practice,charlie’s certificate signing request (CSR) package generated in the last step, is sent to theaggregator, which then signs the certificate with the CA. The signed certificate then has to be sent back tocharlie.
!fx collaborator certify --request-pkg col_charlie_to_agg_cert_request.zip --silent
Show code cell output
2025-06-23 17:31:17.976833: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:17.977565: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:17.980657: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:17.988616: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699878.001380 688 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699878.005145 688 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:18.019737: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
The CSR Hash for file col_charlie.csr is f03b9d8600f63bbfdb54fa01ff877b7ab9c272145c075f47a91d7002fbebb7853454c51e70fabe766137295efe164b95
Signing COLLABORATOR certificate, Warning: manual check of certificate hashes is bypassed in silent mode.
Registering charlie in /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/plan/cols.yaml
✔️ OK
Import CA signed certificates#
This step imports the CA-signed certificates of the collaborator and the aggregator into the workspace. We are back on the individual collaborator nodes for this step.
!fx collaborator certify --import agg_to_col_bob_signed_cert.zip
!fx collaborator certify --import agg_to_col_charlie_signed_cert.zip
Show code cell output
2025-06-23 17:31:21.316616: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:21.317330: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:21.320440: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:21.328393: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699881.341061 694 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699881.344849 694 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:21.359353: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Certificate updated in the PKI directory
✔️ OK
2025-06-23 17:31:24.525593: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:24.526323: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:24.529444: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:24.537410: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699884.550177 700 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699884.553909 700 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:24.568377: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Certificate updated in the PKI directory
✔️ OK
Start the simulation#
This step starts the simulation of the FL experiment. The aggregator will orchestrate the training process between the collaborators.
In practice, this command is executed on respective aggregator/collaborator nodes.
!fx aggregator start &\
fx collaborator start -n bob &\
fx collaborator start -n charlie
Show code cell output
2025-06-23 17:31:28.162777: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:28.163703: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:28.166946: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:28.176280: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699888.189008 708 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699888.192858 708 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:28.207842: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-06-23 17:31:28.224459: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:28.225094: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:28.231948: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:28.234107: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-06-23 17:31:28.234689: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:28.239227: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-06-23 17:31:28.247918: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-06-23 17:31:28.254836: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699888.275414 707 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750699888.277498 706 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750699888.281407 706 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1750699888.283933 707 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-06-23 17:31:28.296086: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-06-23 17:31:28.312066: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
[06/23/25 17:31:31] INFO FL-Plan hash is b5506672795565d7e9a2278a8f721262cb056ca035e481dbb41cd9b822a936c51254ad05042082374bd357a01ac72a47 ]8;id=533342;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=946317;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#305\305]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=574852;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=759557;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#152\152]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=58198;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=583569;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#157\157]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: localhost
agg_port: 54370
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
transport_protocol: grpc
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
INFO Data paths: {'one': '1', 'bob': '0', 'charlie': '1'} ]8;id=528177;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/collaborator.py\collaborator.py]8;;\:]8;id=237852;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/collaborator.py#80\80]8;;\
Data = {'one': '1', 'bob': '0', 'charlie': '1'}
INFO 🧿 Starting a Collaborator Service. ]8;id=298287;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/collaborator.py\collaborator.py]8;;\:]8;id=534272;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/collaborator.py#82\82]8;;\
INFO Building `src.dataloader.KerasMNISTInMemory` Module. ]8;id=765933;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=167322;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#243\243]8;;\
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
8192/11490434 [..............................] - ETA: 0s
[06/23/25 17:31:31] INFO FL-Plan hash is b5506672795565d7e9a2278a8f721262cb056ca035e481dbb41cd9b822a936c51254ad05042082374bd357a01ac72a47 ]8;id=100337;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=994843;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#305\305]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=810381;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=58847;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#152\152]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=392590;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=833153;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#157\157]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: localhost
agg_port: 54370
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
transport_protocol: grpc
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
INFO Data paths: {'one': '1', 'bob': '0', 'charlie': '1'} ]8;id=291566;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/collaborator.py\collaborator.py]8;;\:]8;id=885900;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/collaborator.py#80\80]8;;\
Data = {'one': '1', 'bob': '0', 'charlie': '1'}
INFO 🧿 Starting a Collaborator Service. ]8;id=157077;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/collaborator.py\collaborator.py]8;;\:]8;id=969037;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/collaborator.py#82\82]8;;\
INFO Building `src.dataloader.KerasMNISTInMemory` Module. ]8;id=121876;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=610201;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#243\243]8;;\
A local file was found, but it seems to be incomplete or outdated because the auto file hash does not match the original value of 731c5ac602752760c8e48fbffcf8c3b850d9dc2a2aedcf2cc48468fc17b673d1 so we will re-download the data.
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
212992/11490434 [..............................] - ETA: 3s
3473408/11490434 [========>.....................] - ETA: 0s/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
8192/11490434 [..............................] - ETA: 0s
11493376/11490434 [==============================] - 0s 0us/step
EXCEPTION : Bad magic number for file header
Traceback (most recent call last):
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/bin/fx", line 8, in <module>
sys.exit(entry())
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/cli.py", line 310, in entry
error_handler(e)
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/cli.py", line 229, in error_handler
raise error
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/cli.py", line 308, in entry
cli(max_content_width=120)
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/click/core.py", line 1442, in __call__
return self.main(*args, **kwargs)
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/click/core.py", line 1363, in main
rv = self.invoke(ctx)
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/cli.py", line 131, in invoke
super().invoke(ctx)
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/click/core.py", line 1830, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/click/core.py", line 1830, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/click/core.py", line 1226, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/click/core.py", line 794, in invoke
return callback(*args, **kwargs)
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/collaborator.py", line 84, in start_
collaborator = plan_obj.get_collaborator(collaborator_name)
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py", line 548, in get_collaborator
data_loader = self.get_data_loader(collaborator_name)
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py", line 473, in get_data_loader
self.loader_ = Plan.build(**defaults)
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py", line 250, in build
instance = getattr(module, class_name)(**settings)
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/src/dataloader.py", line 42, in __init__
X_train, y_train, X_valid, y_valid = load_mnist_shard(
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/src/mnist_utils.py", line 89, in load_mnist_shard
(X_train, y_train), (X_valid, y_valid) = _load_raw_datashards(
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/src/mnist_utils.py", line 48, in _load_raw_datashards
X_train_tot = f['x_train']
File "/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 251, in __getitem__
bytes = self.zip.open(key)
File "/home/docs/.asdf/installs/python/3.10.17/lib/python3.10/zipfile.py", line 1546, in open
raise BadZipFile("Bad magic number for file header")
zipfile.BadZipFile: Bad magic number for file header
409600/11490434 [>.............................] - ETA: 1s[06/23/25 17:31:31] INFO FL-Plan hash is b5506672795565d7e9a2278a8f721262cb056ca035e481dbb41cd9b822a936c51254ad05042082374bd357a01ac72a47 ]8;id=768489;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=815501;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#305\305]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=449618;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=739579;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#152\152]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=666220;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=557497;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#157\157]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: localhost
agg_port: 54370
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
transport_protocol: grpc
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
INFO 🧿 Starting the Aggregator Service. ]8;id=81321;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/aggregator.py\aggregator.py]8;;\:]8;id=441249;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/interface/aggregator.py#94\94]8;;\
INFO Building `openfl.component.RandomGroupedAssigner` Module. ]8;id=271590;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=778796;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#243\243]8;;\
5021696/11490434 [============>.................] - ETA: 0s
11493376/11490434 [==============================] - 0s 0us/step
INFO Building `openfl.pipelines.NoCompressionPipeline` Module. ]8;id=636195;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=547709;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#243\243]8;;\
INFO Importing 🡆 Object WaitForAllPolicy from openfl.component.aggregator.straggler_handling Module. ]8;id=597630;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=534437;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#267\267]8;;\
INFO Connector defaults: None ]8;id=254037;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=149405;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#355\355]8;;\
INFO Building `openfl.component.Aggregator` Module. ]8;id=992914;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=479720;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#243\243]8;;\
INFO Persistent checkpoint is enabled, setting persistent db at path local_state/tensor.db ]8;id=416264;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=381922;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/component/aggregator/aggregator.py#157\157]8;;\
INFO Initializing persistent db at local_state/tensor.db ]8;id=500667;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=33517;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/databases/persistent_db.py#36\36]8;;\
INFO MNIST > X_train Shape : (30000, 28, 28, 1) ]8;id=215777;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=172988;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/src/mnist_utils.py#105\105]8;;\
INFO MNIST > y_train Shape : (30000,) ]8;id=651837;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=932056;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/src/mnist_utils.py#106\106]8;;\
INFO MNIST > Train Samples : 30000 ]8;id=53367;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=775477;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/src/mnist_utils.py#107\107]8;;\
INFO MNIST > Valid Samples : 5000 ]8;id=595685;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=322905;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/docs/tutorials/mnist_example/src/mnist_utils.py#108\108]8;;\
INFO Building `src.taskrunner.KerasCNN` Module. ]8;id=564147;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=387075;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#243\243]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py:113: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(activity_regularizer=activity_regularizer, **kwargs)
2025-06-23 17:31:31.861653: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
[06/23/25 17:31:32] INFO Model: "sequential" ]8;id=123290;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/keras/src/utils/summary_utils.py\summary_utils.py]8;;\:]8;id=571501;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/stable/lib/python3.10/site-packages/keras/src/utils/summary_utils.py#389\389]8;;\
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type) ┃ Output Shape ┃ Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d (Conv2D) │ (None, 13, 13, 16) │ 272 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D) │ (None, 5, 5, 32) │ 8,224 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten) │ (None, 800) │ 0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense) │ (None, 100) │ 80,100 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense) │ (None, 10) │ 1,010 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 89,606 (350.02 KB)
Trainable params: 89,606 (350.02 KB)
Non-trainable params: 0 (0.00 B)
INFO Building `openfl.pipelines.NoCompressionPipeline` Module. ]8;id=962671;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=723885;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/federated/plan/plan.py#243\243]8;;\
[06/23/25 17:31:32] INFO Starting Aggregator gRPC Server ]8;id=364958;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/transport/grpc/aggregator_server.py\aggregator_server.py]8;;\:]8;id=699339;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/stable/openfl/transport/grpc/aggregator_server.py#354\354]8;;\
Next steps#
The simulation will run for a few epochs. Once the simulation is complete, the model will be saved under the save directory. The model can be converted to framework native formats using the fx model save command.
For detailed information on options supported via CLI, refer to the CLI reference.
For an in-depth understanding of the TaskRunner architecture, refer to the TaskRunner API.