Quickstart#
In this guide, we will train a simple Convolutional Neural Network (CNN) on MNIST handwritten digits dataset. We will simulate a Federated Learning experiment between two collaborators, orchestrated by an aggregator, via few CLI commands.
OpenFL must be installed for this tutorial. Refer to the installation guide.
(Optional) For a simplified setup without TLS, follow these steps as applicable throughout the process:
Set
use_tls: falsein the FL plan before initialization.Skip all certificate-related commands (incl. CA setup).
Manually populate the collaborator names in
cols.yaml. For example:
collaborators:
- bob
- charlie
Create a workspace#
We will reuse an existing workspace. OpenFL provides many reference model templates. Check them out under workspaces.
!fx workspace create --prefix ./mnist_example --template keras/mnist
%cd ./mnist_example
!pip install -r requirements.txt > /dev/null
Show code cell output
Creating Workspace Directories
Creating Workspace Templates from /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl-workspace/keras/mnist in /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example
[03/26/25 05:52:17] INFO Loading DEFAULTS for section aggregator from file ]8;id=917550;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=107864;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#178\178]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl-workspace/workspace/plan/defaults/aggrega
tor.yaml.
INFO Loading DEFAULTS for section assigner from file ]8;id=683528;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=206046;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#178\178]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl-workspace/workspace/plan/defaults/assigne
r.yaml.
INFO Loading DEFAULTS for section collaborator from file ]8;id=269609;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=124482;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#178\178]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl-workspace/workspace/plan/defaults/collabo
rator.yaml.
INFO Loading DEFAULTS for section compression_pipeline from file ]8;id=933032;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=380286;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#178\178]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl-workspace/workspace/plan/defaults/compres
sion_pipeline.yaml.
INFO Loading DEFAULTS for section data_loader from file ]8;id=617604;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=669074;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#178\178]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl-workspace/workspace/plan/defaults/data_lo
ader.yaml.
INFO Loading DEFAULTS for section network from file ]8;id=652948;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=275541;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#178\178]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl-workspace/workspace/plan/defaults/network
.yaml.
INFO Loading DEFAULTS for section task_runner from file ]8;id=414076;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=241881;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#178\178]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl-workspace/workspace/plan/defaults/task_ru
nner.yaml.
INFO Loading DEFAULTS for section tasks from file ]8;id=583894;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=344139;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#178\178]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl-workspace/workspace/plan/defaults/tasks_k
eras.yaml.
INFO FL-Plan hash is fb9fca411a763d7f8749204d1ababdda2b62a759587c6f838c8f47ded77737e47399ca3de123e1efa2c2099ee7c95707 ]8;id=183716;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=686286;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#297\297]8;;\
INFO FL-Plan hash is e0bfeaf94975e93602f71312783cd28d721573f64abf7be5210bc15aa27710cb1a97b3b8c7e2da019a4a4ef15fe5b8d4 ]8;id=507937;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=299505;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#297\297]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=54222;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=898869;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#144\144]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl-workspace/keras/mnist/plan/plan.yaml.
INFO aggregator: ]8;id=516512;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=695101;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#149\149]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: build-27638422-project-638300-openfl
agg_port: 59905
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
New workspace directory structure:
mnist_example
├── .workspace
├── local_state
├── cert
├── data
├── plan
│ ├── defaults
│ ├── cols.yaml
│ ├── plan.yaml
│ └── data.yaml
├── save
├── requirements.txt
├── src
│ ├── dataloader.py
│ ├── taskrunner.py
│ ├── __init__.py
│ └── mnist_utils.py
└── logs
7 directories, 10 files
✔️ OK
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/IPython/core/magics/osm.py:417: UserWarning: This is now an optional IPython functionality, setting dhist requires you to install the `pickleshare` library.
self.shell.db['dhist'] = compress_dhist(dhist)[-100:]
Initialize a Plan#
This step builds an entire FL experiment plan, along with the initial set of parameters that will be used in the experiment.
We supply localhost as the aggregator address, for simulation purposes. The command below supports models of arbitrary input shapes and initializations (e.g. when using pre-trained models).
For a complete list of supported options, refer to fx plan initialize.
!fx plan initialize -a localhost
Show code cell output
2025-03-26 05:52:52.673966: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:52:52.674849: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:52:52.678003: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:52:52.687084: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968372.703000 665 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968372.708187 665 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:52:52.724719: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
[03/26/25 05:52:55] INFO FL-Plan hash is ed1cbaa585b1b2d3c71cc976c6b2c342965bd51e90b448dca58b076dc9be924c4981c89fd1bf31d6d4de01467383afd8 ]8;id=557993;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=510259;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#297\297]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=682253;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=102097;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#144\144]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=705924;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=163417;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#149\149]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: build-27638422-project-638300-openfl
agg_port: 59905
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
INFO Building `src.dataloader.KerasMNISTInMemory` Module. ]8;id=551892;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=537850;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
8192/11490434 [..............................] - ETA: 0s
278528/11490434 [..............................] - ETA: 2s
6217728/11490434 [===============>..............] - ETA: 0s
11493376/11490434 [==============================] - 0s 0us/step
INFO MNIST > X_train Shape : (30000, 28, 28, 1) ]8;id=442724;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=651807;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#108\108]8;;\
INFO MNIST > y_train Shape : (30000,) ]8;id=452541;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=67501;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#109\109]8;;\
INFO MNIST > Train Samples : 30000 ]8;id=667338;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=640299;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#110\110]8;;\
INFO MNIST > Valid Samples : 5000 ]8;id=532106;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=228095;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#111\111]8;;\
INFO Building `src.taskrunner.KerasCNN` Module. ]8;id=201421;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=389982;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(activity_regularizer=activity_regularizer, **kwargs)
2025-03-26 05:52:55.629488: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
INFO Model: "sequential" ]8;id=954060;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/utils/summary_utils.py\summary_utils.py]8;;\:]8;id=253249;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/utils/summary_utils.py#389\389]8;;\
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type) ┃ Output Shape ┃ Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d (Conv2D) │ (None, 13, 13, 16) │ 272 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D) │ (None, 5, 5, 32) │ 8,224 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten) │ (None, 800) │ 0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense) │ (None, 100) │ 80,100 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense) │ (None, 10) │ 1,010 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 89,606 (350.02 KB)
Trainable params: 89,606 (350.02 KB)
Non-trainable params: 0 (0.00 B)
INFO Train Set Size : 30000 ]8;id=872084;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py\taskrunner.py]8;;\:]8;id=279033;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py#34\34]8;;\
INFO Valid Set Size : 5000 ]8;id=966047;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py\taskrunner.py]8;;\:]8;id=472848;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py#35\35]8;;\
INFO Building `openfl.pipelines.NoCompressionPipeline` Module. ]8;id=955779;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=96928;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
WARNING Following parameters omitted from global initial model, local initialization will determine values: [] ]8;id=869032;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/plan.py\plan.py]8;;\:]8;id=198684;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/plan.py#157\157]8;;\
INFO Saving model state to save/init.pbuf ]8;id=509339;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/plan.py\plan.py]8;;\:]8;id=918421;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/plan.py#165\165]8;;\
WARNING Patching Aggregator Addr in Plan 🠆 localhost ]8;id=666927;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/plan.py\plan.py]8;;\:]8;id=195214;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/plan.py#182\182]8;;\
INFO FL-Plan hash is f9113436022351647107fab944c2673af3a63de34821ad3169a70f7c267a79f706745d8d7b4a360dbb795ceafa836f77 ]8;id=661797;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=564008;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#297\297]8;;\
INFO ['plan_f9113436'] ]8;id=678126;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/plan.py\plan.py]8;;\:]8;id=772733;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/plan.py#196\196]8;;\
✔️ OK
Aggregator Setup#
The fx aggregator subcommand options to manage the aggregator-side certificate signing, generation and verification.
Here, we generate the aggregator’s certificate and private key. The aggregator’s certificate is then signed by the CA generated in the previous step.
For all commands that follow, --silent is used to skip manual confirmation. In practice, users are recommended to verify the details before proceeding.
!fx aggregator generate-cert-request --fqdn localhost
!fx aggregator certify --fqdn localhost --silent
Show code cell output
2025-03-26 05:53:01.622596: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:01.623430: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:01.626443: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:01.635050: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968381.649993 682 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968381.654263 682 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:01.668930: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Creating AGGREGATOR certificate key pair with following settings: CN=localhost, SAN=DNS:localhost
Writing AGGREGATOR certificate key pair to: /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/cert/server
The CSR Hash e9e03b63f08c72e8bc485aedd7b52f75547621b936580dd69e9fb46f550cd9a9a10047f0b322b4340e49aa21f8f25a2c
✔️ OK
2025-03-26 05:53:05.378636: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:05.379343: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:05.382748: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:05.391690: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968385.407663 688 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968385.412003 688 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:05.427681: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
The CSR Hash for file server/agg_localhost.csr = e9e03b63f08c72e8bc485aedd7b52f75547621b936580dd69e9fb46f550cd9a9a10047f0b322b4340e49aa21f8f25a2c
Warning: manual check of certificate hashes is bypassed in silent mode.
Signing AGGREGATOR certificate
✔️ OK
Collaborator Setup#
Following aggregator, fx collaborator subcommand manages the collaborator-side certificate signing, generation and verification. In addition, because collaborators bring their own local datasets, we populate dataset information under plan/data.yaml.
Setup first collaborator bob#
This step registers --data_path against collaborator’s name -n under plan/data.yaml.
!fx collaborator create --data_path 0 -n bob --silent
Show code cell output
2025-03-26 05:53:09.250465: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:09.251181: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:09.254747: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:09.264083: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968389.278524 694 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968389.282769 694 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:09.297426: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
✔️ OK
The next step generates bobs certificate and key pair.
!fx collaborator generate-cert-request -n bob --silent
Show code cell output
2025-03-26 05:53:13.070326: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:13.071022: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:13.074118: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:13.082724: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968393.098009 700 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968393.102329 700 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:13.118339: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Creating COLLABORATOR certificate key pair with following settings: CN=bob, SAN=DNS:bob
Moving COLLABORATOR certificate to: /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/cert/col_bob
The CSR Hash 57899159f115fa8d7f37273673a0ae7518e1024f66b06febe93baebe7542822e67a9fa40297285b7073bfa9dc422b476
Archive col_bob_to_agg_cert_request.zip with certificate signing request created
This file should be sent to the certificate authority (typically hosted by the aggregator) for signing
✔️ OK
Certify bob with the CA.
Note: Here, we assume CA to be the aggregator. Hence the command below runs on the
aggregatorside. In reality,bob’s certificate signing request (CSR) package generated in the last step, is sent to theaggregator, which then signs the certificate with the CA. The signed certificate then has to be sent back tobob.
!fx collaborator certify --request-pkg col_bob_to_agg_cert_request.zip --silent
Show code cell output
2025-03-26 05:53:17.036093: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:17.036784: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:17.039946: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:17.048936: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968397.064120 706 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968397.068419 706 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:17.084167: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
The CSR Hash for file col_bob.csr is 57899159f115fa8d7f37273673a0ae7518e1024f66b06febe93baebe7542822e67a9fa40297285b7073bfa9dc422b476
Signing COLLABORATOR certificate, Warning: manual check of certificate hashes is bypassed in silent mode.
Registering bob in /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/plan/cols.yaml
✔️ OK
Setup second collaborator charlie#
As before, we create a collaborator by registering --data_path against collaborator’s name -n under plan/data.yaml.
!fx collaborator create --data_path 1 -n charlie --silent
Show code cell output
2025-03-26 05:53:20.935590: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:20.936420: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:20.939451: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:20.947993: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968400.962640 712 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968400.966990 712 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:20.981713: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
✔️ OK
Now, generate charlies certificate and key pair.
!fx collaborator generate-cert-request -n charlie --silent
Show code cell output
2025-03-26 05:53:24.690652: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:24.691357: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:24.694409: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:24.703021: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968404.717925 718 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968404.722242 718 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:24.737471: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Creating COLLABORATOR certificate key pair with following settings: CN=charlie, SAN=DNS:charlie
Moving COLLABORATOR certificate to: /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/cert/col_charlie
The CSR Hash 5808e1629c41ae89b487f12f8652e99abdc38d99e967dcfe952ce165e6326618558588b818ba8191d41aa1c8bc6b0c39
Archive col_charlie_to_agg_cert_request.zip with certificate signing request created
This file should be sent to the certificate authority (typically hosted by the aggregator) for signing
✔️ OK
Certify charlie with the CA.
Note: Again, we assume CA to be the aggregator. Hence the command below is expected to be run on the
aggregatorside. In practice,charlie’s certificate signing request (CSR) package generated in the last step, is sent to theaggregator, which then signs the certificate with the CA. The signed certificate then has to be sent back tocharlie.
!fx collaborator certify --request-pkg col_charlie_to_agg_cert_request.zip --silent
Show code cell output
2025-03-26 05:53:28.585442: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:28.586147: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:28.589220: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:28.597884: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968408.612963 724 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968408.617310 724 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:28.632584: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
The CSR Hash for file col_charlie.csr is 5808e1629c41ae89b487f12f8652e99abdc38d99e967dcfe952ce165e6326618558588b818ba8191d41aa1c8bc6b0c39
Signing COLLABORATOR certificate, Warning: manual check of certificate hashes is bypassed in silent mode.
Registering charlie in /home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/plan/cols.yaml
✔️ OK
Import CA signed certificates#
This step imports the CA-signed certificates of the collaborator and the aggregator into the workspace. We are back on the individual collaborator nodes for this step.
!fx collaborator certify --import agg_to_col_bob_signed_cert.zip
!fx collaborator certify --import agg_to_col_charlie_signed_cert.zip
Show code cell output
2025-03-26 05:53:32.521249: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:32.521951: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:32.525032: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:32.533748: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968412.549877 730 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968412.554226 730 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:32.570261: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Certificate updated in the PKI directory
✔️ OK
2025-03-26 05:53:36.266690: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:36.267400: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:36.270477: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:36.280297: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968416.295031 736 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968416.299313 736 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:36.313833: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Certificate updated in the PKI directory
✔️ OK
Start the simulation#
This step starts the simulation of the FL experiment. The aggregator will orchestrate the training process between the collaborators.
In practice, this command is executed on respective aggregator/collaborator nodes.
!fx aggregator start &\
fx collaborator start -n bob &\
fx collaborator start -n charlie
Show code cell output
2025-03-26 05:53:41.033282: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:41.034397: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:41.039118: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:41.053767: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-03-26 05:53:41.055433: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:41.058355: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:41.066872: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968421.079190 742 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968421.088311 742 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:41.099109: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-03-26 05:53:41.112763: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968421.152438 744 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968421.159847 744 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:41.184002: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-03-26 05:53:41.243550: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 05:53:41.244671: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:41.255250: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 05:53:41.284331: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1742968421.316243 743 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1742968421.324097 743 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-03-26 05:53:41.349872: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
[03/26/25 05:53:46] INFO FL-Plan hash is f9113436022351647107fab944c2673af3a63de34821ad3169a70f7c267a79f706745d8d7b4a360dbb795ceafa836f77 ]8;id=476346;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=873569;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#297\297]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=26007;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=34025;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#144\144]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=774406;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=73889;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#149\149]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: localhost
agg_port: 59905
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
Data = {'one': '1', 'bob': '0', 'charlie': '1'}
INFO 🧿 Starting a Collaborator Service. ]8;id=491858;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/collaborator.py\collaborator.py]8;;\:]8;id=924671;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/collaborator.py#82\82]8;;\
INFO Building `src.dataloader.KerasMNISTInMemory` Module. ]8;id=99775;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=791015;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
[03/26/25 05:53:46] INFO FL-Plan hash is f9113436022351647107fab944c2673af3a63de34821ad3169a70f7c267a79f706745d8d7b4a360dbb795ceafa836f77 ]8;id=717591;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=954515;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#297\297]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=216294;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=936278;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#144\144]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=203511;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=780619;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#149\149]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: localhost
agg_port: 59905
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
INFO 🧿 Starting the Aggregator Service. ]8;id=637924;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/aggregator.py\aggregator.py]8;;\:]8;id=402707;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/aggregator.py#94\94]8;;\
INFO Building `openfl.component.RandomGroupedAssigner` Module. ]8;id=723580;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=927011;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
[03/26/25 05:53:47] INFO Building `openfl.pipelines.NoCompressionPipeline` Module. ]8;id=439969;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=253323;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
INFO Importing 🡆 Object WaitForAllPolicy from openfl.component.aggregator.straggler_handling Module. ]8;id=511745;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=500600;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#259\259]8;;\
INFO Connector defaults: None ]8;id=658545;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=218412;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#338\338]8;;\
INFO Building `openfl.component.Aggregator` Module. ]8;id=747312;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=649056;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
INFO Persistent checkpoint is enabled, setting persistent db at path local_state/tensor.db ]8;id=396571;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=772796;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#149\149]8;;\
INFO Initializing persistent db at local_state/tensor.db ]8;id=35307;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=922491;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py#36\36]8;;\
[03/26/25 05:53:47] INFO MNIST > X_train Shape : (30000, 28, 28, 1) ]8;id=927045;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=465874;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#108\108]8;;\
INFO MNIST > y_train Shape : (30000,) ]8;id=4080;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=766369;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#109\109]8;;\
INFO MNIST > Train Samples : 30000 ]8;id=323490;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=413288;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#110\110]8;;\
INFO MNIST > Valid Samples : 5000 ]8;id=657040;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=405427;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#111\111]8;;\
INFO Building `src.taskrunner.KerasCNN` Module. ]8;id=70072;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=745170;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(activity_regularizer=activity_regularizer, **kwargs)
INFO Starting Aggregator gRPC Server ]8;id=182456;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_server.py\aggregator_server.py]8;;\:]8;id=944256;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_server.py#342\342]8;;\
2025-03-26 05:53:47.292191: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
[03/26/25 05:53:47] INFO FL-Plan hash is f9113436022351647107fab944c2673af3a63de34821ad3169a70f7c267a79f706745d8d7b4a360dbb795ceafa836f77 ]8;id=971698;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=405236;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#297\297]8;;\
INFO Parsing Federated Learning Plan : SUCCESS : ]8;id=145501;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=561985;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#144\144]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/plan/plan.yaml.
INFO aggregator: ]8;id=520067;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=132791;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#149\149]8;;\
settings:
best_state_path: save/best.pbuf
db_store_rounds: 2
init_state_path: save/init.pbuf
last_state_path: save/last.pbuf
persist_checkpoint: true
persistent_db_path: local_state/tensor.db
rounds_to_train: 10
template: openfl.component.Aggregator
assigner:
settings:
task_groups:
- name: learning
percentage: 1.0
tasks:
- aggregated_model_validation
- train
- locally_tuned_model_validation
- name: evaluation
percentage: 0
tasks:
- aggregated_model_validation
template: openfl.component.RandomGroupedAssigner
collaborator:
settings:
db_store_rounds: 1
opt_treatment: RESET
use_delta_updates: false
template: openfl.component.Collaborator
compression_pipeline:
settings: {}
template: openfl.pipelines.NoCompressionPipeline
data_loader:
settings:
batch_size: 256
collaborator_count: 2
data_group_name: mnist
template: src.dataloader.KerasMNISTInMemory
network:
settings:
agg_addr: localhost
agg_port: 59905
cert_folder: cert
client_reconnect_interval: 5
enable_atomic_connections: false
hash_salt: auto
require_client_auth: true
use_tls: true
template: openfl.federation.Network
task_runner:
settings: {}
template: src.taskrunner.KerasCNN
tasks:
aggregated_model_validation:
function: validate_task
kwargs:
apply: global
batch_size: 32
metrics:
- accuracy
locally_tuned_model_validation:
function: validate_task
kwargs:
apply: local
batch_size: 32
metrics:
- accuracy
settings: {}
train:
function: train_task
kwargs:
batch_size: 32
epochs: 1
metrics:
- loss
Data = {'one': '1', 'bob': '0', 'charlie': '1'}
INFO 🧿 Starting a Collaborator Service. ]8;id=634252;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/collaborator.py\collaborator.py]8;;\:]8;id=138244;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/interface/collaborator.py#82\82]8;;\
INFO Building `src.dataloader.KerasMNISTInMemory` Module. ]8;id=770178;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=32972;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
INFO Model: "sequential" ]8;id=402991;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/utils/summary_utils.py\summary_utils.py]8;;\:]8;id=123472;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/utils/summary_utils.py#389\389]8;;\
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type) ┃ Output Shape ┃ Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d (Conv2D) │ (None, 13, 13, 16) │ 272 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D) │ (None, 5, 5, 32) │ 8,224 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten) │ (None, 800) │ 0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense) │ (None, 100) │ 80,100 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense) │ (None, 10) │ 1,010 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 89,606 (350.02 KB)
Trainable params: 89,606 (350.02 KB)
Non-trainable params: 0 (0.00 B)
INFO Train Set Size : 30000 ]8;id=629362;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py\taskrunner.py]8;;\:]8;id=689409;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py#34\34]8;;\
INFO Valid Set Size : 5000 ]8;id=207544;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py\taskrunner.py]8;;\:]8;id=973601;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py#35\35]8;;\
INFO Building `openfl.pipelines.NoCompressionPipeline` Module. ]8;id=103939;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=307824;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
INFO Building `openfl.component.Collaborator` Module. ]8;id=17045;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=976395;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
WARNING Argument `device_assignment_policy` is deprecated and will be removed in the future. ]8;id=448927;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=776114;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#128\128]8;;\
INFO Requesting tasks... ]8;id=252855;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=553145;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending tasks to collaborator bob for round 0 ]8;id=852695;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=376478;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=567445;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=151836;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
INFO MNIST > X_train Shape : (30000, 28, 28, 1) ]8;id=196419;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=641761;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#108\108]8;;\
INFO MNIST > y_train Shape : (30000,) ]8;id=582710;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=261758;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#109\109]8;;\
INFO MNIST > Train Samples : 30000 ]8;id=619020;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=587960;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#110\110]8;;\
INFO MNIST > Valid Samples : 5000 ]8;id=248859;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py\mnist_utils.py]8;;\:]8;id=684039;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/mnist_utils.py#111\111]8;;\
INFO Building `src.taskrunner.KerasCNN` Module. ]8;id=189483;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=751751;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.
super().__init__(activity_regularizer=activity_regularizer, **kwargs)
2025-03-26 05:53:47.750547: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
INFO Model: "sequential" ]8;id=427679;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/utils/summary_utils.py\summary_utils.py]8;;\:]8;id=399115;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/utils/summary_utils.py#389\389]8;;\
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type) ┃ Output Shape ┃ Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d (Conv2D) │ (None, 13, 13, 16) │ 272 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D) │ (None, 5, 5, 32) │ 8,224 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten) │ (None, 800) │ 0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense) │ (None, 100) │ 80,100 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense) │ (None, 10) │ 1,010 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 89,606 (350.02 KB)
Trainable params: 89,606 (350.02 KB)
Non-trainable params: 0 (0.00 B)
INFO Train Set Size : 30000 ]8;id=956469;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py\taskrunner.py]8;;\:]8;id=343385;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py#34\34]8;;\
INFO Valid Set Size : 5000 ]8;id=470649;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py\taskrunner.py]8;;\:]8;id=109089;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/docs/tutorials/mnist_example/src/taskrunner.py#35\35]8;;\
INFO Building `openfl.pipelines.NoCompressionPipeline` Module. ]8;id=932149;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=489822;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
[03/26/25 05:53:48] INFO Building `openfl.component.Collaborator` Module. ]8;id=675897;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py\plan.py]8;;\:]8;id=619955;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/plan/plan.py#235\235]8;;\
WARNING Argument `device_assignment_policy` is deprecated and will be removed in the future. ]8;id=385438;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=248401;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#128\128]8;;\
INFO Requesting tasks... ]8;id=437209;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=966922;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:53:48] INFO Sending tasks to collaborator charlie for round 0 ]8;id=300302;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=567863;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=138090;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=844984;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 1s 721ms/step - accuracy: 0.0938 - loss: 2.3225
12/Unknown 1s 5ms/step - accuracy: 0.0793 - loss: 2.3116
26/Unknown 1s 4ms/step - accuracy: 0.0829 - loss: 2.3089
42/Unknown 1s 4ms/step - accuracy: 0.0858 - loss: 2.3078
58/Unknown 1s 4ms/step - accuracy: 0.0861 - loss: 2.3071
70/Unknown 1s 4ms/step - accuracy: 0.0863 - loss: 2.3067
84/Unknown 1s 4ms/step - accuracy: 0.0868 - loss: 2.3063
97/Unknown 1s 4ms/step - accuracy: 0.0871 - loss: 2.3061
112/Unknown 1s 4ms/step - accuracy: 0.0874 - loss: 2.3059
126/Unknown 1s 4ms/step - accuracy: 0.0875 - loss: 2.3057
12/Unknown 1s 5ms/step - accuracy: 0.0667 - loss: 2.3128
155/Unknown 1s 4ms/step - accuracy: 0.0873 - loss: 2.30562025-03-26 05:53:49.111698: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.0874 - loss: 2.3056
25/Unknown 1s 4ms/step - accuracy: 0.0689 - loss: 2.3108
[03/26/25 05:53:49] INFO Collaborator bob is sending task results for aggregated_model_validation, round 0 ]8;id=500452;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=353619;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:53:49] INFO Run 0 epoch of 0 round ]8;id=135307;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=56213;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
39/Unknown 1s 4ms/step - accuracy: 0.0726 - loss: 2.3090
53/Unknown 1s 4ms/step - accuracy: 0.0760 - loss: 2.3081
67/Unknown 1s 4ms/step - accuracy: 0.0779 - loss: 2.3076
82/Unknown 1s 4ms/step - accuracy: 0.0792 - loss: 2.3072
99/Unknown 1s 4ms/step - accuracy: 0.0804 - loss: 2.3069
117/Unknown 1s 4ms/step - accuracy: 0.0815 - loss: 2.3065
135/Unknown 1s 3ms/step - accuracy: 0.0827 - loss: 2.3061
153/Unknown 1s 3ms/step - accuracy: 0.0834 - loss: 2.30592025-03-26 05:53:49.551120: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.0836 - loss: 2.3059
INFO Collaborator charlie is sending task results for aggregated_model_validation, round 0 ]8;id=983639;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=897059;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:53:49] INFO Run 0 epoch of 0 round ]8;id=910060;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=491771;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
2025-03-26 05:53:57.159545: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
938/938 - 8s - 8ms/step - accuracy: 0.9095 - loss: 0.3008
[03/26/25 05:53:57] INFO Collaborator bob is sending task results for train, round 0 ]8;id=115871;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=183686;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 47ms/step - accuracy: 0.9688 - loss: 0.07812025-03-26 05:53:57.458995: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
938/938 - 8s - 8ms/step - accuracy: 0.9408 - loss: 0.2028
15/Unknown 0s 4ms/step - accuracy: 0.9627 - loss: 0.1068 INFO Collaborator charlie is sending task results for train, round 0 ]8;id=260314;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=939609;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
31/Unknown 0s 4ms/step - accuracy: 0.9611 - loss: 0.1112
46/Unknown 0s 4ms/step - accuracy: 0.9617 - loss: 0.1126
1/Unknown 0s 41ms/step - accuracy: 0.9688 - loss: 0.2079
15/Unknown 0s 4ms/step - accuracy: 0.9423 - loss: 0.2237
30/Unknown 0s 4ms/step - accuracy: 0.9381 - loss: 0.2360
44/Unknown 0s 4ms/step - accuracy: 0.9378 - loss: 0.2313
58/Unknown 0s 4ms/step - accuracy: 0.9382 - loss: 0.2255
73/Unknown 0s 4ms/step - accuracy: 0.9392 - loss: 0.2205
88/Unknown 0s 4ms/step - accuracy: 0.9400 - loss: 0.2160
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9645 - loss: 0.1109
[03/26/25 05:53:58] INFO Collaborator bob is sending task results for locally_tuned_model_validation, round 0 ]8;id=606980;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=367821;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 0: Collaborators that have completed all tasks: ['bob'] ]8;id=98479;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=315439;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
[03/26/25 05:53:58] INFO Requesting tasks... ]8;id=423065;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=217357;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
103/Unknown 0s 4ms/step - accuracy: 0.9407 - loss: 0.2122
128/Unknown 0s 3ms/step - accuracy: 0.9413 - loss: 0.2071
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9420 - loss: 0.2029
INFO Collaborator charlie is sending task results for locally_tuned_model_validation, round 0 ]8;id=515625;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=645626;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 0: Collaborators that have completed all tasks: ['bob', 'charlie'] ]8;id=871760;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=91687;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
INFO Round 0: saved the best model with score 0.087900 ]8;id=102589;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=372903;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1119\1119]8;;\
INFO Saving round 0 model... ]8;id=864852;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=195790;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1157\1157]8;;\
INFO Committed model for round 0, saved 8 model tensors and 26 next round model tensors with best_score ]8;id=156043;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=241904;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py#220\220]8;;\
0.08790000155568123
INFO Persist model and clean task result for round 0 ]8;id=953356;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=978878;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#384\384]8;;\
INFO Starting round 1... ]8;id=423539;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=671963;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1170\1170]8;;\
[03/26/25 05:53:58] INFO Requesting tasks... ]8;id=290704;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=77237;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending tasks to collaborator charlie for round 1 ]8;id=27613;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=356210;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=858424;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=958892;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 27ms/step - accuracy: 0.9688 - loss: 0.0667
25/Unknown 0s 2ms/step - accuracy: 0.9705 - loss: 0.0868
49/Unknown 0s 2ms/step - accuracy: 0.9685 - loss: 0.0937
73/Unknown 0s 2ms/step - accuracy: 0.9678 - loss: 0.0972
97/Unknown 0s 2ms/step - accuracy: 0.9672 - loss: 0.0995
121/Unknown 0s 2ms/step - accuracy: 0.9667 - loss: 0.1017
145/Unknown 0s 2ms/step - accuracy: 0.9665 - loss: 0.10332025-03-26 05:53:58.881039: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9664 - loss: 0.1039
INFO Collaborator charlie is sending task results for aggregated_model_validation, round 1 ]8;id=935814;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=780228;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Run 0 epoch of 1 round ]8;id=382369;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=650993;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 3s - 4ms/step - accuracy: 0.9833 - loss: 0.0573
[03/26/25 05:54:02] INFO Collaborator charlie is sending task results for train, round 1 ]8;id=549569;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=528742;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 25ms/step - accuracy: 0.9688 - loss: 0.0942
25/Unknown 0s 2ms/step - accuracy: 0.9738 - loss: 0.0887
48/Unknown 0s 2ms/step - accuracy: 0.9725 - loss: 0.0967
73/Unknown 0s 2ms/step - accuracy: 0.9713 - loss: 0.1018
97/Unknown 0s 2ms/step - accuracy: 0.9703 - loss: 0.1035
122/Unknown 0s 2ms/step - accuracy: 0.9698 - loss: 0.1041
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9694 - loss: 0.1057
INFO Collaborator charlie is sending task results for locally_tuned_model_validation, round 1 ]8;id=490869;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=220116;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 1: Collaborators that have completed all tasks: ['charlie'] ]8;id=205022;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=288462;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
[03/26/25 05:54:02] INFO Requesting tasks... ]8;id=260252;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=293626;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:54:08] INFO Requesting tasks... ]8;id=400895;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=71546;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:54:08] INFO Sending tasks to collaborator bob for round 1 ]8;id=59735;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=520889;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=215246;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=427104;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 26ms/step - accuracy: 1.0000 - loss: 0.0325
25/Unknown 0s 2ms/step - accuracy: 0.9801 - loss: 0.0980
49/Unknown 0s 2ms/step - accuracy: 0.9743 - loss: 0.1065
73/Unknown 0s 2ms/step - accuracy: 0.9726 - loss: 0.1080
97/Unknown 0s 2ms/step - accuracy: 0.9718 - loss: 0.1086
121/Unknown 0s 2ms/step - accuracy: 0.9714 - loss: 0.1081
145/Unknown 0s 2ms/step - accuracy: 0.9709 - loss: 0.10792025-03-26 05:54:08.548651: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9707 - loss: 0.1080
INFO Collaborator bob is sending task results for aggregated_model_validation, round 1 ]8;id=222794;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=397811;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Run 0 epoch of 1 round ]8;id=692977;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=5060;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 3s - 4ms/step - accuracy: 0.9640 - loss: 0.1158
[03/26/25 05:54:11] INFO Collaborator bob is sending task results for train, round 1 ]8;id=792947;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=431613;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 26ms/step - accuracy: 0.9688 - loss: 0.1319
24/Unknown 0s 2ms/step - accuracy: 0.9765 - loss: 0.0683
40/Unknown 0s 3ms/step - accuracy: 0.9771 - loss: 0.0665
61/Unknown 0s 3ms/step - accuracy: 0.9780 - loss: 0.0651
85/Unknown 0s 2ms/step - accuracy: 0.9781 - loss: 0.0646
108/Unknown 0s 2ms/step - accuracy: 0.9781 - loss: 0.0646
133/Unknown 0s 2ms/step - accuracy: 0.9782 - loss: 0.0647
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9782 - loss: 0.0647
[03/26/25 05:54:12] INFO Collaborator bob is sending task results for locally_tuned_model_validation, round 1 ]8;id=786161;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=340226;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 1: Collaborators that have completed all tasks: ['charlie', 'bob'] ]8;id=755024;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=930780;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
INFO Round 1: saved the best model with score 0.966300 ]8;id=474614;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=254200;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1119\1119]8;;\
INFO Saving round 1 model... ]8;id=568358;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=151797;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1157\1157]8;;\
INFO Committed model for round 1, saved 8 model tensors and 26 next round model tensors with best_score ]8;id=106293;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=179690;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py#220\220]8;;\
0.96629998087883
INFO Persist model and clean task result for round 1 ]8;id=37956;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=827927;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#384\384]8;;\
INFO Starting round 2... ]8;id=889539;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=10660;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1170\1170]8;;\
[03/26/25 05:54:12] INFO Requesting tasks... ]8;id=358999;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=977801;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending tasks to collaborator bob for round 2 ]8;id=71045;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=33454;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=530591;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=742802;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
[03/26/25 05:54:12] INFO Requesting tasks... ]8;id=605150;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=813510;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending tasks to collaborator charlie for round 2 ]8;id=368925;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=666091;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=193903;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=800807;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 47ms/step - accuracy: 0.9688 - loss: 0.1354
16/Unknown 0s 4ms/step - accuracy: 0.9874 - loss: 0.0439
32/Unknown 0s 3ms/step - accuracy: 0.9850 - loss: 0.0470
48/Unknown 0s 3ms/step - accuracy: 0.9843 - loss: 0.0502
65/Unknown 0s 3ms/step - accuracy: 0.9842 - loss: 0.0521
80/Unknown 0s 3ms/step - accuracy: 0.9838 - loss: 0.0536
94/Unknown 0s 3ms/step - accuracy: 0.9833 - loss: 0.0547
108/Unknown 0s 3ms/step - accuracy: 0.9831 - loss: 0.0553
122/Unknown 0s 3ms/step - accuracy: 0.9830 - loss: 0.0557
136/Unknown 1s 3ms/step - accuracy: 0.9830 - loss: 0.0559
150/Unknown 1s 3ms/step - accuracy: 0.9830 - loss: 0.0560
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9830 - loss: 0.0561
[03/26/25 05:54:13] INFO Collaborator bob is sending task results for aggregated_model_validation, round 2 ]8;id=527566;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=152604;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
82/Unknown 0s 4ms/step - accuracy: 0.9791 - loss: 0.0684
[03/26/25 05:54:13] INFO Run 0 epoch of 2 round ]8;id=159922;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=963161;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
96/Unknown 0s 4ms/step - accuracy: 0.9790 - loss: 0.0684
112/Unknown 0s 4ms/step - accuracy: 0.9791 - loss: 0.0679
126/Unknown 1s 4ms/step - accuracy: 0.9794 - loss: 0.0672
141/Unknown 1s 4ms/step - accuracy: 0.9796 - loss: 0.0665
155/Unknown 1s 4ms/step - accuracy: 0.9797 - loss: 0.0660/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9797 - loss: 0.0659
INFO Collaborator charlie is sending task results for aggregated_model_validation, round 2 ]8;id=623868;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=916960;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:54:13] INFO Run 0 epoch of 2 round ]8;id=964194;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=547394;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
2025-03-26 05:54:19.769075: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
938/938 - 6s - 7ms/step - accuracy: 0.9734 - loss: 0.0831
[03/26/25 05:54:19] INFO Collaborator bob is sending task results for train, round 2 ]8;id=483450;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=312180;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
2025-03-26 05:54:20.027596: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
1/Unknown 0s 55ms/step - accuracy: 1.0000 - loss: 0.0047938/938 - 6s - 7ms/step - accuracy: 0.9889 - loss: 0.0363
15/Unknown 0s 4ms/step - accuracy: 0.9870 - loss: 0.0514 [03/26/25 05:54:20] INFO Collaborator charlie is sending task results for train, round 2 ]8;id=379577;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=865760;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
30/Unknown 0s 3ms/step - accuracy: 0.9858 - loss: 0.0502
47/Unknown 0s 3ms/step - accuracy: 0.9839 - loss: 0.0523
64/Unknown 0s 3ms/step - accuracy: 0.9822 - loss: 0.0547
79/Unknown 0s 3ms/step - accuracy: 0.9815 - loss: 0.0555
93/Unknown 0s 3ms/step - accuracy: 0.9814 - loss: 0.0556
107/Unknown 0s 3ms/step - accuracy: 0.9813 - loss: 0.0558
121/Unknown 0s 3ms/step - accuracy: 0.9812 - loss: 0.0560
134/Unknown 1s 3ms/step - accuracy: 0.9810 - loss: 0.0565
148/Unknown 1s 3ms/step - accuracy: 0.9808 - loss: 0.0571
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9806 - loss: 0.0576
85/Unknown 0s 4ms/step - accuracy: 0.9676 - loss: 0.0902 INFO Collaborator bob is sending task results for locally_tuned_model_validation, round 2 ]8;id=681753;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=960279;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 2: Collaborators that have completed all tasks: ['bob'] ]8;id=99710;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=578857;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
[03/26/25 05:54:20] INFO Requesting tasks... ]8;id=134093;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=949754;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
107/Unknown 0s 3ms/step - accuracy: 0.9684 - loss: 0.0897
131/Unknown 0s 3ms/step - accuracy: 0.9690 - loss: 0.0897
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9696 - loss: 0.0891
INFO Collaborator charlie is sending task results for locally_tuned_model_validation, round 2 ]8;id=779689;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=286799;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 2: Collaborators that have completed all tasks: ['bob', 'charlie'] ]8;id=878405;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=655210;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
INFO Round 2: saved the best model with score 0.981900 ]8;id=206818;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=607093;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1119\1119]8;;\
INFO Saving round 2 model... ]8;id=31414;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=374731;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1157\1157]8;;\
INFO Committed model for round 2, saved 8 model tensors and 26 next round model tensors with best_score ]8;id=129340;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=940630;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py#220\220]8;;\
0.9819000065326691
INFO Persist model and clean task result for round 2 ]8;id=418430;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=416316;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#384\384]8;;\
INFO Starting round 3... ]8;id=872428;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=413153;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1170\1170]8;;\
[03/26/25 05:54:20] INFO Requesting tasks... ]8;id=675270;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=274879;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending tasks to collaborator charlie for round 3 ]8;id=978530;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=331998;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=477182;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=172879;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 25ms/step - accuracy: 1.0000 - loss: 0.0210
25/Unknown 0s 2ms/step - accuracy: 0.9828 - loss: 0.0636
50/Unknown 0s 2ms/step - accuracy: 0.9836 - loss: 0.0611
74/Unknown 0s 2ms/step - accuracy: 0.9834 - loss: 0.0601
99/Unknown 0s 2ms/step - accuracy: 0.9832 - loss: 0.0597
123/Unknown 0s 2ms/step - accuracy: 0.9830 - loss: 0.0597
147/Unknown 0s 2ms/step - accuracy: 0.9827 - loss: 0.0596
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9826 - loss: 0.0597
[03/26/25 05:54:21] INFO Collaborator charlie is sending task results for aggregated_model_validation, round 3 ]8;id=426316;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=4106;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:54:21] INFO Run 0 epoch of 3 round ]8;id=704056;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=4304;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 3s - 4ms/step - accuracy: 0.9915 - loss: 0.0265
[03/26/25 05:54:26] INFO Collaborator charlie is sending task results for train, round 3 ]8;id=597855;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=524223;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 25ms/step - accuracy: 0.9688 - loss: 0.0665
25/Unknown 0s 2ms/step - accuracy: 0.9695 - loss: 0.0917
49/Unknown 0s 2ms/step - accuracy: 0.9732 - loss: 0.0864
73/Unknown 0s 2ms/step - accuracy: 0.9745 - loss: 0.0855
97/Unknown 0s 2ms/step - accuracy: 0.9752 - loss: 0.0852
121/Unknown 0s 2ms/step - accuracy: 0.9752 - loss: 0.0855
145/Unknown 0s 2ms/step - accuracy: 0.9750 - loss: 0.0862
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9749 - loss: 0.0864
INFO Collaborator charlie is sending task results for locally_tuned_model_validation, round 3 ]8;id=136000;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=186101;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 3: Collaborators that have completed all tasks: ['charlie'] ]8;id=460236;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=855317;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
[03/26/25 05:54:26] INFO Requesting tasks... ]8;id=842725;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=994420;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:54:30] INFO Requesting tasks... ]8;id=623499;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=703880;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:54:30] INFO Sending tasks to collaborator bob for round 3 ]8;id=199531;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=636413;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=818483;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=571922;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 26ms/step - accuracy: 0.9375 - loss: 0.1725
26/Unknown 0s 2ms/step - accuracy: 0.9821 - loss: 0.0568
49/Unknown 0s 2ms/step - accuracy: 0.9837 - loss: 0.0534
73/Unknown 0s 2ms/step - accuracy: 0.9839 - loss: 0.0526
97/Unknown 0s 2ms/step - accuracy: 0.9837 - loss: 0.0516
121/Unknown 0s 2ms/step - accuracy: 0.9835 - loss: 0.0514
145/Unknown 0s 2ms/step - accuracy: 0.9833 - loss: 0.0516/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9832 - loss: 0.0518
[03/26/25 05:54:31] INFO Collaborator bob is sending task results for aggregated_model_validation, round 3 ]8;id=648030;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=15234;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:54:31] INFO Run 0 epoch of 3 round ]8;id=378422;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=744908;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 3s - 4ms/step - accuracy: 0.9794 - loss: 0.0667
[03/26/25 05:54:34] INFO Collaborator bob is sending task results for train, round 3 ]8;id=264270;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=103352;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 27ms/step - accuracy: 1.0000 - loss: 0.0250
25/Unknown 0s 2ms/step - accuracy: 0.9931 - loss: 0.0309
49/Unknown 0s 2ms/step - accuracy: 0.9915 - loss: 0.0323
74/Unknown 0s 2ms/step - accuracy: 0.9907 - loss: 0.0324
99/Unknown 0s 2ms/step - accuracy: 0.9900 - loss: 0.0335
122/Unknown 0s 2ms/step - accuracy: 0.9892 - loss: 0.0347
146/Unknown 0s 2ms/step - accuracy: 0.9884 - loss: 0.0363
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9882 - loss: 0.0369
INFO Collaborator bob is sending task results for locally_tuned_model_validation, round 3 ]8;id=799649;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=978775;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 3: Collaborators that have completed all tasks: ['charlie', 'bob'] ]8;id=608657;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=704270;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
[03/26/25 05:54:35] INFO Saving round 3 model... ]8;id=328842;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=559052;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1157\1157]8;;\
INFO Committed model for round 3, saved 8 model tensors and 26 next round model tensors with best_score ]8;id=365211;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=406429;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py#220\220]8;;\
0.9819000065326691
INFO Persist model and clean task result for round 3 ]8;id=851896;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=287080;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#384\384]8;;\
INFO Starting round 4... ]8;id=869888;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=454527;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1170\1170]8;;\
[03/26/25 05:54:35] INFO Requesting tasks... ]8;id=445418;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=273680;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending tasks to collaborator bob for round 4 ]8;id=654070;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=36438;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=520524;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=897582;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 27ms/step - accuracy: 1.0000 - loss: 0.0241
24/Unknown 0s 2ms/step - accuracy: 0.9861 - loss: 0.0351
48/Unknown 0s 2ms/step - accuracy: 0.9848 - loss: 0.0383
73/Unknown 0s 2ms/step - accuracy: 0.9847 - loss: 0.0394
97/Unknown 0s 2ms/step - accuracy: 0.9846 - loss: 0.0402
122/Unknown 0s 2ms/step - accuracy: 0.9847 - loss: 0.0401
147/Unknown 0s 2ms/step - accuracy: 0.9848 - loss: 0.0402
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9848 - loss: 0.0404
INFO Collaborator bob is sending task results for aggregated_model_validation, round 4 ]8;id=405731;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=868208;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Run 0 epoch of 4 round ]8;id=213013;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=890898;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
[03/26/25 05:54:36] INFO Requesting tasks... ]8;id=152263;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=529079;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:54:36] INFO Sending tasks to collaborator charlie for round 4 ]8;id=982397;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=782550;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=582844;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=966130;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 48ms/step - accuracy: 0.9688 - loss: 0.1773
13/Unknown 0s 4ms/step - accuracy: 0.9775 - loss: 0.0881
25/Unknown 0s 4ms/step - accuracy: 0.9816 - loss: 0.0716
38/Unknown 0s 4ms/step - accuracy: 0.9834 - loss: 0.0631
51/Unknown 0s 4ms/step - accuracy: 0.9846 - loss: 0.0579
63/Unknown 0s 4ms/step - accuracy: 0.9848 - loss: 0.0568
75/Unknown 0s 4ms/step - accuracy: 0.9846 - loss: 0.0566
88/Unknown 0s 4ms/step - accuracy: 0.9845 - loss: 0.0565
100/Unknown 0s 4ms/step - accuracy: 0.9843 - loss: 0.0564
112/Unknown 1s 4ms/step - accuracy: 0.9843 - loss: 0.0564
124/Unknown 1s 4ms/step - accuracy: 0.9843 - loss: 0.0563
136/Unknown 1s 4ms/step - accuracy: 0.9843 - loss: 0.0561
149/Unknown 1s 4ms/step - accuracy: 0.9843 - loss: 0.0558
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9843 - loss: 0.0556
[03/26/25 05:54:37] INFO Collaborator charlie is sending task results for aggregated_model_validation, round 4 ]8;id=342398;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=924137;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:54:37] INFO Run 0 epoch of 4 round ]8;id=382522;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=908237;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 5s - 6ms/step - accuracy: 0.9826 - loss: 0.0531
[03/26/25 05:54:41] INFO Collaborator bob is sending task results for train, round 4 ]8;id=621396;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=887658;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 51ms/step - accuracy: 1.0000 - loss: 0.0043
13/Unknown 0s 4ms/step - accuracy: 0.9910 - loss: 0.0277
26/Unknown 0s 4ms/step - accuracy: 0.9855 - loss: 0.0400
38/Unknown 0s 4ms/step - accuracy: 0.9837 - loss: 0.0484
51/Unknown 0s 4ms/step - accuracy: 0.9829 - loss: 0.0516
63/Unknown 0s 4ms/step - accuracy: 0.9825 - loss: 0.0530
75/Unknown 0s 4ms/step - accuracy: 0.9822 - loss: 0.0539
87/Unknown 0s 4ms/step - accuracy: 0.9820 - loss: 0.0547
99/Unknown 0s 4ms/step - accuracy: 0.9819 - loss: 0.0553
111/Unknown 1s 4ms/step - accuracy: 0.9819 - loss: 0.0555
124/Unknown 1s 4ms/step - accuracy: 0.9819 - loss: 0.0558
136/Unknown 1s 4ms/step - accuracy: 0.9819 - loss: 0.0560
147/Unknown 1s 4ms/step - accuracy: 0.9819 - loss: 0.0564
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9819 - loss: 0.0566
[03/26/25 05:54:42] INFO Collaborator bob is sending task results for locally_tuned_model_validation, round 4 ]8;id=545582;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=647298;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 4: Collaborators that have completed all tasks: ['bob'] ]8;id=405867;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=489709;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
[03/26/25 05:54:42] INFO Requesting tasks... ]8;id=858492;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=372805;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
938/938 - 5s - 6ms/step - accuracy: 0.9931 - loss: 0.0213
[03/26/25 05:54:43] INFO Collaborator charlie is sending task results for train, round 4 ]8;id=657475;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=560803;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 25ms/step - accuracy: 0.9375 - loss: 0.4354
25/Unknown 0s 2ms/step - accuracy: 0.9732 - loss: 0.1297
49/Unknown 0s 2ms/step - accuracy: 0.9751 - loss: 0.1077
73/Unknown 0s 2ms/step - accuracy: 0.9760 - loss: 0.0983
97/Unknown 0s 2ms/step - accuracy: 0.9765 - loss: 0.0929
121/Unknown 0s 2ms/step - accuracy: 0.9769 - loss: 0.0894
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9775 - loss: 0.0855
INFO Collaborator charlie is sending task results for locally_tuned_model_validation, round 4 ]8;id=71933;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=927685;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 4: Collaborators that have completed all tasks: ['bob', 'charlie'] ]8;id=990139;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=665731;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
INFO Round 4: saved the best model with score 0.984500 ]8;id=853892;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=806994;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1119\1119]8;;\
INFO Saving round 4 model... ]8;id=2735;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=389116;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1157\1157]8;;\
INFO Committed model for round 4, saved 8 model tensors and 26 next round model tensors with best_score ]8;id=703275;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=668761;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py#220\220]8;;\
0.984499990940094
INFO Persist model and clean task result for round 4 ]8;id=276321;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=891399;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#384\384]8;;\
INFO Starting round 5... ]8;id=817463;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=654212;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1170\1170]8;;\
[03/26/25 05:54:43] INFO Requesting tasks... ]8;id=628306;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=710722;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending tasks to collaborator charlie for round 5 ]8;id=735436;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=731496;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=131455;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=782537;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 26ms/step - accuracy: 1.0000 - loss: 6.4880e-04
25/Unknown 0s 2ms/step - accuracy: 0.9866 - loss: 0.0494
50/Unknown 0s 2ms/step - accuracy: 0.9863 - loss: 0.0471
75/Unknown 0s 2ms/step - accuracy: 0.9869 - loss: 0.0462
99/Unknown 0s 2ms/step - accuracy: 0.9868 - loss: 0.0475
123/Unknown 0s 2ms/step - accuracy: 0.9864 - loss: 0.0484
147/Unknown 0s 2ms/step - accuracy: 0.9862 - loss: 0.04852025-03-26 05:54:44.290531: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9862 - loss: 0.0484
[03/26/25 05:54:44] INFO Collaborator charlie is sending task results for aggregated_model_validation, round 5 ]8;id=204373;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=181484;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:54:44] INFO Run 0 epoch of 5 round ]8;id=566049;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=200672;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 3s - 4ms/step - accuracy: 0.9946 - loss: 0.0173
[03/26/25 05:54:47] INFO Collaborator charlie is sending task results for train, round 5 ]8;id=126464;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=856614;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 27ms/step - accuracy: 0.9375 - loss: 0.0917
25/Unknown 0s 2ms/step - accuracy: 0.9810 - loss: 0.0436
49/Unknown 0s 2ms/step - accuracy: 0.9837 - loss: 0.0417
73/Unknown 0s 2ms/step - accuracy: 0.9840 - loss: 0.0455
97/Unknown 0s 2ms/step - accuracy: 0.9838 - loss: 0.0497
121/Unknown 0s 2ms/step - accuracy: 0.9837 - loss: 0.0521
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9832 - loss: 0.0565
[03/26/25 05:54:48] INFO Collaborator charlie is sending task results for locally_tuned_model_validation, round 5 ]8;id=529559;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=824302;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 5: Collaborators that have completed all tasks: ['charlie'] ]8;id=391630;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=668124;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
[03/26/25 05:54:48] INFO Requesting tasks... ]8;id=867250;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=278787;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:54:52] INFO Requesting tasks... ]8;id=565492;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=69309;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:54:52] INFO Sending tasks to collaborator bob for round 5 ]8;id=32853;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=528880;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=774173;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=932964;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 26ms/step - accuracy: 1.0000 - loss: 0.0183
25/Unknown 0s 2ms/step - accuracy: 0.9823 - loss: 0.0677
49/Unknown 0s 2ms/step - accuracy: 0.9836 - loss: 0.0580
73/Unknown 0s 2ms/step - accuracy: 0.9841 - loss: 0.0548
96/Unknown 0s 2ms/step - accuracy: 0.9845 - loss: 0.0525
120/Unknown 0s 2ms/step - accuracy: 0.9847 - loss: 0.0514
144/Unknown 0s 2ms/step - accuracy: 0.9850 - loss: 0.05012025-03-26 05:54:52.548232: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence
[[{{node IteratorGetNext}}]]
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9852 - loss: 0.0494
INFO Collaborator bob is sending task results for aggregated_model_validation, round 5 ]8;id=209796;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=400480;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Run 0 epoch of 5 round ]8;id=612551;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=123616;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 3s - 4ms/step - accuracy: 0.9837 - loss: 0.0483
[03/26/25 05:54:56] INFO Collaborator bob is sending task results for train, round 5 ]8;id=484977;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=990434;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 26ms/step - accuracy: 0.9688 - loss: 0.1303
25/Unknown 0s 2ms/step - accuracy: 0.9921 - loss: 0.0351
48/Unknown 0s 2ms/step - accuracy: 0.9900 - loss: 0.0361
72/Unknown 0s 2ms/step - accuracy: 0.9885 - loss: 0.0372
97/Unknown 0s 2ms/step - accuracy: 0.9873 - loss: 0.0392
121/Unknown 0s 2ms/step - accuracy: 0.9867 - loss: 0.0404
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9863 - loss: 0.0411
INFO Collaborator bob is sending task results for locally_tuned_model_validation, round 5 ]8;id=391474;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=715003;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 5: Collaborators that have completed all tasks: ['charlie', 'bob'] ]8;id=226773;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=776561;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
INFO Round 5: saved the best model with score 0.986900 ]8;id=120266;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=926056;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1119\1119]8;;\
INFO Saving round 5 model... ]8;id=949707;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=810584;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1157\1157]8;;\
INFO Committed model for round 5, saved 8 model tensors and 26 next round model tensors with best_score ]8;id=531088;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=958244;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py#220\220]8;;\
0.9869000017642975
INFO Persist model and clean task result for round 5 ]8;id=397163;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=767576;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#384\384]8;;\
INFO Starting round 6... ]8;id=617102;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=246145;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1170\1170]8;;\
[03/26/25 05:54:56] INFO Requesting tasks... ]8;id=110138;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=640682;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending tasks to collaborator bob for round 6 ]8;id=773595;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=867812;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=641989;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=206797;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 27ms/step - accuracy: 1.0000 - loss: 7.9760e-04
24/Unknown 0s 2ms/step - accuracy: 0.9827 - loss: 0.0539
47/Unknown 0s 2ms/step - accuracy: 0.9837 - loss: 0.0492
70/Unknown 0s 2ms/step - accuracy: 0.9844 - loss: 0.0455
94/Unknown 0s 2ms/step - accuracy: 0.9849 - loss: 0.0442
117/Unknown 0s 2ms/step - accuracy: 0.9853 - loss: 0.0432
141/Unknown 0s 2ms/step - accuracy: 0.9857 - loss: 0.0426/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9858 - loss: 0.0423
[03/26/25 05:54:57] INFO Collaborator bob is sending task results for aggregated_model_validation, round 6 ]8;id=972293;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=541717;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:54:57] INFO Run 0 epoch of 6 round ]8;id=628434;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=933249;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
[03/26/25 05:54:58] INFO Requesting tasks... ]8;id=263786;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=60008;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:54:58] INFO Sending tasks to collaborator charlie for round 6 ]8;id=159787;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=882080;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=227822;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=845179;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 52ms/step - accuracy: 0.9688 - loss: 0.1319
13/Unknown 0s 4ms/step - accuracy: 0.9868 - loss: 0.0438
24/Unknown 0s 5ms/step - accuracy: 0.9877 - loss: 0.0373
34/Unknown 0s 5ms/step - accuracy: 0.9879 - loss: 0.0352
45/Unknown 0s 5ms/step - accuracy: 0.9881 - loss: 0.0341
56/Unknown 0s 5ms/step - accuracy: 0.9880 - loss: 0.0348
66/Unknown 0s 5ms/step - accuracy: 0.9876 - loss: 0.0362
77/Unknown 0s 5ms/step - accuracy: 0.9874 - loss: 0.0373
90/Unknown 0s 5ms/step - accuracy: 0.9870 - loss: 0.0385
103/Unknown 1s 5ms/step - accuracy: 0.9867 - loss: 0.0397
116/Unknown 1s 4ms/step - accuracy: 0.9865 - loss: 0.0406
130/Unknown 1s 4ms/step - accuracy: 0.9864 - loss: 0.0411
143/Unknown 1s 4ms/step - accuracy: 0.9862 - loss: 0.0415
155/Unknown 1s 4ms/step - accuracy: 0.9862 - loss: 0.0417/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9862 - loss: 0.0417
[03/26/25 05:54:59] INFO Collaborator charlie is sending task results for aggregated_model_validation, round 6 ]8;id=83017;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=739874;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:54:59] INFO Run 0 epoch of 6 round ]8;id=423030;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=472169;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 6s - 6ms/step - accuracy: 0.9868 - loss: 0.0412
[03/26/25 05:55:03] INFO Collaborator bob is sending task results for train, round 6 ]8;id=802889;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=456070;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 45ms/step - accuracy: 0.9688 - loss: 0.0374
14/Unknown 0s 4ms/step - accuracy: 0.9883 - loss: 0.0324
27/Unknown 0s 4ms/step - accuracy: 0.9901 - loss: 0.0332
40/Unknown 0s 4ms/step - accuracy: 0.9909 - loss: 0.0336
52/Unknown 0s 4ms/step - accuracy: 0.9909 - loss: 0.0348
63/Unknown 0s 4ms/step - accuracy: 0.9909 - loss: 0.0354
75/Unknown 0s 4ms/step - accuracy: 0.9909 - loss: 0.0354
86/Unknown 0s 4ms/step - accuracy: 0.9908 - loss: 0.0354
97/Unknown 0s 4ms/step - accuracy: 0.9907 - loss: 0.0354
109/Unknown 1s 4ms/step - accuracy: 0.9905 - loss: 0.0353
120/Unknown 1s 4ms/step - accuracy: 0.9904 - loss: 0.0354
131/Unknown 1s 4ms/step - accuracy: 0.9903 - loss: 0.0355
143/Unknown 1s 4ms/step - accuracy: 0.9900 - loss: 0.0359
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9896 - loss: 0.0365
INFO Collaborator bob is sending task results for locally_tuned_model_validation, round 6 ]8;id=714288;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=466692;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 6: Collaborators that have completed all tasks: ['bob'] ]8;id=541506;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=19796;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
[03/26/25 05:55:03] INFO Requesting tasks... ]8;id=615124;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=877361;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
938/938 - 6s - 6ms/step - accuracy: 0.9955 - loss: 0.0142
[03/26/25 05:55:05] INFO Collaborator charlie is sending task results for train, round 6 ]8;id=934999;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=425044;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 26ms/step - accuracy: 1.0000 - loss: 0.0083
24/Unknown 0s 2ms/step - accuracy: 0.9886 - loss: 0.0287
47/Unknown 0s 2ms/step - accuracy: 0.9860 - loss: 0.0393
70/Unknown 0s 2ms/step - accuracy: 0.9842 - loss: 0.0440
94/Unknown 0s 2ms/step - accuracy: 0.9826 - loss: 0.0492
118/Unknown 0s 2ms/step - accuracy: 0.9819 - loss: 0.0523
142/Unknown 0s 2ms/step - accuracy: 0.9815 - loss: 0.0548
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9812 - loss: 0.0561
INFO Collaborator charlie is sending task results for locally_tuned_model_validation, round 6 ]8;id=599800;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=303059;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 6: Collaborators that have completed all tasks: ['bob', 'charlie'] ]8;id=345191;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=723780;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
INFO Saving round 6 model... ]8;id=435352;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=313576;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1157\1157]8;;\
INFO Committed model for round 6, saved 8 model tensors and 26 next round model tensors with best_score ]8;id=404948;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=946150;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py#220\220]8;;\
0.9869000017642975
INFO Persist model and clean task result for round 6 ]8;id=572206;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=751027;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#384\384]8;;\
INFO Starting round 7... ]8;id=94854;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=775891;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1170\1170]8;;\
[03/26/25 05:55:05] INFO Requesting tasks... ]8;id=115090;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=640972;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending tasks to collaborator charlie for round 7 ]8;id=870168;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=668334;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=75488;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=66854;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 28ms/step - accuracy: 1.0000 - loss: 0.0306
24/Unknown 0s 2ms/step - accuracy: 0.9939 - loss: 0.0238
48/Unknown 0s 2ms/step - accuracy: 0.9919 - loss: 0.0284
73/Unknown 0s 2ms/step - accuracy: 0.9907 - loss: 0.0305
97/Unknown 0s 2ms/step - accuracy: 0.9899 - loss: 0.0327
121/Unknown 0s 2ms/step - accuracy: 0.9893 - loss: 0.0343
145/Unknown 0s 2ms/step - accuracy: 0.9890 - loss: 0.0354/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9888 - loss: 0.0358
[03/26/25 05:55:06] INFO Collaborator charlie is sending task results for aggregated_model_validation, round 7 ]8;id=924314;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=42754;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:55:06] INFO Run 0 epoch of 7 round ]8;id=288938;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=809925;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 3s - 4ms/step - accuracy: 0.9960 - loss: 0.0128
[03/26/25 05:55:09] INFO Collaborator charlie is sending task results for train, round 7 ]8;id=243079;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=686158;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 28ms/step - accuracy: 1.0000 - loss: 0.0172
25/Unknown 0s 2ms/step - accuracy: 0.9809 - loss: 0.0780
50/Unknown 0s 2ms/step - accuracy: 0.9801 - loss: 0.0811
74/Unknown 0s 2ms/step - accuracy: 0.9804 - loss: 0.0786
97/Unknown 0s 2ms/step - accuracy: 0.9805 - loss: 0.0763
122/Unknown 0s 2ms/step - accuracy: 0.9806 - loss: 0.0743
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9809 - loss: 0.0718
[03/26/25 05:55:10] INFO Collaborator charlie is sending task results for locally_tuned_model_validation, round 7 ]8;id=631416;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=513819;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 7: Collaborators that have completed all tasks: ['charlie'] ]8;id=206208;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=733535;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
[03/26/25 05:55:10] INFO Requesting tasks... ]8;id=333487;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=505012;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:55:13] INFO Requesting tasks... ]8;id=424190;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=278804;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:55:13] INFO Sending tasks to collaborator bob for round 7 ]8;id=145273;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=807579;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=802399;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=17151;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 25ms/step - accuracy: 1.0000 - loss: 0.0023
25/Unknown 0s 2ms/step - accuracy: 0.9852 - loss: 0.0295
49/Unknown 0s 2ms/step - accuracy: 0.9851 - loss: 0.0324
73/Unknown 0s 2ms/step - accuracy: 0.9855 - loss: 0.0338
97/Unknown 0s 2ms/step - accuracy: 0.9859 - loss: 0.0347
121/Unknown 0s 2ms/step - accuracy: 0.9862 - loss: 0.0353
145/Unknown 0s 2ms/step - accuracy: 0.9863 - loss: 0.0360/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9864 - loss: 0.0363
[03/26/25 05:55:14] INFO Collaborator bob is sending task results for aggregated_model_validation, round 7 ]8;id=266921;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=632940;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:55:14] INFO Run 0 epoch of 7 round ]8;id=580738;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=721320;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 3s - 4ms/step - accuracy: 0.9893 - loss: 0.0334
[03/26/25 05:55:17] INFO Collaborator bob is sending task results for train, round 7 ]8;id=121038;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=445839;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 27ms/step - accuracy: 1.0000 - loss: 0.0360
23/Unknown 0s 2ms/step - accuracy: 0.9864 - loss: 0.0482
47/Unknown 0s 2ms/step - accuracy: 0.9869 - loss: 0.0442
71/Unknown 0s 2ms/step - accuracy: 0.9867 - loss: 0.0436
94/Unknown 0s 2ms/step - accuracy: 0.9868 - loss: 0.0436
118/Unknown 0s 2ms/step - accuracy: 0.9869 - loss: 0.0435
142/Unknown 0s 2ms/step - accuracy: 0.9868 - loss: 0.0435
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9868 - loss: 0.0435
[03/26/25 05:55:18] INFO Collaborator bob is sending task results for locally_tuned_model_validation, round 7 ]8;id=351283;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=993499;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 7: Collaborators that have completed all tasks: ['charlie', 'bob'] ]8;id=955994;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=139324;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
INFO Round 7: saved the best model with score 0.987000 ]8;id=288688;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=119591;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1119\1119]8;;\
INFO Saving round 7 model... ]8;id=699387;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=543976;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1157\1157]8;;\
INFO Committed model for round 7, saved 8 model tensors and 26 next round model tensors with best_score ]8;id=659085;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=913852;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py#220\220]8;;\
0.9870000183582306
INFO Persist model and clean task result for round 7 ]8;id=771424;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=61908;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#384\384]8;;\
INFO Starting round 8... ]8;id=637800;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=499722;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1170\1170]8;;\
[03/26/25 05:55:18] INFO Requesting tasks... ]8;id=881865;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=70096;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending tasks to collaborator bob for round 8 ]8;id=144286;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=853526;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=915954;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=385180;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 27ms/step - accuracy: 1.0000 - loss: 0.0120
25/Unknown 0s 2ms/step - accuracy: 0.9899 - loss: 0.0244
49/Unknown 0s 2ms/step - accuracy: 0.9888 - loss: 0.0283
73/Unknown 0s 2ms/step - accuracy: 0.9877 - loss: 0.0337
96/Unknown 0s 2ms/step - accuracy: 0.9871 - loss: 0.0363
119/Unknown 0s 2ms/step - accuracy: 0.9867 - loss: 0.0376
143/Unknown 0s 2ms/step - accuracy: 0.9866 - loss: 0.0382
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9867 - loss: 0.0384
[03/26/25 05:55:19] INFO Collaborator bob is sending task results for aggregated_model_validation, round 8 ]8;id=170296;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=732909;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:55:19] INFO Run 0 epoch of 8 round ]8;id=268207;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=172979;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
[03/26/25 05:55:20] INFO Requesting tasks... ]8;id=787196;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=714739;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:55:20] INFO Sending tasks to collaborator charlie for round 8 ]8;id=881792;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=368224;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=298687;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=500912;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 50ms/step - accuracy: 1.0000 - loss: 0.0049
12/Unknown 0s 5ms/step - accuracy: 0.9812 - loss: 0.0728
23/Unknown 0s 5ms/step - accuracy: 0.9836 - loss: 0.0640
34/Unknown 0s 5ms/step - accuracy: 0.9851 - loss: 0.0573
46/Unknown 0s 5ms/step - accuracy: 0.9858 - loss: 0.0542
59/Unknown 0s 5ms/step - accuracy: 0.9864 - loss: 0.0514
71/Unknown 0s 5ms/step - accuracy: 0.9870 - loss: 0.0491
94/Unknown 0s 4ms/step - accuracy: 0.9875 - loss: 0.0464
105/Unknown 1s 5ms/step - accuracy: 0.9878 - loss: 0.0453
116/Unknown 1s 5ms/step - accuracy: 0.9879 - loss: 0.0448
128/Unknown 1s 5ms/step - accuracy: 0.9879 - loss: 0.0444
140/Unknown 1s 5ms/step - accuracy: 0.9879 - loss: 0.0440
152/Unknown 1s 4ms/step - accuracy: 0.9878 - loss: 0.0437
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9878 - loss: 0.0435
[03/26/25 05:55:21] INFO Collaborator charlie is sending task results for aggregated_model_validation, round 8 ]8;id=428308;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=531941;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
[03/26/25 05:55:21] INFO Run 0 epoch of 8 round ]8;id=140413;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=150337;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 5s - 6ms/step - accuracy: 0.9895 - loss: 0.0304
[03/26/25 05:55:24] INFO Collaborator bob is sending task results for train, round 8 ]8;id=186125;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=821294;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 47ms/step - accuracy: 1.0000 - loss: 0.0081
15/Unknown 0s 4ms/step - accuracy: 0.9918 - loss: 0.0311
28/Unknown 0s 4ms/step - accuracy: 0.9900 - loss: 0.0456
41/Unknown 0s 4ms/step - accuracy: 0.9892 - loss: 0.0475
67/Unknown 0s 4ms/step - accuracy: 0.9887 - loss: 0.0463
80/Unknown 0s 4ms/step - accuracy: 0.9885 - loss: 0.0459
93/Unknown 0s 4ms/step - accuracy: 0.9883 - loss: 0.0457
106/Unknown 0s 4ms/step - accuracy: 0.9881 - loss: 0.0454
118/Unknown 1s 4ms/step - accuracy: 0.9881 - loss: 0.0451
130/Unknown 1s 4ms/step - accuracy: 0.9881 - loss: 0.0447
143/Unknown 1s 4ms/step - accuracy: 0.9881 - loss: 0.0443
157/157 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9881 - loss: 0.0438
[03/26/25 05:55:25] INFO Collaborator bob is sending task results for locally_tuned_model_validation, round 8 ]8;id=994668;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=347;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 8: Collaborators that have completed all tasks: ['bob'] ]8;id=303421;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=201728;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
[03/26/25 05:55:25] INFO Requesting tasks... ]8;id=718214;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=659663;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
938/938 - 5s - 5ms/step - accuracy: 0.9965 - loss: 0.0097
[03/26/25 05:55:26] INFO Collaborator charlie is sending task results for train, round 8 ]8;id=56176;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=510932;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 26ms/step - accuracy: 1.0000 - loss: 0.0039
24/Unknown 0s 2ms/step - accuracy: 0.9872 - loss: 0.0594
47/Unknown 0s 2ms/step - accuracy: 0.9873 - loss: 0.0572
71/Unknown 0s 2ms/step - accuracy: 0.9861 - loss: 0.0603
94/Unknown 0s 2ms/step - accuracy: 0.9854 - loss: 0.0617
118/Unknown 0s 2ms/step - accuracy: 0.9851 - loss: 0.0614
141/Unknown 0s 2ms/step - accuracy: 0.9847 - loss: 0.0616
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9845 - loss: 0.0617
[03/26/25 05:55:27] INFO Collaborator charlie is sending task results for locally_tuned_model_validation, round 8 ]8;id=534174;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=925424;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 8: Collaborators that have completed all tasks: ['bob', 'charlie'] ]8;id=392507;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=56504;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
INFO Round 8: saved the best model with score 0.987300 ]8;id=513532;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=943961;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1119\1119]8;;\
INFO Saving round 8 model... ]8;id=449007;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=296865;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1157\1157]8;;\
INFO Committed model for round 8, saved 8 model tensors and 26 next round model tensors with best_score ]8;id=445617;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=919821;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py#220\220]8;;\
0.9873000085353851
INFO Persist model and clean task result for round 8 ]8;id=33483;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=347353;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#384\384]8;;\
INFO Starting round 9... ]8;id=596099;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=775897;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1170\1170]8;;\
[03/26/25 05:55:27] INFO Requesting tasks... ]8;id=41306;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=234067;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending tasks to collaborator charlie for round 9 ]8;id=177992;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=193532;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=726967;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=612957;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 27ms/step - accuracy: 1.0000 - loss: 8.8434e-04
24/Unknown 0s 2ms/step - accuracy: 0.9866 - loss: 0.0543
48/Unknown 0s 2ms/step - accuracy: 0.9858 - loss: 0.0515
72/Unknown 0s 2ms/step - accuracy: 0.9857 - loss: 0.0490
96/Unknown 0s 2ms/step - accuracy: 0.9856 - loss: 0.0478
119/Unknown 0s 2ms/step - accuracy: 0.9855 - loss: 0.0473
143/Unknown 0s 2ms/step - accuracy: 0.9855 - loss: 0.0470
/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9856 - loss: 0.0465
INFO Collaborator charlie is sending task results for aggregated_model_validation, round 9 ]8;id=432032;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=382740;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Run 0 epoch of 9 round ]8;id=761005;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=899562;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 3s - 4ms/step - accuracy: 0.9975 - loss: 0.0087
[03/26/25 05:55:31] INFO Collaborator charlie is sending task results for train, round 9 ]8;id=416027;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=757320;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 26ms/step - accuracy: 1.0000 - loss: 1.0060e-04
25/Unknown 0s 2ms/step - accuracy: 0.9801 - loss: 0.0721
49/Unknown 0s 2ms/step - accuracy: 0.9757 - loss: 0.0897
73/Unknown 0s 2ms/step - accuracy: 0.9754 - loss: 0.0917
97/Unknown 0s 2ms/step - accuracy: 0.9759 - loss: 0.0890
121/Unknown 0s 2ms/step - accuracy: 0.9766 - loss: 0.0856
145/Unknown 0s 2ms/step - accuracy: 0.9773 - loss: 0.0829
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9775 - loss: 0.0817
INFO Collaborator charlie is sending task results for locally_tuned_model_validation, round 9 ]8;id=580224;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=937235;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 9: Collaborators that have completed all tasks: ['charlie'] ]8;id=155785;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=142404;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
[03/26/25 05:55:31] INFO Requesting tasks... ]8;id=384362;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=411500;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:55:35] INFO Requesting tasks... ]8;id=788264;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=807764;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:55:35] INFO Sending tasks to collaborator bob for round 9 ]8;id=97500;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=687391;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#521\521]8;;\
INFO Received Tasks: [name: "aggregated_model_validation" ]8;id=915701;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=148721;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#172\172]8;;\
, name: "train"
, name: "locally_tuned_model_validation"
]
1/Unknown 0s 26ms/step - accuracy: 1.0000 - loss: 1.4275e-04
25/Unknown 0s 2ms/step - accuracy: 0.9925 - loss: 0.0152
49/Unknown 0s 2ms/step - accuracy: 0.9922 - loss: 0.0230
73/Unknown 0s 2ms/step - accuracy: 0.9914 - loss: 0.0272
98/Unknown 0s 2ms/step - accuracy: 0.9912 - loss: 0.0293
123/Unknown 0s 2ms/step - accuracy: 0.9912 - loss: 0.0303
146/Unknown 0s 2ms/step - accuracy: 0.9912 - loss: 0.0306/home/docs/checkouts/readthedocs.org/user_builds/openfl/envs/v1.8/lib/python3.10/site-packages/keras/src/trainers/epoch_iterator.py:151: UserWarning: Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches. You may need to use the `.repeat()` function when building your dataset.
self._interrupted_warning()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9911 - loss: 0.0308
INFO Collaborator bob is sending task results for aggregated_model_validation, round 9 ]8;id=159657;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=998614;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Run 0 epoch of 9 round ]8;id=216344;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py\runner_keras.py]8;;\:]8;id=204276;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/federated/task/runner_keras.py#107\107]8;;\
938/938 - 3s - 4ms/step - accuracy: 0.9913 - loss: 0.0250
[03/26/25 05:55:39] INFO Collaborator bob is sending task results for train, round 9 ]8;id=221047;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=509342;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
1/Unknown 0s 26ms/step - accuracy: 1.0000 - loss: 0.0014
24/Unknown 0s 2ms/step - accuracy: 0.9848 - loss: 0.0347
48/Unknown 0s 2ms/step - accuracy: 0.9850 - loss: 0.0422
72/Unknown 0s 2ms/step - accuracy: 0.9856 - loss: 0.0434
97/Unknown 0s 2ms/step - accuracy: 0.9859 - loss: 0.0435
121/Unknown 0s 2ms/step - accuracy: 0.9860 - loss: 0.0435
145/Unknown 0s 2ms/step - accuracy: 0.9860 - loss: 0.0440
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.9860 - loss: 0.0442
INFO Collaborator bob is sending task results for locally_tuned_model_validation, round 9 ]8;id=198545;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=549610;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#762\762]8;;\
INFO Round 9: Collaborators that have completed all tasks: ['charlie', 'bob'] ]8;id=307954;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=128351;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1212\1212]8;;\
INFO Round 9: saved the best model with score 0.988500 ]8;id=582870;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=279018;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1119\1119]8;;\
INFO Saving round 9 model... ]8;id=196362;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=171058;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1157\1157]8;;\
INFO Committed model for round 9, saved 8 model tensors and 26 next round model tensors with best_score ]8;id=771767;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py\persistent_db.py]8;;\:]8;id=787223;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/databases/persistent_db.py#220\220]8;;\
0.9884999990463257
INFO Persist model and clean task result for round 9 ]8;id=133492;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=969679;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#384\384]8;;\
INFO Experiment Completed. Cleaning up... ]8;id=156683;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=372420;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#1168\1168]8;;\
[03/26/25 05:55:39] INFO Requesting tasks... ]8;id=746431;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=736789;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
INFO Sending signal to collaborator bob to shutdown... ]8;id=878594;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=905957;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#468\468]8;;\
INFO Received shutdown signal. Exiting... ]8;id=242167;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=293994;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#187\187]8;;\
✔️ OK
[03/26/25 05:55:41] INFO Requesting tasks... ]8;id=122455;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py\aggregator_client.py]8;;\:]8;id=132003;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/transport/grpc/aggregator_client.py#305\305]8;;\
[03/26/25 05:55:41] INFO Sending signal to collaborator charlie to shutdown... ]8;id=267037;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py\aggregator.py]8;;\:]8;id=717166;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/aggregator/aggregator.py#468\468]8;;\
INFO Received shutdown signal. Exiting... ]8;id=539576;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py\collaborator.py]8;;\:]8;id=526482;file:///home/docs/checkouts/readthedocs.org/user_builds/openfl/checkouts/v1.8/openfl/component/collaborator/collaborator.py#187\187]8;;\
✔️ OK
Next steps#
The simulation will run for a few epochs. Once the simulation is complete, the model will be saved under the save directory. The model can be converted to framework native formats using the fx model save command.
For detailed information on options supported via CLI, refer to the CLI reference.
For an in-depth understanding of the TaskRunner architecture, refer to the TaskRunner API.