Director Service Communication Diagrams

The following diagrams depict existing procedure calls to the Director service. Included are interactions with the Director’s inner representations to better understand their signatures.

Director-Envoy Communication

The following diagram depicts a typical process of establishing a Federation and registering an experiment.

sequenceDiagram participant D as Director participant E as Envoy rect rgb(0, 255, 0,.1) Note over D,E: A Federation startup process D->D: Starts E->E: Starts and loads local Shard Descriptor E-->>D: Connects using FQDN and certificate E-->>+D: Communicates dataset info D-->D: Ensures unified data interface D-->>-E: Approves D-->D: Keeps a list of connected Envoys end Note over D,E: We consider a Federation set up rect rgb(0, 0, 255,.05) Note over D,E: An Experiment's start D->D: Registers a model, FL tasks and an FL plan D->D:Starts an Aggregator with a toy plan Note left of D: An Envoy with dummy data is <br>started locally opt Test run failed D-->D: Notifies user that <br>the experiment is inconsistent D-->D: The experiment ends end D->D: Reviews the FL plan D->D: Starts an Aggregator D-->>E: Sends the experiment archive <br>and the FL plan to envoys E-->E: Reviews the FL Plan E-->E: Starts a Collaborator D-->D: Fills the model with the final weights <br>and returns to user end Note over D,E: The Experiment ended. <br> The Federation keeps existing.

Basic Scenario of Director-Envoy Communication

Director Side Envoy Representation and Related Remote Procedure Calls

This diagram shows possible interactions with Envoy handles on the Director side.

sequenceDiagram participant ER as Envoy Representation participant D as Director participant E as Envoy participant F as Frontend API E-)+D: Acknowledge Shard request note over ER,D: Envoy registry is a dict-like object D->>ER: Creates and puts to the Registry activate ER D->>-E: Approves loop Healthcheck timeout E-)+D: Sends status D->>ER: Adds info or updates ER state activate ER D->>-E: Acknowledges end rect rgb(225, 255, 225) Note over D: API requests may overlap <br>with the healthcheck loop F-)+D: Requests Shard Registry D->>ER: Gets required info D->>-F: Replies with the Registry state end D-->E: Disconnect D-->D: Waits for several Healthcheck timeouts D->>ER: Removes from the Registry deactivate ER deactivate ER Note over ER: Will not be included in the next <br>API Registry request

Communications Altering or Requesting Envoy-Related Information

Director Side Experiment Representation and Related Remote Procedure Calls

This diagram shows possible interactions with Experiment handles on the Director side.

sequenceDiagram participant ER as Experiment Representation participant D as Director participant E as Envoy participant F as Frontend API F->>+D: Set New Experiment D->>+ER: Creates ER and puts in the registry note over ER,D: Experiment registry is a queue-like object D->>-F: Approves F->>+D: Get Registered experiments D->>ER: Gets name and status note over ER,D: Registry can also return <br>an experiment list by user id D->>-F: Approves F->>+D: Get Experiment Status D->>+ER: Gets the status of the current experiment note over ER,D: Experiment might be pending, in progress, failed or finished D->>-F: Approves rect rgb(225, 255, 225) note over ER,E: Envoys may start awaiting experiments <br>even if the queue is empty E-)+D: Wait Experiment *STREAM* note over ER,D: Assuming the Experiment is the next in the queue <br>and the Envoy is assigned for this Experiment D->>ER: Gets experiment name (pop operation) D->>-E: Wait Experiment Response E->>+D: Get Experiment Data D->>ER: Gets experiment archive (by name) D-->D: Reviews the FL Plan D->>ER: Sets status to In Progress D->>-E: Experiment Data end rect rgb(255, 225, 225) note over ER,E: The Experiment execution D-->D: Starts an aggregator that orchestrates experiment D->>ER: Removes experiment archive D-->D: Finishes the experiment and releases involved Envoys D->>+ER: Puts training artifacts D->>ER: Sets status to Finished end rect rgb(225, 225, 255) note over ER,F: Restoring experiment on the Frontend <br>from the Director's experiment representation F->>+D: Restore experiment by name D->>ER: Gets required data note over ER,D: The Registry may also return <br>the experiment by its identifier D->>-F: Response with model structure and the plan end F->>+D: Requests training artifacts D->>ER: Gets training artifacts D->>-F: Respond F->>+D: Requests removing the Experiment entry D->>ER: Removes Experiment from registry deactivate ER deactivate ER D->>-F: Acknowledges

Communications Altering or Requesting Experiment-Related Information