Service Discovery

Dynamo components (frontends, workers, planner) need to be able to discover each other and their capabilities at runtime. We refer to this as service discovery. There are 2 kinds of service discovery backends supported on Kubernetes.

Discovery Backends

Kubernetes Discovery (Default)

Kubernetes discovery is the default and recommended backend when running on Kubernetes. It uses native Kubernetes primitives to facilitate discovery of components:

• DynamoWorkerMetadata CRD: Each worker stores its registered endpoints and model cards in a Custom Resource
• EndpointSlices: EndpointSlices signal each component’s readiness status

Implementation Details

Each pod runs a discovery daemon that watches both EndpointSlices and DynamoWorkerMetadata CRs. A pod is only discoverable when it appears as “ready” in an EndpointSlice AND has a corresponding DynamoWorkerMetadata CR. This correlation ensures pods aren’t discoverable until they’re ready, metadata is immediately available, and stale entries are cleaned up when pods terminate.

DynamoWorkerMetadata CRD

Each worker pod creates a DynamoWorkerMetadata CR that stores its discovery metadata:

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoWorkerMetadata
3
metadata:
4
  name: my-worker-pod-abc123
5
  namespace: dynamo-system
6
  ownerReferences:
7
    - apiVersion: v1
8
      kind: Pod
9
      name: my-worker-pod-abc123
10
      uid: <pod-uid>
11
      controller: true
12
spec:
13
  data:
14
    endpoints:
15
      "dynamo/backend/generate":
16
        type: Endpoint
17
        namespace: dynamo
18
        component: backend
19
        endpoint: generate
20
        instance_id: 12345678901234567890
21
        transport:
22
          nats_tcp: "dynamo_backend.generate-abc123"
23
    model_cards: {}

The CR is named after the pod and includes an owner reference for automatic garbage collection when the pod is deleted.

EndpointSlices

While DynamoWorkerMetadata resources provide an up-to-date snapshot of a component’s capabilities, EndpointSlices give a snapshot of health of the various Dynamo components.

The operator creates a Kubernetes Service targeting the Dynamo components. The Kubernetes controller in turn creates and maintains EndpointSlice resources that keep track of the readiness of the pods targeted by the Service. Watching these slices gives us an up-to-date snapshot of which Dynamo components are ready to serve traffic.

Readiness Probes

A pod is marked ready if the readiness probe succeeds. On Dynamo workers, this is when the generate endpoint is available and healthy. These probes are configured by the Dynamo operator for each pod/component.

RBAC

Each Dynamo component pod is automatically given a ServiceAccount that allows it to watch EndpointSlice and DynamoWorkerMetadata resources within its namespace.

Environment Variables

The following environment variables are automatically injected into pods by the operator to facilitate service discovery:

The pod’s instance ID is deterministically generated by hashing the pod name, ensuring consistent identity and correlation between EndpointSlices and CRs.

KV Store Discovery (etcd)

To use etcd-based discovery instead of Kubernetes-native discovery, add the annotation to your DynamoGraphDeployment:

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: my-deployment
5
  annotations:
6
    nvidia.com/dynamo-discovery-backend: etcd
7
spec:
8
  services:
9
    # ...

This requires an etcd cluster to be available. The etcd connection is configured via the platform Helm chart.