Multi-Agent System with Telemetry

Try it yourself! This example is available as an executable Jupyter notebook.

This example demonstrates building a multi-agent system with delegation between agents. You'll see how a coordinator agent delegates to specialist agents.

Understanding the Flow

Prerequisites

• KAOS operator installed (Installation Guide)
• kaos-cli installed
• Access to a Kubernetes cluster

Overview

We'll create:

1. A coordinator agent that delegates to specialists
2. Specialist agents (researcher, analyst) that handle specific tasks
3. Demonstrate agent-to-agent communication via delegation

Setup

First, let's set up the environment and create a namespace:

import os
os.environ['NAMESPACE'] = 'kaos-agent-telemetry-example'

kubectl create namespace $NAMESPACE 2>/dev/null || true
kubectl config set-context --current --namespace=$NAMESPACE

Step 1: Create the ModelAPI

Create a shared ModelAPI for all agents (using mock responses for testing):

kaos modelapi deploy team-api --mode proxy --wait

Wait for the ModelAPI pod to be fully ready (volumes mounted, LiteLLM configured):

kubectl wait --for=condition=available deployment/modelapi-team-api --timeout=120s
kubectl rollout status deployment/modelapi-team-api --timeout=120s

Step 2: Create the Researcher Agent

Create a specialist agent that handles research tasks:

kaos agent deploy researcher \
    --modelapi team-api \
    --model mock-model \
    --instructions "You are a research specialist. You gather and synthesize information on any topic." \
    --mock-response "Here is my research on the topic: AI systems are increasingly being used in enterprise environments. Key trends include automation, decision support, and customer service. Growth is estimated at 40% year-over-year." \
    --expose \
    --wait

Step 3: Create the Analyst Agent

Create another specialist for data analysis:

kaos agent deploy analyst \
    --modelapi team-api \
    --model mock-model \
    --instructions "You are a data analyst. You analyze information and provide insights with statistics." \
    --mock-response "Based on my analysis: The data shows 40% year-over-year growth in AI adoption. The highest impact areas are customer service automation (60%), decision support systems (25%), and predictive analytics (15%)." \
    --expose \
    --wait

Step 4: Create the Coordinator Agent

Create the coordinator that delegates to specialists. The coordinator uses mock responses with native tool_calls format for delegation:

kaos agent deploy coordinator \
    --modelapi team-api \
    --model mock-model \
    --instructions "You are a coordinator that delegates to specialist agents." \
    --mock-response '{"tool_calls": [{"id": "call_1", "name": "delegate_to_researcher", "arguments": {"task": "Research AI adoption trends in enterprises"}}]}' \
    --mock-response '{"tool_calls": [{"id": "call_2", "name": "delegate_to_analyst", "arguments": {"task": "Analyze the growth patterns from the research"}}]}' \
    --mock-response "No more actions needed." \
    --mock-response "Based on input from my team: AI is growing with 40% YoY growth, especially in customer service automation." \
    --sub-agent researcher \
    --sub-agent analyst \
    --expose \
    --wait

Step 5: Test the Multi-Agent System

Ensure all agent deployments are fully ready before sending requests:

kubectl wait --for=condition=available deployment/agent-researcher --timeout=120s
kubectl wait --for=condition=available deployment/agent-analyst --timeout=120s
kubectl wait --for=condition=available deployment/agent-coordinator --timeout=120s

Send a request to the coordinator and watch it delegate:

import subprocess
result = subprocess.run(
    ["kaos", "agent", "invoke", "coordinator", "--message", "What are the current trends in enterprise AI adoption?"],
    capture_output=True, text=True
)
print(result.stdout)
if result.returncode != 0:
    print(result.stderr)
    raise AssertionError(f"Agent invoke failed with code {result.returncode}")

Step 6: Verify the Result

The coordinator should have delegated to both specialists and synthesized their responses:

# Verify we got a meaningful response containing expected content
if "40%" in result.stdout or "growth" in result.stdout.lower():
    print("\nSUCCESS: Multi-agent delegation worked correctly!")
    print("The coordinator synthesized responses from researcher and analyst.")
else:
    raise AssertionError(f"Unexpected response - delegation may have failed: {result.stdout}")

Enabling Telemetry (Production)

For production use, enable OpenTelemetry on your agents using the --otel-endpoint flag:

kaos agent deploy coordinator --modelapi team-api --model gpt-4o \
    --otel-endpoint "http://otel-collector.monitoring.svc:4317"

Or via YAML:

apiVersion: kaos.tools/v1alpha1
kind: Agent
metadata:
  name: coordinator
spec:
  modelAPI: team-api
  model: gpt-4o
  config:
    telemetry:
      enabled: true
      endpoint: "http://otel-collector.monitoring.svc:4317"

This sends traces and metrics to your OTEL collector for observability.

How Delegation Works

The coordinator's mock responses use native tool_calls format for delegation:

{"tool_calls": [{"id": "call_1", "name": "delegate_to_researcher", "arguments": {"task": "Research AI trends"}}]}

When the agent framework sees a delegate_to_{name} tool call, it:

1. Extracts the agent name from the tool name (e.g., delegate_to_researcher → researcher)
2. Sends the task as a message to that agent
3. Waits for the response
4. Includes the response in the conversation context as a tool result
5. Continues to the next model call

Cleanup

kubectl delete namespace $NAMESPACE --wait=false

Next Steps

• Custom MCP Server - Build custom tools
• KAOS Monkey - Kubernetes management agent
• Agent CRD Reference - Full configuration options