@multimodal/agent-interface

Standard protocol, types, event stream and other specifications for @multimodal/agent

Installation

1npm install @multimodal/agent-interface

Overview

The @multimodal/agent-interface package provides the core types, interfaces, and event stream specifications for building intelligent agents in the @multimodal/agent framework. It serves as the foundation for agent communication, tool integration, and real-time event processing.

Key Components

Agent Interface (IAgent)

The core interface that all agent implementations must implement:

1import { IAgent, AgentOptions } from '@multimodal/agent-interface';
2
3class MyAgent implements IAgent {
4  async initialize() {
5    // Initialize your agent
6  }
7
8  async run(input: string) {
9    // Execute agent logic
10  }
11
12  // ... other required methods
13}

Agent Options

Comprehensive configuration options for agent behavior:

1import { AgentOptions } from '@multimodal/agent-interface';
2
3const options: AgentOptions = {
4  // Base configuration
5  id: 'my-agent',
6  name: 'My Custom Agent',
7  instructions: 'You are a helpful assistant...',
8
9  // Model configuration
10  model: {
11    provider: 'openai',
12    id: 'gpt-4',
13  },
14  maxTokens: 4096,
15  temperature: 0.7,
16
17  // Tool configuration
18  tools: [
19    {
20      name: 'calculator',
21      description: 'Perform mathematical calculations',
22      schema: z.object({
23        expression: z.string(),
24      }),
25      function: async (args) => {
26        // Tool implementation
27      },
28    },
29  ],
30
31  // Loop control
32  maxIterations: 50,
33
34  // Memory and context
35  context: {
36    maxImagesCount: 10,
37  },
38  eventStreamOptions: {
39    maxEvents: 1000,
40    autoTrim: true,
41  },
42
43  // Logging
44  logLevel: LogLevel.INFO,
45};

Tool System

Define and register tools for agent capabilities:

1import { Tool } from '@multimodal/agent-interface';
2import { z } from 'zod';
3
4const weatherTool: Tool = {
5  name: 'get_weather',
6  description: 'Get current weather for a location',
7  schema: z.object({
8    location: z.string().describe('The city and state/country'),
9    unit: z.enum(['celsius', 'fahrenheit']).default('celsius'),
10  }),
11  function: async ({ location, unit }) => {
12    // Fetch weather data
13    return {
14      location,
15      temperature: 22,
16      unit,
17      condition: 'sunny',
18    };
19  },
20};

Event Stream

The event stream system provides real-time visibility into agent execution, conversation flow, and internal reasoning processes. It's designed for both monitoring and building reactive user interfaces.

Core Event Types

The event stream supports various categories of events:

Conversation Events

• user_message - User input to the agent
• assistant_message - Agent's response
• assistant_thinking_message - Agent's reasoning process

Streaming Events

• assistant_streaming_message - Real-time content updates
• assistant_streaming_thinking_message - Real-time reasoning updates
• final_answer_streaming - Streaming final answers

Tool Execution Events

• tool_call - Tool invocation
• tool_result - Tool execution result

Planning Events

• plan_start - Beginning of planning session
• plan_update - Plan state changes
• plan_finish - Completion of plan

System Events

• system - Logs, warnings, errors
• agent_run_start - Agent execution start
• agent_run_end - Agent execution completion
• environment_input - External context injection
• final_answer - Structured final response

Using the Event Stream

Basic Event Subscription

1import { AgentEventStream } from '@multimodal/agent-interface';
2
3// Get the event stream from your agent
4const eventStream = agent.getEventStream();
5
6// Subscribe to all events
7const unsubscribe = eventStream.subscribe((event) => {
8  console.log('Event:', event.type, event);
9});
10
11// Subscribe to specific event types
12const unsubscribeSpecific = eventStream.subscribeToTypes(
13  ['assistant_message', 'tool_call'],
14  (event) => {
15    console.log('Specific event:', event);
16  }
17);
18
19// Subscribe to streaming events only
20const unsubscribeStreaming = eventStream.subscribeToStreamingEvents((event) => {
21  if (event.type === 'assistant_streaming_message') {
22    process.stdout.write(event.content);
23  }
24});

Event Stream Processing

1// Create custom events
2const customEvent = eventStream.createEvent('user_message', {
3  content: 'Hello, agent!',
4});
5
6// Send events manually
7eventStream.sendEvent(customEvent);
8
9// Query historical events
10const recentEvents = eventStream.getEvents(['assistant_message'], 10);
11const toolEvents = eventStream.getEventsByType(['tool_call', 'tool_result']);
12
13// Get latest assistant response
14const latestResponse = eventStream.getLatestAssistantResponse();
15
16// Get recent tool results
17const toolResults = eventStream.getLatestToolResults();

Streaming Agent Execution

1// Run agent in streaming mode
2const streamingEvents = await agent.run({
3  input: 'Analyze the weather data and create a report',
4  stream: true,
5});
6
7// Process streaming events
8for await (const event of streamingEvents) {
9  switch (event.type) {
10    case 'assistant_streaming_message':
11      // Update UI with incremental content
12      updateMessageUI(event.messageId, event.content);
13      break;
14
15    case 'assistant_streaming_thinking_message':
16      // Show reasoning process
17      updateThinkingUI(event.content);
18      break;
19
20    case 'tool_call':
21      // Show tool being executed
22      showToolExecution(event.name, event.arguments);
23      break;
24
25    case 'final_answer':
26      // Display final structured answer
27      showFinalAnswer(event.content, event.format);
28      break;
29  }
30}

Custom Event Extensions

Extend the event system with custom event types:

1// Define custom event interface
2interface MyCustomEventInterface extends AgentEventStream.BaseEvent {
3  type: 'custom_analysis';
4  analysisType: 'sentiment' | 'classification';
5  confidence: number;
6  result: any;
7}
8
9// Extend the event mapping through module augmentation
10declare module '@multimodal/agent-interface' {
11  namespace AgentEventStream {
12    interface ExtendedEventMapping {
13      custom_analysis: MyCustomEventInterface;
14    }
15  }
16}
17
18// Now you can use the custom event type
19const customEvent = eventStream.createEvent('custom_analysis', {
20  analysisType: 'sentiment',
21  confidence: 0.95,
22  result: { sentiment: 'positive', score: 0.85 },
23});
24
25// Type-safe subscription
26eventStream.subscribeToTypes(['custom_analysis'], (event) => {
27  // TypeScript knows this is MyCustomEventInterface
28  console.log('Analysis result:', event.result);
29});

Event Stream Configuration

Configure event stream behavior:

1const eventStreamOptions: AgentEventStream.ProcessorOptions = {
2  maxEvents: 1000,        // Keep last 1000 events in memory
3  autoTrim: true,         // Automatically remove old events
4};
5
6const agentOptions: AgentOptions = {
7  eventStreamOptions,
8  // ... other options
9};

Agent Run Options

Basic Text Input

1// Simple string input
2const response = await agent.run('What is the weather in New York?');
3console.log(response.content);

Advanced Options

1// Object-based options with configuration
2const response = await agent.run({
3  input: [
4    { type: 'text', text: 'Analyze this image:' },
5    { type: 'image_url', image_url: { url: 'data:image/jpeg;base64,...' } },
6  ],
7  model: 'gpt-4-vision-preview',
8  provider: 'openai',
9  sessionId: 'conversation-123',
10  toolCallEngine: 'native',
11});

Streaming Mode

1// Enable streaming for real-time updates
2const events = await agent.run({
3  input: 'Create a detailed analysis report',
4  stream: true,
5});
6
7for await (const event of events) {
8  // Handle streaming events
9}

Tool Call Engines

Configure how tools are executed:

Native Engine (Default)

Uses LLM's built-in function calling capabilities:

1const options: AgentOptions = {
2  toolCallEngine: 'native',
3  tools: [weatherTool],
4};

Prompt Engineering Engine

Uses prompt-based tool calling for models without native function calling:

1const options: AgentOptions = {
2  toolCallEngine: 'prompt_engineering',
3  tools: [weatherTool],
4};

Structured Outputs Engine

Uses structured JSON outputs for tool calling:

1const options: AgentOptions = {
2  toolCallEngine: 'structured_outputs',
3  tools: [weatherTool],
4};

Agent Lifecycle Hooks

Implement hooks to customize agent behavior:

1class CustomAgent implements IAgent {
2  // Called before each LLM request
3  async onLLMRequest(sessionId: string, payload: LLMRequestHookPayload) {
4    console.log('Sending request to:', payload.provider);
5  }
6
7  // Called after LLM response
8  async onLLMResponse(sessionId: string, payload: LLMResponseHookPayload) {
9    console.log('Received response from:', payload.provider);
10  }
11
12  // Called before tool execution
13  async onBeforeToolCall(sessionId: string, toolCall: any, args: any) {
14    console.log('Executing tool:', toolCall.name);
15    return args; // Can modify arguments
16  }
17
18  // Called after tool execution
19  async onAfterToolCall(sessionId: string, toolCall: any, result: any) {
20    console.log('Tool result:', result);
21    return result; // Can modify result
22  }
23
24  // Called before loop termination
25  async onBeforeLoopTermination(sessionId: string, finalEvent: any) {
26    // Decide whether to continue or finish
27    return { finished: true };
28  }
29
30  // Called at start of each iteration
31  async onEachAgentLoopStart(sessionId: string) {
32    console.log('Starting new iteration');
33  }
34
35  // Called when agent loop ends
36  async onAgentLoopEnd(sessionId: string) {
37    console.log('Agent execution completed');
38  }
39}

Context Management

Configure how the agent manages conversation context:

1const contextOptions: AgentContextAwarenessOptions = {
2  maxImagesCount: 5, // Limit images in context to prevent token overflow
3};
4
5const agentOptions: AgentOptions = {
6  context: contextOptions,
7};

Error Handling

Handle tool execution errors:

1class RobustAgent implements IAgent {
2  async onToolCallError(sessionId: string, toolCall: any, error: any) {
3    console.error('Tool execution failed:', error);
4    
5    // Return a recovery value or re-throw
6    return {
7      error: true,
8      message: 'Tool execution failed, please try again',
9    };
10  }
11}

TypeScript Support

The package is fully typed with TypeScript support:

• Complete type safety for all interfaces and options
• Generic support for custom agent implementations
• Module augmentation for extending event types
• Strict typing for tool definitions and parameters

Best Practices

1. Event Stream Management: Use appropriate maxEvents limits to prevent memory leaks
2. Tool Design: Keep tools focused and well-documented with clear schemas
3. Context Awareness: Configure maxImagesCount for multimodal conversations
4. Error Handling: Implement proper error handling in tool functions and hooks
5. Streaming: Use streaming mode for long-running or interactive applications
6. Custom Events: Extend event types for domain-specific monitoring needs

License

Apache-2.0