# Multi LLM Configuration Developers can define configurations for multiple AI language models and specify strategies to optimize their usage. This documentation explains the properties and strategies available for configuring multiple LLMs in the AI Agent State. ### Multi LLM Configuration Properties The multi LLM configuration allows developers to specify multiple AI language models and define strategies for using them. Below are the key properties that can be defined, along with examples for each: #### Strategy The `strategy` property defines how multiple AI language models should be used. It includes the mode of operation and the status codes that trigger the strategy. * **Name**: `strategy` * **Type**: `object` * **Description**: The strategy to use when multiple AI language models are employed. This can include strategies such as fallback mechanisms, load balancing, and other techniques to ensure robust and reliable AI performance. * **Properties**: * **Mode**: Defines the mode for handling the request. It can be `single`, `fallback`, or `loadbalance`. * **OnStatusCodes**: An array of status codes that trigger the strategy. * **Example**: ```yaml strategy: mode: "fallback" onStatusCodes: [429, 500] ``` #### Targets The `targets` property specifies the list of AI language models to use. Each target includes details about the provider, API key, and optional configuration parameters. * **Name**: `targets` * **Type**: `array` * **Description**: The list of AI language models to use. * **Properties**: * **Provider**: The name of the provider offering the AI language model services. * **API Key**: The API key used to authenticate and access the AI language model services provided by the chosen provider. * **Weight**: (Optional) The weight of the provider, used for load balancing. * **OverrideParams**: (Optional) The parameters to override for the provider. See [**LLm Configuration OverrideParams**](/xflow/developer-guide/ai-agent-development/ai-agent/llm-configuration.md#override-parameters)**.** * **Strategy**: (Optional) The strategy to use for the target. * **Targets**: (Optional) The list of LLM provider configurations for the target. * **Example**: ```yaml targets: - provider: "openai" apiKey: "your-openai-api-key" weight: 1.0 overrideParams: model: "gpt-4o" temperature: 0.7 max_tokens: 1000 - provider: "anthropic" apiKey: "your-anthropic-api-key" weight: 0.5 overrideParams: model: "claude-2" temperature: 0.8 max_tokens: 800 ``` ### Supported Strategies #### Single Mode In `single` mode, the AI agent uses a single AI language model for handling requests. This is the simplest strategy and does not involve any fallback or load balancing. * **Example**: ```yaml strategy: mode: "single" onStatusCodes: [200] ``` #### Fallback Mode In `fallback` mode, the AI agent attempts to use the primary AI language model. If it encounters specified status codes (e.g., errors or rate limits), it automatically switches to the next model in the list. * **Example**: ```yaml strategy: mode: "fallback" onStatusCodes: [429, 500] targets: - provider: "openai" apiKey: "primary-api-key" overrideParams: model: "gpt-4o" - provider: "anthropic" apiKey: "secondary-api-key" overrideParams: model: "claude-2" ``` #### Loadbalance Mode In `loadbalance` mode, the AI agent distributes requests across multiple AI language models based on their defined weights. This ensures efficient utilization of resources and prevents any single model from becoming a bottleneck. * **Example**: ```yaml strategy: mode: "loadbalance" onStatusCodes: [200] targets: - provider: "openai" apiKey: "api-key-1" weight: 0.7 overrideParams: model: "gpt-4o" - provider: "groq" apiKey: "api-key-2" weight: 0.3 overrideParams: model: "llama3-70b" ``` ### Usage Example To use a multi LLM configuration in your AI Agent State, you can define the configuration in the state definition. Here is an example of how to configure an AI Agent State using multiple LLMs with a fallback strategy: ```yaml - name: MultiLLMAgentState type: aiagent agentName: MultiLLMAgent aiModel: custom systemMessage: "You are an assistant designed to provide accurate answers." userMessage: '${ "User: " + .request.question }' output: { "type": "object", "properties": { "response": { "type": "string", "description": "The AI's response to the user question" } }, "required": ["response"] } maxToolExecutions: 5 memory: memoryId: "session123" memoryType: "message_window" maxMessages: 10 tools: - name: SEARCH_DOCUMENTS description: "Search for relevant documents based on the user's query." parameters: { "type": "object", "properties": { "query": { "type": "string", "description": "The search query" } }, "required": ["query"] } output: { "type": "object", "properties": { "documents": { "type": "array", "items": { "type": "string", "format": "uri" } } }, "required": ["documents"] } llmConfig: strategy: mode: "fallback" onStatusCodes: [429, 500] targets: - provider: "openai" apiKey: "primary-api-key" overrideParams: model: "gpt-4o" - provider: "anthropic" apiKey: "secondary-api-key" overrideParams: model: "claude-2" agentOutcomes: - condition: '${ $agentOutcome.returnValues.response != null }' transition: SuccessState - condition: '${ $agentOutcome.returnValues.response == null }' transition: ErrorState ``` For more detailed information and advanced configurations, refer to the [AI Agent State spec](/xflow/developer-guide/workflow-states-reference/aiagent-state.md). --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.a4b.vn/xflow/developer-guide/ai-agent-development/ai-agent/multi-llm-configuration.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.