feat: add Together AI integration and provider implementation guide

[mouimet-infinisoft]

Related issue: #307

• Create detailed provider implementation guide with:
  • Architecture overview and implementation steps
  • Configuration patterns and best practices
  • Testing checklist and Docker integration guide
  • Example using Together AI implementation
• Add Together AI as new provider with:
  • Environment variables and Docker configuration
  • Support for Qwen, Llama, and Mixtral models
  • API key and base URL management
  • OpenAI-compatible API integration

Question ?

I wrote a small documentation about the AI Provider Architecture and implementation steps to add new provider in the future. Where should it be stored, is there a knowledge base, wiki or storage somewhere?

---

AI Provider Implementation Guide

Architecture Overview

The system follows a modular architecture for managing LLM providers and their models. Here's how the components interact:

1. Application Layer: Orchestrates the initialization and dynamic loading of providers and models
2. PROVIDER_LIST: Contains static configurations and dynamic loading capabilities for each provider
3. ModelList: Maintains the combined list of static and dynamically loaded models
4. Dynamic Providers: Handles runtime model discovery and API interactions
5. External APIs: The actual LLM provider services

Diagram

Implementation Steps

1. Environment Variables Setup

First, add the necessary environment variables in .env.example:

# Provider API Base URL (if required)
PROVIDER_NAME_API_BASE_URL=

# Provider API Key (if required)
PROVIDER_NAME_API_KEY=

Example from Together AI implementation:

TOGETHER_API_BASE_URL=
TOGETHER_API_KEY=

2. Docker Configuration

Update the Docker-related files to include the new provider's environment variables:

1. Update Dockerfile:

# Add in both build and runtime stages
ARG PROVIDER_NAME_API_KEY
ARG PROVIDER_NAME_API_BASE_URL

# Add to ENV section in both stages
ENV PROVIDER_NAME_API_KEY=${PROVIDER_NAME_API_KEY} \
    PROVIDER_NAME_API_BASE_URL=${PROVIDER_NAME_API_BASE_URL}

2. Update docker-compose.yaml:

services:
  app:
    environment:
      - PROVIDER_NAME_API_KEY=${PROVIDER_NAME_API_KEY}
      - PROVIDER_NAME_API_BASE_URL=${PROVIDER_NAME_API_BASE_URL}

3. Type Definitions

Add the environment variable types to worker-configuration.d.ts:

interface Env {
  PROVIDER_NAME_API_KEY: string;
  PROVIDER_NAME_API_BASE_URL: string;
}

4. Provider List Configuration

Add the provider information in app/utils/constants.ts. This is the entry point for both static and dynamic model configurations:

const PROVIDER_LIST: ProviderInfo[] = [
  {
    name: 'ProviderName',
    staticModels: [
      {
        name: 'model-identifier',
        label: 'Display Name',
        provider: 'ProviderName',
        maxTokenAllowed: 8000,
      }
    ],
    getDynamicModels: async (apiKey: string, baseURL: string) => {
      // Implementation for dynamic model loading
      return [/* ModelInfo[] */];
    }
  }
];

5. API Key and Base URL Handling

Modify app/lib/.server/llm/api-key.ts to include the new provider:

// API Key Resolution
export function getAPIKey(cloudflareEnv: Env, provider: string, userApiKeys?: Record<string, string>) {
  switch (provider) {
    case 'ProviderName':
      return env.PROVIDER_NAME_API_KEY || cloudflareEnv.PROVIDER_NAME_API_KEY;
    // ... other providers
  }
}

// Base URL Resolution
export function getBaseURL(cloudflareEnv: Env, provider: string) {
  switch (provider) {
    case 'ProviderName':
      return env.PROVIDER_NAME_API_BASE_URL || cloudflareEnv.PROVIDER_NAME_API_BASE_URL;
    // ... other providers
  }
}

6. Model Implementation

Update app/lib/.server/llm/model.ts to handle the new provider:

export function getModel(provider: string, model: string, env: Env, apiKeys?: Record<string, string>) {
  const apiKey = getAPIKey(env, provider, apiKeys);
  const baseURL = getBaseURL(env, provider);

  switch (provider) {
    case 'ProviderName':
      return getProviderNameModel(baseURL, apiKey, model);
    // ... other providers
  }
}

// Provider-specific implementation
function getProviderNameModel(baseURL: string, apiKey: string, model: string) {
  return {
    generate: async (prompt: string, options: GenerateOptions) => {
      // Implementation
    },
    // Additional methods
  };
}

7. Dynamic Model Loading (Optional)

If your provider supports dynamic model discovery, implement the getDynamicModels function:

async function getProviderNameModels(apiKey: string, baseURL: string): Promise<ModelInfo[]> {
  try {
    const response = await fetch(`${baseURL}/models`, {
      headers: {
        'Authorization': `Bearer ${apiKey}`,
      },
    });
    
    const data = await response.json();
    return data.models.map(model => ({
      name: model.id,
      label: model.name,
      provider: 'ProviderName',
      maxTokenAllowed: model.context_length,
    }));
  } catch (error) {
    console.error('Failed to fetch models:', error);
    return [];
  }
}

Implementation Patterns

1. OpenAI-Compatible APIs

For providers offering OpenAI-compatible APIs (like Together AI):

case 'ProviderName':
  return getOpenAILikeModel(baseURL, apiKey, model);

2. Custom APIs

For providers with unique APIs:

function getProviderNameModel(baseURL: string, apiKey: string, model: string) {
  return {
    generate: async (prompt: string, options: GenerateOptions) => {
      const response = await fetch(`${baseURL}/generate`, {
        method: 'POST',
        headers: {
          'Authorization': `Bearer ${apiKey}`,
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({
          model,
          prompt,
          // Provider-specific options
        }),
      });
      
      return handleProviderResponse(await response.json());
    },
  };
}

Example: Together AI Implementation

The Together AI implementation demonstrates using the OpenAI-compatible API pattern:

1. Environment Variables:

TOGETHER_API_BASE_URL=
TOGETHER_API_KEY=

2. Docker Configuration:

ARG TOGETHER_API_KEY
ARG TOGETHER_API_BASE_URL

ENV TOGETHER_API_KEY=${TOGETHER_API_KEY} \
    TOGETHER_API_BASE_URL=${TOGETHER_API_BASE_URL}

3. Provider List Configuration:

{
  name: 'Together',
  staticModels: [
    { 
      name: 'meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo',
      label: 'meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo',
      provider: 'Together',
      maxTokenAllowed: 8000,
    }
  ],
}

4. API Integration:

case 'Together':
  return getOpenAILikeModel(baseURL, apiKey, model);

Testing Checklist

1. Environment Setup

   • Add required environment variables
   • Configure API keys and base URLs
   • Update Docker configurations

2. Static Configuration

   • Verify provider list entry
   • Confirm static model definitions

3. Dynamic Loading

   • Test dynamic model fetching
   • Verify error handling

4. API Integration

   • Test model generation
   • Verify response handling
   • Check token limits

5. Error Handling

   • Test invalid API keys
   • Verify rate limit handling
   • Check error messages

6. Docker Integration

   • Test building Docker image with new variables
   • Verify environment variable passing
   • Test Docker Compose configuration

Best Practices

1. Error Handling: Implement comprehensive error handling for API calls
2. Rate Limiting: Consider implementing rate limiting for API calls
3. Validation: Validate API responses and model configurations
4. Documentation: Document provider-specific features and limitations
5. Testing: Create tests for both success and failure scenarios
6. Docker: Follow the existing pattern for environment variable handling in Docker configurations

Notes

• Follow existing patterns for error handling and response formatting
• Consider rate limiting and quota management
• Document any provider-specific limitations or requirements
• Update relevant configuration types and interfaces
• Ensure Docker configurations are consistent with existing providers

[wonderwhy-er]

wrote a small documentation about the AI Provider Architecture and implementation steps to add new provider in the future. Where should it be stored, is there a knowledge base, wiki or storage somewhere?

I would say, separate md file linked from readme?
may be add documentation folder

[wonderwhy-er]

Got to test together ai and does not work unless env has https://api.together.xyz/v1 setup
But why do we need it in env? It could be hardcoded no?