Feature: Configuring OpenAI-Compatible (LiteLLM) Models (#2166)

Co-authored-by: bbeiler <bbeiler@ridgelineintl.com>

CONTRIBUTING.md

@@ -16,6 +16,7 @@
     - [Your First Code Contribution](#your-first-code-contribution)
     - [Improving The Documentation](#improving-the-documentation)
   - [Styleguides](#styleguides)
+    - [Pre Commit Hooks](#pre-commit-hooks)
     - [Commit Messages](#commit-messages)
   - [Join The Project Team](#join-the-project-team)
   - [Attribution](#attribution)
@@ -170,6 +171,18 @@ Updating, improving and correcting the documentation
 -->
 
 ## Styleguides
+
+### Pre Commit Hooks
+Make sure to run the pre-commit hooks before committing your code. 
+This will help you to automatically format your code and catch cicd failures early.
+```bash
+# Make sure `pre-commit` is installed
+pip install pre-commit
+
+# Run pre-commit hooks on files in your commit
+pre-commit run --all-files
+```
+
 ### Commit Messages
 <!-- TODO
 

README.md

@@ -307,6 +307,7 @@ More extensive documentation can be found on our [docs page](https://docs.skyver
 | Gemini | Coming soon (contributions welcome) |
 | Llama 3.2 | Coming soon (contributions welcome) | 
 | Novita AI | Llama 3.1 (8B, 70B), Llama 3.2 (1B, 3B, 11B Vision) |
+| OpenAI-compatible | Any custom API endpoint that follows OpenAI's API format (via [liteLLM](https://docs.litellm.ai/docs/providers/openai_compatible)) |
 
 #### Environment Variables
 | Variable | Description| Type | Sample Value|
@@ -317,8 +318,9 @@ More extensive documentation can be found on our [docs page](https://docs.skyver
 | `ENABLE_BEDROCK` | Register AWS Bedrock models. To use AWS Bedrock, you need to make sure your [AWS configurations](https://github.com/boto/boto3?tab=readme-ov-file#using-boto3) are set up correctly first. | Boolean | `true`, `false` |
 | `ENABLE_GEMINI` | Register Gemini models| Boolean | `true`, `false` |
 | `ENABLE_NOVITA`| Register Novita AI models | Boolean | `true`, `false` |
-| `LLM_KEY` | The name of the model you want to use | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `BEDROCK_AMAZON_NOVA_PRO`, `BEDROCK_AMAZON_NOVA_LITE`|
+| `ENABLE_OPENAI_COMPATIBLE`| Register a custom OpenAI-compatible API endpoint | Boolean | `true`, `false` |
-| `SECONDARY_LLM_KEY` | The name of the model for mini agents skyvern runs with | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `NOVITA_DEEPSEEK_R1`, `NOVITA_DEEPSEEK_V3`, `NOVITA_LLAMA_3_3_70B`, `NOVITA_LLAMA_3_2_1B`, `NOVITA_LLAMA_3_2_3B`, `NOVITA_LLAMA_3_2_11B_VISION`, `NOVITA_LLAMA_3_1_8B`, `NOVITA_LLAMA_3_1_70B`, `NOVITA_LLAMA_3_1_405B`, `NOVITA_LLAMA_3_8B`, `NOVITA_LLAMA_3_70B`|
+| `LLM_KEY` | The name of the model you want to use | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `BEDROCK_AMAZON_NOVA_PRO`, `BEDROCK_AMAZON_NOVA_LITE`, `OPENAI_COMPATIBLE`|
+| `SECONDARY_LLM_KEY` | The name of the model for mini agents skyvern runs with | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `NOVITA_DEEPSEEK_R1`, `NOVITA_DEEPSEEK_V3`, `NOVITA_LLAMA_3_3_70B`, `NOVITA_LLAMA_3_2_1B`, `NOVITA_LLAMA_3_2_3B`, `NOVITA_LLAMA_3_2_11B_VISION`, `NOVITA_LLAMA_3_1_8B`, `NOVITA_LLAMA_3_1_70B`, `NOVITA_LLAMA_3_1_405B`, `NOVITA_LLAMA_3_8B`, `NOVITA_LLAMA_3_70B`, `OPENAI_COMPATIBLE`|
 | `OPENAI_API_KEY` | OpenAI API Key | String | `sk-1234567890` |
 | `OPENAI_API_BASE` | OpenAI API Base, optional | String | `https://openai.api.base` |
 | `OPENAI_ORGANIZATION` | OpenAI Organization ID, optional | String | `your-org-id` |
@@ -328,6 +330,18 @@ More extensive documentation can be found on our [docs page](https://docs.skyver
 | `AZURE_API_BASE` | Azure deployment api base url| String | `https://skyvern-deployment.openai.azure.com/`|
 | `AZURE_API_VERSION` | Azure API Version| String | `2024-02-01`|
 | `GEMINI_API_KEY` | Gemini API Key| String | `your_google_gemini_api_key`|
+| `NOVITA_API_KEY` | Novita AI API Key| String | `your_novita_api_key`|
+| `OPENAI_COMPATIBLE_MODEL_NAME` | Model name for OpenAI-compatible endpoint | String | `yi-34b`, `gpt-3.5-turbo`, `mistral-large`, etc.|
+| `OPENAI_COMPATIBLE_API_KEY` | API key for OpenAI-compatible endpoint | String | `sk-1234567890`|
+| `OPENAI_COMPATIBLE_API_BASE` | Base URL for OpenAI-compatible endpoint | String | `https://api.together.xyz/v1`, `http://localhost:8000/v1`, etc.|
+
+#### Environment Variables (OpenAI-Compatible model - additional config)
+| Variable | Description| Type | Sample Value|
+| -------- | ------- | ------- | ------- |
+| `OPENAI_COMPATIBLE_API_VERSION` | API version for OpenAI-compatible endpoint, optional| String | `2023-05-15`|
+| `OPENAI_COMPATIBLE_MAX_TOKENS` | Maximum tokens for completion, optional| Integer | `4096`, `8192`, etc.|
+| `OPENAI_COMPATIBLE_TEMPERATURE` | Temperature setting, optional| Float | `0.0`, `0.5`, `0.7`, etc.|
+| `OPENAI_COMPATIBLE_SUPPORTS_VISION` | Whether model supports vision, optional| Boolean | `true`, `false`|
 
 # Feature Roadmap
 This is our planned roadmap for the next few months. If you have any suggestions or would like to see a feature added, please don't hesitate to reach out to us [via email](mailto:founders@skyvern.com) or [discord](https://discord.gg/fG2XXEuQX3).

docs/development.mdx

@@ -96,3 +96,69 @@ Repeat step 3.
 </AccordionGroup>
 
 Curious about what changed in a CLI version? [Check out the CLI changelog.](/changelog/command-line)
+
+## Environment Variable Configuration
+
+The simplest way to get started with Skyvern is to set environment variables in a `.env` file at the root of the project. This file is loaded by the application at runtime, and the values are used to configure the system.
+
+### LLM Configuration
+
+Skyvern works with multiple LLM providers. You need to set the following environment variables to configure your LLM provider:
+
+```bash
+# One of the below must be set to true
+ENABLE_OPENAI=true
+# ENABLE_ANTHROPIC=true
+# ENABLE_AZURE=true
+# ENABLE_BEDROCK=true
+# ENABLE_GEMINI=true
+# ENABLE_NOVITA=true
+# ENABLE_OPENAI_COMPATIBLE=true
+
+# Set your LLM provider API key
+OPENAI_API_KEY=sk-xxxxxxxxxxxxx
+
+# Set which model to use
+LLM_KEY=OPENAI_GPT4O
+```
+
+If you're using OpenAI, you'll need to set `ENABLE_OPENAI=true` and provide your `OPENAI_API_KEY`. The `LLM_KEY` specifies which model to use, and should match one of the registered models in the `LLMConfigRegistry`.
+
+#### Using Custom OpenAI-compatible Models
+
+Skyvern can also use any OpenAI-compatible LLM API endpoint. This is useful for connecting to alternative providers that follow the OpenAI API format or to self-hosted models. This feature is implemented using [liteLLM's OpenAI-compatible provider support](https://docs.litellm.ai/docs/providers/openai_compatible).
+
+To enable this:
+
+```bash
+# Enable OpenAI-compatible mode
+ENABLE_OPENAI_COMPATIBLE=true
+
+# Required configuration
+OPENAI_COMPATIBLE_MODEL_NAME=gpt-3.5-turbo  # The model name supported by your endpoint (REQUIRED)
+OPENAI_COMPATIBLE_API_KEY=your-api-key      # API key for your endpoint (REQUIRED)
+OPENAI_COMPATIBLE_API_BASE=https://your-api-endpoint.com/v1  # Base URL for your endpoint (REQUIRED)
+
+# Optional configuration
+OPENAI_COMPATIBLE_API_VERSION=2023-05-15  # Optional API version
+OPENAI_COMPATIBLE_MAX_TOKENS=4096         # Optional, defaults to LLM_CONFIG_MAX_TOKENS
+OPENAI_COMPATIBLE_TEMPERATURE=0.0         # Optional, defaults to LLM_CONFIG_TEMPERATURE 
+OPENAI_COMPATIBLE_SUPPORTS_VISION=false   # Optional, defaults to false
+OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX=false  # Optional, defaults to false
+OPENAI_COMPATIBLE_MODEL_KEY=OPENAI_COMPATIBLE  # Optional custom key to register the model with
+```
+
+**Important Note**: When using this feature, the model name you provide will be prefixed with "openai/" in the liteLLM configuration. This is how liteLLM determines the routing for OpenAI-compatible providers.
+
+This feature allows you to use models from providers like:
+- Together.ai
+- Anyscale
+- Mistral
+- Self-hosted models (e.g., using LM Studio or a local vLLM server)
+- Any API endpoint that follows the OpenAI API format
+
+Once configured, you can set `LLM_KEY=OPENAI_COMPATIBLE` to use this model as your primary LLM.
+
+For more detailed information about OpenAI-compatible providers supported by liteLLM, see their [documentation](https://docs.litellm.ai/docs/providers/openai_compatible).
+
+### Running the Server

setup.sh

@@ -161,6 +161,39 @@ setup_llm_providers() {
         update_or_add_env_var "ENABLE_NOVITA" "false"
     fi
 
+    # OpenAI Compatible Configuration
+    echo "To enable an OpenAI-compatible provider, you must have a model name, API key, and API base URL."
+    read -p "Do you want to enable an OpenAI-compatible provider (y/n)? " enable_openai_compatible
+    if [[ "$enable_openai_compatible" == "y" ]]; then
+        read -p "Enter the model name (e.g., 'yi-34b', 'mistral-large'): " openai_compatible_model_name
+        read -p "Enter your API key: " openai_compatible_api_key
+        read -p "Enter the API base URL (e.g., 'https://api.together.xyz/v1'): " openai_compatible_api_base
+        read -p "Does this model support vision (y/n)? " openai_compatible_vision
+        
+        if [ -z "$openai_compatible_model_name" ] || [ -z "$openai_compatible_api_key" ] || [ -z "$openai_compatible_api_base" ]; then
+            echo "Error: All required fields must be populated."
+            echo "OpenAI-compatible provider will not be enabled."
+        else
+            update_or_add_env_var "OPENAI_COMPATIBLE_MODEL_NAME" "$openai_compatible_model_name"
+            update_or_add_env_var "OPENAI_COMPATIBLE_API_KEY" "$openai_compatible_api_key"
+            update_or_add_env_var "OPENAI_COMPATIBLE_API_BASE" "$openai_compatible_api_base"
+            
+            # Set vision support
+            if [[ "$openai_compatible_vision" == "y" ]]; then
+                update_or_add_env_var "OPENAI_COMPATIBLE_SUPPORTS_VISION" "true"
+            else
+                update_or_add_env_var "OPENAI_COMPATIBLE_SUPPORTS_VISION" "false"
+            fi
+
+            update_or_add_env_var "ENABLE_OPENAI_COMPATIBLE" "true"
+            model_options+=(
+                "OPENAI_COMPATIBLE"
+            )
+        fi
+    else
+        update_or_add_env_var "ENABLE_OPENAI_COMPATIBLE" "false"
+    fi
+
     # Model Selection
     if [ ${#model_options[@]} -eq 0 ]; then
         echo "No LLM providers enabled. You won't be able to run Skyvern unless you enable at least one provider. You can re-run this script to enable providers or manually update the .env file."

skyvern/cli/commands.py

@@ -326,6 +326,39 @@ def setup_llm_providers() -> None:
     else:
         update_or_add_env_var("ENABLE_NOVITA", "false")
 
+    # OpenAI Compatible Configuration
+    print("To enable an OpenAI-compatible provider, you must have a model name, API key, and API base URL.")
+    enable_openai_compatible = input("Do you want to enable an OpenAI-compatible provider (y/n)? ").lower() == "y"
+    if enable_openai_compatible:
+        openai_compatible_model_name = input("Enter the model name (e.g., 'yi-34b', 'mistral-large'): ")
+        openai_compatible_api_key = input("Enter your API key: ")
+        openai_compatible_api_base = input("Enter the API base URL (e.g., 'https://api.together.xyz/v1'): ")
+        openai_compatible_vision = input("Does this model support vision (y/n)? ").lower() == "y"
+
+        if not all([openai_compatible_model_name, openai_compatible_api_key, openai_compatible_api_base]):
+            print("Error: All required fields must be populated.")
+            print("OpenAI-compatible provider will not be enabled.")
+        else:
+            update_or_add_env_var("OPENAI_COMPATIBLE_MODEL_NAME", openai_compatible_model_name)
+            update_or_add_env_var("OPENAI_COMPATIBLE_API_KEY", openai_compatible_api_key)
+            update_or_add_env_var("OPENAI_COMPATIBLE_API_BASE", openai_compatible_api_base)
+
+            # Set vision support
+            if openai_compatible_vision:
+                update_or_add_env_var("OPENAI_COMPATIBLE_SUPPORTS_VISION", "true")
+            else:
+                update_or_add_env_var("OPENAI_COMPATIBLE_SUPPORTS_VISION", "false")
+
+            # Optional: Ask for API version
+            openai_compatible_api_version = input("Enter API version (optional, press enter to skip): ")
+            if openai_compatible_api_version:
+                update_or_add_env_var("OPENAI_COMPATIBLE_API_VERSION", openai_compatible_api_version)
+
+            update_or_add_env_var("ENABLE_OPENAI_COMPATIBLE", "true")
+            model_options.append("OPENAI_COMPATIBLE")
+    else:
+        update_or_add_env_var("ENABLE_OPENAI_COMPATIBLE", "false")
+
     # Model Selection
     if not model_options:
         print(

skyvern/config.py

@@ -124,10 +124,24 @@ class Settings(BaseSettings):
     ENABLE_BEDROCK: bool = False
     ENABLE_GEMINI: bool = False
     ENABLE_AZURE_CUA: bool = False
+    ENABLE_OPENAI_COMPATIBLE: bool = False
     # OPENAI
     OPENAI_API_KEY: str | None = None
     # ANTHROPIC
     ANTHROPIC_API_KEY: str | None = None
+
+    # OPENAI COMPATIBLE
+    OPENAI_COMPATIBLE_MODEL_NAME: str | None = None
+    OPENAI_COMPATIBLE_API_KEY: str | None = None
+    OPENAI_COMPATIBLE_API_BASE: str | None = None
+    OPENAI_COMPATIBLE_API_VERSION: str | None = None
+    OPENAI_COMPATIBLE_MAX_TOKENS: int | None = None
+    OPENAI_COMPATIBLE_TEMPERATURE: float | None = None
+    OPENAI_COMPATIBLE_SUPPORTS_VISION: bool = False
+    OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX: bool = False
+    OPENAI_COMPATIBLE_MODEL_KEY: str = "OPENAI_COMPATIBLE"
+    OPENAI_COMPATIBLE_REASONING_EFFORT: str | None = None
+
     # AZURE
     AZURE_DEPLOYMENT: str | None = None
     AZURE_API_KEY: str | None = None

skyvern/forge/sdk/api/llm/config_registry.py

@@ -606,3 +606,45 @@ if settings.ENABLE_NOVITA:
             ),
         ),
     )
+
+# Add support for dynamically configuring OpenAI-compatible LLM models
+# Based on liteLLM's support for OpenAI-compatible APIs
+# See documentation: https://docs.litellm.ai/docs/providers/openai_compatible
+if settings.ENABLE_OPENAI_COMPATIBLE:
+    # Check for required model name
+    model_key = settings.OPENAI_COMPATIBLE_MODEL_KEY
+    model_name = settings.OPENAI_COMPATIBLE_MODEL_NAME
+
+    if not model_name:
+        raise InvalidLLMConfigError(
+            "OPENAI_COMPATIBLE_MODEL_NAME is required but not set. OpenAI-compatible model will not be registered."
+        )
+    else:
+        # Required environment variables to check
+        required_env_vars = ["OPENAI_COMPATIBLE_API_KEY", "OPENAI_COMPATIBLE_MODEL_NAME", "OPENAI_COMPATIBLE_API_BASE"]
+
+        # Configure litellm parameters - note the "openai/" prefix required for liteLLM routing
+        litellm_params = LiteLLMParams(
+            api_key=settings.OPENAI_COMPATIBLE_API_KEY,
+            api_base=settings.OPENAI_COMPATIBLE_API_BASE,
+            api_version=settings.OPENAI_COMPATIBLE_API_VERSION,
+            model_info={"model_name": f"openai/{model_name}"},
+        )
+
+        # Configure LLMConfig
+        LLMConfigRegistry.register_config(
+            model_key,
+            LLMConfig(
+                f"openai/{model_name}",  # Add openai/ prefix for liteLLM
+                required_env_vars,
+                supports_vision=settings.OPENAI_COMPATIBLE_SUPPORTS_VISION,
+                add_assistant_prefix=settings.OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX,
+                max_completion_tokens=settings.OPENAI_COMPATIBLE_MAX_TOKENS or settings.LLM_CONFIG_MAX_TOKENS,
+                temperature=settings.OPENAI_COMPATIBLE_TEMPERATURE
+                if settings.OPENAI_COMPATIBLE_TEMPERATURE is not None
+                else settings.LLM_CONFIG_TEMPERATURE,
+                litellm_params=litellm_params,
+                reasoning_effort=settings.OPENAI_COMPATIBLE_REASONING_EFFORT,
+            ),
+        )
+        LOG.info(f"Registered OpenAI-compatible model with key {model_key}", model_name=model_name)