vodorod/Dorod-Sky
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Feature: Configuring OpenAI-Compatible (LiteLLM) Models (#2166)

Browse Source 
Co-authored-by: bbeiler <bbeiler@ridgelineintl.com>
This commit is contained in:
Brandon Beiler
2025-04-20 20:25:59 -04:00
committed by GitHub
parent 9038e6e047
commit 3d381a60f0
7 changed files with 217 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
CONTRIBUTING.md
View File

@@ -16,6 +16,7 @@
- [Your First Code Contribution](#your-first-code-contribution)
- [Improving The Documentation](#improving-the-documentation)
- [Styleguides](#styleguides)
- [Pre Commit Hooks](#pre-commit-hooks)
- [Commit Messages](#commit-messages)
- [Join The Project Team](#join-the-project-team)
- [Attribution](#attribution)
@@ -170,6 +171,18 @@ Updating, improving and correcting the documentation
-->
## Styleguides
### Pre Commit Hooks
Make sure to run the pre-commit hooks before committing your code.
This will help you to automatically format your code and catch cicd failures early.
```bash
# Make sure `pre-commit` is installed
pip install pre-commit
# Run pre-commit hooks on files in your commit
pre-commit run --all-files
```
### Commit Messages
<!-- TODO

18
README.md
View File

@@ -307,6 +307,7 @@ More extensive documentation can be found on our [docs page](https://docs.skyver
| Gemini | Coming soon (contributions welcome) |
| Llama 3.2 | Coming soon (contributions welcome) |
| Novita AI | Llama 3.1 (8B, 70B), Llama 3.2 (1B, 3B, 11B Vision) |
| OpenAI-compatible | Any custom API endpoint that follows OpenAI's API format (via [liteLLM](https://docs.litellm.ai/docs/providers/openai_compatible)) |
#### Environment Variables
| Variable | Description| Type | Sample Value|
@@ -317,8 +318,9 @@ More extensive documentation can be found on our [docs page](https://docs.skyver
| `ENABLE_BEDROCK` | Register AWS Bedrock models. To use AWS Bedrock, you need to make sure your [AWS configurations](https://github.com/boto/boto3?tab=readme-ov-file#using-boto3) are set up correctly first. | Boolean | `true`, `false` |
| `ENABLE_GEMINI` | Register Gemini models| Boolean | `true`, `false` |
| `ENABLE_NOVITA`| Register Novita AI models | Boolean | `true`, `false` |
| `LLM_KEY` | The name of the model you want to use | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `BEDROCK_AMAZON_NOVA_PRO`, `BEDROCK_AMAZON_NOVA_LITE`|
| `SECONDARY_LLM_KEY` | The name of the model for mini agents skyvern runs with | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `NOVITA_DEEPSEEK_R1`, `NOVITA_DEEPSEEK_V3`, `NOVITA_LLAMA_3_3_70B`, `NOVITA_LLAMA_3_2_1B`, `NOVITA_LLAMA_3_2_3B`, `NOVITA_LLAMA_3_2_11B_VISION`, `NOVITA_LLAMA_3_1_8B`, `NOVITA_LLAMA_3_1_70B`, `NOVITA_LLAMA_3_1_405B`, `NOVITA_LLAMA_3_8B`, `NOVITA_LLAMA_3_70B`|
| `ENABLE_OPENAI_COMPATIBLE`| Register a custom OpenAI-compatible API endpoint | Boolean | `true`, `false` |
| `LLM_KEY` | The name of the model you want to use | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `BEDROCK_AMAZON_NOVA_PRO`, `BEDROCK_AMAZON_NOVA_LITE`, `OPENAI_COMPATIBLE`|
| `SECONDARY_LLM_KEY` | The name of the model for mini agents skyvern runs with | String | Currently supported llm keys: `OPENAI_GPT4_TURBO`, `OPENAI_GPT4V`, `OPENAI_GPT4O`, `OPENAI_GPT4O_MINI`, `ANTHROPIC_CLAUDE3`, `ANTHROPIC_CLAUDE3_OPUS`, `ANTHROPIC_CLAUDE3_SONNET`, `ANTHROPIC_CLAUDE3_HAIKU`, `ANTHROPIC_CLAUDE3.5_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_OPUS`, `BEDROCK_ANTHROPIC_CLAUDE3_SONNET`, `BEDROCK_ANTHROPIC_CLAUDE3_HAIKU`, `BEDROCK_ANTHROPIC_CLAUDE3.5_SONNET`, `AZURE_OPENAI`, `GEMINI_PRO`, `GEMINI_FLASH`, `NOVITA_DEEPSEEK_R1`, `NOVITA_DEEPSEEK_V3`, `NOVITA_LLAMA_3_3_70B`, `NOVITA_LLAMA_3_2_1B`, `NOVITA_LLAMA_3_2_3B`, `NOVITA_LLAMA_3_2_11B_VISION`, `NOVITA_LLAMA_3_1_8B`, `NOVITA_LLAMA_3_1_70B`, `NOVITA_LLAMA_3_1_405B`, `NOVITA_LLAMA_3_8B`, `NOVITA_LLAMA_3_70B`, `OPENAI_COMPATIBLE`|
| `OPENAI_API_KEY` | OpenAI API Key | String | `sk-1234567890` |
| `OPENAI_API_BASE` | OpenAI API Base, optional | String | `https://openai.api.base` |
| `OPENAI_ORGANIZATION` | OpenAI Organization ID, optional | String | `your-org-id` |
@@ -328,6 +330,18 @@ More extensive documentation can be found on our [docs page](https://docs.skyver
| `AZURE_API_BASE` | Azure deployment api base url| String | `https://skyvern-deployment.openai.azure.com/`|
| `AZURE_API_VERSION` | Azure API Version| String | `2024-02-01`|
| `GEMINI_API_KEY` | Gemini API Key| String | `your_google_gemini_api_key`|
| `NOVITA_API_KEY` | Novita AI API Key| String | `your_novita_api_key`|
| `OPENAI_COMPATIBLE_MODEL_NAME` | Model name for OpenAI-compatible endpoint | String | `yi-34b`, `gpt-3.5-turbo`, `mistral-large`, etc.|
| `OPENAI_COMPATIBLE_API_KEY` | API key for OpenAI-compatible endpoint | String | `sk-1234567890`|
| `OPENAI_COMPATIBLE_API_BASE` | Base URL for OpenAI-compatible endpoint | String | `https://api.together.xyz/v1`, `http://localhost:8000/v1`, etc.|
#### Environment Variables (OpenAI-Compatible model - additional config)
| Variable | Description| Type | Sample Value|
| -------- | ------- | ------- | ------- |
| `OPENAI_COMPATIBLE_API_VERSION` | API version for OpenAI-compatible endpoint, optional| String | `2023-05-15`|
| `OPENAI_COMPATIBLE_MAX_TOKENS` | Maximum tokens for completion, optional| Integer | `4096`, `8192`, etc.|
| `OPENAI_COMPATIBLE_TEMPERATURE` | Temperature setting, optional| Float | `0.0`, `0.5`, `0.7`, etc.|
| `OPENAI_COMPATIBLE_SUPPORTS_VISION` | Whether model supports vision, optional| Boolean | `true`, `false`|
# Feature Roadmap
This is our planned roadmap for the next few months. If you have any suggestions or would like to see a feature added, please don't hesitate to reach out to us [via email](mailto:founders@skyvern.com) or [discord](https://discord.gg/fG2XXEuQX3).

66
docs/development.mdx
View File

@@ -96,3 +96,69 @@ Repeat step 3.
</AccordionGroup>
Curious about what changed in a CLI version? [Check out the CLI changelog.](/changelog/command-line)
## Environment Variable Configuration
The simplest way to get started with Skyvern is to set environment variables in a `.env` file at the root of the project. This file is loaded by the application at runtime, and the values are used to configure the system.
### LLM Configuration
Skyvern works with multiple LLM providers. You need to set the following environment variables to configure your LLM provider:
```bash
# One of the below must be set to true
ENABLE_OPENAI=true
# ENABLE_ANTHROPIC=true
# ENABLE_AZURE=true
# ENABLE_BEDROCK=true
# ENABLE_GEMINI=true
# ENABLE_NOVITA=true
# ENABLE_OPENAI_COMPATIBLE=true
# Set your LLM provider API key
OPENAI_API_KEY=sk-xxxxxxxxxxxxx
# Set which model to use
LLM_KEY=OPENAI_GPT4O
```
If you're using OpenAI, you'll need to set `ENABLE_OPENAI=true` and provide your `OPENAI_API_KEY`. The `LLM_KEY` specifies which model to use, and should match one of the registered models in the `LLMConfigRegistry`.
#### Using Custom OpenAI-compatible Models
Skyvern can also use any OpenAI-compatible LLM API endpoint. This is useful for connecting to alternative providers that follow the OpenAI API format or to self-hosted models. This feature is implemented using [liteLLM's OpenAI-compatible provider support](https://docs.litellm.ai/docs/providers/openai_compatible).
To enable this:
```bash
# Enable OpenAI-compatible mode
ENABLE_OPENAI_COMPATIBLE=true
# Required configuration
OPENAI_COMPATIBLE_MODEL_NAME=gpt-3.5-turbo # The model name supported by your endpoint (REQUIRED)
OPENAI_COMPATIBLE_API_KEY=your-api-key # API key for your endpoint (REQUIRED)
OPENAI_COMPATIBLE_API_BASE=https://your-api-endpoint.com/v1 # Base URL for your endpoint (REQUIRED)
# Optional configuration
OPENAI_COMPATIBLE_API_VERSION=2023-05-15 # Optional API version
OPENAI_COMPATIBLE_MAX_TOKENS=4096 # Optional, defaults to LLM_CONFIG_MAX_TOKENS
OPENAI_COMPATIBLE_TEMPERATURE=0.0 # Optional, defaults to LLM_CONFIG_TEMPERATURE
OPENAI_COMPATIBLE_SUPPORTS_VISION=false # Optional, defaults to false
OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX=false # Optional, defaults to false
OPENAI_COMPATIBLE_MODEL_KEY=OPENAI_COMPATIBLE # Optional custom key to register the model with
```
**Important Note**: When using this feature, the model name you provide will be prefixed with "openai/" in the liteLLM configuration. This is how liteLLM determines the routing for OpenAI-compatible providers.
This feature allows you to use models from providers like:
- Together.ai
- Anyscale
- Mistral
- Self-hosted models (e.g., using LM Studio or a local vLLM server)
- Any API endpoint that follows the OpenAI API format
Once configured, you can set `LLM_KEY=OPENAI_COMPATIBLE` to use this model as your primary LLM.
For more detailed information about OpenAI-compatible providers supported by liteLLM, see their [documentation](https://docs.litellm.ai/docs/providers/openai_compatible).
### Running the Server

33
setup.sh
View File

@@ -161,6 +161,39 @@ setup_llm_providers() {
update_or_add_env_var "ENABLE_NOVITA" "false"
fi
# OpenAI Compatible Configuration
echo "To enable an OpenAI-compatible provider, you must have a model name, API key, and API base URL."
read -p "Do you want to enable an OpenAI-compatible provider (y/n)? " enable_openai_compatible
if [[ "$enable_openai_compatible" == "y" ]]; then
read -p "Enter the model name (e.g., 'yi-34b', 'mistral-large'): " openai_compatible_model_name
read -p "Enter your API key: " openai_compatible_api_key
read -p "Enter the API base URL (e.g., 'https://api.together.xyz/v1'): " openai_compatible_api_base
read -p "Does this model support vision (y/n)? " openai_compatible_vision
if [ -z "$openai_compatible_model_name" ] || [ -z "$openai_compatible_api_key" ] || [ -z "$openai_compatible_api_base" ]; then
echo "Error: All required fields must be populated."
echo "OpenAI-compatible provider will not be enabled."
else
update_or_add_env_var "OPENAI_COMPATIBLE_MODEL_NAME" "$openai_compatible_model_name"
update_or_add_env_var "OPENAI_COMPATIBLE_API_KEY" "$openai_compatible_api_key"
update_or_add_env_var "OPENAI_COMPATIBLE_API_BASE" "$openai_compatible_api_base"
# Set vision support
if [[ "$openai_compatible_vision" == "y" ]]; then
update_or_add_env_var "OPENAI_COMPATIBLE_SUPPORTS_VISION" "true"
else
update_or_add_env_var "OPENAI_COMPATIBLE_SUPPORTS_VISION" "false"
fi
update_or_add_env_var "ENABLE_OPENAI_COMPATIBLE" "true"
model_options+=(
"OPENAI_COMPATIBLE"
)
fi
else
update_or_add_env_var "ENABLE_OPENAI_COMPATIBLE" "false"
fi
# Model Selection
if [ ${#model_options[@]} -eq 0 ]; then
echo "No LLM providers enabled. You won't be able to run Skyvern unless you enable at least one provider. You can re-run this script to enable providers or manually update the .env file."

33
skyvern/cli/commands.py
View File

@@ -326,6 +326,39 @@ def setup_llm_providers() -> None:
else:
update_or_add_env_var("ENABLE_NOVITA", "false")
# OpenAI Compatible Configuration
print("To enable an OpenAI-compatible provider, you must have a model name, API key, and API base URL.")
enable_openai_compatible = input("Do you want to enable an OpenAI-compatible provider (y/n)? ").lower() == "y"
if enable_openai_compatible:
openai_compatible_model_name = input("Enter the model name (e.g., 'yi-34b', 'mistral-large'): ")
openai_compatible_api_key = input("Enter your API key: ")
openai_compatible_api_base = input("Enter the API base URL (e.g., 'https://api.together.xyz/v1'): ")
openai_compatible_vision = input("Does this model support vision (y/n)? ").lower() == "y"
if not all([openai_compatible_model_name, openai_compatible_api_key, openai_compatible_api_base]):
print("Error: All required fields must be populated.")
print("OpenAI-compatible provider will not be enabled.")
else:
update_or_add_env_var("OPENAI_COMPATIBLE_MODEL_NAME", openai_compatible_model_name)
update_or_add_env_var("OPENAI_COMPATIBLE_API_KEY", openai_compatible_api_key)
update_or_add_env_var("OPENAI_COMPATIBLE_API_BASE", openai_compatible_api_base)
# Set vision support
if openai_compatible_vision:
update_or_add_env_var("OPENAI_COMPATIBLE_SUPPORTS_VISION", "true")
else:
update_or_add_env_var("OPENAI_COMPATIBLE_SUPPORTS_VISION", "false")
# Optional: Ask for API version
openai_compatible_api_version = input("Enter API version (optional, press enter to skip): ")
if openai_compatible_api_version:
update_or_add_env_var("OPENAI_COMPATIBLE_API_VERSION", openai_compatible_api_version)
update_or_add_env_var("ENABLE_OPENAI_COMPATIBLE", "true")
model_options.append("OPENAI_COMPATIBLE")
else:
update_or_add_env_var("ENABLE_OPENAI_COMPATIBLE", "false")
# Model Selection
if not model_options:
print(

14
skyvern/config.py
View File

@@ -124,10 +124,24 @@ class Settings(BaseSettings):
ENABLE_BEDROCK: bool = False
ENABLE_GEMINI: bool = False
ENABLE_AZURE_CUA: bool = False
ENABLE_OPENAI_COMPATIBLE: bool = False
# OPENAI
OPENAI_API_KEY: str | None = None
# ANTHROPIC
ANTHROPIC_API_KEY: str | None = None
# OPENAI COMPATIBLE
OPENAI_COMPATIBLE_MODEL_NAME: str | None = None
OPENAI_COMPATIBLE_API_KEY: str | None = None
OPENAI_COMPATIBLE_API_BASE: str | None = None
OPENAI_COMPATIBLE_API_VERSION: str | None = None
OPENAI_COMPATIBLE_MAX_TOKENS: int | None = None
OPENAI_COMPATIBLE_TEMPERATURE: float | None = None
OPENAI_COMPATIBLE_SUPPORTS_VISION: bool = False
OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX: bool = False
OPENAI_COMPATIBLE_MODEL_KEY: str = "OPENAI_COMPATIBLE"
OPENAI_COMPATIBLE_REASONING_EFFORT: str | None = None
# AZURE
AZURE_DEPLOYMENT: str | None = None
AZURE_API_KEY: str | None = None

42
skyvern/forge/sdk/api/llm/config_registry.py
View File

@@ -606,3 +606,45 @@ if settings.ENABLE_NOVITA:
),
),
)
# Add support for dynamically configuring OpenAI-compatible LLM models
# Based on liteLLM's support for OpenAI-compatible APIs
# See documentation: https://docs.litellm.ai/docs/providers/openai_compatible
if settings.ENABLE_OPENAI_COMPATIBLE:
# Check for required model name
model_key = settings.OPENAI_COMPATIBLE_MODEL_KEY
model_name = settings.OPENAI_COMPATIBLE_MODEL_NAME
if not model_name:
raise InvalidLLMConfigError(
"OPENAI_COMPATIBLE_MODEL_NAME is required but not set. OpenAI-compatible model will not be registered."
)
else:
# Required environment variables to check
required_env_vars = ["OPENAI_COMPATIBLE_API_KEY", "OPENAI_COMPATIBLE_MODEL_NAME", "OPENAI_COMPATIBLE_API_BASE"]
# Configure litellm parameters - note the "openai/" prefix required for liteLLM routing
litellm_params = LiteLLMParams(
api_key=settings.OPENAI_COMPATIBLE_API_KEY,
api_base=settings.OPENAI_COMPATIBLE_API_BASE,
api_version=settings.OPENAI_COMPATIBLE_API_VERSION,
model_info={"model_name": f"openai/{model_name}"},
)
# Configure LLMConfig
LLMConfigRegistry.register_config(
model_key,
LLMConfig(
f"openai/{model_name}", # Add openai/ prefix for liteLLM
required_env_vars,
supports_vision=settings.OPENAI_COMPATIBLE_SUPPORTS_VISION,
add_assistant_prefix=settings.OPENAI_COMPATIBLE_ADD_ASSISTANT_PREFIX,
max_completion_tokens=settings.OPENAI_COMPATIBLE_MAX_TOKENS or settings.LLM_CONFIG_MAX_TOKENS,
temperature=settings.OPENAI_COMPATIBLE_TEMPERATURE
if settings.OPENAI_COMPATIBLE_TEMPERATURE is not None
else settings.LLM_CONFIG_TEMPERATURE,
litellm_params=litellm_params,
reasoning_effort=settings.OPENAI_COMPATIBLE_REASONING_EFFORT,
),
)
LOG.info(f"Registered OpenAI-compatible model with key {model_key}", model_name=model_name)