124 lines
3.5 KiB
Plaintext
124 lines
3.5 KiB
Plaintext
|
---
|
|||
|
|
title: Ollama + LiteLLM
|
|||
|
|
description: Run Skyvern with a local LLM via Ollama or an OpenAI-compatible proxy via LiteLLM.
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Overview
|
|||
|
|
|
|||
|
|
Skyvern can use local models via **Ollama** or any **OpenAI-compatible** endpoint (e.g., **LiteLLM**). Two paths:
|
|||
|
|
|
|||
|
|
- **(A) Direct Ollama** — use the Ollama API (`/v1/chat/completions`)
|
|||
|
|
- **(B) OpenAI-compatible (LiteLLM)** — Skyvern connects to a proxy that exposes an OpenAI-style API
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## A) Direct Ollama
|
|||
|
|
|
|||
|
|
### 1) Start Ollama locally
|
|||
|
|
Install Ollama and run a model (example: `llama3.1`):
|
|||
|
|
```bash
|
|||
|
|
ollama pull llama3.1
|
|||
|
|
ollama serve
|
|||
|
|
```
|
|||
|
|
The API is usually at `http://localhost:11434`.
|
|||
|
|
|
|||
|
|
### 2) Configure Skyvern (ENV)
|
|||
|
|
Add to your `.env`:
|
|||
|
|
```bash
|
|||
|
|
# Enable Ollama integration
|
|||
|
|
ENABLE_OLLAMA=true
|
|||
|
|
|
|||
|
|
# Ollama server URL
|
|||
|
|
OLLAMA_SERVER_URL=http://localhost:11434
|
|||
|
|
|
|||
|
|
# Model name in Ollama (check with `ollama list`)
|
|||
|
|
OLLAMA_MODEL=llama3.1
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
> Note: Ollama may not support `max_completion_tokens` — Skyvern handles this internally.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## B) OpenAI-compatible via LiteLLM
|
|||
|
|
|
|||
|
|
### 1) Run LiteLLM as a proxy
|
|||
|
|
Minimal example (see LiteLLM docs for more options):
|
|||
|
|
```bash
|
|||
|
|
# Map an Ollama model behind an OpenAI-compatible endpoint
|
|||
|
|
litellm --model ollama/llama3.1 --host 0.0.0.0 --port 4000
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### 2) Configure Skyvern (ENV)
|
|||
|
|
Add to your `.env`:
|
|||
|
|
```bash
|
|||
|
|
# Enable OpenAI-compatible provider
|
|||
|
|
ENABLE_OPENAI_COMPATIBLE=true
|
|||
|
|
|
|||
|
|
# The "model" exposed by the proxy (any identifier your proxy accepts)
|
|||
|
|
OPENAI_COMPATIBLE_MODEL_NAME=llama3.1
|
|||
|
|
|
|||
|
|
# API key required by the proxy (use any placeholder if proxy doesn't enforce)
|
|||
|
|
OPENAI_COMPATIBLE_API_KEY=sk-test
|
|||
|
|
|
|||
|
|
# Base URL for the LiteLLM proxy (OpenAI-compatible)
|
|||
|
|
OPENAI_COMPATIBLE_API_BASE=http://localhost:4000/v1
|
|||
|
|
|
|||
|
|
# (optional)
|
|||
|
|
# OPENAI_COMPATIBLE_API_VERSION=2024-06-01
|
|||
|
|
# OPENAI_COMPATIBLE_SUPPORTS_VISION=false
|
|||
|
|
# OPENAI_COMPATIBLE_REASONING_EFFORT=low|medium|high
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Start Skyvern (local)
|
|||
|
|
|
|||
|
|
After setting the environment variables in `.env`, start backend + UI:
|
|||
|
|
```bash
|
|||
|
|
# with Docker (recommended)
|
|||
|
|
docker compose up -d
|
|||
|
|
|
|||
|
|
# or locally (dev):
|
|||
|
|
# poetry install
|
|||
|
|
# ./run_skyvern.sh
|
|||
|
|
# ./run_ui.sh
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Open the UI and pick the model (or keep default); if only Ollama/LiteLLM are enabled, Skyvern will use that.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Verify your setup
|
|||
|
|
|
|||
|
|
Before starting Skyvern, quickly verify that your LLM endpoint is reachable.
|
|||
|
|
|
|||
|
|
### Ollama
|
|||
|
|
```bash
|
|||
|
|
# Should return the list of local models
|
|||
|
|
curl -s http://localhost:11434/api/tags | jq .
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### LiteLLM (OpenAI-compatible)
|
|||
|
|
```bash
|
|||
|
|
# Should return available models exposed by your proxy
|
|||
|
|
curl -s http://localhost:4000/v1/models -H "Authorization: Bearer sk-test" | jq .
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
> If your model doesn’t appear, re-check the proxy flags and your `.env` values (`OPENAI_COMPATIBLE_API_BASE`, `OPENAI_COMPATIBLE_MODEL_NAME`, etc.).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Troubleshooting
|
|||
|
|
|
|||
|
|
- **Model not responding / timeout**: ensure `ollama serve` is running and `OLLAMA_MODEL` exists (`ollama list`).
|
|||
|
|
- **LiteLLM 401**: set `OPENAI_COMPATIBLE_API_KEY` to a value accepted by the proxy or disable auth on the proxy.
|
|||
|
|
- **CORS / wrong base URL**: confirm `OPENAI_COMPATIBLE_API_BASE` and that it ends with `/v1`.
|
|||
|
|
- **Model not visible**: ensure `ENABLE_OLLAMA=true` or `ENABLE_OPENAI_COMPATIBLE=true` in `.env`, then restart services.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Internal References
|
|||
|
|
|
|||
|
|
- Ollama vars: `ENABLE_OLLAMA`, `OLLAMA_SERVER_URL`, `OLLAMA_MODEL`
|
|||
|
|
- OpenAI-compatible vars: `ENABLE_OPENAI_COMPATIBLE`, `OPENAI_COMPATIBLE_MODEL_NAME`, `OPENAI_COMPATIBLE_API_KEY`, `OPENAI_COMPATIBLE_API_BASE`, `OPENAI_COMPATIBLE_API_VERSION`, `OPENAI_COMPATIBLE_SUPPORTS_VISION`, `OPENAI_COMPATIBLE_REASONING_EFFORT`
|