124 lines
3.5 KiB
Plaintext
124 lines
3.5 KiB
Plaintext
---
|
||
title: Ollama + LiteLLM
|
||
description: Run Skyvern with a local LLM via Ollama or an OpenAI-compatible proxy via LiteLLM.
|
||
---
|
||
|
||
## Overview
|
||
|
||
Skyvern can use local models via **Ollama** or any **OpenAI-compatible** endpoint (e.g., **LiteLLM**). Two paths:
|
||
|
||
- **(A) Direct Ollama** — use the Ollama API (`/v1/chat/completions`)
|
||
- **(B) OpenAI-compatible (LiteLLM)** — Skyvern connects to a proxy that exposes an OpenAI-style API
|
||
|
||
---
|
||
|
||
## A) Direct Ollama
|
||
|
||
### 1) Start Ollama locally
|
||
Install Ollama and run a model (example: `llama3.1`):
|
||
```bash
|
||
ollama pull llama3.1
|
||
ollama serve
|
||
```
|
||
The API is usually at `http://localhost:11434`.
|
||
|
||
### 2) Configure Skyvern (ENV)
|
||
Add to your `.env`:
|
||
```bash
|
||
# Enable Ollama integration
|
||
ENABLE_OLLAMA=true
|
||
|
||
# Ollama server URL
|
||
OLLAMA_SERVER_URL=http://localhost:11434
|
||
|
||
# Model name in Ollama (check with `ollama list`)
|
||
OLLAMA_MODEL=llama3.1
|
||
```
|
||
|
||
> Note: Ollama may not support `max_completion_tokens` — Skyvern handles this internally.
|
||
|
||
---
|
||
|
||
## B) OpenAI-compatible via LiteLLM
|
||
|
||
### 1) Run LiteLLM as a proxy
|
||
Minimal example (see LiteLLM docs for more options):
|
||
```bash
|
||
# Map an Ollama model behind an OpenAI-compatible endpoint
|
||
litellm --model ollama/llama3.1 --host 0.0.0.0 --port 4000
|
||
```
|
||
|
||
### 2) Configure Skyvern (ENV)
|
||
Add to your `.env`:
|
||
```bash
|
||
# Enable OpenAI-compatible provider
|
||
ENABLE_OPENAI_COMPATIBLE=true
|
||
|
||
# The "model" exposed by the proxy (any identifier your proxy accepts)
|
||
OPENAI_COMPATIBLE_MODEL_NAME=llama3.1
|
||
|
||
# API key required by the proxy (use any placeholder if proxy doesn't enforce)
|
||
OPENAI_COMPATIBLE_API_KEY=sk-test
|
||
|
||
# Base URL for the LiteLLM proxy (OpenAI-compatible)
|
||
OPENAI_COMPATIBLE_API_BASE=http://localhost:4000/v1
|
||
|
||
# (optional)
|
||
# OPENAI_COMPATIBLE_API_VERSION=2024-06-01
|
||
# OPENAI_COMPATIBLE_SUPPORTS_VISION=false
|
||
# OPENAI_COMPATIBLE_REASONING_EFFORT=low|medium|high
|
||
```
|
||
|
||
---
|
||
|
||
## Start Skyvern (local)
|
||
|
||
After setting the environment variables in `.env`, start backend + UI:
|
||
```bash
|
||
# with Docker (recommended)
|
||
docker compose up -d
|
||
|
||
# or locally (dev):
|
||
# poetry install
|
||
# ./run_skyvern.sh
|
||
# ./run_ui.sh
|
||
```
|
||
|
||
Open the UI and pick the model (or keep default); if only Ollama/LiteLLM are enabled, Skyvern will use that.
|
||
|
||
---
|
||
|
||
## Verify your setup
|
||
|
||
Before starting Skyvern, quickly verify that your LLM endpoint is reachable.
|
||
|
||
### Ollama
|
||
```bash
|
||
# Should return the list of local models
|
||
curl -s http://localhost:11434/api/tags | jq .
|
||
```
|
||
|
||
### LiteLLM (OpenAI-compatible)
|
||
```bash
|
||
# Should return available models exposed by your proxy
|
||
curl -s http://localhost:4000/v1/models -H "Authorization: Bearer sk-test" | jq .
|
||
```
|
||
|
||
> If your model doesn’t appear, re-check the proxy flags and your `.env` values (`OPENAI_COMPATIBLE_API_BASE`, `OPENAI_COMPATIBLE_MODEL_NAME`, etc.).
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
- **Model not responding / timeout**: ensure `ollama serve` is running and `OLLAMA_MODEL` exists (`ollama list`).
|
||
- **LiteLLM 401**: set `OPENAI_COMPATIBLE_API_KEY` to a value accepted by the proxy or disable auth on the proxy.
|
||
- **CORS / wrong base URL**: confirm `OPENAI_COMPATIBLE_API_BASE` and that it ends with `/v1`.
|
||
- **Model not visible**: ensure `ENABLE_OLLAMA=true` or `ENABLE_OPENAI_COMPATIBLE=true` in `.env`, then restart services.
|
||
|
||
---
|
||
|
||
## Internal References
|
||
|
||
- Ollama vars: `ENABLE_OLLAMA`, `OLLAMA_SERVER_URL`, `OLLAMA_MODEL`
|
||
- OpenAI-compatible vars: `ENABLE_OPENAI_COMPATIBLE`, `OPENAI_COMPATIBLE_MODEL_NAME`, `OPENAI_COMPATIBLE_API_KEY`, `OPENAI_COMPATIBLE_API_BASE`, `OPENAI_COMPATIBLE_API_VERSION`, `OPENAI_COMPATIBLE_SUPPORTS_VISION`, `OPENAI_COMPATIBLE_REASONING_EFFORT`
|