Give text-only LLMs vision through a tiny OpenAI-compatible proxy.
VisionBridge sits between your chat UI and your models. It lets a reasoning model
from LM Studio, Ollama, vLLM, Z.ai, OpenRouter, or any OpenAI-compatible backend
inspect images by asking a separate vision model targeted questions.
No training. No weights. Bring your own reasoning model and your own vision model.
OpenWebUI / LibreChat / app
|
v
VisionBridge at /v1/chat/completions
|
+– reasoning model: LM Studio / Ollama / vLLM / GLM / DeepSeek
|
+– vision model: LLaVA / Qwen-VL / GLM-V / GPT-4o-mini / any VLM
VisionBridge receives an OpenAI-style chat request. If the request contains
images, it stores the images, rewrites the prompt for the reasoning model, and
lets the reasoning model call tools:
look(image_id, question) for targeted visual inspection
ocr(image_id) for text extraction
scan(image_id) for a tiled full-document text sweep of dense pages
crop_and_look(image_id, box, question) for zooming into a region
(small crops are automatically upscaled before the VLM sees them)
compare(image_id_a, image_id_b, question) for judging two images together
The vision model does the seeing. The reasoning model does the thinking.
Before the loop starts, every image gets a one-shot scene description from
the vision model (in parallel, cached by content), so the reasoning model
begins with global context and spends its tool budget on details. Multiple
tool calls in one turn run concurrently, and repeated images across a
conversation are served from a content-addressed cache instead of being
re-fetched and re-decoded.
Tool calling uses the backend’s native OpenAI function calling when
available. In the default TOOL_MODE=auto, VisionBridge probes the reasoning
backend once and transparently falls back to a prompt-JSON protocol for
backends and models that don’t support tools — so it works with strong hosted
models and weak local ones alike.
The loop is built to survive imperfect models: hallucinated image ids, bad
crop boxes, malformed JSON, and vision-backend hiccups are fed back to the
reasoning model as observations so it can correct itself, and hitting the
inspection limit triggers a final best-effort answer instead of an error.
docker run -p 8080:8080 \
-e REASONING_BASE_URL=http://host.docker.internal:1234/v1 \
-e REASONING_API_KEY=lm-studio \
-e REASONING_MODEL=local-model \
-e VISION_BASE_URL=http://host.docker.internal:11434/v1 \
-e VISION_API_KEY=ollama \
-e VISION_MODEL=llava:13b \
ghcr.io/eekosystems/visionbridge:latest
Then point your OpenAI-compatible client at:
Base URL: http://localhost:8080/v1
API key: anything
Model: visionbridge
cd visionbridge
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e “.(dev)”
cp .env.example .env # Windows: copy .env.example .env
visionbridge serve –host 0.0.0.0 –port 8080
Check your backend configuration any time with:
It verifies both backends are reachable, that the configured models exist, and
that a test completion succeeds.
All settings come from environment variables (or .env):
Variable
Default
Purpose
REASONING_BASE_URL
http://localhost:1234/v1
Reasoning backend
REASONING_API_KEY
local
Reasoning API key
REASONING_MODEL
local-model
Reasoning model id
VISION_BASE_URL
http://localhost:11434/v1
Vision backend
VISION_API_KEY
local
Vision API key
VISION_MODEL
llava:13b
Vision model id
BRIDGE_MODEL_NAME
visionbridge
Model name the proxy advertises
TOOL_MODE
auto
auto, native, or prompt tool calling
MAX_VISION_CALLS
8
Reasoning turns before a forced final answer
REQUEST_TIMEOUT_SECONDS
120
Per-backend-call timeout
BACKEND_MAX_RETRIES
2
Retries on 429/5xx/transport errors
SCENE_PRIMING
true
Describe each image once before the loop starts
ALLOW_FILE_URLS
false
Allow file:// image URLs
ALLOW_PRIVATE_URLS
false
Allow image fetches from private/internal hosts
MAX_IMAGE_BYTES
20971520
Max bytes per image
MAX_IMAGES_PER_REQUEST
16
Max images per request
IMAGE_CACHE_ENTRIES
64
Content-addressed image cache size
ENABLE_TRACES
true
Keep tool-loop traces for /v1/traces/{id}
BRIDGE_API_KEYS
(empty)
Comma-separated keys; when set, /v1/* requires one
EXTRA_MODELS
(empty)
JSON map of extra model names to backend pairs
LOG_JSON
false
Emit JSON log lines
Serving multiple model pairs
One instance can serve several reasoning+vision pairs under different model
names. Unspecified fields inherit from the default pair:
EXTRA_MODELS={“visionbridge-fast”: {“reasoning”: {“model”: “qwen3:8b”}, “vision”: {“model”: “llava:7b”}}}
(or point EXTRA_MODELS_FILE at a JSON file). All names appear in
GET /v1/models, and clients select a pair with the model field.
Open LM Studio.
Load your reasoning model.
Go to the Developer tab and start the local server.
Use the default base URL:
REASONING_BASE_URL=http://localhost:1234/v1
REASONING_API_KEY=lm-studio
REASONING_MODEL=local-model
LM Studio can also be used as the vision backend if you load a vision-capable
model and expose it through the same OpenAI-compatible server.
REASONING_BASE_URL=http://localhost:11434/v1
REASONING_API_KEY=ollama
REASONING_MODEL=qwen3:32b
VISION_BASE_URL=http://localhost:11434/v1
VISION_API_KEY=ollama
VISION_MODEL=llava:13b
REASONING_BASE_URL=http://localhost:8000/v1
REASONING_API_KEY=token-abc123
REASONING_MODEL=Qwen/Qwen3-32B
VISION_BASE_URL=http://localhost:8001/v1
VISION_API_KEY=token-abc123
VISION_MODEL=Qwen/Qwen3-VL-8B-Instruct
In OpenWebUI:
Admin Panel -> Settings -> Connections -> OpenAI-compatible
Base URL: http://host.docker.internal:8080/v1
API Key: anything
Model: visionbridge
VisionBridge intentionally exposes the small subset most clients need:
GET /health (add ?probe=true to verify both backends are reachable)
GET /v1/models
POST /v1/chat/completions
GET /v1/traces/{completion_id} — the full tool-loop trace for a recent
request: every tool call, its arguments, the vision model’s observation, and
timings. Invaluable for answering “why did it say that?”
Responses include a usage block aggregated across every reasoning and vision
call made for the request, and an X-Request-ID header for correlating logs.
Streaming: image-free requests are streamed token-by-token straight from
the reasoning backend. Requests with images stream too: SSE heartbeats keep
the connection alive while tools run, and in native tool-calling mode the
final answer is forwarded token-by-token as the reasoning model produces it
(prompt-JSON mode buffers the loop and streams the answer in chunks).
Errors: backend failures return HTTP 502, invalid inputs 400.
Auth: set BRIDGE_API_KEYS=key1,key2 to require a Bearer key on /v1/*
endpoints (/health stays open). For anything internet-facing, still prefer a
reverse proxy with TLS in front.
Image URL fetching is hardened by default: hosts resolving to private,
loopback, or link-local addresses are rejected (SSRF guard), redirects are
validated hop-by-hop, downloads are size-capped, and payloads must decode as
real images. Run VisionBridge behind your own reverse proxy / auth layer if
you expose it beyond localhost — it deliberately ships without built-in auth.
benchmarks/ contains a harness that scores VisionBridge against a direct
VLM (relaxed accuracy + ANLS) on DocVQA-style JSONL task sets. See
benchmarks/README.md.
Actively developed. See CHANGELOG.md for what’s new in 0.2.0.
Next up:
published benchmark numbers against caption-only prompting
persistent on-disk image/primer cache
OpenWebUI and LibreChat screenshots
Contributions welcome — see CONTRIBUTING.md.
MIT
Source link
Leave a Reply