Compatibility
Polyglot needs an OpenAI-compatible chat-completions endpoint, or the Anthropic API. Beyond “does it connect”, the thing that varies is whether a given model produces tool calls reliably enough to get work done - and whether a given server supports structured output as a fallback for weaker models.
Servers
Section titled “Servers”| Server | Connects | Structured output | Notes |
|---|---|---|---|
| Ollama | ✅ | ✅ (recent versions) | Default baseURL if unset. |
| LM Studio | ✅ | ✅ | Start the server from the Developer tab. |
| llama.cpp server | ✅ | ✅ (GBNF grammars) | llama-server. |
| vLLM | ✅ | ✅ (guided decoding) | |
| TGI (Text Generation Inference) | ✅ | ⚠️ depends on version | Free-text protocol always works. |
| LocalAI | ✅ | ⚠️ depends on backend | |
| Jan | ✅ | ⚠️ | Uses an OpenAI-compatible local server under the hood. |
| KoboldCpp | ✅ | ⚠️ | |
| Anthropic API | ✅ | n/a - native tool use | Structured-output setting is ignored. |
| Hosted OpenAI API | ⚠️ | ❌ not supported | Chat works; strict structured mode requires all-required schemas, which Polyglot’s tools don’t satisfy. Other OpenAI-compatible gateways are fine. |
Models
Section titled “Models”Rough guidance, not a hard list - try yours:
| Model class | Free-text tool calls | With structured output |
|---|---|---|
| Claude, GPT-4-class | reliable | n/a / reliable |
| Qwen2.5-Coder / Qwen3-Coder (7B+) | usually reliable | reliable |
| DeepSeek-Coder, GLM-4 (coder variants) | usually reliable | reliable |
| Llama 3.x instruct (8B+) | mixed | reliable |
| General (non-coding) small models (<7B) | often unreliable | usually usable |
If a model keeps producing invalid calls, Polyglot detects it and stops the turn rather than looping. Switch to a stronger model or turn on structured output.
Tested combinations and reports are welcome - open an issue on GitHub.