Orla is a project of 
Build Multi-Agent Systems That Scale
Orla is a runtime-adaptive execution layer for agentic systems. It optimizes the infrastructure behind your agents for cost and accuracy, learning from production data in real time.
You define
Your Agents
Your Models
Orla's Runtime
-
Lower cost
-
Higher accuracy
-
Adapts at runtime
LLM Inference Engines
Why Use Orla For Your Agentic Applications
-
Optimizes cost and accuracy
Route each stage of your agent to the model that fits it. Orla learns from real outcomes and shifts work to the cheapest model that holds quality.
-
Backend-agnostic
Route stages to SGLang, vLLM, Ollama, or any OpenAI-compatible cloud API. Mix and match backends without changing your agent.
-
Adapts at runtime
Orla shadow-tests model changes against real traffic and promotes only the ones that improve cost and quality, so your agents keep getting better with no redeploys.
Works with
Drop-in integration
Add Orla in One Change
Before
from openai import OpenAI
client = OpenAI(api_key=KEY)
def call(prompt):
return client.chat.completions.create(
model="gpt-5.1",
messages=[{"role": "user", "content": prompt}],
)After
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8081/v1",
api_key=KEY,
)
def call(prompt, stage):
return client.chat.completions.create(
model="set-by-orla",
messages=[{"role": "user", "content": prompt}],
extra_headers={"X-Orla-Stage": stage},
)Under the Hood
Your agent tags each call with a stage and sends it to Orla. Orla routes the call to the mapped backend, records what it cost and how it went, and a mapper retunes the mapping from those outcomes. The result is lower cost and higher accuracy with no changes to your agent.
Three core components make this work:
-
Stage Router
An OpenAI-compatible proxy that routes each tagged call to the backend currently mapped to that stage, across heterogeneous infrastructure.
-
Telemetry
Records cost, latency, and the quality ratings you report for every call, giving the mapper a live signal to learn from.
-
Runtime Mapper
Reads the telemetry and remaps each stage to the model that best balances cost and accuracy, shadow-testing changes before it promotes them.
Team
-
Hayder Tirmazi
Creator and Lead Developer
-
Rana Shahout
Primary Researcher
-
Michael Mitzenmacher
Principal Investigator
-
Minlan Yu
Principal Investigator
Portable deployments
Build Your Agents Once, Ship Them Anywhere
Build your agents once. Test and run them consistently across your stack.
Mix private infrastructure and cloud models in the same agent when you need both.
-
Local machine
Develop and debug with the same runtime you ship to production.
-
CI
Gate merges with the same agent runtime you run locally.
-
Private infrastructure
Keep models and data on your network and call hosted APIs from the same agent when you need them.
-
Cloud models
Use hosted APIs alone or beside private backends. One agent, mixed backends, tuned for cost and accuracy.
Cite this work
If you use Orla in your research, please cite our paper.
@misc{shahout2026orlalibraryservingllmbased,
title={Orla: A Library for Serving LLM-Based Multi-Agent Systems},
author={Rana Shahout and Hayder Tirmazi and Minlan Yu and Michael Mitzenmacher},
year={2026},
eprint={2603.13605},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2603.13605},
}
Help Orla grow
A GitHub star helps other developers discover Orla and keeps the project going.
Star on GitHub