Cost-efficient routing

Cost-efficient routing uses the Flex service tier to lower inference cost on eligible requests. It is on by default. You can turn it off from the Strategies page, or pass allow_flex_tier: false to disable it for a specific request.

{
  "model": "your-model-id",
  "input": "Summarize this document.",
  "allow_flex_tier": false
}

An explicit allow_flex_tier always wins over the account setting, so true opts a single request in while the toggle is off.

What Router chooses

When cost-efficient routing runs on its own, Router keeps the same provider and model and only chooses between that model's standard and Flex service tiers. It sends a request to Flex when doing so is not expected to degrade it, based on how each tier has been performing recently. Flex costs less, but its latency is higher and less predictable.

If the selected tier fails or times out, Router can try the other tier.

Combine with NVIDIA NeMo Switchyard

NVIDIA NeMo Switchyard is a separate, opt-in routing strategy for coding-agent traffic. It treats the model in the request as the capable tier and routes eligible turns between that model and a configurable efficient model. Router scores recent conversation signals, including errors, context compaction, tool-call patterns, passing tests, and turn depth, to choose a tier. Switching sensitivity controls which tier handles ambiguous turns.

When Switchyard and cost-efficient routing are both enabled, Router can compose them. Switchyard chooses the capable or efficient model tier, while cost-efficient routing orders the eligible standard and Flex service-tier variants within each tier. Composition applies when the request leaves both allow_flex_tier unset and service_tier omitted, null, or "auto". An explicit allow_flex_tier setting or non-neutral service_tier prevents this composition and keeps the request's tier intent.

Enable and configure NVIDIA NeMo Switchyard from the Strategies page.

Where Flex applies

Cost-efficient routing applies only when the request uses an eligible direct model, rather than a models list, and does not set service_tier.

Anything else runs unchanged. Sending allow_flex_tier explicitly where Flex is unavailable returns 400 rather than being ignored. See allow_flex_tier.

Set latency guardrails

Because Flex latency varies, bound it explicitly:

See Timeouts for the full semantics. The request detail in Logs shows which model and service tier handled the request.