Engineering

Try a cheaper model before switching

Delegate a model change to Merlin in Linear. It runs the feature's eval cases on cheaper models through OpenRouter and opens a pull request with the comparison, or says none held up.

@Merlin from now on, when an issue asks you to try a cheaper model for a feature, run that feature's eval cases from acme/app through OpenRouter on the current model and on two or three cheaper candidates that support what the feature needs. Compare how many cases each gets right, what the run cost and the median latency. If a candidate gets within two cases of the current model, open a pull request switching to the cheapest one, with the comparison in its description. If none does, say so on the issue with the numbers and change nothing.

ENG · AI features
Get started for free
  1. Starts whenA teammate delegates an issue asking for a cheaper model for one feature
  2. Merlin worksClaude Code or Codex runs the feature's eval cases from the repository on the current model and cheaper candidates through OpenRouter, then compares quality, cost and latency
  3. You getA pull request switching the model with the comparison in its description, linked on the issue

Tools

  • Linear
  • OpenRouter
  • GitHub

Capabilities

  • Integrations
  • Your agent's MCP

How it works

From a model issue to a pull request with the numbers.

Delegate an issue that names the feature to make cheaper. It can name candidate models or leave the choice to Merlin.

ENENG › Issues › ENG-482

Try a cheaper model for ticket triage

Triage sends every new support ticket to claude-sonnet-5. Try cheaper models on the cases in evals/triage and switch if they hold up.

Activity

Omardelegated the issue to Merlin·now

Leave a comment…

Properties

Todo

Omar

Merlin

Labels

Improvement

Before and after

Model switches get decided on your own cases, not on a hunch.

Without Merlin

An engineer reruns the evals on each candidate by hand, adds up the costs in a sheet and then makes the switch.

  1. 01Pick up the model issue
  2. 02Shortlist cheaper models
  3. 03Find the triage eval cases
  4. 04Run them on the current model
  5. 05Rerun them on each candidate
  6. 06Look up what each run cost
  7. 07Compare right answers, cost and latency
  8. 08Change the model id
  9. 09Run the tests
  10. 10Open a pull request with the numbers
  11. 11Link it on the issue

With Merlin

Delegate the issue and review a switch that comes with its own comparison.

  1. Model issue delegated to MerlinNames the feature to make cheaper
  2. Runs the evals on Omar's MacBookThree models through OpenRouter
  3. Pull request with the comparisonLinked on the issue

Setup

Connect Linear and GitHub, then add OpenRouter to your agent.

Set these up once and anyone can ask for a cheaper model from an issue. Merlin never merges, so a person decides on the switch.

  1. 01Connect a machineRuntimeInstall the Merlin desktop app or CLI on a computer running Claude Code or Codex. You can also use a machine a teammate shares with your workspace.
  2. 02Connect LinearIntegrationsConnect your Linear workspace so issues can be delegated to Merlin.
  3. 03Add OpenRouter to your agentYour agent's MCPConnect the OpenRouter MCP server to Claude Code or Codex on the machine that runs this. Its credentials stay on that machine.
  4. 04Install the GitHub appIntegrationsPick the repositories Merlin can access, so it can read and link the pull requests there. Pushes use the machine owner's own Git access.
  5. 05Keep your code on the machineRuntimeMerlin reads the code in a local project on the runtime, so keep the repository checked out there.
  6. 06Keep eval cases next to the featureRuntime

    Merlin scores each model against the cases your team keeps in the repository, such as evals/triage/cases.jsonl with the expected answer for each. Without them, it says so on the issue instead of guessing.

  7. 07Size the key for the evalYour agent's MCP

    Each case is a billable send-message call. The key OpenRouter mints when you connect starts with a $10 spend cap and expires after 7 days (how it works), so raise the cap on the approval screen for a large eval set and reconnect once the key has expired.

  8. 08Paste the setup promptGuidanceAn owner or admin sends the prompt below to @Merlin once, and it becomes workspace guidance for every future run.

Prompt

Tell Merlin how to try a cheaper model.

@Merlin from now on, when an issue asks you to try a cheaper model for a feature, run that feature's eval cases from acme/app through OpenRouter on the current model and on two or three cheaper candidates that support what the feature needs. Compare how many cases each gets right, what the run cost and the median latency. If a candidate gets within two cases of the current model, open a pull request switching to the cheapest one, with the comparison in its description. If none does, say so on the issue with the numbers and change nothing.

ENG · AI features

Once, when you paste it

  1. PromptYou paste the prompt to @Merlin
  2. GuidanceMerlin saves your rules as workspace guidance

Every time an issue is delegated to Merlin

  1. IntegrationLinear hands Merlin the issue
  2. RuntimeMerlin picks an available machine
  3. ContextThe run gathers context
  4. WorkClaude Code or Codex runs the evals on each model
  5. SteeringYour team steers it from the issue
  6. ResultMerlin links the pull request with the comparison

Bring your coding agents into the tools your team already uses.

Let your whole team hand work to Claude Code and Codex today. Connect one machine and add Merlin to Slack, Linear or GitHub. 14-day free trial, no API keys, no card.