The Cost of Defaults: Re-Architecting Pipeline Efficiency with RLMs

For a long time, every task in our mobile agentic pipeline went to the same model. Writing an architecture doc? Opus. Pulling email addresses out of a config file? Also, Opus. Nobody decided that on purpose. It was the default, and defaults tend to stick around. It cost us about $25.57 per user story. The Flutter project rebuilds had 43 stories, so that project came to $1,099.57. Not a disaster. But we wanted other teams using the pipeline, and when I did that multiplication in a spreadsheet, I didn't love what I saw.

Honestly the gates annoyed me more than the money. After each phase the pipeline stopped and waited for someone to read the output and click approve. Maybe 45 seconds if that person was at their desk. If they weren't, the run just sat there. Most of the time the output was fine, and everyone knew it. I found the Recursive Language Models(RLM) paper (Zhang, Kraska and Khattab, MIT CSAIL) not long after it went up on arXiv at the end of December 2025. My first read was a skim, I'll admit. I went back to it a week or so later because one part kept bugging me.

The pitch, roughly: stop forcing the whole input into one giant prompt. Let a root model look at the task, split it up, and fire off smaller sub-calls. Their answers land in a REPL as ordinary variables, and the root model works with those instead of the raw text. What got me wasn't the recursion. Nothing in that setup says the sub-calls have to use the same model as the root. If most sub-calls are just reading and summarising, why pay frontier prices for them? That was when the cost problem and the design problem stopped being two separate things in my head.

oMLX on Apple Silicon handles the local side. Qwen 35B runs fine on a Mac with enough RAM, and the 7B and 14B Coder models cover the lighter stuff. The catch is context. Hand a local model a big repo, ask it where authentication lives, and it falls over pretty fast. RLM fixed most of that. The engine splits an investigation into sub-queries, walks the code a piece at a time, and stores what it finds as structured results. It runs read-only, so it can't break anything while it pokes around.

I added one rule I wouldn't budge on: every finding needs a file and a line number, or it doesn't get reported. If I'm being straight about it, I added that because I didn't trust the model. It ended up being the thing people liked most. They could click through to the exact line and judge for themselves.

Phase 1 wrapped at the end of September. CLI, desktop app, terminal UI, oMLX, the RLM engine, all in our GitLab and working offline. Going through the logs also confirmed something I'd half suspected from the paper. Most of the tokens in an investigation go on exploring, not on the answer. Hybrid mode, which we're starting now, comes straight out of that.

Which raised an obvious question. Who decides where each sub-call goes? Asking an LLM would mean paying for a model call to save money on model calls. A bit silly. So we built Laya, and it's boring on purpose. Plain rule-based pattern matching, no model involved. The router checks a prompt in a fixed order. Security words like auth, token or encrypt send it to Opus, and so do risky code patterns like eval or os.system. Ordinary code terms go to Qwen Coder, simple extraction goes to the free model, and whatever's left goes to Sonnet. It looks at file paths too, so anything under an auth or login folder gets careful treatment even when the prompt sounds harmless.

The gate checker works the same way. Each phase has a short list of structured questions. Does the vision doc actually have a problem statement, an audience, a value proposition and a goal? Is anything contradicting itself? If every score is 4 or higher and nothing's flagged, the gate approves itself. A score of 2 or lower, or a critical flag, and it blocks with a reason. Anything in between goes to a person, and risky architecture goes to a senior reviewer. Each decision takes around 33 milliseconds.

Laya used to lean on an external library, and it was optional. We dropped both in version 1.1. Now it's a module in the repo and the pipeline won't start without it. Strict, sure, but we stopped maintaining two code paths. It also triages code reviews (a diff with eval(user_input) in it goes straight to a security reviewer, a typo fix in a comment just gets approved) and classifies files during brownfield analysis.

Speculative decoding was the bit I was nervous about. The original idea comes from Leviathan and others in 2022. In the pipeline we run a response-level version: GPT-5 nano drafts, Sonnet scores the draft, and we keep it if the score is 0.9 or above. Where it's allowed, things run roughly 1.5 to 2 times faster. It isn't allowed on architecture, code generation, code review or anything near security. Those are exactly where an answer that's slightly off but looks right does real damage. So it gets summaries, status text, that kind of thing.

For AMSI-AI we're going further in December with proper token-level speculative decoding inside oMLX. A small Qwen model proposes tokens and the 35B checks them in one pass. Output stays identical to what the big model would produce alone, and we're hoping for 2 to 3 times faster local inference.

Per story that's $11.95, down from $25.57, so about 53% cheaper. Around 85% of gates now approve themselves, which saves something like 14 hours of waiting on a project of that size. Review scores went up as well.

I'd be careful with that 53% though. A decent chunk of it is just the newer model generation being cheaper per token. Routing to specialised and free models is another big piece. Laya's gate automation and RLM batching make up the rest. If someone tells you one trick halved their bill, ask them how they measured it.

Laya has flaws, and I'd rather list them here than have someone find them in production. It matches substrings, so "keyboard" contains "key" and gets sent to Opus for no good reason. The reverse is worse. A security task described without any security words can end up on a cheap model, which is why file paths are always checked and why we push people to be explicit in prompts. And gate scores check structure, not substance. A well organised bad idea will pass. That's the price of a 33 ms decision, and it's why architecture still gets a human look.

Next on the AMSI-AI side is hybrid mode. File reading, code search, drafts and RLM sub-calls stay local on Qwen, while Claude handles reasoning, architecture, security, final review and whatever the user ends up reading. We're aiming for 70 to 90% fewer Claude tokens. After that comes speculative decoding in December, domain specialist agents early next year, a knowledge cache, and a fully offline mode by April 2027.

None of these ideas are mine, to be clear. RLM came from a paper, speculative decoding has been around for years, and keyword routing is ancient. I mostly got annoyed at a bill and started connecting things.

Blog By

Sreejith Rajan

Senior Associate
adesso mobile solutions (ams)

adesso India