Javier Rodeiro

Qwen3.8 vs Ornith 1.5 on a Mac: speed and code review

Case studies8 min#Local LLMs#Benchmarks

In this article

On a Mac Studio M4 Max with 64 GB, Qwen3.8-27B with MTP does not replace the 35B mixture-of-experts model I already run. Measured on 12 October 2026 under oMLX 0.7.0, Qwen generated 55 tokens per second against 90, and took 3.46 seconds to produce its first token on an 800-token prompt against 0.49. On a code review test the two caught the same bugs with reasoning off, and Qwen caught fewer with it on.

The number that decides it is time to first token

Qwen3.8-27B needs seven times longer than Ornith 1.5 to start answering an 800-token prompt: 3.46 seconds against 0.49, averaged over 8 requests per model. Output speed is closer, 55.1 tokens per second against 90.4. In a chat that gap is a nuisance. A coding agent or a code review sends long prompts many times in a row, and there the wait before the first token is most of the cost.

oMLX is an inference server for Apple Silicon Macs that serves MLX models behind an OpenAI-compatible API. I run both models on it, and each one has its own trick for generating faster.

Qwen3.8-27B is a dense model: all 27 billion parameters work on every token. It ships with MTP, multi-token prediction, an extra head trained with the model that drafts the next few tokens so the model can verify them in one pass. Ornith 1.5 35B-A3B is a mixture of experts: 35 billion parameters, about 3 billion active per token. I run it with DFlash, where a separate small draft model proposes a block of tokens and the main model checks them.

Both tricks are the same idea, speculative decoding. A cheap guesser drafts a few tokens, and the big model checks all of them in one pass. It keeps the guesses that match, replaces the first wrong one with its own token and drops the rest. The words in the diagram are an example I made up to show the mechanism, not a measured run.

Draft and verify. A small model drafts four tokens: Paris, comma, a, city. The big model checks all four in one pass, keeps Paris and the comma, replaces the third with its own token, the, and drops the fourth. One pass gives three tokens instead of one.

The race below plays one 400-token answer from each model at the speeds I measured, and a button restarts it. Qwen spends 3.46 seconds reading the prompt before it says anything, and Ornith is already about two thirds of the way through its answer by then.

The chart puts both measurements side by side. The first pair of bars is the one I care about.

Bar chart of two measurements on oMLX 0.7.0. Time to first token on an 800-token prompt: Ornith with DFlash 0.49 seconds, Qwen3.8-27B with MTP 3.46 seconds. Output speed: Ornith 90.4 tokens per second, Qwen 55.1. Ornith wins both.

I measured with aiperf 0.12.0, one model loaded at a time, the server restarted before each run and no other traffic. This is the command. It works against any OpenAI-compatible server:

aiperf profile --url http://localhost:8010 --model <model-id> \
  --endpoint-type chat --streaming \
  --concurrency 1 --request-count 8 --num-warmup-requests 2 \
  --prompt-input-tokens-mean 800 --prompt-input-tokens-stddev 0 \
  --output-tokens-mean 400 --output-tokens-stddev 0 \
  --extra-inputs '{"temperature":0.6,"chat_template_kwargs":{"enable_thinking":false}}' \
  --tokenizer <model-snapshot-dir> --random-seed 7

Upgrading oMLX helped Qwen by about 20%, and it was not enough

Moving from oMLX 0.6.4 to 0.7.0 raised Qwen3.8-27B from 57-59 to 71.7 tokens per second on a code prompt and from 46-48 to 55-56 on prose, two runs per cell with 400 output tokens. The oMLX 0.7.0 release notes list faster MTP verify and decode for this model, and the numbers agree. Ornith stayed where it was: 103-121 tokens per second on code, 99-102 on prose.

I also tried the obvious knob. MTP drafts up to 4 tokens per cycle by default on this model, and oMLX lets you raise the ceiling. At 6 the output speed was 53.9 tokens per second, at 8 it was 52.4, against 55.1 at the default. The adaptive controller already picks a depth that pays, so I put the setting back.

On a code review test the two models tie

With reasoning off, Qwen3.8-27B caught 9 of 20 planted bugs and Ornith 1.5 caught 8. With 20 cases, one bug of difference is noise. The test takes 20 real commits from four of my repositories, flips one operator on an added line (< to <=, == to !=, a dropped not) and asks the model to review the diff in a single call, without tools. Each case is paired with the clean diff, and a model only gets credit if it flags the planted line on the mutant and not on the clean version.

Model Reasoning Bugs caught Unreadable answers Findings per clean diff Median time
Qwen3.8-27B + MTP off 9 of 20 (45%) 4 of 40 0.7 10 s
Ornith 1.5 + DFlash off 8 of 20 (40%) 2 of 40 1.6 3 s
Qwen3.8-27B + MTP on 6 of 20 (30%) 12 of 40 0.3 194 s
Ornith 1.5 + DFlash on 8 of 20 (40%) 13 of 40 0.0 102 s

The Ornith rows were measured on 2 October 2026 on oMLX 0.6.4, the Qwen rows on 12 October on 0.7.0. With reasoning off, where they differ is false alarms. Qwen reports half as many problems on code that has none, and it takes three times as long per review.

Turning reasoning on does not help Qwen. It caught 6 of 20 against 9 without it, and half of its 40 answers used up the whole 12,000-token output limit, which is why 12 of them never reached valid JSON. Ornith overthinks in the same way and loses 13 of 40 answers, but it holds at 8 of 20 and takes about half the time per review. A review that needs three minutes and fails to parse a third of the time is no use to me from either model.

The two models do not fit in memory together

Qwen3.8-27B takes 16.1 GB once loaded and Ornith 1.5 takes 19.8 GB. On a 64 GB machine that sounds fine, and under oMLX 0.6.4 both did load, with swap climbing to 15.3 of 16 GB. The memory guard in 0.7.0 refuses instead. While Qwen was busy, every request for Ornith got this:

POST /v1/responses → 507: Cannot load Ornith-1.5-35B-A3B-MLX-4bit: projected memory
38.24GB would exceed the dynamic memory ceiling 34.51GB

When Qwen is idle, oMLX evicts one model to load the other, and the log marks the unload as an immediate abort, so a request in flight on the evicted model is lost. I found out what that looks like under load by accident. I ran the review test on Qwen with reasoning on for 90 minutes while another tool kept sending requests to Ornith. oMLX loaded Ornith, evicted it six seconds later to make room for Qwen’s prompt, loaded it again, and eventually evicted Qwen in the middle of an answer. Of Qwen’s 40 requests, 19 failed.

I had hoped to keep Qwen beside Ornith as a second opinion, and that is off the table. I have to pick one, and on these numbers I pick Ornith.

Qwen sometimes cuts its own JSON, and the fix costs half the speed

In 5 of 40 review answers Qwen3.8-27B returned {"findings":[] and stopped, with the closing brace missing. It only happened on empty answers of five or six tokens. I reran the five affected diffs three times each under four settings, with MTP on and off and with the presence penalty at 1.5 and 0. The cut answers appeared with MTP off too, and without the penalty, in up to 4 of 15 runs. As far as I can tell the model just ends early now and then.

Constrained decoding removes it. oMLX has it built in through xgrammar, but the Homebrew formula leaves it out unless you ask:

brew reinstall jundot/omlx/omlx --with-grammar

With a JSON schema in the request, 0 of 30 answers were unparseable. The diagram shows what that costs. A grammar and MTP cannot run together, so those requests fall back to plain decoding at about 25 tokens per second.

Flow diagram of a review request to Qwen3.8-27B. Without a JSON schema it uses MTP decoding at about 55 tokens per second and up to 4 of 15 answers are unparseable. With a JSON schema it uses grammar-constrained decoding with MTP switched off at about 25 tokens per second and 0 of 30 answers are unparseable.

What is still open

Qwen’s row with reasoning on mixes two runs. 21 answers come from the morning run described above, with other traffic on the server, and I repeated the 19 failed requests in the afternoon with everything else paused. I counted the morning answers because they were complete, and I did not rerun all 40 in one clean pass. The speed test used 8 requests and one prompt size. I did not test Qwen’s vision, and I do not know what a JSON schema costs Ornith with DFlash.

I removed Qwen from the proxy that evening and deleted its weights, so none of that will be measured. The next candidate is Muse Glimmer 30B, which is being measured now.

For interactive work, where you wait on every answer, Ornith stays. If you are choosing a local model for a Mac, measure time to first token at the prompt length you really send before you look at tokens per second. For unattended jobs that gap matters much less.

Keep reading

NEWSLETTER - PUPPETS & SCRIPTS