Fable 5: Does the smartest LLM decompile better?

I benchmarked Fable 5 against Sonnet 4.6 on 55 functions. It wins on the hard ones, but it costs 4x more

We have a new model that everyone is talking about: Fable 5. Within the matching decompilation community, there are mixed opinions about it. Some say it’s too expensive for little to no gain, while others say that it’s very helpful on the hardest functions.

In the previous chapter, I introduced Mizuchi, a tool that guides LLMs to do matching decompilation, along with a benchmark I ran with it. The dataset was 60 functions from two retro games. It used a leading model at the time: Claude Sonnet 4.6.

⚙️ What is Matching Decompilation?

Matching decompilation is the art of converting assembly back into C source code that, when compiled, produces byte-for-byte identical machine code. It’s popular in the retro gaming community for recreating the source code of classic games. For example, Super Mario 64 and The Legend of Zelda: Ocarina of Time have been fully match-decompiled.

Well, I think it's good to have numbers™️ to guide this discussion. So, let's run Mizuchi to benchmark Fable 5!

Spoiler: Fable 5 is actually good, but a drop-in replacement might not be the right call.

Dataset and settings

The dataset is the same as in the previous benchmark. I really recommend reading the previous chapter for how the functions were picked, but in short: 30 functions from Sonic Advance 3 (SA3, a GBA game) and 30 from Animal Forest (AF, an N64 game).

One caveat for AF: 5 of its functions get matched without AI, solved by m2c in the programmatic phase. So, for measuring the model, AF effectively contributes 25 functions, not 30 like SA3.

Although the dataset is the same, the settings need adjusting because of Fable's token cost.

Run 1 - All functions

• Goal: Confirm that Fable doesn’t introduce regressions on the functions that Sonnet already matches

• Retry limit: 6 (Sonnet used 12)

• Soft timeout: 3:40 (Sonnet used 7:00)

Run 2 & 3 - Sonnet's always failing and flaky functions

• Goal: Check whether Fable can get a new match consistently

• Retry limit: 6

• Soft timeout: 7:00

Results

Let’s explore the benchmark result!

You can explore the full report on this webapp.
Check the full results in this repository.

Yay! After running for 3 days, it’s all done now! A few highlights:

• Fable matched 5 functions that Sonnet always failed to match.

• When Fable matches a function, it always does so within the first 3 attempts.

• Fable is more stable. It stabilized almost all of Sonnet’s flaky matches.

Regressions

Table comparing the match outcome across the runs on Sonnet and Fable

Run 1 confirmed that Fable introduced no regressions: the functions that Sonnet always matched still match on Fable, and except in one case, it got the best outcome: if a Sonnet match is flaky, Fable likely stabilizes it, even with this stricter budget!

Under this strict budget, only one Sonnet-flaky function went unmatched by Fable in Run 1, and Fable even matched one extra function that Sonnet always failed to match.

New matches

Table comparing the match outcome across the functions that Sonnet could not match consistently on SA3

Table comparing the match outcome across the functions that Sonnet could not match consistently on AF

On the gain side: beyond the one function Fable surprisingly matched back in Run 1, the larger budget in Runs 2 and 3 let it add 4 new matches: 2 from SA3 and 2 from AF.

On the miss side: Fable missed one SA3 function that Sonnet matched 1 out of 3 times. AF didn't have any misses.

Attempts to match

How many functions match per attempt

In the previous benchmark using Sonnet, we learned that most functions match by the 3rd attempt, and few match on later retries.

It's an even stronger statement for Fable, since all matches happened within the first 3 attempts.

Mismatch Counts

SA3 mismatch counts per function. Lower is better

AF mismatch counts per function. Lower is better

Matching a function is the best result, but does Fable at least reduce the number of mismatches in the functions it couldn't match?

Not quite. Sometimes Fable finds better code, and sometimes it doesn't.

Cost

Fable emits fewer tokens than Sonnet, but it's much more expensive

All the above results are great for Fable: more matches in fewer attempts. But now, let's talk about the main limitation: its cost.

Since Sonnet and Fable benchmark settings differ, we can only compare cost on the subset of data that’s equivalent across both runs. Then, I decided to compare the cost for matching a function that didn't hit the soft timeout.

Using this data, we can see that Fable is 3.5-4.6x more expensive than Sonnet in this case. It produces fewer output tokens, but these tokens are more expensive.

It's worth mentioning that its session limit is 50% smaller too (at least, during the release phase). I had to split the runs into smaller turns because Fable’s usage exceeded its session limit. That’s a downside next to Sonnet, where I could run a full benchmark without hitting the limit.

Conclusion

Although the results show a clear gain for Fable on hard functions, that gain comes at a steep price. It can fit well into your workflow, but swapping in Fable wholesale may not pay off if a cheaper model already matches many of your functions.

An approach that I've seen other people doing (as Chris Lewis wrote months ago) is using a more expensive model as a last resort for the genuinely difficult work.

Another idea that I saw some folks doing is using Fable not to match the function directly but to build tooling and research documents that a less-capable model will use.

So, that's all for now! This post was a short detour from my current work on decompiling “Klonoa: Empire of Dreams”. In the next post I’ll explain the LLM-assisted workflow I’m developing to decompile it!

---

Follow me on Twitter and Bluesky to stay in the loop. I’m active on the decomp.me Discord server as well, my username is trickster.42.

I’m posting the next chapters on my Substack. See you on the next chapter there! 👋