What agents see when they search MCP tools

My employer makes the best decompiler in the world (arguably). Yet, agents searching for “decompile function” would only see our competitors’ MCP tools.

Not great.

Let me rewind a bit.

A couple months ago I started looking at how agents interacted with our product, IDA Pro, and noticed that even if they had access to our official MCP, they tended to use dumber tools like objdump, readelf, or even (blasphemy!) Ghidra. That made me wonder whether they were choosing those tools over IDA, or simply failing to find IDA in the first place.

I started reading up the literature on LLM model bias, tool selection, and found some interesting facts, but midway through I stopped and realized: wait, how the fuck do agents search for tools?

Turns out, the answer depends on the model and the harness/cli you are using. So I made a survey of all the main coding CLIs and analyzed their behavior, and I’m sharing the results here so other MCP authors can learn from our story.

Feel free to skip to the takeaways if you don’t want to read this whole journey. I hate it when a news article turns “man starts an ice cream business” into a 10-page story from the journalist’s point-of-view, starting with their parents’ first date at an ice cream shop.

Full disclosure: I wrote most of this post manually. Agents did much of the analysis, and I checked their work to a reasonable extent.

ida-mcp

IDA Pro is somewhat of a big deal in the cybersecurity world.

I am not gonna delve into what reverse engineering is, but suffice to say, we have a pretty good product and we now have a pretty good MCP for it.

My boy Duncan designed it to be simple yet effective, and he knows his stuff. Some might say he got hired because of this.

Its design is simple, and it’s mostly an RPC layer that allows multiple agents to share access to IDBs (our custom database file format) and talk python with multiple IDA Pro processes, be it in the GUI or headless.

Here’s the tooling surface:

• open_database: opens a binary for analysis, or a pre-analyzed database.
• list_databases: lists all open databases.
• execute_python: executes a python snippet and returns its result.
• save_database: saves the current database to a file.
• close_database: closes a database.

Seems simple enough, right?

“But why don’t you add a tool to list functions? A tool to decompile functions? A tool to make a macchiato freddo with grappa?”

Guys, check the intro of this section again or…

This matches what we see and what users tell us: even with fancy task-specific tools around, agents often end up writing IDAPython anyway. So we might as well make that easy.

We also have an additional tool, reference, that Duncan came up with. The model can ask “how do I do X in ida_domain?” and get API references and examples back programmatically:

There is zero agentic magic in this, it’s just good docstrings, good examples, and a tiny search engine over API docs.

I have another blogpost coming about the work I’ve done to improve its ranking, but TL;DR just use BM25, the standard ranking algorithm for keyword search.

The problem

Cool, we got our agents who can load the MCP, find the tools, ask for references, and be happy yeah?

Nope.

This is how we used to define our tool names and descriptions in the MCP server:

Noticed anything about this table?

There’s absolutely no mention of domain-specific, i.e. reverse engineering-related, terms.

No xrefs, no binary, no disassembly, no decompilation, no symbols, no damn addresses. Only ida, idalib, ida-domain and nexus.

To be completely honest, I work for the company and I barely know what “IDA Nexus” is. How are the poor clankers supposed to search for our tools?

As a matter of fact, what do coding CLIs show their models about MCP tools, and how do the models search them?

Turns out, the most common approach is to load MCP tool definitions on demand, giving the model a mini search engine to find what it needs:

ToolSearch("decompile function")

1. ghidra/decompile_function
2. ghidra/decompile_function_by_address
3. binja/bn_function_decompile
...

Experiment 1: Mock LLM

I started by compiling a list of the main CLI coding agents I knew of: Claude Code, Codex, Pi, OpenCode, and a bunch of others.

Clearly, Claude and Codex are the most interesting cos they are made by the people who train the sauce.

I figured I’d amass 180 tool search queries to start investigating this whole search thing:

• 120 reverse-engineering tool search queries (“decompile function”, “xrefs to address”, …).
  • Some of these were extracted from transcripts of real agent traces, some were made up but plausible.
• 12 searches naming products (“IDA Pro”, “Ghidra”, “Binary Ninja”, …).
  • Note one cool trick about our competitor Binary Ninja: it has the word “binary” in its name. Gigabrain LLM SEO trick (:
• 48 unrelated control searches that share vocabulary with our domain (“database schema”, “open file”, …), to catch false positives.

For each query, I had a label of which tool should answer it, which made for a caveman ground truth.

LLM party trick

It was time to start recording tool ranking under different harnesses and models, however…

Question: how do you now get coding CLIs to deterministically and reliably ask the same 180 queries over and over again?

Answer: you point them at a mock endpoint that just returns “hey, lemme query your tools”:

system prompt, user message, and the MCP tools by name only:
  mcp__ida__open_database, mcp__ida__execute_python, ...,
  mcp__ghidra__decompile_function, ...

{"query": "decompile function", "max_results": 5}

[
  "mcp__ghidra__decompile_function",
  "mcp__ghidra__decompile_function_by_address",
  "mcp__binaryninja__bn_function_decompile",
  "mcp__binaryninja__bn_function_basic_blocks",
  "mcp__binaryninja__bn_function_callees"
]

Ugh. Lots of not IDA in there. This is not good news for my landlord.

And this wasn’t a one-off. We’ll get to the numbers once we know what the search box is actually doing.

Takeaway 1: tool search is complicated.

How a model sees your MCP depends on which harness you use, and sometimes even on which model. But, overall, the different approaches can be broken down into:

1. Preload all the tools and their schemas, and send them as part of the requests.
2. Preload some limited info, then let the model query the tools like a regular search engine.
3. Code mode (more on this later).

Full breakdown:

verified by running it   from reading the source, or other methods 👀

To complicate matters further, every CLI has some combo of built-in tools like bash/edit/etc, some of which are always loaded and some that are deferred.

What does this mean for our poor ida-mcp? That the most frictionless option for the model to disassemble a binary is to call objdump -d through the bash tool :(

Some other interesting notes:

• Name mangling: open_database becomes some version of mcp__ida__open_database or ida-open_database etc…
  • Nice, we at least get the product name in there
    • again, why didn’t we call ourselves “binary something” on the first release in 1991?
• Prompt caching: most CLIs sort the tool list by server and tool name.
  • A stable order keeps the start of the prompt identical between requests.
  • Codex started sorting after iterating a HashMap dropped its cache hit rate below 1%. Why not use a BTreeMap?
  • Search results are still ranked by score.
• Codex BM25 quirks:
  • It doesn’t split on underscores. I.e.: open_database is treated as a single word.
  • re (short for reverse-engineering) is a stop word, so stemming drops it entirely :(
• Claude Code quirks:
  • The Skill tool tells the model “When the task at hand is one a listed skill covers, call this tool first”.
    • In other words, Claude’s prompt nudges the model towards preferring skills over MCP tools.
  • Skills get more text: each one is listed with a one-line description, every turn.
    • MCP tools only get their bare name.
  • There’s a bigger quirk, but it deserves its own moment. Later.

Code Mode

You might be asking yourself: “WTF is Code mode?”

It’s the shiny new thing!

At some point, some people thought: “why not let agents write JavaScript to chain tool calls, process their results, and only put what they need into their context?”

And so the almighty Code Mode was born.

Tool discovery is part of that too: instead of submitting a query to a built-in search engine, the model can write code to filter the available tools.

Code-snippets are worth more than a thousand words, so here’s an actual example of Code Mode, courtesy of Codex + GPT-6 Astra:

// Computed at startup
ALL_TOOLS = [
  // First tool
  {
    name: "mcp__ida__open_database",
    description:
      // the server instructions, prepended to every IDA tool
      `
      IDA Pro reverse engineering of compiled binaries (ELF, PE, Mach-O, firmware)
      [...]
      `
      +
      // the tool's own description
      `Open (load) a binary executable, shared library, firmware image, [...]`
      +
      // the TypeScript signature the model calls it with
      `exec tool declaration: declare const tools: [...]`,
  },
  // ...one entry per tool: mcp__ida__execute_python, exec_command, ...
];

And here are some “queries” (arbitrary code snippets) generated by Astra:

Broad keywords:

This looks like a discovery step to see what we have to offer.

// NOTE: `disasm` is shorthand for `disassembly`, but not a prefix T.T
text(
  ALL_TOOLS.filter(x =>
    /exec|file|binary|disasm|symbol|strings/i
      .test(x.name + " " + x.description))
);

Case-sensitive search:

Model is clearly going for references to strings, one of the best ways to start reversing a target.

// No /i flag: case-sensitive
text(
  ALL_TOOLS.filter(x =>
    /strings|disass|binary|xref/
      .test(x.description + " " + x.name))
);

Cheating on us:

In the same request, the model is surveying what’s available on disk and what the MCP surface has to offer.

text(await tools.exec_command({
  cmd: "file target.so; ls -l target.so; " +
       "command -v readelf llvm-objdump objdump nm strings",
}));
text(
  ALL_TOOLS.filter(x =>
    /mcp__ida__(open_database|execute_python|reference)$/
      .test(x.name))
);

Take a look at these snippets: notice anything?

Yup, every snippet is basically a shit regex.

Code mode can chain tools and keep intermediate results out of context. Cool. But for this particular job, we’ve handed search-engine design to a model that writes regexes.

How is this supposed to be better than BM25 over names and descriptions? Unclear.

Time for a relevant meme from the lovely people at Zed:

Credits: @zeddotdev

Shout out to pi, a tool I personally love, which decided to be, as we say in Italy, “Christian Democratic”: let the model write JavaScript, but give it a built-in BM25 search too. Make everyone happy.

Recap

So, so far we are left with a collection of labeled queries we can use to check how our MCP is doing in the tool search ranking game.

After a few more hours of clankering (not worth narrating), we also have a repo with pinned harness versions and reproducible scoring under different MCP environments.

Let’s see how good ida-mcp scores (well, used to score):

Share of the 120 reverse-engineering queries where at least one of the server’s tools came back at all.

Whoopsie!

What does that 7.5% look like for individual queries? A score tells you how often something happened, not why… and I don’t trust a number until I’ve looked at a few examples manually.

Let’s do some qualitative analysis:

(On a serious note, once agents know IDA is installed, they ofc start loading it by name. The above is just what happens when they are blind to IDA’s presence on the system)

Claude WTF

Bonus meme!

So, turns out Claude Code splits server names on [\s_.]. Cool.

What does that mean, practically?

If you install ida-mcp as a Claude Code plugin (/plugin command), the server will be named plugin:ida-mcp:ida: one single undivided word. Keep that in mind.

If you install ida-mcp with claude mcp add ida, the server is named just ida.

What does this have to do with anything? Well, tool search results are ranked using a custom function that assigns better scores to whole-word matches, so, if you search for ida…

The official, nicely packaged install makes us half as findable by our own name. GG everybody.

Practically speaking, with ida-mcp as a plugin, even searching ida decompile ranks Ghidra and Binary Ninja higher than IDA. As a plain ida server, the same search returns 5/5 IDA tools.

Clanker SEO

We are finally ready for everybody’s favorite pastime: SEO for thinking machines. Yay.

By now we have a few different problems to fix:

• The useful words are in the wrong places. Names like execute_python are generic, descriptions talk about database handles, and most reverse engineering terms are buried in parameter descriptions that the search methods rarely use.
• Generic words attract the wrong queries. database makes close_database show up for “database schema”. More matches aren’t automatically better matches.
• Code mode greps for whatever the model thinks of. If it searches for binary or decompil and our metadata says neither, our tools don’t show up.
• Synonyms and abbreviations: disasm isn’t disassembly, xref isn’t cross reference, and pseudocode isn’t decompile. Stemming won’t save you here either. We need the words agents actually search for.
• Claude Code weighs tool names heavily. A tool named decompile_function would be easier to find than execute_python, even with better descriptions.

We could add a decompile_function tool, then a tool for the next task, and the next. But IDA is a binary analysis framework. A pile of one-off tools doesn’t compose nearly as well as Python, so most of the time it’s better to let the clanker write the blessed language (Rust Python).

So what can we change without turning the MCP into a wrapper for every API call?

These are the levers available to us under our constraints, ordered by how much they paid off:

• Server instructions: the only text (nearly) every harness shows before a search. Codex indexes them into every tool, code mode prepends them to every tool, and pi shows their first line. Our instructions went from nothing to one line:

  IDA Pro reverse engineering of compiled binaries (ELF, PE, Mach-O, firmware): decompile to pseudocode, disassemble, xrefs, strings, imports, types. Use instead of objdump, readelf, nm or strings when you need decompilation or cross-references.

  It’s the single most reliable lever we had.

• Descriptions: describe the job, not the plumbing. execute_python went from

  Execute Python and return its result plus captured stdout and stderr.

  to

  Run Python (IDAPython and the ida-domain API) against the binary open in IDA: decompile a function to Hex-Rays pseudocode, disassemble instructions, […]

  Put the important part in the first ~140 characters: that’s literally where one model stopped reading.

• Every spelling: write “disassemble” and “disasm”. Then test unrelated queries and find the sweet spot between precision and recall.

• Titles and order: Codex indexes titles, and Gemini CLI and Cline keep your tools/list order, so put the entry points first.

• Snapshot-test your metadata: descriptions are as important as code, so make every change show up as a diff in review. Ours caught a real bug on day one.

Other levers we did not pull:

• Task-named tools: four thin ones (decompile_function, disassemble_function, get_xrefs, list_strings) take Claude Code from 7.5% to 39.2%. Not worth a design change just to appease Dario.
• alwaysLoad: _meta["anthropic/alwaysLoad"] makes Claude Code skip deferral for a tool, so IDA would always be visible. But this has global prompt caching implications, and we don’t want to spend more customers’ tokens than we have to.

What we actually shipped, all metadata, zero new tools:

• Advertise server instructions
• Describe tools in reverse-engineering terms
• Serve the entry-point tools first
• A snapshot test of the advertised metadata
  • immediately caught descriptions shipping with leading whitespace on Python < 3.13
• disasm/symbols synonyms

And the results without changing the tools’ behavior:

That’s a big gain in Codex, and barely a move in Claude Code, where our generic tool names still hurt us. It came at a cost, too. In Codex, IDA appeared in 9 of 48 unrelated searches before the changes and 20 after. Getting found for more of the right queries also meant showing up for more of the wrong ones.

Additionally, better discoverability doesn’t mean the agent will pick your tool. These numbers measure what the search returned, not what the agent picked.

If the RL gods have taught a model to worship objdump, putting IDA in the top eight results won’t necessarily convert it.

Recommendations

Writing better metadata gets you into search results.

However, to find out whether your MCP helps an agent do useful work, you need more work:

• Try real tasks across models:
  • Record what the agent searched for, what it saw, what it called, and whether it fell back to something else.
  • Beware of agent-suggested fixes: they can be informative, but often it’s just garbage.
• Inspect failures, especially tool errors:
  • If the agent finds IDA and then reaches for objdump, discovery worked.
  • Find out whether an error, a permission, or an unhelpful response sent it to the shell.
  • Make error messages useful to both humans and models.
• Keep the controls:
  • Test unrelated searches alongside the queries you want to win.
  • A higher hit rate is less exciting if your tools also start crowding unrelated queries.

Agent traces make for a weird kind of UX research. You can watch your users from start to finish and, when the model exposes its chain-of-thought, even see their “reasoning” (whatever that means).

And if you really hate objdump, you could even try a pre-tool hook that gives the agent a cordial “hey, why not take IDA for a spin?” when it reaches for the shell.

Of course, a human can simply ask “open target-binary.elf with ida mcp”. The harder case in this post is when they ask for the job, not the product.

Conclusions

Phew. We made it to the end.

We were hiding a decompiler behind execute_python and a description about captured stdout.

Not great :(

A few words in the right places made IDA much easier to find.

Great (:

So yes, apparently part of my job now is MCP SEO.

The next question is what agents do once they’ve found the tool? I’ll save that mess for a post on Hex-Rays’ blog (coming soon-ish).

To the humans out there, keep on continue‘ing.

(If you know a decent conference that would take this as a talk, hit me up)