Benchmark Results

sysml-bench is an exploratory benchmark evaluating how tool-augmented LLMs perform on structured engineering tasks in SysML v2. 132 tasks across 8 categories (discovery, reasoning, explanation, layer, boundary, vector-sensitive, structural trace, corpus scaling), tested with 4 models and 40+ experimental conditions.

The study generated 14 observations. Three achieved nominal statistical significance. None survive correction for running 14 tests simultaneously. We want to be clear about what this study is and isn't: it is a well-characterized exploratory study that identifies patterns and estimates effect sizes. It is not a confirmatory study that proves those patterns are real. The contribution is the benchmark methodology, the identification of task-tool interaction as a key variable, and the effect size estimates that make confirmatory follow-up designable.

That said, the patterns are consistent and the effect sizes are large. We think they are worth sharing and worth testing further.

Representation matters more than retrieval

For AI systems working with structured knowledge, the form in which information is presented to the model matters more than the mechanism used to find it.

Most of the energy in the tool-augmented LLM space is going into retrieval infrastructure: vector databases, graph traversal engines, multi-hop reasoning chains. On our benchmark, every retrieval intervention we tested produced null results:

• Vector search added nothing (O5, exact tie 0.880 vs 0.880)
• Graph traversal added nothing at 2–3 hops (O9) or 4–5 hops (O14)
• Planning tools added nothing on hard tasks (O6, +0.035)

Every representation and guidance intervention produced large effects:

• Rendering model elements into structured views doubled accuracy on explanation tasks (O4, d=0.83)
• One sentence of tool selection guidance eliminated a 13-point penalty on discovery tasks (O12, d=0.75)
• CLI tool-based search dominated bulk context injection on discovery (O8, d=0.64)

The implication, if these patterns hold on other corpora, is that the marginal investment in retrieval infrastructure has near-zero return on structured engineering corpora of this scale. The marginal investment in representation infrastructure (views, summaries, structured renderings of domain artifacts) has large return. This is testable across every domain where LLMs operate on structured data.

Aggregate benchmarks hide task-level structure

Standard benchmark reporting (mean accuracy across tasks) actively misleads when task-level variance dominates condition-level variance.

Our aggregate tool comparison showed no significant difference (O1, p=0.391). A naive reading: tools don't matter. But per-task analysis revealed effect sizes ranging from −0.400 to +0.800. Enormous, opposite-signed effects that cancel in the mean. The aggregate null is not "no effect." It is "large effects in both directions, hidden by averaging."

This pattern recurs throughout the data:

• CLI outperforms RAG on discovery by 29 points but RAG is roughly equivalent on reasoning (O8). An aggregate would hide both signals.
• Scaling collapse is bimodal: 5 of 20 tasks score 1.000 on the larger corpus while 11 of 20 score below 0.333 (O10). The mean of 0.423 describes no actual task.
• O4's render advantage ranges from +1.000 (tasks E4, E7) to −0.200 (task E5). The mean of +0.383 understates the wins and hides the reversal.

Every benchmark that reports a single accuracy number is potentially hiding this structure. The methodological contribution is demonstrating that per-task analysis with paired effect sizes is necessary to surface real patterns in tool-augmented LLM evaluation.

O12 — Context engineering outperforms tool restriction

The naive response to "too many tools hurt performance" is to restrict the tool set. The better response is a sentence in the system prompt. When agents are instructed to start with search and read_file, escalating to graph tools only when search is insufficient, the 13-point discovery penalty from over-tooling disappears entirely. Performance with 6 tools matches and marginally exceeds the 2-tool baseline (0.887 vs 0.880).

The affected tasks (D11, D12, D16, D6) are those where unguided agents select structurally complex tools for attribute-lookup tasks that search handles trivially. The agent doesn't need graph traversal to find a part's mass. It needs search. But without guidance, it reaches for the most powerful tool available, and the overhead of using it (more tokens, more turns, more opportunities to go off track) costs accuracy.

This is the only adequately powered observation in the study (power=0.80). It is also the lowest nominal p-value (0.009). If we had to pick one finding to bet on replicating, this would be it.

O4 — Pre-rendered views outperform agent-assembled context

On explanation tasks, pre-rendered model views scored 0.873 vs 0.490 for letting the agent assemble its own context. A 38-point gap. Two tasks collapsed entirely without rendering: 1.000 with a pre-rendered view, 0.000 with agent assembly. The agent-assembled approach exhausts turns building context that a single render call provides.

The advantage is explanation-specific. On discovery tasks, pre-rendering scored 0.719, worse than search (0.880). Pre-rendering the wrong view adds noise, not signal. This matters: it means pre-rendering is not a universal improvement. It is a task-dependent one, and the task type determines whether it helps or hurts.

The cost difference is striking: $0.008 per task with rendering vs $0.053 with agent assembly. Better results at 6.6× lower cost. The pre-rendered view does the work at index time that the agent would otherwise do at query time, and it does it once instead of per-query.

This has the largest effect size in the study (d=0.83) but is underpowered (power=0.53, needs 14 tasks for 80%). The effect is real but we can't be confident in its magnitude yet.

O8 — Retrieval strategy interacts with task type

CLI tool-based search dominated structured lookup (+29 points over RAG, p=0.021, d=0.64, N=16 tasks). RAG edged ahead on cross-file reasoning (+14 points, p=0.403, not significant), likely because it injects all relevant context at once, avoiding the problem where the agent runs out of turns before it can chain together enough tool calls to answer multi-step questions.

The CLI advantage on discovery is driven by 5 tasks where RAG scores 0.000: tasks requiring iterative tool-mediated retrieval that single-shot context injection cannot perform. The model needs to search, read a result, search again based on what it found, and repeat. RAG gives it everything at once, which is the right coverage but the wrong format for these tasks.

Neither retrieval architecture is universally better. This is interesting because it suggests that the right approach is not to pick one, but to route queries to the right strategy based on task type. Whether that routing can be done cheaply enough to be practical is an open question.

O1 — Tool-task interaction is heterogeneous

Graph tools hurt discovery tasks, help layer tasks, and are near-neutral on reasoning. The aggregate difference is not statistically significant (paired t-test p=0.391, N=16) because the effect is task-dependent: graph tools help on tasks requiring structural completeness checking (D10, D13: +0.300 to +0.400) and hurt on tasks where search retrieves the answer directly (D11, D6: −0.600 to −0.800).

The pattern holds across all four models tested, making it one of the most robust qualitative observations in the benchmark despite the null aggregate test. This is the lead evidence for Thesis 2: the aggregate null is not "tools don't matter." It is "tools matter enormously, but in opposite directions on different tasks, and the average hides everything interesting."

O10 — Corpus scale is the dominant difficulty factor

Performance roughly halves from 19 to 95 files (0.880 to 0.423). Graph tools and vector search make things worse at scale: schema overhead and retrieval noise compound without compensating signal. 11 of 20 scaling tasks fall below 0.333. The distribution is bimodal: easy tasks remain easy, hard tasks become impossible.

The failure mode is revealing: 55% of the time the agent ran out of turns before finishing. 27% were reasoning errors. 0% were search failures. The agent can find the information. It just can't process enough of it within the turn budget to reach the right answer. This suggests the path forward is better orchestration (smarter turn allocation, hierarchical planning) rather than better search.

This is the observation that keeps us honest. Our other results come from a 19-file corpus. Real engineering repositories are hundreds or thousands of files. The scaling problem is unsolved by any method we tested, and small-corpus benchmarks produce optimistic estimates that may not transfer.

Other observations

Scoring

Per-field structured scoring. Each task defines expected fields with typed scorers: Bool (exact match), Float (numeric within tolerance), Str (exact string match), StrContains (case-insensitive substring), ListStr (F1 score with 0.8 threshold, binarized). Task score = mean of field scores. Condition score = mean of task scores across N runs.

Tool sets

Known confounds

cli_search (2 tools, ~250 schema tokens) vs cli_graph (6 tools, ~1120 schema tokens) confounds tool count, schema overhead, and selection complexity. When graph tools "hurt," the cause could be the tools themselves, the schema overhead consuming context window space, or the difficulty of choosing the right tool from a larger set. O12 (guided selection) partially disentangles selection difficulty. A schema ablation experiment (same tools, reduced schema) would isolate overhead.

Ground truth

Created by the primary author from SysML v2 model inspection. Two corrections applied during experimentation: D16 (35.0→37.0), R5 (3→2). Structural trace scoring schema corrected in v2 (ST2/ST7 scores changed 0.542→0.865). No independent verification. Single-author ground truth is a limitation.

Multiple comparison correction

14 observations tested at α=0.05 yields a family-wise error rate of about 51%. When corrected (Holm-Bonferroni step-down), no observation remains significant:

If O4 and O12 are designated as the only two hypotheses under test (Holm-Bonferroni at m=2), both survive: O12 adjusted p=0.018, O4 adjusted p=0.047. This designation was made after seeing the data. It becomes pre-registered only if declared before collecting new data on a second corpus.

Power analysis

Only one observation (O12) has enough statistical power to reliably detect its effect. The power analysis tells us exactly how large a follow-up study needs to be. This is itself a contribution: it makes the confirmatory work designable.

T1: Single corpus. All primary observations derive from one 19-file SysML v2 model. The corpus is small enough that exhaustive search may substitute for structured traversal, potentially explaining why graph tools show no advantage. All claims are scoped to "on our benchmark corpus."

T2: Confounded tool sets. cli_search vs cli_graph differs in tool count, schema overhead, and selection complexity simultaneously. O12 partially disentangles selection difficulty.

T3: Multiple comparisons. 14 observations at α=0.05 yields ~51% family-wise error rate. No observation survives full correction.

T4: Underpowered tests. Only O12 achieves 80% power. Most observations would need 30–300+ tasks to detect their effects reliably.

T5: Scoring methodology. ListStr binarization at 0.8 creates cliff effects. StrContains scoring for explanation tasks may be too lenient.

T6: Ground truth. Created and verified by a single author. Two corrections applied mid-experiment. No inter-rater reliability assessment.