Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

 

History

17 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

JumpTableBench

Ground truth for jump table recovery, used to measure the accuracy of B2R2's middle-end on every push.

A jump table is how compilers implement a dense switch: an indirect branch reads a target out of a table in the binary. Recovering those tables is what decides whether a reconstructed CFG is complete, and getting it wrong is silent. This repository pins down the right answer for real binaries per architecture (x86, x64, 32-bit ARM, AArch64, MIPS, RISC-V, PowerPC and m68k) so that a regression shows up as a failing check rather than as a quietly worse CFG.

What is here

Ground truth lives in git; binaries do not. The JSON is small, reviewable, and belongs in history. The binaries are large and change a few times a year, so they ship as one tarball per configuration attached to a release. A configuration is an architecture and a compiler, <arch>-<compiler> (e.g., x64-gcc), and both ground-truth/ and binaries/ are split by it.

SHA256SUMS                        one line per binary; the integrity anchor
exclusions.json                   cases left out of scoring, and why
ground-truth/<config>/<case>.json one file per binary
schema/jumptable.schema.json      JSON Schema for the above
scripts/                          generate shasums, pack tarballs, find redundant cases

binaries/<config>/ is where a local build lands; it is gitignored.

Dataset Overview

Every case is a stripped binary, built both as a PIE and as a non-PIE, named <project>-<version>-<compiler>-<pie|nopie>-<opt>-<program>:

projects GNU coreutils 9.9 and GNU binutils 2.45
compilers gcc 16.2.0 and clang 23.0.0
architectures x86, x64, ARMv7 as armhf (hard float) and armel (soft float), aarch64, big-endian MIPS32r2 (o32) and MIPS64r2 (n64) as mips32 and mips64, RV32GC (ilp32d) and RV64GC (lp64d) as riscv32 and riscv64, big-endian 32-bit PowerPC (SysV, hard float) and 64-bit PowerPC (ELFv1) as ppc32 and ppc64, and the Motorola 68020 as m68k
binaries 546 (x86), 548 (x64), 397 each (armhf, armel), 548 (aarch64), 409 (mips32), 542 (mips64), 546 each (riscv32, riscv64), 548 each (ppc32, ppc64) and 276 (m68k), of which 24, 24, 6, 24, 18, 18, 24, 24 and 12 are from binutils
tables 10,458 (x86), 9,582 (x64), 5,279 each (armhf, armel), 5,581 (aarch64), 9,287 (mips32), 9,024 (mips64), 8,964 (riscv32), 8,273 (riscv64), 4,948 (ppc32), 4,528 (ppc64) and 4,736 (m68k)
  • We exclude 37 coreutils programs (e.g., ls, vdir, test, true, yes) whose jump tables are already covered by another included program, to avoid redundant evaluation. Two tables count as the same when their kind, entry size, entry count, and targets relative to the indirect branch all match; scripts/find-redundant.py lists any case that adds no distinct table.
  • We keep three binutils programs: as-new, objdump and readelf. The other seven link the same libbfd, so they add only about 10% of the distinct binutils tables but would take over 60% of the binutils scoring time.

Intel

x86 and x64 take both compilers at -O2 and -Os:

  • We leave out -O0 and -O1 because they mostly repeat -O2: gcc builds almost no jump table at -O0, and 85-89% of the tables at -O1 have an -O2 counterpart of the same kind, entry size, entry count, and target count in the same program.
  • We leave out -O3 and -Oz to bound the scoring time. Every jump table bug B2R2 has hit at those levels so far also showed up at -O2 or -Os.
  • clang builds no stdbuf, and neither does gcc for the non-PIE x86 build.

ARM

armhf and armel take only the cells, a compiler at an optimization level, that B2R2 already gets entirely right on both. The check passes only when nothing is missed or invented, and one that fails from the start catches no regression. A cell joins once B2R2 gets all of it right:

  • gcc at -O2 and clang at -Os, coreutils only. gcc at -Os and clang at -O2 still miss a table or two in five and six programs, and of the binutils programs only the clang PIE as-new passes at -O2 and -Os.
  • clang at -O0, binutils included. Only a third of its coreutils tables have an -Os counterpart of the same kind, entry size, entry count, and target count in the same program, and clang -O2, which would repeat two thirds of them, is not in yet.
  • clang builds no stdbuf, and neither does gcc for the non-PIE build at -O2.

AArch64

aarch64 takes both compilers at -O2 and -Os, as x86 and x64 do, binutils included. As with ARM, a cell joins only once B2R2 gets all of it right; these four already pass:

  • We leave out -O1 and -O3 because they mostly repeat -O2: 73-75% (gcc) and 88-95% (clang) of their tables have an -O2 counterpart of the same kind, entry size, entry count, and target count in the same program. gcc builds no jump table at -O0, and its -Oz binaries are byte for byte its -Os ones.
  • clang at -O0 and -Oz is not in yet. None of its -O0 tables and only 41% of its -Oz ones have an -Os counterpart, but B2R2 still invents targets in the non-PIE as-new and objdump at -O0, and misses or invents some in as-new, objdump and the non-PIE expr at -Oz.
  • clang builds no stdbuf.

MIPS

mips32 and mips64 take both compilers at -O2 and -Os, as x86 and x64 do, binutils included. As with ARM, a cell joins only once B2R2 gets all of it right, which leaves two out for now:

  • gcc at -Os is not in yet for mips32, as B2R2 still misses a table in the PIE sort, and gcc at -O2 takes coreutils only for mips64, as B2R2 still misses one in readelf.
  • We leave out -O1 and -O3 because they mostly repeat -O2: 83-90% (gcc) and 91-97% (clang) of their tables have an -O2 counterpart of the same kind, entry size, entry count, and target count in the same program. gcc builds almost no jump table at -O0 (none for mips32, 14 for mips64), and its -Oz binaries are byte for byte its -Os ones.
  • We leave out clang at -O0 and -Oz to bound the scoring time: 71-74% of the -O0 tables and 98% of the -Oz ones have an -O2 or an -Os counterpart, respectively.
  • clang builds no stdbuf, and neither does gcc for the non-PIE mips32 build.

RISC-V

riscv32 and riscv64 take both compilers at -O2 and -Os, as x86 and x64 do, binutils included. B2R2 already gets every cell of both right, so, as with Intel, the levels left out only bound the scoring time:

  • We leave out -O1 and -O3 because they mostly repeat -O2: 69-92% (gcc) and 95-97% (clang) of their tables have an -O2 counterpart of the same kind, entry size, entry count, and target count in the same program. gcc builds almost no jump table at -O0 (14 per architecture), and its -Oz binaries are byte for byte its -Os ones.
  • We leave out clang at -O0 and -Oz: 67-69% of the -O0 tables and 99% of the -Oz ones have an -O2 or an -Os counterpart, respectively.
  • clang builds no stdbuf, and neither does gcc for the non-PIE build.

PowerPC

ppc32 and ppc64 take both compilers at -O2 and -Os, as x86 and x64 do, binutils included. B2R2 already gets every cell of both right, so, as with Intel, the levels left out only bound the scoring time:

  • We leave out -O1 and -O3 because they mostly repeat -O2: 64-93% (gcc) and 96-100% (clang) of their tables have an -O2 counterpart of the same kind, entry size, entry count, and target count in the same program. gcc builds no jump table at -O0, and its -Oz binaries are byte for byte its -Os ones.
  • We leave out clang at -O0 and -Oz: 55-59% of the -O0 tables and all of the -Oz ones have an -O2 or an -Os counterpart, respectively.
  • clang builds a jump table on PowerPC only for a switch of 64 cases or more, so its configurations hold 132 (ppc32) and 140 (ppc64) tables, all but 8 of them in binutils. The 252 cases of each that hold none still check that no table is invented.
  • clang builds no stdbuf.

m68k

m68k takes gcc only, at -O2 and -Os, binutils included. B2R2 already gets every cell right, so, as with Intel, the levels left out only bound the scoring time:

  • We leave out -O1 and -O3 because they mostly repeat -O2: 78-93% of their tables have an -O2 counterpart of the same kind, entry size, entry count, and target count in the same program. gcc builds no jump table at -O0, and its -Oz binaries are byte for byte its -Os ones.
  • gcc builds stdbuf for both the PIE and the non-PIE build.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages