What Happened
There's been a vulnpocalypse, to which I don't because I'm not a security person, but I haven't seen much discussion on the closely related (, less serious, issue), the benchmarkpocalypse.
While it's become easier serious performance gains, it's also become easier reward hack a benchmark performance gains. The former is probably happening quietly across many different companies, but the latter is something I a week nowadays. Someone optimized X performance improvement over existing software, but, , optimization that improves benchmark performance without actually improving real-world performance. "we rewrote X in Rust"1 project or a new startup that's looking to either fundraise or sell something, but it happens projects as well.
Of course, people have always trumpeted unrepresentative microbenchmarks project is great. It's always unrepresentative microbenchmark and that's change. What's changed a a large benchmark suite, . a few famous examples of gaming large benchmark suites . For example, people cared about SPECint / SPECfp as proxies for workstation performance, CPU vendors compiler "optimizations" calculation in the benchmark, finding a way to improve 179.art by 12x in SPECfp2000. Skilled engineers spent a trying to find benchmark . trivial.
Rather someone's bad claim, I', this regex engine I , which I 's fastest regex engine because fairly comprehensive rebar regex benchmark suite. created by putting a loop for a month with instructions to not overfit to the benchmark supervision. , getting a good benchmark score is fairly easy, different; it took a couple weeks to roughly performance and then another couple 1.4x faster2 on rebar. But agents reward hack and overfit unless you put serious guardrails , which I didn't experiment.
Why It Matters
overfitting, I somewhat arbitrarily3 used the ripgrep benchmark corpus as a holdout benchmark it was 10x slower benchmark didn't take forever algorithmic blow-up, 't reasonable benchmark to complete. 40% faster!
Andrew Gallant (aka BurntSushi)'s rebar benchmark suite is fairly comprehensive as benchmaark suites go, a fairly comprehensive benchmark suite, agents have no problem getting a overfitting in a 't necessarily give good general performance.
a trick we talked about before telling , 's a holdout benchmark 's judged against. After that, the LLM moderately generalized performance 's about 2.4x slower overall on the holdout. That sounds pretty good considering that we're comparing fastest general purpose regex engine in existence. But, recall that these benchmarks a coding agent. On looking benchmarks measure, really don't include, weight. benchmarks matter, FRE is 4x slower on the holdout0, which is a lot better than before applying ' " a holdout" trick, but still pretty 40% faster.
There are a few things I thought were interesting about this:
What Comes Next
On (1), no wonder I'm seeing claims. , to build something performance well enough bogusly claim a 40% speedup, a fair amount of expertise. At a minimum, you'd a pretty good understanding of string matching algorithms, regex engines, decent general code optimization and SIMD optimization skills. a compiles machine code, so you'd compiler expertise. benchmark cheating (whether cheating) with a few minutes of typing.
On (2), I'm curious if this generalizes but haven't tried enough examples .
On (3), there's no reason to use a library that was almost no human effort that's slower than a robust, existing, well-tested, library, so I artifact uninteresting. The thing I find interesting substitute , specialized, and expensive, knowledge.
, knowledge, you probably wouldn't write a custom regex engine that's optimized for your particular workload. - people customization, e.g., when I worked , the code contained multiple different compilers because someone who worked on it wanted maximal performance; compile time and compiled performance in a search engine -offs are different in different places, you get better performance by writing a custom compiler a normal project interpreter or directly structure with "normal code". The person compilers, working on regex- multiple custom regex engines, people expertise and the inclination , freedom specialized . engineer (then a Partner-level engineer, promoted to Distinguished Engineer search index) compared running a loop, writing specialized orders of magnitude.
Even though the overall FRE regex engine has worse performance , specializing to your workload , , reasonable to insert your own specialized regex engine somewhere, various -level software. You don't maximalist 's plausible that, within some number of years, happening for larger things, like databases.
Thanks to Yossi Kreinin, Jamie Brandon, Peter Geoghegan, Luke Burton, John Spurling, and Max Bittker for comments/corrections/discussion.
P.S. Per the discussion here, with LLMs, something for a bit and satisfy my curiosity , something rigorous enough to publish 't really changed (for a variety of reasons, I think it's actually gone up). The result I'm doing a lot more analyses sharing results with a few friends but not publishing them. As an experiment, I'm trying things very quickly, with a much lower standard for how cleaned up and rigorous things are than I'd normally have for something that appears ; I'd tell a friend in a casual conversation than what I'd normally put in a blog post. -, so it's something I really . opinions on this, !
Explore more: Software & AI Guide