Winner’s Curse

The phenomenon where implementing a “winning” test actually decreases performance, usually because the test was stopped too early or the results weren’t actually significant. Often accompanied by confused muttering about “statistical flukes.”