49f1827633
## Objective Reduce the time spent on ECS benchmarks without significantly compromising coverage. ## Background A `cargo bench -p benches --bench ecs` takes about 45 minutes. I'm guessing this bench is mainly used to check for regressions after ECS changes, and requiring 2x45 minute tests means that most people will skip benchmarking entirely. I noticed that some benches are repeated with sizes from long linear progressions (10, 20, ..., 100). This might be nice for detailed profiling, but seems too much for a overall regression check. ## Solution The PR follows the principles of "three or four different sizes is fine" and "powers of ten where it fits". The number of benches is reduced from 394 to 238 (-40%), and time from 46.2 minutes to 32.8 (-30%). While some coverage is lost, I think it's reasonable for anyone doing detailed profiling of a particular feature to temporarily add more benches. There's a couple of changes to avoid leading zeroes. I felt that `0010, 0100, 1000` is harder to read than `10, 100, 1000`. ## Is That Enough? 32 minutes is still too much. Possible future options: - Reduce measurement and warmup times. I suspect the current times (mostly 4-5 seconds total) are too conservative, and 1 second would be fine for spotting significant regressions. - Split the bench into quick and detailed variants. ## Testing ``` cargo bench -p benches --bench ecs ``` |
||
|---|---|---|
| .. | ||
| benches | ||
| src | ||
| Cargo.toml | ||
| README.md | ||
Bevy Benchmarks
This is a crate with a collection of benchmarks for Bevy.
Running benchmarks
Benchmarks can be run through Cargo:
# Run all benchmarks. (This will take a while!)
cargo bench -p benches
# Just compile the benchmarks, do not run them.
cargo bench -p benches --no-run
# Run the benchmarks for a specific crate. (See `Cargo.toml` for a complete list of crates
# tracked.)
cargo bench -p benches --bench ecs
# Filter which benchmarks are run based on the name. This will only run benchmarks whose name
# contains "name_fragment".
cargo bench -p benches -- name_fragment
# List all available benchmarks.
cargo bench -p benches -- --list
# Save a baseline to be compared against later.
cargo bench -p benches -- --save-baseline before
# Compare the current benchmarks against a baseline to find performance gains and regressions.
cargo bench -p benches -- --baseline before
Criterion
Bevy's benchmarks use Criterion. If you want to learn more about using Criterion for comparing performance against a baseline or generating detailed reports, you can read the Criterion.rs documentation.