mirror of
https://github.com/zed-industries/zed.git
synced 2026-10-19 11:04:06 +00:00
gpui: Add bench_metrics for multi-metric Criterion benchmarks (#64753)
GPUI benchmarks measured only wall time, which is noisy enough on shared
CI runners that a regression must be large before Criterion flags it.
This PR adds a `bench_metrics` crate that records several metrics per
Criterion sample and reports them alongside Criterion's own analysis,
and wires GPUI's benchmark harness to it. Nothing in `bench_metrics`
depends on GPUI, so any Criterion benchmark in the workspace can adopt
it.
**What a benchmark now prints** (Linux, hardware counters available):
```
closed_inspector_grid/case/100-nodes-0-cycles
time: [273.99 µs 274.78 µs 276.04 µs]
GPUI bench report (all observed iterations): closed_inspector_grid/case/100-nodes-0-cycles
instructions per iteration: median 5.791 M instructions (min 5.788, max 5.880, samples 22)
foreground instructions per iteration: median 5.791 M instructions (...)
cycles per iteration: median 1.453 M cycles (...)
branch misses per iteration: median 1.856 K branch misses (...)
cache misses per iteration: median 123.008 cache misses (...)
foreground context switches per iteration: median 0.000 context switches (...)
page faults per iteration: median 0.000 page faults (...)
IPC: median 3.996 (min 3.349, max 4.063, samples 22)
foreground share of instructions: median 1.000 (...)
```
Criterion analyzes one metric (the *primary*; wall time by default) and
the rest are *secondaries* taken over the same iterations.
`BENCH_MEASUREMENT=instructions` makes process-wide retired instructions
the primary, and `BENCH_MEASUREMENT=foreground-instructions` the
benchmark thread's alone; both are near-deterministic, so a ~0.01%
interval is typical and small regressions are detectable, with wall time
demoted to the report. `BENCH_MEASUREMENT=wall-time` disables all
counters. Counters that can't be opened (no `CAP_PERFMON`, Docker
seccomp, VMs without a PMU, Intel Macs) are skipped with one note, so
`cargo bench` never fails by default; the instruction modes fail fast so
CI can't silently measure the wrong thing.
**Sources.** `HardwareCounter` is platform-neutral over per-OS backends.
On Linux it uses `perf_event_open`: counters are opened once per process
with `inherit` so every later thread is covered, read as deltas per
sample (~150 ns per counter), with hybrid-CPU encodings parsed from
sysfs and multiplexing scaled with a one-time warning. On Apple Silicon
it reads `proc_pid_rusage` and `thread_selfcounts`, which are
unprivileged, for instructions and cycles; branch and cache events need
`kperf` and stay Linux-only. `ResourceCounter` reads `getrusage` for
voluntary context switches on the foreground thread (the waits
instructions can't see) and minor page faults, with no privileges.
Foreground-thread instructions are counted separately from the process
total, so work shifting between GPUI's foreground and background threads
is visible even when the total is flat; with a headless renderer the gap
is driver-thread submission work (about 7% of a small Metal frame).
**Design.** `BenchMeasurement` type-erases the primary so
`#[gpui::bench]` functions and `BenchAppContext` need no type parameter;
`with_secondary`/`with_ratio` add metrics; `bench_group!` still
delegates to `criterion_group!`. Plain Criterion benches opt in with a
`config =` line and `MetricReport::iter` (documented in the gpui-bench
skill; wiring `rope` etc. is a follow-up). Reports are now emitted per
input for `inputs = ...` benchmarks, since per-iteration metrics blended
across inputs are meaningless. Criterion is bumped to 0.8.2, which also
randomizes stack alignment per sample.
**Verified** on Linux (hybrid Alder Lake) and an M-series Mac: every
mode on `inspector_render` and `editor_render`; counter denial via
seccomp on Linux; `bench_metrics` and `gpui` `bench_context` tests
including thread-scope inclusion/exclusion through real counters on both
platforms.
**Follow-ups:** counting-allocator metric (peak heap, allocations),
wiring non-GPUI benches, per-secondary baseline comparison.
Release Notes:
- [GPUI] Improved benchmarks to report retired instructions, cycles,
IPC, foreground-thread share, cache and branch misses, context switches,
and page faults per iteration alongside wall time, with
`BENCH_MEASUREMENT=instructions` or `=foreground-instructions` selecting
near-deterministic instruction counts as the Criterion-analyzed metric
for CI regression detection
This commit is contained in:
Anthony Eid
GitHub
parent
4c841aaf1c
commit
e683fd7b46
19 changed files with 2136 additions and 48 deletions
|
|
@ -132,6 +132,58 @@ The attribute supports options such as:
|
|||
|
||||
Inspect the current macro before depending on an option because the API is evolving.
|
||||
|
||||
Register benchmark functions with `gpui::bench_group!` and `gpui::bench_main!` (drop-in for Criterion's `criterion_group!`/`criterion_main!`, including the `name = ...; config = ...; targets = ...` form). The generated functions take `&mut criterion::Criterion<gpui::BenchMeasurement>`; `bench_group!` supplies that measurement from the environment.
|
||||
|
||||
### Measurements: wall time, hardware counters, and rusage
|
||||
|
||||
`gpui::BenchMeasurement` (from the gpui-independent `bench_metrics` crate) records several metrics per Criterion sample. Criterion analyzes one of them (the *primary*, shown on its `time:` line with confidence intervals and baseline comparison); the rest are *secondaries*, printed per iteration in the GPUI bench report as median/min/max. `BENCH_MEASUREMENT` selects the arrangement:
|
||||
|
||||
| `BENCH_MEASUREMENT` | Criterion analyzes | Reported as secondaries |
|
||||
|---|---|---|
|
||||
| unset (default) | wall time | every counter the machine supports (below) |
|
||||
| `instructions` | process-wide retired instructions; fails fast if counters are unavailable | the rest, plus wall time |
|
||||
| `foreground-instructions` | the benchmark thread's retired instructions; fails fast if unavailable | the rest, plus wall time |
|
||||
| `wall-time` | wall time | nothing; no counters are opened |
|
||||
|
||||
For a CI gate on frame cost prefer `foreground-instructions`: it excludes background parsing and GPU-driver submission threads, whose work varies run to run, and on Apple Silicon the process-wide count includes driver jitter that widens the interval (about ±0.6% versus ±0.01% for the foreground thread).
|
||||
|
||||
Default secondaries, each skipped with a one-time note when unavailable:
|
||||
|
||||
- **instructions** (process-wide) and **foreground instructions** (the benchmark thread only, where GPUI polls tasks, lays out, paints, and submits). Near-deterministic, so a ~0.1% spread is typical and small regressions are detectable. Blind to waits, memory stalls, and GPU shader work. Their ratio, **foreground share of instructions**, shows work moving between the foreground and background threads even when the total is flat; with a headless renderer it also exposes driver-thread submission work (about 7% of a small Metal frame).
|
||||
- **cycles** and **IPC** (`instructions / cycles`). Falling IPC with flat instructions means worse cache or branch behavior, not more work.
|
||||
- **branch misses** and **cache misses** (last-level), for diagnosing an IPC drop. Linux only.
|
||||
- **foreground context switches**: voluntary switches on the benchmark thread per iteration, i.e. how many times a frame blocked on a lock, channel, or fence. This is the metric that catches the waits instructions cannot see. Linux only.
|
||||
- **page faults**: minor faults per iteration, a privilege-free proxy for fresh memory touched.
|
||||
|
||||
Run one benchmark process per machine at a time. Hardware counters are per task, so instruction counts stay correct next to a concurrent `cargo bench`, but wall time, IPC, and cache misses share the hardware and degrade, and two processes requesting counters can exceed a core's counter slots, at which point the kernel multiplexes and the counts become scaled estimates (a warning is printed once). Within one process, benchmarks already run sequentially; process-wide counters can still attribute a previous benchmark's trailing background work to the next benchmark's first sample, which the foreground-thread metrics are immune to.
|
||||
|
||||
Platform support: on Linux, hardware counters come from `perf_event_open` and need `CAP_PERFMON` or `/proc/sys/kernel/perf_event_paranoid` at 2 or below; Docker's default seccomp profile and many cloud VMs block it, and the default mode then reports wall time plus the `getrusage` metrics. On Apple Silicon, instructions and cycles (both scopes) come from `proc_pid_rusage` and `thread_selfcounts` with no privileges; Intel Macs have no source. Criterion baselines are keyed by benchmark id, not measurement, so use distinct `--save-baseline` names when switching the primary on one machine.
|
||||
|
||||
Never present instructions alone for a latency claim; pair them with wall time and the frame metrics from the same run. Other Criterion measurements plug in with `criterion_group!` and `config = Criterion::default().with_measurement(gpui::BenchMeasurement::new(primary).with_secondary("name", other))`.
|
||||
|
||||
#### Benchmarks outside GPUI
|
||||
|
||||
`bench_metrics` has no GPUI dependency, so any Criterion benchmark in the workspace (`rope`, `sum_tree`, `text`, ...) gets the same metrics and the same `BENCH_MEASUREMENT` switch. Add `bench_metrics.workspace = true` to `[dev-dependencies]`, take `&mut Criterion<bench_metrics::BenchMeasurement>` in the benchmark functions, and register them with:
|
||||
|
||||
```rust
|
||||
criterion_group! {
|
||||
name = benches;
|
||||
config = Criterion::default()
|
||||
.with_measurement(bench_metrics::BenchMeasurement::from_env_or_exit());
|
||||
targets = rope_benchmarks
|
||||
}
|
||||
```
|
||||
|
||||
That alone switches Criterion's analyzed metric with `BENCH_MEASUREMENT=instructions`. To also print the per-iteration secondaries, run each routine through a `MetricReport`, which counts iterations for you:
|
||||
|
||||
```rust
|
||||
let report = bench_metrics::MetricReport::new();
|
||||
group.bench_function("append", |b| report.iter(b, || rope.append(small.clone())));
|
||||
report.print(" ");
|
||||
```
|
||||
|
||||
Criterion's `Throughput` lines keep working alongside. Page faults per iteration are a useful allocation-pressure signal for data-structure crates even without a counting allocator.
|
||||
|
||||
### `bench_iter`: synchronous functions and compute work
|
||||
|
||||
Use `bench_iter` for synchronous application or compute code:
|
||||
|
|
@ -197,8 +249,9 @@ On macOS, the headless renderer uses Metal without showing a window. `bench_rend
|
|||
|
||||
On platforms without a headless renderer, GPUI still performs CPU-side window work, but presenting discards the scene and does not measure real GPU submission. State this limitation in results.
|
||||
|
||||
Depending on the pinned revision, `BenchReport` can include:
|
||||
`BenchReport` is printed once per benchmark, and once per input for `inputs = ...` benchmarks, labeled with the Criterion benchmark id so the two can be correlated. Depending on the pinned revision, it can include:
|
||||
|
||||
- secondary measurements per iteration (median, min, max): instructions, foreground instructions, cycles, IPC, branch and cache misses, foreground context switches, page faults;
|
||||
- dirty-to-draw duration;
|
||||
- draw and present intervals;
|
||||
- invalidations per frame;
|
||||
|
|
@ -359,6 +412,7 @@ Before accepting a GPUI benchmark, verify:
|
|||
- [ ] Cache state is controlled and documented.
|
||||
- [ ] A responsiveness signal competes with the heavy work when foreground starvation matters.
|
||||
- [ ] Frame metrics and completion throughput are both reported.
|
||||
- [ ] Instructions and IPC (Linux and Apple Silicon) and foreground context switches (Linux) are reported next to wall time, and none is presented alone as the whole result.
|
||||
- [ ] Final correctness, ordering, and work counts are asserted.
|
||||
- [ ] The workload completes under bounded smoke and measured runs.
|
||||
- [ ] The same benchmark code runs on baseline and candidate.
|
||||
|
|
|
|||
1
.rules
1
.rules
|
|
@ -148,6 +148,7 @@ When an agent opens or updates a pull request, it must:
|
|||
- Use one bullet under `Release Notes:`:
|
||||
- `- Added ...`, `- Fixed ...`, or `- Improved ...` for user-facing changes, or
|
||||
- `- N/A` for docs-only and other non-user-facing changes.
|
||||
- Changes to `gpui` (including its benchmark and test harnesses) must have a bullet prefixed with `[GPUI]` describing the improvement even when nothing is user-facing in Zed itself, since GPUI is consumed as a library (CI enforces this), for example `- [GPUI] Improved benchmarks to report retired CPU instructions alongside wall time on Linux`.
|
||||
- Format release notes exactly with a blank line after the heading, for example:
|
||||
|
||||
```
|
||||
|
|
|
|||
84
Cargo.lock
generated
84
Cargo.lock
generated
|
|
@ -657,6 +657,15 @@ dependencies = [
|
|||
"equator",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "alloca"
|
||||
version = "0.4.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e5a7d05ea6aea7e9e64d25b9156ba2fee3fdd659e34e41063cd2fc7cd020d7f4"
|
||||
dependencies = [
|
||||
"cc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "allocator-api2"
|
||||
version = "0.2.21"
|
||||
|
|
@ -2112,6 +2121,16 @@ version = "0.5.2"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3a8241f3ebb85c056b509d4327ad0358fbbba6ffb340bf388f26350aeda225b1"
|
||||
|
||||
[[package]]
|
||||
name = "bench_metrics"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"criterion",
|
||||
"libc",
|
||||
"perf-event2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "benchmarks"
|
||||
version = "0.1.0"
|
||||
|
|
@ -2187,7 +2206,7 @@ dependencies = [
|
|||
"bitflags 2.13.1",
|
||||
"cexpr",
|
||||
"clang-sys",
|
||||
"itertools 0.11.0",
|
||||
"itertools 0.13.0",
|
||||
"log",
|
||||
"prettyplease",
|
||||
"proc-macro2",
|
||||
|
|
@ -2592,6 +2611,12 @@ dependencies = [
|
|||
"pkg-config",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "c-enum"
|
||||
version = "0.2.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cd17eb909a8c6a894926bfcc3400a4bb0e732f5a57d37b1f14e8b29e329bace8"
|
||||
|
||||
[[package]]
|
||||
name = "cached"
|
||||
version = "0.56.0"
|
||||
|
|
@ -4341,25 +4366,24 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "criterion"
|
||||
version = "0.5.1"
|
||||
version = "0.8.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f2b12d017a929603d80db1831cd3a24082f8137ce19c69e6447f54f5fc8d692f"
|
||||
checksum = "950046b2aa2492f9a536f5f4f9a3de7b9e2476e575e05bd6c333371add4d98f3"
|
||||
dependencies = [
|
||||
"alloca",
|
||||
"anes",
|
||||
"cast",
|
||||
"ciborium",
|
||||
"clap",
|
||||
"criterion-plot",
|
||||
"is-terminal",
|
||||
"itertools 0.10.5",
|
||||
"itertools 0.13.0",
|
||||
"num-traits",
|
||||
"once_cell",
|
||||
"oorandom",
|
||||
"page_size",
|
||||
"plotters",
|
||||
"rayon",
|
||||
"regex",
|
||||
"serde",
|
||||
"serde_derive",
|
||||
"serde_json",
|
||||
"tinytemplate",
|
||||
"walkdir",
|
||||
|
|
@ -4367,12 +4391,12 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "criterion-plot"
|
||||
version = "0.5.0"
|
||||
version = "0.8.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6b50826342786a51a89e2da3a28f1c32b06e387201bc2d19791f622c673706b1"
|
||||
checksum = "d8d80a2f4f5b554395e47b5d8305bc3d27813bacb73493eb1001e8f76dae29ea"
|
||||
dependencies = [
|
||||
"cast",
|
||||
"itertools 0.10.5",
|
||||
"itertools 0.13.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -7507,6 +7531,7 @@ dependencies = [
|
|||
"async-channel 2.5.0",
|
||||
"async-task",
|
||||
"backtrace",
|
||||
"bench_metrics",
|
||||
"bindgen",
|
||||
"bitflags 2.13.1",
|
||||
"chrono",
|
||||
|
|
@ -9131,9 +9156,9 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "itertools"
|
||||
version = "0.11.0"
|
||||
version = "0.13.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b1c173a5686ce8bfa551b3563d0c2170bf24ca44da99c7ca4bfdab5418c3fe57"
|
||||
checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186"
|
||||
dependencies = [
|
||||
"either",
|
||||
]
|
||||
|
|
@ -12965,6 +12990,41 @@ dependencies = [
|
|||
"serde_json",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "perf-event-data"
|
||||
version = "0.1.8"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "575828d9d7d205188048eb1508560607a03d21eafdbba47b8cade1736c1c28e1"
|
||||
dependencies = [
|
||||
"bitflags 2.13.1",
|
||||
"c-enum",
|
||||
"perf-event-open-sys2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "perf-event-open-sys2"
|
||||
version = "5.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d9c25955321465255e437600b54296983fab1feac2cd0c38958adeb26dbae49e"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"memoffset",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "perf-event2"
|
||||
version = "0.7.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0939b8fad77dfaeb29ebbd35faaeaadbf833167f30975f1b8993bbba09ea0a0f"
|
||||
dependencies = [
|
||||
"bitflags 2.13.1",
|
||||
"c-enum",
|
||||
"libc",
|
||||
"memmap2",
|
||||
"perf-event-data",
|
||||
"perf-event-open-sys2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pest"
|
||||
version = "2.8.3"
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ members = [
|
|||
"crates/auto_update_ui",
|
||||
"crates/aws_http_client",
|
||||
"crates/bedrock",
|
||||
"crates/bench_metrics",
|
||||
"crates/benchmarks",
|
||||
"crates/breadcrumbs",
|
||||
"crates/buffer_diff",
|
||||
|
|
@ -291,6 +292,7 @@ anthropic = { path = "crates/anthropic" }
|
|||
askpass = { path = "crates/askpass" }
|
||||
assets = { path = "crates/assets" }
|
||||
audio = { path = "crates/audio" }
|
||||
bench_metrics = { path = "crates/bench_metrics" }
|
||||
auto_update = { path = "crates/auto_update" }
|
||||
auto_update_ui = { path = "crates/auto_update_ui" }
|
||||
aws_http_client = { path = "crates/aws_http_client" }
|
||||
|
|
@ -597,7 +599,8 @@ core-text = "21"
|
|||
core-video = { version = "0.5.2", features = ["metal"] }
|
||||
cpal = "0.17"
|
||||
crash-handler = "0.8"
|
||||
criterion = { version = "0.5", features = ["html_reports"] }
|
||||
criterion = { version = "0.8.2", features = ["html_reports"] }
|
||||
perf-event = { package = "perf-event2", version = "0.7.4" }
|
||||
crossbeam = "0.8.4"
|
||||
ctor = "1.0.12"
|
||||
dap-types = { git = "https://github.com/zed-industries/dap-types", rev = "1b461b310481d01e02b2603c16d7144b926339f8" }
|
||||
|
|
|
|||
24
crates/bench_metrics/Cargo.toml
Normal file
24
crates/bench_metrics/Cargo.toml
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
[package]
|
||||
name = "bench_metrics"
|
||||
version = "0.1.0"
|
||||
edition.workspace = true
|
||||
publish = false
|
||||
license = "Apache-2.0"
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
|
||||
[lib]
|
||||
path = "src/bench_metrics.rs"
|
||||
doctest = false
|
||||
|
||||
[features]
|
||||
test-support = []
|
||||
|
||||
[dependencies]
|
||||
anyhow.workspace = true
|
||||
criterion.workspace = true
|
||||
libc.workspace = true
|
||||
|
||||
[target.'cfg(target_os = "linux")'.dependencies]
|
||||
perf-event.workspace = true
|
||||
1
crates/bench_metrics/LICENSE-APACHE
Symbolic link
1
crates/bench_metrics/LICENSE-APACHE
Symbolic link
|
|
@ -0,0 +1 @@
|
|||
../../LICENSE-APACHE
|
||||
1801
crates/bench_metrics/src/bench_metrics.rs
Normal file
1801
crates/bench_metrics/src/bench_metrics.rs
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -1,4 +1,4 @@
|
|||
use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};
|
||||
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
|
||||
use editor::{EditorStyle, MultiBuffer, display_map::*};
|
||||
use gpui::{AppContext as _, HighlightStyle, Hsla, TestDispatcher, font, px};
|
||||
use itertools::Itertools;
|
||||
|
|
@ -6,6 +6,7 @@ use multi_buffer::MultiBufferOffset;
|
|||
use project::project_settings::DiagnosticSeverity;
|
||||
use rand::{Rng, SeedableRng, rngs::StdRng};
|
||||
use settings::SettingsStore;
|
||||
use std::hint::black_box;
|
||||
use std::{num::NonZeroU32, time::Duration};
|
||||
use text::Bias;
|
||||
use util::RandomCharIter;
|
||||
|
|
|
|||
|
|
@ -16,9 +16,7 @@ use benchmarks::bench_utils::{
|
|||
RUST_FUNCTION_BODY_LINES, RUST_FUNCTION_LINES, RUST_MODULE_HEADER_LINES, random_rust_file,
|
||||
rust_file_line_count, rust_identifier as identifier,
|
||||
};
|
||||
use criterion::{
|
||||
BatchSize, BenchmarkId, Criterion, Throughput, black_box, criterion_group, criterion_main,
|
||||
};
|
||||
use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};
|
||||
use editor::{Editor, EditorStyle};
|
||||
use futures::{StreamExt as _, pin_mut, task::noop_waker};
|
||||
use gpui::{
|
||||
|
|
@ -32,6 +30,7 @@ use prompt_store::ProjectContext;
|
|||
use rand::{Rng as _, SeedableRng as _, rngs::StdRng};
|
||||
use serde_json::{Value, json};
|
||||
use settings::{Settings as _, SettingsStore};
|
||||
use std::hint::black_box;
|
||||
use ui::IntoElement as _;
|
||||
|
||||
const SEED: u64 = 0x5EED_5EED;
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};
|
||||
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
|
||||
use edit_prediction::metrics::compute_kept_rate;
|
||||
use std::hint::black_box;
|
||||
|
||||
fn repeated_function_lines(line_count: usize) -> String {
|
||||
let mut text = String::with_capacity(line_count * 32);
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
use std::sync::Arc;
|
||||
|
||||
use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};
|
||||
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
|
||||
use edit_prediction::metrics::count_tree_sitter_errors;
|
||||
use fs::FakeFs;
|
||||
use gpui::{AppContext as _, TestAppContext};
|
||||
|
|
@ -8,6 +8,7 @@ use language::{Buffer, BufferSnapshot, LanguageRegistry};
|
|||
use languages::init as init_languages;
|
||||
use node_runtime::NodeRuntime;
|
||||
use settings::SettingsStore;
|
||||
use std::hint::black_box;
|
||||
|
||||
struct ParsedCase {
|
||||
label: String,
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ test-support = [
|
|||
"x11",
|
||||
"proptest",
|
||||
]
|
||||
bench-support = ["profiler", "dep:criterion"]
|
||||
bench-support = ["profiler", "dep:criterion", "dep:bench_metrics"]
|
||||
# Preserve the published feature name while consumers migrate to `bench-support`.
|
||||
bench = ["bench-support"]
|
||||
inspector = ["gpui_macros/inspector"]
|
||||
|
|
@ -53,6 +53,7 @@ backtrace = { workspace = true, optional = true }
|
|||
bitflags.workspace = true
|
||||
|
||||
collections.workspace = true
|
||||
bench_metrics = { workspace = true, optional = true }
|
||||
criterion = { workspace = true, optional = true }
|
||||
ctor.workspace = true
|
||||
derive_more.workspace = true
|
||||
|
|
@ -131,13 +132,13 @@ objc2-metal = { version = "0.3", optional = true }
|
|||
scap = { workspace = true, optional = true }
|
||||
|
||||
|
||||
|
||||
[target.'cfg(target_os = "windows")'.dependencies]
|
||||
windows = { workspace = true, features = ["Win32_Foundation", "Win32_System_Power"] }
|
||||
|
||||
|
||||
[dev-dependencies]
|
||||
backtrace.workspace = true
|
||||
bench_metrics = { workspace = true, features = ["test-support"] }
|
||||
collections = { workspace = true, features = ["test-support"] }
|
||||
env_logger.workspace = true
|
||||
gpui_platform = { workspace = true, features = ["font-kit", "wayland", "x11"] }
|
||||
|
|
|
|||
|
|
@ -21,7 +21,10 @@ use slotmap::SlotMap;
|
|||
|
||||
pub use async_context::*;
|
||||
#[cfg(feature = "bench-support")]
|
||||
pub use bench_context::{BenchAppContext, BenchReport, BenchWindowContext, bench_platform};
|
||||
pub use bench_context::{
|
||||
BenchAppContext, BenchMeasurement, BenchReport, BenchWindowContext, MetricReport,
|
||||
bench_platform,
|
||||
};
|
||||
use collections::{FxHashMap, FxHashSet, HashMap, TypeIdHashMap, TypeIdHashSet, VecDeque};
|
||||
pub use context::*;
|
||||
pub use entity_map::*;
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ use std::{
|
|||
};
|
||||
|
||||
use anyhow::{Result, anyhow};
|
||||
pub use bench_metrics::{BenchMeasurement, MetricReport};
|
||||
use hdrhistogram::Histogram;
|
||||
|
||||
use crate::{
|
||||
|
|
@ -101,6 +102,7 @@ pub struct ForegroundWorkSummary {
|
|||
#[derive(Clone)]
|
||||
pub struct BenchReport {
|
||||
frame_snapshot: Rc<RefCell<WindowFrameSnapshot>>,
|
||||
metrics: MetricReport,
|
||||
frame_budget_nanos: u128,
|
||||
}
|
||||
|
||||
|
|
@ -127,10 +129,18 @@ impl BenchReport {
|
|||
);
|
||||
Self {
|
||||
frame_snapshot: Rc::new(RefCell::new(WindowFrameSnapshot::new())),
|
||||
metrics: MetricReport::new(),
|
||||
frame_budget_nanos,
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the per-iteration secondary metrics (retired instructions,
|
||||
/// cycles, context switches, ...) recorded alongside Criterion's primary
|
||||
/// measurement.
|
||||
pub fn metrics(&self) -> &MetricReport {
|
||||
&self.metrics
|
||||
}
|
||||
|
||||
fn record_frame_timings<'i>(&self, events: impl IntoIterator<Item = &'i FrameEvent>) {
|
||||
let mut snapshot = self.frame_snapshot.borrow_mut();
|
||||
// `.ok()` on `record`: this operation is infallible (the histograms auto-resize).
|
||||
|
|
@ -239,15 +249,15 @@ impl BenchReport {
|
|||
}
|
||||
|
||||
/// Prints this report to stderr.
|
||||
pub fn print(&self, benchmark_name: Option<&'static str>) {
|
||||
pub fn print(&self, benchmark_name: &str) {
|
||||
let frame_snapshot = self.frame_snapshot.borrow();
|
||||
if frame_snapshot.is_empty() {
|
||||
if frame_snapshot.is_empty() && self.metrics.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
let benchmark_name = benchmark_name.unwrap_or("unknown benchmark");
|
||||
eprintln!("GPUI bench report (all observed iterations): {benchmark_name}");
|
||||
eprintln!(" note: includes Criterion warmup/calibration");
|
||||
self.metrics.print(" ");
|
||||
self.print_histogram("window dirty-to-draw", &frame_snapshot.dirty_to_draw);
|
||||
self.print_histogram("window draw", &frame_snapshot.draw);
|
||||
self.print_histogram("window present interval", &frame_snapshot.present_interval);
|
||||
|
|
@ -488,7 +498,7 @@ pub struct BenchAppContext<'a, 'measurement> {
|
|||
background_executor: BackgroundExecutor,
|
||||
foreground_executor: ForegroundExecutor,
|
||||
benchmark_name: Option<&'static str>,
|
||||
bencher: Rc<RefCell<Option<&'a mut criterion::Bencher<'measurement>>>>,
|
||||
bencher: Rc<RefCell<Option<&'a mut criterion::Bencher<'measurement, BenchMeasurement>>>>,
|
||||
report: BenchReport,
|
||||
}
|
||||
|
||||
|
|
@ -501,7 +511,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
pub fn new(
|
||||
platform: Rc<dyn Platform>,
|
||||
benchmark_name: Option<&'static str>,
|
||||
bencher: &'a mut criterion::Bencher<'measurement>,
|
||||
bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>,
|
||||
) -> Self {
|
||||
Self::build(platform, benchmark_name, bencher, BenchReport::default())
|
||||
}
|
||||
|
|
@ -515,7 +525,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
pub fn new_with_platform_and_report(
|
||||
platform: Rc<dyn Platform>,
|
||||
benchmark_name: Option<&'static str>,
|
||||
bencher: &'a mut criterion::Bencher<'measurement>,
|
||||
bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>,
|
||||
report: BenchReport,
|
||||
) -> Self {
|
||||
Self::build(platform, benchmark_name, bencher, report)
|
||||
|
|
@ -524,7 +534,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
fn build(
|
||||
platform: Rc<dyn Platform>,
|
||||
benchmark_name: Option<&'static str>,
|
||||
bencher: &'a mut criterion::Bencher<'measurement>,
|
||||
bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>,
|
||||
report: BenchReport,
|
||||
) -> Self {
|
||||
let background_executor = platform.background_executor();
|
||||
|
|
@ -662,12 +672,16 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
let bencher = self.take_bencher("bench_iter");
|
||||
self.dispatch_pending_frames(|| true);
|
||||
let collector = TraceScope::start(self.foreground_journal_collector());
|
||||
self.report.metrics.discard_pending();
|
||||
let mut iterations = 0;
|
||||
let mut benchmark = || {
|
||||
iterations += 1;
|
||||
benchmark(self);
|
||||
self.dispatch_pending_frames(|| true);
|
||||
};
|
||||
bencher.iter(&mut benchmark);
|
||||
let events = collector.finish();
|
||||
self.report.metrics.record_sample(iterations);
|
||||
self.report.record_frame_timings(events.frame_events.iter());
|
||||
self.report
|
||||
.record_foreground_events(events.foreground_events());
|
||||
|
|
@ -722,6 +736,8 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
let mut benchmark_context = self.clone();
|
||||
let foreground_executor = self.foreground_executor.clone();
|
||||
let report = self.report.clone();
|
||||
report.metrics.discard_pending();
|
||||
let iterations = std::cell::Cell::new(0);
|
||||
|
||||
bencher.iter_batched_ref(
|
||||
|| {
|
||||
|
|
@ -740,6 +756,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
}
|
||||
},
|
||||
|measured_input| {
|
||||
iterations.set(iterations.get() + 1);
|
||||
let task = benchmark(&mut measured_input.input, &mut benchmark_context);
|
||||
benchmark_context.dispatch_pending_frames(|| true);
|
||||
let output = Rc::new(RefCell::new(None));
|
||||
|
|
@ -759,6 +776,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
},
|
||||
criterion::BatchSize::PerIteration,
|
||||
);
|
||||
report.metrics.record_sample(iterations.get());
|
||||
self.replace_bencher(bencher);
|
||||
}
|
||||
|
||||
|
|
@ -789,7 +807,10 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
let dispatcher = self.background_executor.dispatcher().clone();
|
||||
self.dispatch_pending_frames(|| true);
|
||||
let collector = TraceScope::start(self.foreground_journal_collector());
|
||||
self.report.metrics.discard_pending();
|
||||
let mut iterations = 0;
|
||||
let mut benchmark = || {
|
||||
iterations += 1;
|
||||
dispatcher
|
||||
.as_threaded()
|
||||
.expect("validated in BenchAppContext::build")
|
||||
|
|
@ -808,6 +829,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
bencher.iter(&mut benchmark);
|
||||
|
||||
let events = collector.finish();
|
||||
self.report.metrics.record_sample(iterations);
|
||||
self.report.record_frame_timings(events.frame_events.iter());
|
||||
self.report
|
||||
.record_foreground_events(events.foreground_events());
|
||||
|
|
@ -853,6 +875,8 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
.as_threaded()
|
||||
.expect("validated in BenchAppContext::build");
|
||||
let report = self.report.clone();
|
||||
report.metrics.discard_pending();
|
||||
let iterations = std::cell::Cell::new(0);
|
||||
|
||||
bencher.iter_batched_ref(
|
||||
|| {
|
||||
|
|
@ -867,6 +891,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
}
|
||||
},
|
||||
|measured_input| {
|
||||
iterations.set(iterations.get() + 1);
|
||||
let (state, window, stopped) = &mut measured_input.input;
|
||||
let started = Instant::now();
|
||||
let check_deadline = || {
|
||||
|
|
@ -932,6 +957,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
},
|
||||
criterion::BatchSize::PerIteration,
|
||||
);
|
||||
report.metrics.record_sample(iterations.get());
|
||||
self.replace_bencher(bencher);
|
||||
}
|
||||
|
||||
|
|
@ -1009,13 +1035,16 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> {
|
|||
}
|
||||
}
|
||||
|
||||
fn take_bencher(&self, benchmark_kind: &str) -> &'a mut criterion::Bencher<'measurement> {
|
||||
fn take_bencher(
|
||||
&self,
|
||||
benchmark_kind: &str,
|
||||
) -> &'a mut criterion::Bencher<'measurement, BenchMeasurement> {
|
||||
self.bencher.borrow_mut().take().unwrap_or_else(|| {
|
||||
panic!("cannot start {benchmark_kind}: benchmark measurement is already running")
|
||||
})
|
||||
}
|
||||
|
||||
fn replace_bencher(&self, bencher: &'a mut criterion::Bencher<'measurement>) {
|
||||
fn replace_bencher(&self, bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>) {
|
||||
let previous = self.bencher.borrow_mut().replace(bencher);
|
||||
assert!(
|
||||
previous.is_none(),
|
||||
|
|
@ -1323,11 +1352,96 @@ impl VisualContext for BenchWindowContext<'_, '_> {
|
|||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::{rc::Rc, sync::Arc};
|
||||
use std::{
|
||||
rc::Rc,
|
||||
sync::{
|
||||
Arc,
|
||||
atomic::{AtomicU64, Ordering},
|
||||
},
|
||||
};
|
||||
|
||||
use super::*;
|
||||
use crate::profiler::journal::install_test_foreground_journal;
|
||||
|
||||
/// Runs `benchmark` under a wall-time primary with a fake counter as the
|
||||
/// secondary metric and returns the per-iteration values the report
|
||||
/// recorded for it.
|
||||
fn secondary_values_per_iteration(
|
||||
counter: Arc<AtomicU64>,
|
||||
benchmark: impl Fn(&mut BenchAppContext, &Arc<AtomicU64>),
|
||||
) -> Vec<u64> {
|
||||
let platform = bench_platform(None, Arc::new(crate::NoopTextSystem::new()));
|
||||
let report = BenchReport::default();
|
||||
let mut criterion = criterion::Criterion::default()
|
||||
.with_measurement(
|
||||
BenchMeasurement::new(criterion::measurement::WallTime).with_secondary(
|
||||
"fake counter",
|
||||
bench_metrics::FakeCounter::new(counter.clone()),
|
||||
),
|
||||
)
|
||||
.without_plots()
|
||||
.sample_size(10)
|
||||
.warm_up_time(Duration::from_millis(1))
|
||||
.measurement_time(Duration::from_millis(1));
|
||||
|
||||
criterion.bench_function("secondary_values_per_iteration", |bencher| {
|
||||
let mut cx = BenchAppContext::new_with_platform_and_report(
|
||||
platform.clone(),
|
||||
Some("secondary_values_per_iteration"),
|
||||
bencher,
|
||||
report.clone(),
|
||||
);
|
||||
// Fixture work outside the measured loop must not be attributed
|
||||
// to any iteration.
|
||||
counter.fetch_add(1_000_000, Ordering::SeqCst);
|
||||
benchmark(&mut cx, &counter);
|
||||
cx.teardown();
|
||||
});
|
||||
|
||||
report
|
||||
.metrics()
|
||||
.values("fake counter")
|
||||
.expect("the fake counter was recorded")
|
||||
.into_iter()
|
||||
.map(|value| value as u64)
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn secondary_metric_is_normalized_per_iteration_for_bench_iter() {
|
||||
let values = secondary_values_per_iteration(Arc::new(AtomicU64::new(0)), |cx, counter| {
|
||||
cx.bench_iter(|_| {
|
||||
counter.fetch_add(11, Ordering::SeqCst);
|
||||
});
|
||||
});
|
||||
assert!(!values.is_empty());
|
||||
assert!(
|
||||
values.iter().all(|value| *value == 11),
|
||||
"each sample's total should be divided by its iteration count: {values:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn secondary_metric_is_accumulated_across_per_iteration_batches() {
|
||||
let values = secondary_values_per_iteration(Arc::new(AtomicU64::new(0)), |cx, counter| {
|
||||
cx.bench_batched_task(
|
||||
|_| {
|
||||
// Setup runs outside every measured interval.
|
||||
counter.fetch_add(1_000, Ordering::SeqCst);
|
||||
},
|
||||
|(), cx| {
|
||||
counter.fetch_add(11, Ordering::SeqCst);
|
||||
cx.background_spawn(async {})
|
||||
},
|
||||
);
|
||||
});
|
||||
assert!(!values.is_empty());
|
||||
assert!(
|
||||
values.iter().all(|value| *value == 11),
|
||||
"PerIteration batches should sum to one per-iteration value: {values:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn foreground_work_reports_long_task_without_window_draw() {
|
||||
let (journal, _journal_guard) = install_test_foreground_journal(1024, 64);
|
||||
|
|
@ -1421,6 +1535,7 @@ mod tests {
|
|||
let name = "bench_task_reports_long_task_without_window";
|
||||
|
||||
let mut criterion = criterion::Criterion::default()
|
||||
.with_measurement(BenchMeasurement::new(criterion::measurement::WallTime))
|
||||
.without_plots()
|
||||
.sample_size(10)
|
||||
.warm_up_time(Duration::from_millis(1))
|
||||
|
|
|
|||
|
|
@ -112,14 +112,35 @@ pub use spring::*;
|
|||
|
||||
/// Defines a Criterion benchmark group for benchmarks annotated with [`gpui::bench`].
|
||||
///
|
||||
/// This mirrors `criterion::criterion_group!` so GPUI benchmark files can keep the
|
||||
/// same shape as ordinary Criterion benchmarks.
|
||||
/// This mirrors `criterion::criterion_group!`, but the group measures with the
|
||||
/// `gpui::BenchMeasurement` configured by `BENCH_MEASUREMENT` (see
|
||||
/// `BenchMeasurement::from_env`). By default Criterion analyzes wall time
|
||||
/// while retired instructions, cycles, IPC, context switches, and the other
|
||||
/// counters this machine supports are printed per iteration in the GPUI bench
|
||||
/// report. `BENCH_MEASUREMENT=instructions` makes Criterion analyze
|
||||
/// process-wide instructions instead, `foreground-instructions` the benchmark
|
||||
/// thread's alone, and `wall-time` disables all counters. A
|
||||
/// `config = ...` expression may set any other Criterion option; its
|
||||
/// measurement is replaced. To measure with something else, call
|
||||
/// `criterion::criterion_group!` directly with
|
||||
/// `config = criterion::Criterion::default().with_measurement(gpui::BenchMeasurement::new(...))`.
|
||||
///
|
||||
/// [`gpui::bench`]: crate::bench
|
||||
#[macro_export]
|
||||
macro_rules! bench_group {
|
||||
($($tokens:tt)*) => {
|
||||
criterion::criterion_group!($($tokens)*);
|
||||
(name = $name:ident; config = $config:expr; targets = $($target:path),+ $(,)?) => {
|
||||
criterion::criterion_group! {
|
||||
name = $name;
|
||||
config = ($config).with_measurement($crate::BenchMeasurement::from_env_or_exit());
|
||||
targets = $($target),+
|
||||
}
|
||||
};
|
||||
($name:ident, $($target:path),+ $(,)?) => {
|
||||
$crate::bench_group! {
|
||||
name = $name;
|
||||
config = criterion::Criterion::default();
|
||||
targets = $($target),+
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -83,10 +83,13 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {
|
|||
let sample_size =
|
||||
sample_size.map(|sample_size| quote! { group.sample_size(#sample_size); });
|
||||
quote! {
|
||||
let report = #report_expr;
|
||||
let mut group = criterion.benchmark_group(#group_name);
|
||||
#sample_size
|
||||
for input in #inputs {
|
||||
// One report per input: per-iteration metrics differ across
|
||||
// inputs, so blending them would make the summary meaningless.
|
||||
let report = #report_expr;
|
||||
let report_name = format!("{}/{}/{}", #group_name, #input_name, input);
|
||||
group.bench_with_input(criterion::BenchmarkId::new(#input_name, &input), &input, {
|
||||
let report = report.clone();
|
||||
move |bencher, input| {
|
||||
|
|
@ -105,9 +108,9 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {
|
|||
cx.teardown();
|
||||
}
|
||||
});
|
||||
report.print(&report_name);
|
||||
}
|
||||
group.finish();
|
||||
report.print(Some(stringify!(#outer_fn_name)));
|
||||
}
|
||||
} else {
|
||||
if let Some(input_name) = input_name {
|
||||
|
|
@ -148,14 +151,14 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {
|
|||
cx.teardown();
|
||||
}
|
||||
});
|
||||
report.print(Some(stringify!(#outer_fn_name)));
|
||||
report.print(stringify!(#outer_fn_name));
|
||||
}
|
||||
};
|
||||
|
||||
TokenStream::from(quote! {
|
||||
#inner_fn
|
||||
|
||||
fn #outer_fn_name(criterion: &mut criterion::Criterion) {
|
||||
fn #outer_fn_name(criterion: &mut criterion::Criterion<gpui::BenchMeasurement>) {
|
||||
#benchmark
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main};
|
||||
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
|
||||
use gpui::rgba;
|
||||
use language::build_highlight_map;
|
||||
use std::hint::black_box;
|
||||
use theme::SyntaxTheme;
|
||||
|
||||
fn syntax_theme(highlight_names: &[&str]) -> SyntaxTheme {
|
||||
|
|
|
|||
|
|
@ -1,11 +1,10 @@
|
|||
use std::ops::Range;
|
||||
|
||||
use criterion::{
|
||||
BatchSize, BenchmarkId, Criterion, Throughput, black_box, criterion_group, criterion_main,
|
||||
};
|
||||
use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};
|
||||
use rand::prelude::*;
|
||||
use rand::rngs::StdRng;
|
||||
use rope::{Point, Rope};
|
||||
use std::hint::black_box;
|
||||
use sum_tree::Bias;
|
||||
use util::RandomCharIter;
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,6 @@
|
|||
use criterion::{
|
||||
BatchSize, BenchmarkId, Criterion, Throughput, black_box, criterion_group, criterion_main,
|
||||
};
|
||||
use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};
|
||||
use rand::{Rng as _, SeedableRng as _, rngs::StdRng};
|
||||
use std::hint::black_box;
|
||||
use streaming_diff::StreamingDiff;
|
||||
|
||||
const SEED: u64 = 0x5EED_5EED;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue