diff --git a/.agents/skills/gpui-bench/SKILL.md b/.agents/skills/gpui-bench/SKILL.md index 8aa142b8364..8a1a2777835 100644 --- a/.agents/skills/gpui-bench/SKILL.md +++ b/.agents/skills/gpui-bench/SKILL.md @@ -132,6 +132,58 @@ The attribute supports options such as: Inspect the current macro before depending on an option because the API is evolving. +Register benchmark functions with `gpui::bench_group!` and `gpui::bench_main!` (drop-in for Criterion's `criterion_group!`/`criterion_main!`, including the `name = ...; config = ...; targets = ...` form). The generated functions take `&mut criterion::Criterion`; `bench_group!` supplies that measurement from the environment. + +### Measurements: wall time, hardware counters, and rusage + +`gpui::BenchMeasurement` (from the gpui-independent `bench_metrics` crate) records several metrics per Criterion sample. Criterion analyzes one of them (the *primary*, shown on its `time:` line with confidence intervals and baseline comparison); the rest are *secondaries*, printed per iteration in the GPUI bench report as median/min/max. `BENCH_MEASUREMENT` selects the arrangement: + +| `BENCH_MEASUREMENT` | Criterion analyzes | Reported as secondaries | +|---|---|---| +| unset (default) | wall time | every counter the machine supports (below) | +| `instructions` | process-wide retired instructions; fails fast if counters are unavailable | the rest, plus wall time | +| `foreground-instructions` | the benchmark thread's retired instructions; fails fast if unavailable | the rest, plus wall time | +| `wall-time` | wall time | nothing; no counters are opened | + +For a CI gate on frame cost prefer `foreground-instructions`: it excludes background parsing and GPU-driver submission threads, whose work varies run to run, and on Apple Silicon the process-wide count includes driver jitter that widens the interval (about ±0.6% versus ±0.01% for the foreground thread). + +Default secondaries, each skipped with a one-time note when unavailable: + +- **instructions** (process-wide) and **foreground instructions** (the benchmark thread only, where GPUI polls tasks, lays out, paints, and submits). Near-deterministic, so a ~0.1% spread is typical and small regressions are detectable. Blind to waits, memory stalls, and GPU shader work. Their ratio, **foreground share of instructions**, shows work moving between the foreground and background threads even when the total is flat; with a headless renderer it also exposes driver-thread submission work (about 7% of a small Metal frame). +- **cycles** and **IPC** (`instructions / cycles`). Falling IPC with flat instructions means worse cache or branch behavior, not more work. +- **branch misses** and **cache misses** (last-level), for diagnosing an IPC drop. Linux only. +- **foreground context switches**: voluntary switches on the benchmark thread per iteration, i.e. how many times a frame blocked on a lock, channel, or fence. This is the metric that catches the waits instructions cannot see. Linux only. +- **page faults**: minor faults per iteration, a privilege-free proxy for fresh memory touched. + +Run one benchmark process per machine at a time. Hardware counters are per task, so instruction counts stay correct next to a concurrent `cargo bench`, but wall time, IPC, and cache misses share the hardware and degrade, and two processes requesting counters can exceed a core's counter slots, at which point the kernel multiplexes and the counts become scaled estimates (a warning is printed once). Within one process, benchmarks already run sequentially; process-wide counters can still attribute a previous benchmark's trailing background work to the next benchmark's first sample, which the foreground-thread metrics are immune to. + +Platform support: on Linux, hardware counters come from `perf_event_open` and need `CAP_PERFMON` or `/proc/sys/kernel/perf_event_paranoid` at 2 or below; Docker's default seccomp profile and many cloud VMs block it, and the default mode then reports wall time plus the `getrusage` metrics. On Apple Silicon, instructions and cycles (both scopes) come from `proc_pid_rusage` and `thread_selfcounts` with no privileges; Intel Macs have no source. Criterion baselines are keyed by benchmark id, not measurement, so use distinct `--save-baseline` names when switching the primary on one machine. + +Never present instructions alone for a latency claim; pair them with wall time and the frame metrics from the same run. Other Criterion measurements plug in with `criterion_group!` and `config = Criterion::default().with_measurement(gpui::BenchMeasurement::new(primary).with_secondary("name", other))`. + +#### Benchmarks outside GPUI + +`bench_metrics` has no GPUI dependency, so any Criterion benchmark in the workspace (`rope`, `sum_tree`, `text`, ...) gets the same metrics and the same `BENCH_MEASUREMENT` switch. Add `bench_metrics.workspace = true` to `[dev-dependencies]`, take `&mut Criterion` in the benchmark functions, and register them with: + +```rust +criterion_group! { + name = benches; + config = Criterion::default() + .with_measurement(bench_metrics::BenchMeasurement::from_env_or_exit()); + targets = rope_benchmarks +} +``` + +That alone switches Criterion's analyzed metric with `BENCH_MEASUREMENT=instructions`. To also print the per-iteration secondaries, run each routine through a `MetricReport`, which counts iterations for you: + +```rust +let report = bench_metrics::MetricReport::new(); +group.bench_function("append", |b| report.iter(b, || rope.append(small.clone()))); +report.print(" "); +``` + +Criterion's `Throughput` lines keep working alongside. Page faults per iteration are a useful allocation-pressure signal for data-structure crates even without a counting allocator. + ### `bench_iter`: synchronous functions and compute work Use `bench_iter` for synchronous application or compute code: @@ -197,8 +249,9 @@ On macOS, the headless renderer uses Metal without showing a window. `bench_rend On platforms without a headless renderer, GPUI still performs CPU-side window work, but presenting discards the scene and does not measure real GPU submission. State this limitation in results. -Depending on the pinned revision, `BenchReport` can include: +`BenchReport` is printed once per benchmark, and once per input for `inputs = ...` benchmarks, labeled with the Criterion benchmark id so the two can be correlated. Depending on the pinned revision, it can include: +- secondary measurements per iteration (median, min, max): instructions, foreground instructions, cycles, IPC, branch and cache misses, foreground context switches, page faults; - dirty-to-draw duration; - draw and present intervals; - invalidations per frame; @@ -359,6 +412,7 @@ Before accepting a GPUI benchmark, verify: - [ ] Cache state is controlled and documented. - [ ] A responsiveness signal competes with the heavy work when foreground starvation matters. - [ ] Frame metrics and completion throughput are both reported. +- [ ] Instructions and IPC (Linux and Apple Silicon) and foreground context switches (Linux) are reported next to wall time, and none is presented alone as the whole result. - [ ] Final correctness, ordering, and work counts are asserted. - [ ] The workload completes under bounded smoke and measured runs. - [ ] The same benchmark code runs on baseline and candidate. diff --git a/.rules b/.rules index c0d62fcdbe6..4f07c39e70c 100644 --- a/.rules +++ b/.rules @@ -148,6 +148,7 @@ When an agent opens or updates a pull request, it must: - Use one bullet under `Release Notes:`: - `- Added ...`, `- Fixed ...`, or `- Improved ...` for user-facing changes, or - `- N/A` for docs-only and other non-user-facing changes. +- Changes to `gpui` (including its benchmark and test harnesses) must have a bullet prefixed with `[GPUI]` describing the improvement even when nothing is user-facing in Zed itself, since GPUI is consumed as a library (CI enforces this), for example `- [GPUI] Improved benchmarks to report retired CPU instructions alongside wall time on Linux`. - Format release notes exactly with a blank line after the heading, for example: ``` diff --git a/Cargo.lock b/Cargo.lock index 1ed2119bc9e..633872e6929 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -657,6 +657,15 @@ dependencies = [ "equator", ] +[[package]] +name = "alloca" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5a7d05ea6aea7e9e64d25b9156ba2fee3fdd659e34e41063cd2fc7cd020d7f4" +dependencies = [ + "cc", +] + [[package]] name = "allocator-api2" version = "0.2.21" @@ -2112,6 +2121,16 @@ version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3a8241f3ebb85c056b509d4327ad0358fbbba6ffb340bf388f26350aeda225b1" +[[package]] +name = "bench_metrics" +version = "0.1.0" +dependencies = [ + "anyhow", + "criterion", + "libc", + "perf-event2", +] + [[package]] name = "benchmarks" version = "0.1.0" @@ -2187,7 +2206,7 @@ dependencies = [ "bitflags 2.13.1", "cexpr", "clang-sys", - "itertools 0.11.0", + "itertools 0.13.0", "log", "prettyplease", "proc-macro2", @@ -2592,6 +2611,12 @@ dependencies = [ "pkg-config", ] +[[package]] +name = "c-enum" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd17eb909a8c6a894926bfcc3400a4bb0e732f5a57d37b1f14e8b29e329bace8" + [[package]] name = "cached" version = "0.56.0" @@ -4341,25 +4366,24 @@ dependencies = [ [[package]] name = "criterion" -version = "0.5.1" +version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2b12d017a929603d80db1831cd3a24082f8137ce19c69e6447f54f5fc8d692f" +checksum = "950046b2aa2492f9a536f5f4f9a3de7b9e2476e575e05bd6c333371add4d98f3" dependencies = [ + "alloca", "anes", "cast", "ciborium", "clap", "criterion-plot", - "is-terminal", - "itertools 0.10.5", + "itertools 0.13.0", "num-traits", - "once_cell", "oorandom", + "page_size", "plotters", "rayon", "regex", "serde", - "serde_derive", "serde_json", "tinytemplate", "walkdir", @@ -4367,12 +4391,12 @@ dependencies = [ [[package]] name = "criterion-plot" -version = "0.5.0" +version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6b50826342786a51a89e2da3a28f1c32b06e387201bc2d19791f622c673706b1" +checksum = "d8d80a2f4f5b554395e47b5d8305bc3d27813bacb73493eb1001e8f76dae29ea" dependencies = [ "cast", - "itertools 0.10.5", + "itertools 0.13.0", ] [[package]] @@ -7507,6 +7531,7 @@ dependencies = [ "async-channel 2.5.0", "async-task", "backtrace", + "bench_metrics", "bindgen", "bitflags 2.13.1", "chrono", @@ -9131,9 +9156,9 @@ dependencies = [ [[package]] name = "itertools" -version = "0.11.0" +version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b1c173a5686ce8bfa551b3563d0c2170bf24ca44da99c7ca4bfdab5418c3fe57" +checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186" dependencies = [ "either", ] @@ -12965,6 +12990,41 @@ dependencies = [ "serde_json", ] +[[package]] +name = "perf-event-data" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "575828d9d7d205188048eb1508560607a03d21eafdbba47b8cade1736c1c28e1" +dependencies = [ + "bitflags 2.13.1", + "c-enum", + "perf-event-open-sys2", +] + +[[package]] +name = "perf-event-open-sys2" +version = "5.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c25955321465255e437600b54296983fab1feac2cd0c38958adeb26dbae49e" +dependencies = [ + "libc", + "memoffset", +] + +[[package]] +name = "perf-event2" +version = "0.7.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0939b8fad77dfaeb29ebbd35faaeaadbf833167f30975f1b8993bbba09ea0a0f" +dependencies = [ + "bitflags 2.13.1", + "c-enum", + "libc", + "memmap2", + "perf-event-data", + "perf-event-open-sys2", +] + [[package]] name = "pest" version = "2.8.3" diff --git a/Cargo.toml b/Cargo.toml index 0e5908bbe45..41db7bfee5a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -20,6 +20,7 @@ members = [ "crates/auto_update_ui", "crates/aws_http_client", "crates/bedrock", + "crates/bench_metrics", "crates/benchmarks", "crates/breadcrumbs", "crates/buffer_diff", @@ -291,6 +292,7 @@ anthropic = { path = "crates/anthropic" } askpass = { path = "crates/askpass" } assets = { path = "crates/assets" } audio = { path = "crates/audio" } +bench_metrics = { path = "crates/bench_metrics" } auto_update = { path = "crates/auto_update" } auto_update_ui = { path = "crates/auto_update_ui" } aws_http_client = { path = "crates/aws_http_client" } @@ -597,7 +599,8 @@ core-text = "21" core-video = { version = "0.5.2", features = ["metal"] } cpal = "0.17" crash-handler = "0.8" -criterion = { version = "0.5", features = ["html_reports"] } +criterion = { version = "0.8.2", features = ["html_reports"] } +perf-event = { package = "perf-event2", version = "0.7.4" } crossbeam = "0.8.4" ctor = "1.0.12" dap-types = { git = "https://github.com/zed-industries/dap-types", rev = "1b461b310481d01e02b2603c16d7144b926339f8" } diff --git a/crates/bench_metrics/Cargo.toml b/crates/bench_metrics/Cargo.toml new file mode 100644 index 00000000000..8715ba0da5a --- /dev/null +++ b/crates/bench_metrics/Cargo.toml @@ -0,0 +1,24 @@ +[package] +name = "bench_metrics" +version = "0.1.0" +edition.workspace = true +publish = false +license = "Apache-2.0" + +[lints] +workspace = true + +[lib] +path = "src/bench_metrics.rs" +doctest = false + +[features] +test-support = [] + +[dependencies] +anyhow.workspace = true +criterion.workspace = true +libc.workspace = true + +[target.'cfg(target_os = "linux")'.dependencies] +perf-event.workspace = true diff --git a/crates/bench_metrics/LICENSE-APACHE b/crates/bench_metrics/LICENSE-APACHE new file mode 120000 index 00000000000..1cd601d0a3a --- /dev/null +++ b/crates/bench_metrics/LICENSE-APACHE @@ -0,0 +1 @@ +../../LICENSE-APACHE \ No newline at end of file diff --git a/crates/bench_metrics/src/bench_metrics.rs b/crates/bench_metrics/src/bench_metrics.rs new file mode 100644 index 00000000000..3d157b629d3 --- /dev/null +++ b/crates/bench_metrics/src/bench_metrics.rs @@ -0,0 +1,1801 @@ +//! Criterion measurements beyond wall time, and a way to report several of +//! them from one benchmark run. +//! +//! Criterion analyzes exactly one scalar per benchmark. [`BenchMeasurement`] +//! wraps that *primary* measurement together with any number of *secondary* +//! measurements taken over the same iterations, whose per-iteration values a +//! benchmark harness records into a [`MetricReport`]. The measurements here +//! are process- or thread-scoped counters: Linux hardware performance +//! counters ([`HardwareCounter`]) and `getrusage` statistics +//! ([`ResourceCounter`]). Nothing in this crate depends on GPUI, so any +//! Criterion benchmark can use it: +//! +//! ```ignore +//! fn benches(c: &mut Criterion) { +//! let report = bench_metrics::MetricReport::new(); +//! c.bench_function("append", |b| report.iter(b, || work())); +//! report.print(" "); +//! } +//! +//! criterion_group! { +//! name = group; +//! config = Criterion::default() +//! .with_measurement(bench_metrics::BenchMeasurement::from_env_or_exit()); +//! targets = benches +//! } +//! criterion_main!(group); +//! ``` + +use std::{ + cell::{Cell, RefCell}, + rc::Rc, +}; + +use anyhow::{Result, anyhow}; +use criterion::measurement::{Measurement, ValueFormatter}; + +/// The environment variable [`BenchMeasurement::from_env`] reads. +pub const BENCH_MEASUREMENT_ENV_VAR: &str = "BENCH_MEASUREMENT"; + +/// Which threads a counter covers. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum MetricScope { + /// Every thread in the benchmark process, including threads created + /// after the counter was opened. + Process, + /// Only the thread that constructed the counter. In a Criterion benchmark + /// that is the thread running the benchmark routine; in GPUI benchmarks it + /// is also the foreground thread where task polls, layout, and paint run. + CallingThread, +} + +impl MetricScope { + fn label(self) -> &'static str { + match self { + Self::Process => "the benchmark process", + Self::CallingThread => "the calling thread", + } + } +} + +/// Formats event counts with K/M/G prefixes and a unit noun. +#[derive(Clone, Copy)] +pub struct CountFormatter { + unit: &'static str, +} + +impl CountFormatter { + /// Creates a formatter whose values are counts of `unit`, e.g. `"cycles"`. + pub const fn new(unit: &'static str) -> Self { + Self { unit } + } +} + +impl ValueFormatter for CountFormatter { + fn scale_values(&self, typical_value: f64, values: &mut [f64]) -> &'static str { + let (scale, prefix) = if typical_value < 1_000.0 { + (1.0, "") + } else if typical_value < 1_000_000.0 { + (1_000.0, "K ") + } else if typical_value < 1_000_000_000.0 { + (1_000_000.0, "M ") + } else { + (1_000_000_000.0, "G ") + }; + for value in values { + *value /= scale; + } + // `&'static str` is required, so the prefixed unit is interned once. + intern(prefix, self.unit) + } + + fn scale_throughputs( + &self, + _typical_value: f64, + throughput: &criterion::Throughput, + values: &mut [f64], + ) -> &'static str { + let (units, per) = match throughput { + criterion::Throughput::Bits(units) => (*units, "/bit"), + criterion::Throughput::Bytes(units) | criterion::Throughput::BytesDecimal(units) => { + (*units, "/byte") + } + criterion::Throughput::Elements(units) + | criterion::Throughput::ElementsAndBytes { + elements: units, .. + } => (*units, "/element"), + }; + for value in values { + *value /= units as f64; + } + intern(self.unit, per) + } + + fn scale_for_machines(&self, _values: &mut [f64]) -> &'static str { + self.unit + } +} + +/// Returns a `'static` concatenation of two strings, leaking each distinct +/// combination once. Criterion's `ValueFormatter` requires `&'static str` +/// units, and the set of combinations is a handful of fixed labels. +fn intern(first: &str, second: &str) -> &'static str { + use std::sync::{Mutex, OnceLock}; + + static INTERNED: OnceLock>> = OnceLock::new(); + let combined = format!("{first}{second}"); + let mut interned = INTERNED + .get_or_init(Default::default) + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + if let Some(existing) = interned.iter().find(|existing| **existing == combined) { + return existing; + } + let leaked: &'static str = Box::leak(combined.into_boxed_str()); + interned.push(leaked); + leaked +} + +// --------------------------------------------------------------------------- +// Hardware performance counters +// --------------------------------------------------------------------------- + +/// A CPU event counted by [`HardwareCounter`]. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum HardwareEvent { + /// Retired userspace instructions: near-deterministic, so small + /// regressions are detectable, but blind to waits and memory stalls. + Instructions, + /// Userspace CPU cycles. `instructions / cycles` is IPC; a falling IPC with + /// flat instructions means worse cache or branch behavior. + Cycles, + /// Mispredicted branches. Linux only. + BranchMisses, + /// Last-level cache misses, i.e. accesses served from memory. Linux only. + CacheMisses, + /// Last-level cache accesses. Linux only. + CacheReferences, +} + +impl HardwareEvent { + /// The unit noun used in reports. + pub fn unit(self) -> &'static str { + match self { + Self::Instructions => "instructions", + Self::Cycles => "cycles", + Self::BranchMisses => "branch misses", + Self::CacheMisses => "cache misses", + Self::CacheReferences => "cache references", + } + } + + /// The event's `perf` name, also used in error messages on every platform. + fn name(self) -> &'static str { + match self { + Self::Instructions => "instructions", + Self::Cycles => "cpu-cycles", + Self::BranchMisses => "branch-misses", + Self::CacheMisses => "cache-misses", + Self::CacheReferences => "cache-references", + } + } +} + +/// Criterion measurement for a CPU hardware event. +/// +/// The counter is opened once, at construction; each `Measurement::start`/`end` +/// reads it and reports the difference, so a sample costs one read per counter +/// (tens to hundreds of nanoseconds). Which events and scopes are available +/// depends on the platform backend: +/// +/// * **Linux** (`perf_event_open`): every event, both scopes. Needs +/// `CAP_PERFMON` or `perf_event_paranoid <= 2`. See [`perf`] for how +/// process scope, hybrid CPUs, and multiplexing are handled. +/// * **macOS** (`proc_pid_rusage` and `thread_selfcounts`): instructions and +/// cycles, both scopes, on Apple Silicon without privileges. Intel Macs and +/// the cache/branch events are unsupported. +/// +/// Unsupported combinations fail at construction with an actionable error, so +/// callers can skip them (as [`BenchMeasurement::from_env`] does) rather than +/// measure zeros. +pub struct HardwareCounter { + event: HardwareEvent, + scope: MetricScope, + backend: backend::Counter, + formatter: CountFormatter, +} + +impl HardwareCounter { + /// Opens a counter for `event` over `scope`. + pub fn new(event: HardwareEvent, scope: MetricScope) -> Result { + let backend = backend::Counter::open(event, scope)?; + let this = Self { + event, + scope, + backend, + formatter: CountFormatter::new(event.unit()), + }; + this.verify_counting()?; + Ok(this) + } + + /// The counted event. + pub fn event(&self) -> HardwareEvent { + self.event + } + + /// The covered threads. + pub fn scope(&self) -> MetricScope { + self.scope + } + + /// Confirms the counter advances for work on this thread. A virtual machine + /// can accept the open yet never schedule the event, and Intel Macs report + /// zero; failing here is clearer than measuring zeros. + fn verify_counting(&self) -> Result<()> { + let before = self.backend.snapshot(); + std::hint::black_box((0..10_000_u64).fold(0_u64, |total, value| total.wrapping_add(value))); + let after = self.backend.snapshot(); + if self.backend.advanced(&before, &after) { + Ok(()) + } else { + Err(anyhow!( + "the {} counter for {} opened but does not advance; hardware performance \ + counters are unavailable to this machine or CI runner", + self.event.name(), + self.scope.label() + )) + } + } +} + +impl Measurement for HardwareCounter { + type Intermediate = backend::Snapshot; + type Value = f64; + + fn start(&self) -> Self::Intermediate { + self.backend.snapshot() + } + + fn end(&self, start: Self::Intermediate) -> Self::Value { + let end = self.backend.snapshot(); + self.backend.delta(&start, &end) + } + + fn add(&self, first: &Self::Value, second: &Self::Value) -> Self::Value { + first + second + } + + fn zero(&self) -> Self::Value { + 0.0 + } + + fn to_f64(&self, value: &Self::Value) -> f64 { + *value + } + + fn formatter(&self) -> &dyn ValueFormatter { + &self.formatter + } +} + +#[cfg(target_os = "macos")] +use darwin as backend; +/// The platform backend behind [`HardwareCounter`]. Each provides a `Counter` +/// with `open`, `snapshot`, `advanced`, and `delta`, and a `Snapshot` type. +#[cfg(target_os = "linux")] +use perf as backend; +#[cfg(not(any(target_os = "linux", target_os = "macos")))] +use unsupported as backend; + +/// Linux backend: `perf_event_open` through the `perf-event2` crate, whose +/// items are referenced through the `perf_event` path so this module's own +/// helpers are distinguishable from the crate's. +/// +/// With [`MetricScope::Process`], one inherited counter is opened for every +/// thread that exists at construction. Inheritance extends each counter to +/// the threads that thread later creates, and live descendants are summed on +/// read, so the whole process is covered for the counter's lifetime. With +/// [`MetricScope::CallingThread`], a single non-inherited counter observes the +/// constructing thread only. +/// +/// Hybrid CPUs (separate performance and efficiency core PMUs) need one event +/// per PMU, of which only the one matching the CPU a thread runs on can be +/// scheduled; each PMU's encoding is read from sysfs. If the kernel +/// multiplexes a conventional counter with other profiling, its count is +/// scaled by `time_enabled / time_running` and a warning is printed once. +#[cfg(target_os = "linux")] +pub mod perf { + use super::*; + + pub(super) struct Counter { + event: HardwareEvent, + counters: RefCell>, + multiplexing_reported: Cell, + } + + struct PerfCounter { + counter: perf_event::Counter, + /// Generic events on conventional CPUs are scaled when multiplexed. + /// Hybrid per-PMU events are not, since a thread on the other core type + /// shows as "not running" for that PMU by design. + scale_for_multiplexing: bool, + } + + /// Every counter's state at one instant. + #[doc(hidden)] + pub struct Snapshot(Vec); + + #[derive(Clone, Copy)] + struct CounterState { + count: u64, + time_enabled: std::time::Duration, + time_running: std::time::Duration, + } + + impl Counter { + pub(super) fn open(event: HardwareEvent, scope: MetricScope) -> Result { + let hybrid_events = perf::hybrid_pmu_events(event)?; + let mut counters = Vec::new(); + match scope { + MetricScope::CallingThread => { + Self::open_target( + event, + CounterTarget::CallingThread, + &hybrid_events, + &mut counters, + )?; + } + MetricScope::Process => { + for thread_id in perf::process_thread_ids()? { + Self::open_target( + event, + CounterTarget::ThreadAndDescendants(thread_id), + &hybrid_events, + &mut counters, + )?; + } + } + } + if counters.is_empty() { + return Err(anyhow!( + "no Linux {} counter could be opened for {}: the CPU does not expose this event", + event.name(), + scope.label() + )); + } + for counter in &mut counters { + counter + .counter + .enable() + .map_err(|error| Self::operation_error(event, "enable", error))?; + } + Ok(Self { + event, + counters: RefCell::new(counters), + multiplexing_reported: Cell::new(false), + }) + } + + fn open_target( + event: HardwareEvent, + target: CounterTarget, + hybrid_events: &[(u32, u64)], + counters: &mut Vec, + ) -> Result<()> { + let opened: Vec<(std::io::Result, bool)> = + if hybrid_events.is_empty() { + let mut builder = perf_event::Builder::new(generic_event(event)); + target.configure(&mut builder); + vec![(builder.build(), true)] + } else { + hybrid_events + .iter() + .map(|&(pmu_type, config)| { + let mut builder = + perf_event::Builder::new(perf_event::events::Raw::new(config)); + builder.attrs_mut().type_ = pmu_type; + target.configure(&mut builder); + (builder.build(), false) + }) + .collect() + }; + for (result, scale_for_multiplexing) in opened { + match result { + Ok(counter) => counters.push(PerfCounter { + counter, + scale_for_multiplexing, + }), + // The thread exited between enumeration and open. + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => return Err(Self::open_error(event, target, error)), + } + } + Ok(()) + } + + fn open_error( + event: HardwareEvent, + target: CounterTarget, + error: std::io::Error, + ) -> anyhow::Error { + anyhow!( + "failed to open Linux {} counter for {target}: {error}. Grant this benchmark \ + CAP_PERFMON or adjust /proc/sys/kernel/perf_event_paranoid according to your CI \ + security policy", + event.name() + ) + } + + fn operation_error( + event: HardwareEvent, + operation: &str, + error: std::io::Error, + ) -> anyhow::Error { + anyhow!("Linux {} counter {operation} failed: {error}", event.name()) + } + + pub(super) fn snapshot(&self) -> Snapshot { + Snapshot( + self.counters + .borrow_mut() + .iter_mut() + .map(|counter| { + let data = counter.counter.read_full().unwrap_or_else(|error| { + panic!("{}", Self::operation_error(self.event, "read", error)) + }); + CounterState { + count: data.count(), + time_enabled: data + .time_enabled() + .expect("time-enabled counter data was requested"), + time_running: data + .time_running() + .expect("time-running counter data was requested"), + } + }) + .collect(), + ) + } + + pub(super) fn advanced(&self, before: &Snapshot, after: &Snapshot) -> bool { + before + .0 + .iter() + .zip(&after.0) + .any(|(before, after)| after.time_running > before.time_running) + } + + pub(super) fn delta(&self, start: &Snapshot, end: &Snapshot) -> f64 { + let counters = self.counters.borrow(); + let mut total = 0.0; + let mut any_counter_ran = false; + for ((counter, start), end) in counters.iter().zip(&start.0).zip(&end.0) { + let count = end.count.saturating_sub(start.count) as f64; + let time_enabled = end.time_enabled.saturating_sub(start.time_enabled); + let time_running = end.time_running.saturating_sub(start.time_running); + if time_running.is_zero() { + // Process-wide coverage includes threads that slept for the + // whole interval, and hybrid PMUs whose core type this thread + // never ran on. + continue; + } + any_counter_ran = true; + if counter.scale_for_multiplexing && time_running < time_enabled { + if !self.multiplexing_reported.replace(true) { + eprintln!( + "{} counters were multiplexed by the kernel; counts are scaled using \ + time_enabled/time_running", + self.event.name() + ); + } + total += count * time_enabled.as_secs_f64() / time_running.as_secs_f64(); + } else { + total += count; + } + } + if !any_counter_ran { + panic!( + "Linux {} counters did not run during the measured interval; ensure hardware \ + performance counters are available to this machine or CI runner", + self.event.name() + ); + } + total + } + } + + fn generic_event(event: HardwareEvent) -> perf_event::events::Hardware { + use perf_event::events::Hardware; + match event { + HardwareEvent::Instructions => Hardware::INSTRUCTIONS, + HardwareEvent::Cycles => Hardware::CPU_CYCLES, + HardwareEvent::BranchMisses => Hardware::BRANCH_MISSES, + HardwareEvent::CacheMisses => Hardware::CACHE_MISSES, + HardwareEvent::CacheReferences => Hardware::CACHE_REFERENCES, + } + } + + #[derive(Clone, Copy)] + enum CounterTarget { + /// `pid = 0` to `perf_event_open`: the calling thread, with no inheritance. + CallingThread, + /// A specific thread plus every thread it creates afterwards. + ThreadAndDescendants(i32), + } + + impl CounterTarget { + fn configure(self, builder: &mut perf_event::Builder) { + use perf_event::ReadFormat; + + match self { + Self::CallingThread => builder.observe_self().inherit(false), + Self::ThreadAndDescendants(thread_id) => { + builder.observe_pid(thread_id).inherit(true) + } + }; + builder.read_format(ReadFormat::TOTAL_TIME_ENABLED | ReadFormat::TOTAL_TIME_RUNNING); + } + } + + impl std::fmt::Display for CounterTarget { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::CallingThread => write!(f, "the calling thread"), + Self::ThreadAndDescendants(thread_id) => write!(f, "thread {thread_id}"), + } + } + } + + fn process_thread_ids() -> Result> { + let mut thread_ids = Vec::new(); + for entry in std::fs::read_dir("/proc/self/task") + .map_err(|error| anyhow!("failed to enumerate benchmark process threads: {error}"))? + { + let entry = entry.map_err(|error| { + anyhow!("failed to enumerate a benchmark process thread: {error}") + })?; + if let Some(thread_id) = entry + .file_name() + .to_str() + .and_then(|thread_id| thread_id.parse().ok()) + { + thread_ids.push(thread_id); + } + } + Ok(thread_ids) + } + + /// Returns `(pmu_type, config)` for `event` on each hybrid CPU PMU, or an + /// empty list on conventional CPUs where the generic hardware event suffices. + /// + /// A PMU (performance monitoring unit) is the per-core block of counter + /// registers; Linux exposes each as a device under + /// `/sys/bus/event_source/devices/`. Conventional CPUs have one, `cpu`, and + /// the generic events target it. Hybrid CPUs have `cpu_core` (performance + /// cores) and `cpu_atom` (efficiency cores) with distinct event encodings, + /// and the kernel refuses the generic event for a task that may run on + /// either, so one raw event per PMU is needed. + /// + /// Each PMU publishes its events as `term=value,...` strings and the bit + /// layout of each term under `format/`, e.g. `cache-misses` is + /// `event=0x2e,umask=0x41` with `event` at `config:0-7` and `umask` at + /// `config:8-15`. + fn hybrid_pmu_events(event: HardwareEvent) -> Result> { + use anyhow::Context as _; + + let mut events = Vec::new(); + for pmu in ["cpu_core", "cpu_atom"] { + let pmu_path = std::path::Path::new("/sys/bus/event_source/devices").join(pmu); + if !pmu_path.exists() { + continue; + } + let pmu_type = std::fs::read_to_string(pmu_path.join("type")) + .with_context(|| format!("failed to read Linux {pmu} performance-counter type"))? + .trim() + .parse::() + .with_context(|| format!("invalid Linux {pmu} performance-counter type"))?; + let encoding = std::fs::read_to_string(pmu_path.join("events").join(event.name())) + .with_context(|| format!("failed to read Linux {pmu} {} event", event.name()))?; + let mut config = 0_u64; + for term in encoding.trim().split(',') { + let (name, value) = term.split_once('=').unwrap_or((term, "1")); + let value = value + .strip_prefix("0x") + .map_or_else(|| value.parse::(), |hex| u64::from_str_radix(hex, 16)) + .with_context(|| { + format!("invalid Linux {pmu} {} event term {term:?}", event.name()) + })?; + let layout = std::fs::read_to_string(pmu_path.join("format").join(name)) + .with_context(|| format!("failed to read Linux {pmu} format for {name:?}"))?; + let bits = layout.trim().strip_prefix("config:").ok_or_else(|| { + anyhow!( + "unsupported Linux {pmu} format {:?} for {name:?}", + layout.trim() + ) + })?; + let (low, high) = match bits.split_once('-') { + Some((low, high)) => (low.parse::()?, high.parse::()?), + None => { + let bit = bits.parse::()?; + (bit, bit) + } + }; + let width = high - low + 1; + let mask = if width >= 64 { + u64::MAX + } else { + (1_u64 << width) - 1 + }; + config |= (value & mask) << low; + } + events.push((pmu_type, config)); + } + Ok(events) + } + + #[cfg(test)] + mod tests { + use super::*; + + #[test] + fn perf_permission_error_is_actionable() { + let error = Counter::open_error( + HardwareEvent::Instructions, + CounterTarget::ThreadAndDescendants(42), + std::io::Error::from(std::io::ErrorKind::PermissionDenied), + ); + let message = error.to_string(); + assert!(message.contains("CAP_PERFMON")); + assert!(message.contains("perf_event_paranoid")); + assert!(message.contains("thread 42")); + } + } +} + +/// macOS backend. Apple Silicon exposes retired instructions and cycles to +/// unprivileged processes through two XNU interfaces: `proc_pid_rusage` with +/// `RUSAGE_INFO_V4` for the whole process (every thread, including exited +/// ones), and `thread_selfcounts` for the calling thread. Both are plain +/// reads with no counter to open or close. Intel Macs report zero for both, +/// which [`HardwareCounter::new`] detects, and macOS virtual machines (CI +/// runners included) reject `thread_selfcounts` with `ENOTSUP` while still +/// serving `proc_pid_rusage`. Branch and cache events require the private +/// `kperf` framework and root, so they are unsupported here. +#[cfg(target_os = "macos")] +pub mod darwin { + use super::*; + + pub(super) struct Counter { + event: HardwareEvent, + scope: MetricScope, + } + + /// `[instructions, cycles]` at one instant. + #[doc(hidden)] + pub struct Snapshot([u64; 2]); + + /// The leading fields of `struct rusage_info_v4` from ``, + /// up to and including the two counters this backend reads. Only a + /// prefix is declared: the kernel copies out `sizeof` the flavor's full + /// struct, so the buffer below is padded to that size. + #[repr(C)] + struct RusageInfoV4Prefix { + ri_uuid: [u8; 16], + ri_user_time: u64, + ri_system_time: u64, + ri_pkg_idle_wkups: u64, + ri_interrupt_wkups: u64, + ri_pageins: u64, + ri_wired_size: u64, + ri_resident_size: u64, + ri_phys_footprint: u64, + ri_proc_start_abstime: u64, + ri_proc_exit_abstime: u64, + ri_child_user_time: u64, + ri_child_system_time: u64, + ri_child_pkg_idle_wkups: u64, + ri_child_interrupt_wkups: u64, + ri_child_pageins: u64, + ri_child_elapsed_abstime: u64, + ri_diskio_bytesread: u64, + ri_diskio_byteswritten: u64, + ri_cpu_time_qos_default: u64, + ri_cpu_time_qos_maintenance: u64, + ri_cpu_time_qos_background: u64, + ri_cpu_time_qos_utility: u64, + ri_cpu_time_qos_legacy: u64, + ri_cpu_time_qos_user_initiated: u64, + ri_cpu_time_qos_user_interactive: u64, + ri_billed_system_time: u64, + ri_serviced_system_time: u64, + ri_logical_writes: u64, + ri_lifetime_max_phys_footprint: u64, + ri_instructions: u64, + ri_cycles: u64, + } + + /// `sizeof(struct rusage_info_v4)`: the prefix above plus the four trailing + /// `u64` fields (`ri_billed_energy`, `ri_serviced_energy`, + /// `ri_interval_max_phys_footprint`, `ri_runnable_time`). + const RUSAGE_INFO_V4_SIZE: usize = std::mem::size_of::() + 4 * 8; + const RUSAGE_INFO_V4: libc::c_int = 4; + /// `THSC_CPI` in `` (not in `libc`). + const THREAD_SELFCOUNTS_CPI: libc::c_int = 1; + + #[repr(C, align(8))] + struct RusageInfoV4Buffer([u8; RUSAGE_INFO_V4_SIZE]); + + unsafe extern "C" { + fn proc_pid_rusage( + pid: libc::c_int, + flavor: libc::c_int, + buffer: *mut RusageInfoV4Buffer, + ) -> libc::c_int; + fn thread_selfcounts( + kind: libc::c_int, + buffer: *mut u64, + size: libc::size_t, + ) -> libc::c_int; + } + + impl Counter { + pub(super) fn open(event: HardwareEvent, scope: MetricScope) -> Result { + if !matches!(event, HardwareEvent::Instructions | HardwareEvent::Cycles) { + return Err(anyhow!( + "macOS exposes only instructions and cycles without root; {} needs kperf", + event.name() + )); + } + let this = Self { event, scope }; + this.read().map_err(|error| { + anyhow!( + "{error}; hardware counters are unavailable for {} on this Mac (Intel Macs and \ + macOS virtual machines such as CI runners do not expose them)", + scope.label() + ) + })?; + Ok(this) + } + + fn read(&self) -> Result<[u64; 2]> { + match self.scope { + MetricScope::Process => { + let mut buffer = RusageInfoV4Buffer([0; RUSAGE_INFO_V4_SIZE]); + // SAFETY: the buffer is the flavor's full size and aligned + // for its `u64` fields; the kernel fills it on success. + let status = unsafe { + proc_pid_rusage( + std::process::id() as libc::c_int, + RUSAGE_INFO_V4, + &mut buffer, + ) + }; + if status != 0 { + return Err(anyhow!( + "proc_pid_rusage failed: {}", + std::io::Error::last_os_error() + )); + } + // SAFETY: the buffer is at least as large and as aligned as + // the prefix struct, and every field is a plain integer. + let info = unsafe { &*(buffer.0.as_ptr() as *const RusageInfoV4Prefix) }; + Ok([info.ri_instructions, info.ri_cycles]) + } + MetricScope::CallingThread => { + let mut counts = [0_u64; 2]; + // SAFETY: `counts` holds the two `u64`s the CPI flavor writes. + let status = unsafe { + thread_selfcounts( + THREAD_SELFCOUNTS_CPI, + counts.as_mut_ptr(), + std::mem::size_of_val(&counts), + ) + }; + if status != 0 { + return Err(anyhow!( + "thread_selfcounts failed: {}", + std::io::Error::last_os_error() + )); + } + Ok(counts) + } + } + } + + fn value(&self, snapshot: &Snapshot) -> u64 { + match self.event { + HardwareEvent::Instructions => snapshot.0[0], + _ => snapshot.0[1], + } + } + + pub(super) fn snapshot(&self) -> Snapshot { + Snapshot(self.read().unwrap_or_else(|error| panic!("{error}"))) + } + + pub(super) fn advanced(&self, before: &Snapshot, after: &Snapshot) -> bool { + self.value(after) > self.value(before) + } + + pub(super) fn delta(&self, start: &Snapshot, end: &Snapshot) -> f64 { + self.value(end).saturating_sub(self.value(start)) as f64 + } + } +} + +/// Fallback backend for platforms without a hardware-counter source. +#[cfg(not(any(target_os = "linux", target_os = "macos")))] +pub mod unsupported { + use super::*; + + pub(super) struct Counter; + + #[doc(hidden)] + pub struct Snapshot; + + impl Counter { + pub(super) fn open(event: HardwareEvent, scope: MetricScope) -> Result { + Err(anyhow!( + "hardware counters ({} for {}) are unavailable on this platform", + event.name(), + scope.label() + )) + } + + pub(super) fn snapshot(&self) -> Snapshot { + Snapshot + } + + pub(super) fn advanced(&self, _: &Snapshot, _: &Snapshot) -> bool { + false + } + + pub(super) fn delta(&self, _: &Snapshot, _: &Snapshot) -> f64 { + 0.0 + } + } +} + +// --------------------------------------------------------------------------- +// getrusage statistics +// --------------------------------------------------------------------------- + +/// A statistic reported by `getrusage`, counted by [`ResourceCounter`]. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum ResourceMetric { + /// Times the thread blocked and gave up the CPU: lock waits, channel + /// receives, sleeps, and I/O. Preemptions are not included, since they + /// reflect the machine rather than the code. + VoluntaryContextSwitches, + /// Page faults served without disk I/O: first touches of freshly mapped + /// memory, a privilege-free proxy for allocation pressure. + MinorPageFaults, +} + +impl ResourceMetric { + /// The unit noun used in reports. + pub fn unit(self) -> &'static str { + match self { + Self::VoluntaryContextSwitches => "context switches", + Self::MinorPageFaults => "page faults", + } + } + + #[cfg(unix)] + fn read(self, usage: &libc::rusage) -> u64 { + let value = match self { + Self::VoluntaryContextSwitches => usage.ru_nvcsw, + Self::MinorPageFaults => usage.ru_minflt, + }; + value.max(0) as u64 + } +} + +/// Criterion measurement for a `getrusage` statistic. +/// +/// Unlike hardware counters this needs no privileges and works on every Unix. +/// [`MetricScope::CallingThread`] uses `RUSAGE_THREAD`, which only Linux +/// provides; [`MetricScope::Process`] uses `RUSAGE_SELF`, which sums every +/// thread in the process. +pub struct ResourceCounter { + metric: ResourceMetric, + scope: MetricScope, + formatter: CountFormatter, +} + +impl ResourceCounter { + /// Creates a counter for `metric` over `scope`. + pub fn new(metric: ResourceMetric, scope: MetricScope) -> Result { + let this = Self { + metric, + scope, + formatter: CountFormatter::new(metric.unit()), + }; + this.read()?; + Ok(this) + } + + /// The counted statistic. + pub fn metric(&self) -> ResourceMetric { + self.metric + } + + /// The covered threads. + pub fn scope(&self) -> MetricScope { + self.scope + } + + #[cfg(unix)] + fn read(&self) -> Result { + let who = match self.scope { + MetricScope::Process => libc::RUSAGE_SELF, + #[cfg(target_os = "linux")] + MetricScope::CallingThread => libc::RUSAGE_THREAD, + #[cfg(not(target_os = "linux"))] + MetricScope::CallingThread => { + return Err(anyhow!( + "getrusage statistics for {} need RUSAGE_THREAD, which only Linux provides", + self.scope.label() + )); + } + }; + // SAFETY: `rusage` is plain old data that `getrusage` fully initializes + // on success, and `who` is one of the constants the call accepts. + let mut usage: libc::rusage = unsafe { std::mem::zeroed() }; + let status = unsafe { libc::getrusage(who, &mut usage) }; + if status != 0 { + return Err(anyhow!( + "getrusage failed: {}", + std::io::Error::last_os_error() + )); + } + Ok(self.metric.read(&usage)) + } + + #[cfg(not(unix))] + fn read(&self) -> Result { + Err(anyhow!( + "getrusage statistics for {} are unavailable on this platform", + self.scope.label() + )) + } +} + +impl Measurement for ResourceCounter { + type Intermediate = u64; + type Value = f64; + + fn start(&self) -> Self::Intermediate { + self.read().unwrap_or_else(|error| panic!("{error}")) + } + + fn end(&self, start: Self::Intermediate) -> Self::Value { + let end = self.read().unwrap_or_else(|error| panic!("{error}")); + end.saturating_sub(start) as f64 + } + + fn add(&self, first: &Self::Value, second: &Self::Value) -> Self::Value { + first + second + } + + fn zero(&self) -> Self::Value { + 0.0 + } + + fn to_f64(&self, value: &Self::Value) -> f64 { + *value + } + + fn formatter(&self) -> &dyn ValueFormatter { + &self.formatter + } +} + +// --------------------------------------------------------------------------- +// Composite measurement +// --------------------------------------------------------------------------- + +/// A Criterion measurement that records several metrics per sample. +/// +/// Criterion analyzes one scalar per benchmark, fixed in the `Criterion` +/// type. This type erases the concrete [`Measurement`] Criterion analyzes (the +/// *primary*) so benchmark code needs no type parameter, and takes any number +/// of *secondary* measurements over the same iterations. A harness records +/// their per-iteration values into a [`MetricReport`]; see +/// [`MetricReport::iter`] for plain Criterion benchmarks. +/// +/// Every measurement already reduces to `f64` for Criterion's statistics, so +/// the erased value is that `f64` and each measurement's own formatter still +/// labels it. +pub struct BenchMeasurement { + primary: Box, + secondaries: Rc, +} + +/// The name under which the primary's per-sample total is available to +/// [`BenchMeasurement::with_ratio`] until [`BenchMeasurement::named`] renames it. +pub const PRIMARY_METRIC_NAME: &str = "primary"; + +impl BenchMeasurement { + /// Wraps the Criterion measurement whose values Criterion analyzes. + pub fn new(primary: M) -> Self + where + M: Measurement + 'static, + M::Intermediate: 'static, + { + Self { + primary: Box::new(ErasedMeasurementCell::new(primary)), + secondaries: Rc::new(SecondaryMeasurements { + primary_name: PRIMARY_METRIC_NAME, + ..Default::default() + }), + } + } + + /// Names the primary so ratios can refer to it, e.g. `named("cycles")` + /// then `with_ratio("IPC", "instructions", "cycles")`. The primary is not + /// itself reported as a secondary; Criterion already reports it. + pub fn named(mut self, name: &'static str) -> Self { + self.secondaries_mut().primary_name = name; + self + } + + /// Adds a measurement taken alongside the primary on every sample. + /// + /// `name` labels the metric in reports, e.g. `"instructions"`. + /// Secondaries start before and end after the primary, so their own + /// readout stays outside the primary's measured interval. + pub fn with_secondary(mut self, name: &'static str, measurement: M) -> Self + where + M: Measurement + 'static, + M::Intermediate: 'static, + { + self.secondaries_mut() + .metrics + .push(Rc::new(SecondaryMetric { + name, + measurement: Box::new(ErasedMeasurementCell::new(measurement)), + total: Cell::new(0.0), + })); + self + } + + /// Adds `name` to reports as the ratio of two secondaries' totals over + /// each sample, e.g. `with_ratio("IPC", "instructions", "cycles")`. + /// Unknown names are reported as unavailable rather than failing, so a + /// ratio can be declared even when one input was skipped on this machine. + pub fn with_ratio( + mut self, + name: &'static str, + numerator: &'static str, + denominator: &'static str, + ) -> Self { + self.secondaries_mut().ratios.push(RatioMetric { + name, + numerator, + denominator, + }); + self + } + + fn secondaries_mut(&mut self) -> &mut SecondaryMeasurements { + Rc::get_mut(&mut self.secondaries) + .expect("secondaries are only shared once measurement starts") + } + + /// Returns the measurement configured by [`BENCH_MEASUREMENT_ENV_VAR`]. + /// + /// * unset: Criterion analyzes wall time; every counter this machine + /// supports is a secondary: process and calling-thread instructions, + /// cycles, and their ratios where [`HardwareCounter`] has a backend + /// (Linux, Apple Silicon); branch and cache misses on Linux; voluntary + /// context switches and minor page faults from `getrusage`. Counters + /// that fail to open are skipped with a note printed once. + /// * `wall-time`: wall time only, no counters. + /// * `instructions`: Criterion analyzes process-wide retired instructions. + /// * `foreground-instructions`: Criterion analyzes the benchmark thread's + /// retired instructions, which excludes background and driver threads + /// and is the tighter regression gate for frame cost. + /// + /// In the instruction modes wall time becomes a secondary, and construction + /// fails when counters are unavailable so CI does not silently measure + /// something else. + pub fn from_env() -> Result { + match std::env::var(BENCH_MEASUREMENT_ENV_VAR).as_deref() { + Err(std::env::VarError::NotPresent) => { + let measurement = Self::new(criterion::measurement::WallTime); + Ok(measurement.with_default_secondaries(None)) + } + Ok("wall-time") => Ok(Self::new(criterion::measurement::WallTime)), + Ok("instructions") => Self::with_instructions_primary(MetricScope::Process), + Ok("foreground-instructions") => { + Self::with_instructions_primary(MetricScope::CallingThread) + } + Ok(value) => Err(anyhow!( + "unsupported {BENCH_MEASUREMENT_ENV_VAR} value {value:?}; expected \"wall-time\", \ + \"instructions\", or \"foreground-instructions\"" + )), + Err(error) => Err(anyhow!( + "{BENCH_MEASUREMENT_ENV_VAR} is not valid Unicode: {error}" + )), + } + } + + /// [`Self::from_env`], exiting the process with an actionable message when + /// the requested measurement can't be used on this machine. For use in + /// `criterion_group!` `config` expressions, where there is nothing to + /// propagate an error to. + pub fn from_env_or_exit() -> Self { + Self::from_env().unwrap_or_else(|error| { + eprintln!("failed to select benchmark measurement: {error:#}"); + std::process::exit(2); + }) + } + + fn with_instructions_primary(scope: MetricScope) -> Result { + let instructions = HardwareCounter::new(HardwareEvent::Instructions, scope)?; + let name = match scope { + MetricScope::Process => "instructions", + MetricScope::CallingThread => "foreground instructions", + }; + Ok(Self::new(instructions) + .named(name) + .with_default_secondaries(Some(scope)) + .with_secondary("wall time", criterion::measurement::WallTime)) + } + + /// Adds every counter this machine supports, leaving out the instruction + /// count for `primary_instructions` when that scope is already the + /// primary. + fn with_default_secondaries(mut self, primary_instructions: Option) -> Self { + // Every hardware counter needs the same access, so a denial of the + // first one is reported once for all of them rather than once per + // event. Events a platform lacks are still noted individually. + let probe = if primary_instructions.is_some() { + Ok(None) + } else { + HardwareCounter::new(HardwareEvent::Instructions, MetricScope::Process).map(Some) + }; + match probe { + Err(error) => note_unavailable("hardware counters", &error), + Ok(instructions) => { + if let Some(instructions) = instructions { + self = self.with_secondary("instructions", instructions); + } + let hardware = + |this: Self, name, event, scope| match HardwareCounter::new(event, scope) { + Ok(counter) => this.with_secondary(name, counter), + Err(error) => { + note_unavailable(name, &error); + this + } + }; + // Whichever instruction scope is not the primary is reported + // as a secondary, so both are always present for the ratio. + let (other_scope, other_name) = match primary_instructions { + Some(MetricScope::CallingThread) => (MetricScope::Process, "instructions"), + _ => (MetricScope::CallingThread, "foreground instructions"), + }; + self = hardware(self, other_name, HardwareEvent::Instructions, other_scope); + self = hardware(self, "cycles", HardwareEvent::Cycles, MetricScope::Process); + self = hardware( + self, + "branch misses", + HardwareEvent::BranchMisses, + MetricScope::Process, + ); + self = hardware( + self, + "cache misses", + HardwareEvent::CacheMisses, + MetricScope::Process, + ); + self = self.with_ratio("IPC", "instructions", "cycles").with_ratio( + "foreground share of instructions", + "foreground instructions", + "instructions", + ); + } + } + let resource = |this: Self, name, metric, scope| match ResourceCounter::new(metric, scope) { + Ok(counter) => this.with_secondary(name, counter), + Err(error) => { + note_unavailable(name, &error); + this + } + }; + self = resource( + self, + "foreground context switches", + ResourceMetric::VoluntaryContextSwitches, + MetricScope::CallingThread, + ); + self = resource( + self, + "page faults", + ResourceMetric::MinorPageFaults, + MetricScope::Process, + ); + self + } +} + +/// Prints why a default secondary is unavailable, once per metric per process, +/// so a machine without perf access reports it without spamming every group. +fn note_unavailable(name: &str, error: &anyhow::Error) { + use std::sync::{Mutex, OnceLock}; + + static NOTED: OnceLock>> = OnceLock::new(); + let mut noted = NOTED + .get_or_init(Default::default) + .lock() + .unwrap_or_else(|poisoned| poisoned.into_inner()); + if noted.iter().any(|noted| noted == name) { + return; + } + noted.push(name.to_owned()); + eprintln!("benchmark metric {name:?} unavailable on this machine: {error:#}"); +} + +/// Object-safe view of a Criterion measurement. +/// +/// Criterion always pairs `start` with `end` on the same thread and never +/// nests them, so the intermediate value can be parked in the measurement +/// between the two calls instead of flowing through the generic +/// `Intermediate` type. +trait ErasedMeasurement { + fn start(&self); + fn end(&self) -> f64; + fn formatter(&self) -> &dyn ValueFormatter; +} + +struct ErasedMeasurementCell { + measurement: M, + intermediate: RefCell>, +} + +impl ErasedMeasurementCell { + fn new(measurement: M) -> Self { + Self { + measurement, + intermediate: RefCell::new(None), + } + } +} + +impl ErasedMeasurement for ErasedMeasurementCell { + fn start(&self) { + let previous = self + .intermediate + .borrow_mut() + .replace(self.measurement.start()); + assert!( + previous.is_none(), + "Measurement::start called again before Measurement::end" + ); + } + + fn end(&self) -> f64 { + let intermediate = self + .intermediate + .borrow_mut() + .take() + .expect("Measurement::end called without a matching Measurement::start"); + self.measurement.to_f64(&self.measurement.end(intermediate)) + } + + fn formatter(&self) -> &dyn ValueFormatter { + self.measurement.formatter() + } +} + +/// A secondary metric and its total since the report last drained it. +struct SecondaryMetric { + name: &'static str, + measurement: Box, + total: Cell, +} + +struct RatioMetric { + name: &'static str, + numerator: &'static str, + denominator: &'static str, +} + +#[derive(Default)] +struct SecondaryMeasurements { + metrics: Vec>, + ratios: Vec, + primary_name: &'static str, + /// The primary's total since the report last drained it, so ratios can + /// combine it with secondaries. + primary_total: Cell, +} + +impl SecondaryMeasurements { + /// Returns each metric's total since the previous call and resets it. + fn take_totals(&self) -> Vec<(Rc, f64)> { + self.metrics + .iter() + .map(|metric| (metric.clone(), metric.total.replace(0.0))) + .collect() + } + + fn take_primary_total(&self) -> (&'static str, f64) { + (self.primary_name, self.primary_total.replace(0.0)) + } +} + +thread_local! { + /// The secondaries of the `BenchMeasurement` most recently started on this + /// thread. Criterion owns the measurement and hands benchmark code only a + /// `Bencher`, so this is how a harness reaches the secondary totals after + /// Criterion's iteration loop returns. + static ACTIVE_SECONDARIES: RefCell>> = + const { RefCell::new(None) }; +} + +fn active_secondaries() -> Option> { + ACTIVE_SECONDARIES.with(|active| active.borrow().clone()) +} + +impl Measurement for BenchMeasurement { + type Intermediate = (); + type Value = f64; + + fn start(&self) -> Self::Intermediate { + ACTIVE_SECONDARIES.with(|active| *active.borrow_mut() = Some(self.secondaries.clone())); + for metric in &self.secondaries.metrics { + metric.measurement.start(); + } + self.primary.start(); + } + + fn end(&self, (): Self::Intermediate) -> Self::Value { + let value = self.primary.end(); + self.secondaries + .primary_total + .set(self.secondaries.primary_total.get() + value); + for metric in self.secondaries.metrics.iter().rev() { + metric + .total + .set(metric.total.get() + metric.measurement.end()); + } + value + } + + fn add(&self, first: &Self::Value, second: &Self::Value) -> Self::Value { + first + second + } + + fn zero(&self) -> Self::Value { + 0.0 + } + + fn to_f64(&self, value: &Self::Value) -> f64 { + *value + } + + fn formatter(&self) -> &dyn ValueFormatter { + self.primary.formatter() + } +} + +// --------------------------------------------------------------------------- +// Report +// --------------------------------------------------------------------------- + +/// Per-iteration values of every secondary metric and ratio, one entry per +/// Criterion sample. +#[derive(Clone, Default)] +pub struct MetricReport { + samples: Rc>>, +} + +struct MetricSamples { + name: &'static str, + /// `None` for ratios, which are unitless. + formatter: Option>, + values: Vec, +} + +impl MetricReport { + /// Creates an empty report. + pub fn new() -> Self { + Self::default() + } + + /// Returns whether any sample has been recorded. + pub fn is_empty(&self) -> bool { + self.samples.borrow().is_empty() + } + + /// Returns every recorded per-iteration value (or ratio) for `name`, in + /// sample order, for programmatic checks on top of the printed summary. + pub fn values(&self, name: &str) -> Option> { + self.samples + .borrow() + .iter() + .find(|samples| samples.name == name) + .map(|samples| samples.values.clone()) + } + + /// Records the per-iteration value of every active secondary metric for + /// the Criterion sample that just finished, using the totals accumulated + /// since the previous call, and each ratio of those totals. + /// + /// `iterations` is counted by the caller because Criterion's `Bencher` + /// does not expose it, and `iter_batched` with `BatchSize::PerIteration` + /// ends the measurement once per iteration rather than once per sample. + pub fn record_sample(&self, iterations: u64) { + let Some(secondaries) = active_secondaries() else { + return; + }; + let totals = secondaries.take_totals(); + let (primary_name, primary_total) = secondaries.take_primary_total(); + if iterations == 0 { + return; + } + let mut samples = self.samples.borrow_mut(); + for (metric, total) in &totals { + Self::push( + &mut samples, + metric.name, + Some(metric), + total / iterations as f64, + ); + } + for ratio in &secondaries.ratios { + let total_named = |name| { + if name == primary_name { + return Some(primary_total); + } + totals + .iter() + .find(|(metric, _)| metric.name == name) + .map(|(_, total)| *total) + }; + if let (Some(numerator), Some(denominator)) = + (total_named(ratio.numerator), total_named(ratio.denominator)) + && denominator > 0.0 + { + Self::push(&mut samples, ratio.name, None, numerator / denominator); + } + } + } + + fn push( + samples: &mut Vec, + name: &'static str, + metric: Option<&Rc>, + value: f64, + ) { + match samples.iter_mut().find(|samples| samples.name == name) { + Some(samples) => samples.values.push(value), + None => samples.push(MetricSamples { + name, + formatter: metric.cloned(), + values: vec![value], + }), + } + } + + /// Discards secondary totals accumulated outside a measured interval, + /// e.g. by Criterion iterating a different benchmark on this thread. + pub fn discard_pending(&self) { + if let Some(secondaries) = active_secondaries() { + secondaries.take_totals(); + secondaries.take_primary_total(); + } + } + + /// `Bencher::iter` for plain Criterion benchmarks: runs `routine` under + /// Criterion's loop and records this sample's secondary metrics. + pub fn iter( + &self, + bencher: &mut criterion::Bencher<'_, BenchMeasurement>, + mut routine: impl FnMut() -> O, + ) { + self.discard_pending(); + let mut iterations = 0; + bencher.iter(|| { + iterations += 1; + routine() + }); + self.record_sample(iterations); + } + + /// Prints one line per metric, each prefixed with `indent`, to stderr. + pub fn print(&self, indent: &str) { + for samples in self.samples.borrow().iter() { + let mut sorted = samples.values.clone(); + sorted.sort_by(f64::total_cmp); + let Some((&min, &max)) = sorted.first().zip(sorted.last()) else { + continue; + }; + let median = sorted[sorted.len() / 2]; + match &samples.formatter { + Some(metric) => { + let mut scaled = [median, min, max]; + let unit = metric + .measurement + .formatter() + .scale_values(median, &mut scaled); + let [median, min, max] = scaled; + eprintln!( + "{indent}{} per iteration: median {median:.3} {unit} (min {min:.3}, max \ + {max:.3}, samples {})", + samples.name, + sorted.len() + ); + } + None => eprintln!( + "{indent}{}: median {median:.3} (min {min:.3}, max {max:.3}, samples {})", + samples.name, + sorted.len() + ), + } + } + } +} + +/// A deterministic measurement for tests: counts increments of a shared +/// counter between `start` and `end`. +#[cfg(any(test, feature = "test-support"))] +pub struct FakeCounter { + counter: std::sync::Arc, + formatter: CountFormatter, +} + +#[cfg(any(test, feature = "test-support"))] +impl FakeCounter { + /// Creates a measurement reading `counter`. + pub fn new(counter: std::sync::Arc) -> Self { + Self { + counter, + formatter: CountFormatter::new("ticks"), + } + } +} + +#[cfg(any(test, feature = "test-support"))] +impl Measurement for FakeCounter { + type Intermediate = u64; + type Value = u64; + + fn start(&self) -> u64 { + self.counter.load(std::sync::atomic::Ordering::SeqCst) + } + + fn end(&self, start: u64) -> u64 { + self.counter.load(std::sync::atomic::Ordering::SeqCst) - start + } + + fn add(&self, first: &u64, second: &u64) -> u64 { + first + second + } + + fn zero(&self) -> u64 { + 0 + } + + fn to_f64(&self, value: &u64) -> f64 { + *value as f64 + } + + fn formatter(&self) -> &dyn ValueFormatter { + &self.formatter + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::{ + Arc, + atomic::{AtomicU64, Ordering}, + }; + + fn fast_criterion(measurement: BenchMeasurement) -> criterion::Criterion { + criterion::Criterion::default() + .with_measurement(measurement) + .without_plots() + .sample_size(10) + .warm_up_time(std::time::Duration::from_millis(1)) + .measurement_time(std::time::Duration::from_millis(1)) + } + + #[test] + fn erased_measurement_preserves_wrapped_values() { + let counter = Arc::new(AtomicU64::new(0)); + let measurement = BenchMeasurement::new(FakeCounter::new(counter.clone())); + + let intermediate = measurement.start(); + counter.fetch_add(5, Ordering::SeqCst); + assert_eq!(measurement.end(intermediate), 5.0); + + let intermediate = measurement.start(); + counter.fetch_add(3, Ordering::SeqCst); + assert_eq!(measurement.end(intermediate), 3.0); + } + + #[test] + fn report_normalizes_secondaries_per_iteration_and_computes_ratios() { + let primary = Arc::new(AtomicU64::new(0)); + let secondary = Arc::new(AtomicU64::new(0)); + let report = MetricReport::new(); + let mut criterion = fast_criterion( + BenchMeasurement::new(FakeCounter::new(primary.clone())) + .named("primary ticks") + .with_secondary("ticks", FakeCounter::new(secondary.clone())) + .with_secondary("double ticks", FakeCounter::new(secondary.clone())) + .with_ratio("tick ratio", "double ticks", "ticks") + .with_ratio("ticks per primary", "ticks", "primary ticks") + .with_ratio("unavailable ratio", "ticks", "missing"), + ); + + criterion.bench_function("report", |bencher| { + // Fixture work outside the measured loop is not attributed to any + // iteration. + secondary.fetch_add(1_000_000, Ordering::SeqCst); + report.iter(bencher, || { + primary.fetch_add(1, Ordering::SeqCst); + secondary.fetch_add(11, Ordering::SeqCst); + }); + }); + + let samples = report.samples.borrow(); + let values = |name: &str| { + samples + .iter() + .find(|samples| samples.name == name) + .map(|samples| samples.values.clone()) + }; + let ticks = values("ticks").expect("ticks were recorded"); + assert!(!ticks.is_empty()); + assert!( + ticks.iter().all(|value| *value == 11.0), + "each sample's total should be divided by its iteration count: {ticks:?}" + ); + let ratio = values("tick ratio").expect("ratio was recorded"); + assert!( + ratio.iter().all(|value| *value == 1.0), + "ratio of identical counters should be 1: {ratio:?}" + ); + let per_primary = + values("ticks per primary").expect("ratio against the primary was recorded"); + assert!( + per_primary.iter().all(|value| *value == 11.0), + "secondary 11 per iteration over primary 1 per iteration: {per_primary:?}" + ); + assert!(values("unavailable ratio").is_none()); + } + + #[test] + fn count_formatter_scales_with_prefixes() { + let formatter = CountFormatter::new("cycles"); + let mut values = [2_500_000.0]; + assert_eq!(formatter.scale_values(2_500_000.0, &mut values), "M cycles"); + assert_eq!(values, [2.5]); + let mut values = [12.0]; + assert_eq!(formatter.scale_values(12.0, &mut values), "cycles"); + assert_eq!(formatter.scale_for_machines(&mut values), "cycles"); + } + + #[cfg(unix)] + #[test] + fn resource_counter_counts_page_faults_and_context_switches() { + let faults = ResourceCounter::new(ResourceMetric::MinorPageFaults, MetricScope::Process) + .expect("RUSAGE_SELF is always available"); + let start = faults.start(); + // Touch fresh pages: a 4 MiB allocation faults in ~1000 pages. + let buffer = vec![1_u8; 4 << 20]; + std::hint::black_box(&buffer); + assert!(faults.end(start) > 100.0); + + #[cfg(target_os = "linux")] + { + let switches = ResourceCounter::new( + ResourceMetric::VoluntaryContextSwitches, + MetricScope::CallingThread, + ) + .expect("RUSAGE_THREAD is available on Linux"); + let start = switches.start(); + for _ in 0..5 { + std::thread::sleep(std::time::Duration::from_micros(100)); + } + assert!(switches.end(start) >= 5.0); + } + } + + /// Returns a closure measuring a background thread spinning `iterations` + /// times, or `None` when this machine has no usable hardware counters. + fn background_spin_instructions(scope: MetricScope) -> Option f64> { + let counter = match HardwareCounter::new(HardwareEvent::Instructions, scope) { + Ok(counter) => counter, + Err(error) => { + let message = error.to_string(); + assert!( + message.contains("CAP_PERFMON") + || message.contains("does not advance") + || message.contains("unavailable on this platform") + || message.contains("unavailable for"), + "unavailable counters should have an actionable error: {message}" + ); + return None; + } + }; + Some(move |iterations: u64| { + let start = counter.start(); + let worker = std::thread::spawn(move || { + let mut total = 0_u64; + for value in 0..iterations { + total = std::hint::black_box(total.wrapping_add(value)); + } + total + }); + std::hint::black_box(worker.join().expect("spin thread should finish")); + counter.end(start) + }) + } + + #[test] + fn process_instructions_include_threads_spawned_after_open() { + let Some(measure) = background_spin_instructions(MetricScope::Process) else { + return; + }; + let idle = measure(0); + let busy = measure(1_000_000); + assert!( + busy > idle + 500_000.0, + "work on a thread spawned after the counter opened should be counted: \ + idle={idle}, busy={busy}" + ); + } + + #[test] + fn calling_thread_instructions_exclude_other_threads() { + let Some(measure) = background_spin_instructions(MetricScope::CallingThread) else { + return; + }; + let idle = measure(0); + let busy = measure(1_000_000); + assert!( + busy < idle + 100_000.0, + "another thread's work should not count toward the calling thread: \ + idle={idle}, busy={busy}" + ); + } + + #[test] + fn hardware_events_open_or_fail_actionably() { + for event in [ + HardwareEvent::Cycles, + HardwareEvent::BranchMisses, + HardwareEvent::CacheMisses, + HardwareEvent::CacheReferences, + ] { + match HardwareCounter::new(event, MetricScope::Process) { + Ok(counter) => { + let start = counter.start(); + std::hint::black_box((0..100_000_u64).fold(0_u64, |t, v| t.wrapping_add(v))); + let count = counter.end(start); + assert!(count >= 0.0); + } + Err(error) => { + let message = error.to_string(); + assert!( + message.contains("CAP_PERFMON") + || message.contains("does not expose") + || message.contains("does not advance") + || message.contains("needs kperf") + || message.contains("unavailable on this platform") + || message.contains("unavailable for"), + "{event:?}: {message}" + ); + } + } + } + } +} diff --git a/crates/benchmarks/benches/display_map.rs b/crates/benchmarks/benches/display_map.rs index bad8d232772..06117a665f0 100644 --- a/crates/benchmarks/benches/display_map.rs +++ b/crates/benchmarks/benches/display_map.rs @@ -1,4 +1,4 @@ -use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main}; +use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; use editor::{EditorStyle, MultiBuffer, display_map::*}; use gpui::{AppContext as _, HighlightStyle, Hsla, TestDispatcher, font, px}; use itertools::Itertools; @@ -6,6 +6,7 @@ use multi_buffer::MultiBufferOffset; use project::project_settings::DiagnosticSeverity; use rand::{Rng, SeedableRng, rngs::StdRng}; use settings::SettingsStore; +use std::hint::black_box; use std::{num::NonZeroU32, time::Duration}; use text::Bias; use util::RandomCharIter; diff --git a/crates/benchmarks/benches/edit_file_tool.rs b/crates/benchmarks/benches/edit_file_tool.rs index 204628cfe80..f364e260729 100644 --- a/crates/benchmarks/benches/edit_file_tool.rs +++ b/crates/benchmarks/benches/edit_file_tool.rs @@ -16,9 +16,7 @@ use benchmarks::bench_utils::{ RUST_FUNCTION_BODY_LINES, RUST_FUNCTION_LINES, RUST_MODULE_HEADER_LINES, random_rust_file, rust_file_line_count, rust_identifier as identifier, }; -use criterion::{ - BatchSize, BenchmarkId, Criterion, Throughput, black_box, criterion_group, criterion_main, -}; +use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main}; use editor::{Editor, EditorStyle}; use futures::{StreamExt as _, pin_mut, task::noop_waker}; use gpui::{ @@ -32,6 +30,7 @@ use prompt_store::ProjectContext; use rand::{Rng as _, SeedableRng as _, rngs::StdRng}; use serde_json::{Value, json}; use settings::{Settings as _, SettingsStore}; +use std::hint::black_box; use ui::IntoElement as _; const SEED: u64 = 0x5EED_5EED; diff --git a/crates/edit_prediction/benches/kept_rate.rs b/crates/edit_prediction/benches/kept_rate.rs index 44defe5d19d..8db3c274d22 100644 --- a/crates/edit_prediction/benches/kept_rate.rs +++ b/crates/edit_prediction/benches/kept_rate.rs @@ -1,5 +1,6 @@ -use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main}; +use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; use edit_prediction::metrics::compute_kept_rate; +use std::hint::black_box; fn repeated_function_lines(line_count: usize) -> String { let mut text = String::with_capacity(line_count * 32); diff --git a/crates/edit_prediction/benches/ts_error_count.rs b/crates/edit_prediction/benches/ts_error_count.rs index 518f9ab5bc2..c6e77f609a9 100644 --- a/crates/edit_prediction/benches/ts_error_count.rs +++ b/crates/edit_prediction/benches/ts_error_count.rs @@ -1,6 +1,6 @@ use std::sync::Arc; -use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main}; +use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; use edit_prediction::metrics::count_tree_sitter_errors; use fs::FakeFs; use gpui::{AppContext as _, TestAppContext}; @@ -8,6 +8,7 @@ use language::{Buffer, BufferSnapshot, LanguageRegistry}; use languages::init as init_languages; use node_runtime::NodeRuntime; use settings::SettingsStore; +use std::hint::black_box; struct ParsedCase { label: String, diff --git a/crates/gpui/Cargo.toml b/crates/gpui/Cargo.toml index e767de582ff..1cebc98f702 100644 --- a/crates/gpui/Cargo.toml +++ b/crates/gpui/Cargo.toml @@ -26,7 +26,7 @@ test-support = [ "x11", "proptest", ] -bench-support = ["profiler", "dep:criterion"] +bench-support = ["profiler", "dep:criterion", "dep:bench_metrics"] # Preserve the published feature name while consumers migrate to `bench-support`. bench = ["bench-support"] inspector = ["gpui_macros/inspector"] @@ -53,6 +53,7 @@ backtrace = { workspace = true, optional = true } bitflags.workspace = true collections.workspace = true +bench_metrics = { workspace = true, optional = true } criterion = { workspace = true, optional = true } ctor.workspace = true derive_more.workspace = true @@ -131,13 +132,13 @@ objc2-metal = { version = "0.3", optional = true } scap = { workspace = true, optional = true } - [target.'cfg(target_os = "windows")'.dependencies] windows = { workspace = true, features = ["Win32_Foundation", "Win32_System_Power"] } [dev-dependencies] backtrace.workspace = true +bench_metrics = { workspace = true, features = ["test-support"] } collections = { workspace = true, features = ["test-support"] } env_logger.workspace = true gpui_platform = { workspace = true, features = ["font-kit", "wayland", "x11"] } diff --git a/crates/gpui/src/app.rs b/crates/gpui/src/app.rs index b62cb7e6764..612d3df5882 100644 --- a/crates/gpui/src/app.rs +++ b/crates/gpui/src/app.rs @@ -21,7 +21,10 @@ use slotmap::SlotMap; pub use async_context::*; #[cfg(feature = "bench-support")] -pub use bench_context::{BenchAppContext, BenchReport, BenchWindowContext, bench_platform}; +pub use bench_context::{ + BenchAppContext, BenchMeasurement, BenchReport, BenchWindowContext, MetricReport, + bench_platform, +}; use collections::{FxHashMap, FxHashSet, HashMap, TypeIdHashMap, TypeIdHashSet, VecDeque}; pub use context::*; pub use entity_map::*; diff --git a/crates/gpui/src/app/bench_context.rs b/crates/gpui/src/app/bench_context.rs index c9e5484d298..15eef07eb57 100644 --- a/crates/gpui/src/app/bench_context.rs +++ b/crates/gpui/src/app/bench_context.rs @@ -10,6 +10,7 @@ use std::{ }; use anyhow::{Result, anyhow}; +pub use bench_metrics::{BenchMeasurement, MetricReport}; use hdrhistogram::Histogram; use crate::{ @@ -101,6 +102,7 @@ pub struct ForegroundWorkSummary { #[derive(Clone)] pub struct BenchReport { frame_snapshot: Rc>, + metrics: MetricReport, frame_budget_nanos: u128, } @@ -127,10 +129,18 @@ impl BenchReport { ); Self { frame_snapshot: Rc::new(RefCell::new(WindowFrameSnapshot::new())), + metrics: MetricReport::new(), frame_budget_nanos, } } + /// Returns the per-iteration secondary metrics (retired instructions, + /// cycles, context switches, ...) recorded alongside Criterion's primary + /// measurement. + pub fn metrics(&self) -> &MetricReport { + &self.metrics + } + fn record_frame_timings<'i>(&self, events: impl IntoIterator) { let mut snapshot = self.frame_snapshot.borrow_mut(); // `.ok()` on `record`: this operation is infallible (the histograms auto-resize). @@ -239,15 +249,15 @@ impl BenchReport { } /// Prints this report to stderr. - pub fn print(&self, benchmark_name: Option<&'static str>) { + pub fn print(&self, benchmark_name: &str) { let frame_snapshot = self.frame_snapshot.borrow(); - if frame_snapshot.is_empty() { + if frame_snapshot.is_empty() && self.metrics.is_empty() { return; } - let benchmark_name = benchmark_name.unwrap_or("unknown benchmark"); eprintln!("GPUI bench report (all observed iterations): {benchmark_name}"); eprintln!(" note: includes Criterion warmup/calibration"); + self.metrics.print(" "); self.print_histogram("window dirty-to-draw", &frame_snapshot.dirty_to_draw); self.print_histogram("window draw", &frame_snapshot.draw); self.print_histogram("window present interval", &frame_snapshot.present_interval); @@ -488,7 +498,7 @@ pub struct BenchAppContext<'a, 'measurement> { background_executor: BackgroundExecutor, foreground_executor: ForegroundExecutor, benchmark_name: Option<&'static str>, - bencher: Rc>>>, + bencher: Rc>>>, report: BenchReport, } @@ -501,7 +511,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { pub fn new( platform: Rc, benchmark_name: Option<&'static str>, - bencher: &'a mut criterion::Bencher<'measurement>, + bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>, ) -> Self { Self::build(platform, benchmark_name, bencher, BenchReport::default()) } @@ -515,7 +525,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { pub fn new_with_platform_and_report( platform: Rc, benchmark_name: Option<&'static str>, - bencher: &'a mut criterion::Bencher<'measurement>, + bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>, report: BenchReport, ) -> Self { Self::build(platform, benchmark_name, bencher, report) @@ -524,7 +534,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { fn build( platform: Rc, benchmark_name: Option<&'static str>, - bencher: &'a mut criterion::Bencher<'measurement>, + bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>, report: BenchReport, ) -> Self { let background_executor = platform.background_executor(); @@ -662,12 +672,16 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { let bencher = self.take_bencher("bench_iter"); self.dispatch_pending_frames(|| true); let collector = TraceScope::start(self.foreground_journal_collector()); + self.report.metrics.discard_pending(); + let mut iterations = 0; let mut benchmark = || { + iterations += 1; benchmark(self); self.dispatch_pending_frames(|| true); }; bencher.iter(&mut benchmark); let events = collector.finish(); + self.report.metrics.record_sample(iterations); self.report.record_frame_timings(events.frame_events.iter()); self.report .record_foreground_events(events.foreground_events()); @@ -722,6 +736,8 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { let mut benchmark_context = self.clone(); let foreground_executor = self.foreground_executor.clone(); let report = self.report.clone(); + report.metrics.discard_pending(); + let iterations = std::cell::Cell::new(0); bencher.iter_batched_ref( || { @@ -740,6 +756,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { } }, |measured_input| { + iterations.set(iterations.get() + 1); let task = benchmark(&mut measured_input.input, &mut benchmark_context); benchmark_context.dispatch_pending_frames(|| true); let output = Rc::new(RefCell::new(None)); @@ -759,6 +776,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { }, criterion::BatchSize::PerIteration, ); + report.metrics.record_sample(iterations.get()); self.replace_bencher(bencher); } @@ -789,7 +807,10 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { let dispatcher = self.background_executor.dispatcher().clone(); self.dispatch_pending_frames(|| true); let collector = TraceScope::start(self.foreground_journal_collector()); + self.report.metrics.discard_pending(); + let mut iterations = 0; let mut benchmark = || { + iterations += 1; dispatcher .as_threaded() .expect("validated in BenchAppContext::build") @@ -808,6 +829,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { bencher.iter(&mut benchmark); let events = collector.finish(); + self.report.metrics.record_sample(iterations); self.report.record_frame_timings(events.frame_events.iter()); self.report .record_foreground_events(events.foreground_events()); @@ -853,6 +875,8 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { .as_threaded() .expect("validated in BenchAppContext::build"); let report = self.report.clone(); + report.metrics.discard_pending(); + let iterations = std::cell::Cell::new(0); bencher.iter_batched_ref( || { @@ -867,6 +891,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { } }, |measured_input| { + iterations.set(iterations.get() + 1); let (state, window, stopped) = &mut measured_input.input; let started = Instant::now(); let check_deadline = || { @@ -932,6 +957,7 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { }, criterion::BatchSize::PerIteration, ); + report.metrics.record_sample(iterations.get()); self.replace_bencher(bencher); } @@ -1009,13 +1035,16 @@ impl<'a, 'measurement> BenchAppContext<'a, 'measurement> { } } - fn take_bencher(&self, benchmark_kind: &str) -> &'a mut criterion::Bencher<'measurement> { + fn take_bencher( + &self, + benchmark_kind: &str, + ) -> &'a mut criterion::Bencher<'measurement, BenchMeasurement> { self.bencher.borrow_mut().take().unwrap_or_else(|| { panic!("cannot start {benchmark_kind}: benchmark measurement is already running") }) } - fn replace_bencher(&self, bencher: &'a mut criterion::Bencher<'measurement>) { + fn replace_bencher(&self, bencher: &'a mut criterion::Bencher<'measurement, BenchMeasurement>) { let previous = self.bencher.borrow_mut().replace(bencher); assert!( previous.is_none(), @@ -1323,11 +1352,96 @@ impl VisualContext for BenchWindowContext<'_, '_> { #[cfg(test)] mod tests { - use std::{rc::Rc, sync::Arc}; + use std::{ + rc::Rc, + sync::{ + Arc, + atomic::{AtomicU64, Ordering}, + }, + }; use super::*; use crate::profiler::journal::install_test_foreground_journal; + /// Runs `benchmark` under a wall-time primary with a fake counter as the + /// secondary metric and returns the per-iteration values the report + /// recorded for it. + fn secondary_values_per_iteration( + counter: Arc, + benchmark: impl Fn(&mut BenchAppContext, &Arc), + ) -> Vec { + let platform = bench_platform(None, Arc::new(crate::NoopTextSystem::new())); + let report = BenchReport::default(); + let mut criterion = criterion::Criterion::default() + .with_measurement( + BenchMeasurement::new(criterion::measurement::WallTime).with_secondary( + "fake counter", + bench_metrics::FakeCounter::new(counter.clone()), + ), + ) + .without_plots() + .sample_size(10) + .warm_up_time(Duration::from_millis(1)) + .measurement_time(Duration::from_millis(1)); + + criterion.bench_function("secondary_values_per_iteration", |bencher| { + let mut cx = BenchAppContext::new_with_platform_and_report( + platform.clone(), + Some("secondary_values_per_iteration"), + bencher, + report.clone(), + ); + // Fixture work outside the measured loop must not be attributed + // to any iteration. + counter.fetch_add(1_000_000, Ordering::SeqCst); + benchmark(&mut cx, &counter); + cx.teardown(); + }); + + report + .metrics() + .values("fake counter") + .expect("the fake counter was recorded") + .into_iter() + .map(|value| value as u64) + .collect() + } + + #[test] + fn secondary_metric_is_normalized_per_iteration_for_bench_iter() { + let values = secondary_values_per_iteration(Arc::new(AtomicU64::new(0)), |cx, counter| { + cx.bench_iter(|_| { + counter.fetch_add(11, Ordering::SeqCst); + }); + }); + assert!(!values.is_empty()); + assert!( + values.iter().all(|value| *value == 11), + "each sample's total should be divided by its iteration count: {values:?}" + ); + } + + #[test] + fn secondary_metric_is_accumulated_across_per_iteration_batches() { + let values = secondary_values_per_iteration(Arc::new(AtomicU64::new(0)), |cx, counter| { + cx.bench_batched_task( + |_| { + // Setup runs outside every measured interval. + counter.fetch_add(1_000, Ordering::SeqCst); + }, + |(), cx| { + counter.fetch_add(11, Ordering::SeqCst); + cx.background_spawn(async {}) + }, + ); + }); + assert!(!values.is_empty()); + assert!( + values.iter().all(|value| *value == 11), + "PerIteration batches should sum to one per-iteration value: {values:?}" + ); + } + #[test] fn foreground_work_reports_long_task_without_window_draw() { let (journal, _journal_guard) = install_test_foreground_journal(1024, 64); @@ -1421,6 +1535,7 @@ mod tests { let name = "bench_task_reports_long_task_without_window"; let mut criterion = criterion::Criterion::default() + .with_measurement(BenchMeasurement::new(criterion::measurement::WallTime)) .without_plots() .sample_size(10) .warm_up_time(Duration::from_millis(1)) diff --git a/crates/gpui/src/gpui.rs b/crates/gpui/src/gpui.rs index 51585b5e5ef..387729e8a3e 100644 --- a/crates/gpui/src/gpui.rs +++ b/crates/gpui/src/gpui.rs @@ -112,14 +112,35 @@ pub use spring::*; /// Defines a Criterion benchmark group for benchmarks annotated with [`gpui::bench`]. /// -/// This mirrors `criterion::criterion_group!` so GPUI benchmark files can keep the -/// same shape as ordinary Criterion benchmarks. +/// This mirrors `criterion::criterion_group!`, but the group measures with the +/// `gpui::BenchMeasurement` configured by `BENCH_MEASUREMENT` (see +/// `BenchMeasurement::from_env`). By default Criterion analyzes wall time +/// while retired instructions, cycles, IPC, context switches, and the other +/// counters this machine supports are printed per iteration in the GPUI bench +/// report. `BENCH_MEASUREMENT=instructions` makes Criterion analyze +/// process-wide instructions instead, `foreground-instructions` the benchmark +/// thread's alone, and `wall-time` disables all counters. A +/// `config = ...` expression may set any other Criterion option; its +/// measurement is replaced. To measure with something else, call +/// `criterion::criterion_group!` directly with +/// `config = criterion::Criterion::default().with_measurement(gpui::BenchMeasurement::new(...))`. /// /// [`gpui::bench`]: crate::bench #[macro_export] macro_rules! bench_group { - ($($tokens:tt)*) => { - criterion::criterion_group!($($tokens)*); + (name = $name:ident; config = $config:expr; targets = $($target:path),+ $(,)?) => { + criterion::criterion_group! { + name = $name; + config = ($config).with_measurement($crate::BenchMeasurement::from_env_or_exit()); + targets = $($target),+ + } + }; + ($name:ident, $($target:path),+ $(,)?) => { + $crate::bench_group! { + name = $name; + config = criterion::Criterion::default(); + targets = $($target),+ + } }; } diff --git a/crates/gpui_macros/src/bench.rs b/crates/gpui_macros/src/bench.rs index 8fea39ef0ec..eb1350bc781 100644 --- a/crates/gpui_macros/src/bench.rs +++ b/crates/gpui_macros/src/bench.rs @@ -83,10 +83,13 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream { let sample_size = sample_size.map(|sample_size| quote! { group.sample_size(#sample_size); }); quote! { - let report = #report_expr; let mut group = criterion.benchmark_group(#group_name); #sample_size for input in #inputs { + // One report per input: per-iteration metrics differ across + // inputs, so blending them would make the summary meaningless. + let report = #report_expr; + let report_name = format!("{}/{}/{}", #group_name, #input_name, input); group.bench_with_input(criterion::BenchmarkId::new(#input_name, &input), &input, { let report = report.clone(); move |bencher, input| { @@ -105,9 +108,9 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream { cx.teardown(); } }); + report.print(&report_name); } group.finish(); - report.print(Some(stringify!(#outer_fn_name))); } } else { if let Some(input_name) = input_name { @@ -148,14 +151,14 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream { cx.teardown(); } }); - report.print(Some(stringify!(#outer_fn_name))); + report.print(stringify!(#outer_fn_name)); } }; TokenStream::from(quote! { #inner_fn - fn #outer_fn_name(criterion: &mut criterion::Criterion) { + fn #outer_fn_name(criterion: &mut criterion::Criterion) { #benchmark } diff --git a/crates/language/benches/highlight_map.rs b/crates/language/benches/highlight_map.rs index 678bd08a8db..ab449d51e61 100644 --- a/crates/language/benches/highlight_map.rs +++ b/crates/language/benches/highlight_map.rs @@ -1,6 +1,7 @@ -use criterion::{BenchmarkId, Criterion, black_box, criterion_group, criterion_main}; +use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; use gpui::rgba; use language::build_highlight_map; +use std::hint::black_box; use theme::SyntaxTheme; fn syntax_theme(highlight_names: &[&str]) -> SyntaxTheme { diff --git a/crates/rope/benches/rope_benchmark.rs b/crates/rope/benches/rope_benchmark.rs index 8599328aacf..707acef1d5d 100644 --- a/crates/rope/benches/rope_benchmark.rs +++ b/crates/rope/benches/rope_benchmark.rs @@ -1,11 +1,10 @@ use std::ops::Range; -use criterion::{ - BatchSize, BenchmarkId, Criterion, Throughput, black_box, criterion_group, criterion_main, -}; +use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main}; use rand::prelude::*; use rand::rngs::StdRng; use rope::{Point, Rope}; +use std::hint::black_box; use sum_tree::Bias; use util::RandomCharIter; diff --git a/crates/streaming_diff/benches/streaming_diff.rs b/crates/streaming_diff/benches/streaming_diff.rs index cbf83b7557e..3067ba4ee56 100644 --- a/crates/streaming_diff/benches/streaming_diff.rs +++ b/crates/streaming_diff/benches/streaming_diff.rs @@ -1,7 +1,6 @@ -use criterion::{ - BatchSize, BenchmarkId, Criterion, Throughput, black_box, criterion_group, criterion_main, -}; +use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main}; use rand::{Rng as _, SeedableRng as _, rngs::StdRng}; +use std::hint::black_box; use streaming_diff::StreamingDiff; const SEED: u64 = 0x5EED_5EED;