Michael Neale
27958c4c29
Remove open-model-gym eval suite ( #10174 )
Canary / Prepare Version (push) Waiting to run
Canary / build-cli (push) Blocked by required conditions
Canary / Upload Install Script (push) Blocked by required conditions
Canary / bundle-desktop (push) Blocked by required conditions
Canary / bundle-desktop-intel (push) Blocked by required conditions
Canary / bundle-desktop-linux (push) Blocked by required conditions
Canary / bundle-desktop-windows (push) Blocked by required conditions
Canary / bundle-desktop-windows-cuda (push) Blocked by required conditions
Canary / Release (push) Blocked by required conditions
Unused Dependencies / machete (push) Waiting to run
CI / changes (push) Waiting to run
CI / Check Rust Code Format (push) Blocked by required conditions
CI / Build and Test Rust Project (push) Blocked by required conditions
CI / Build Rust Project on Windows (push) Waiting to run
CI / Check MSRV (push) Blocked by required conditions
CI / Lint Rust Code (push) Blocked by required conditions
CI / Check Generated Schemas are Up-to-Date (push) Blocked by required conditions
CI / Test and Lint Electron Desktop App (push) Blocked by required conditions
Create Minor Release PR / check-version-bump-pr (push) Waiting to run
Create Minor Release PR / release (push) Blocked by required conditions
Live Provider Tests / check-fork (push) Waiting to run
Live Provider Tests / changes (push) Blocked by required conditions
Live Provider Tests / Build Binary (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (Code Execution) (push) Blocked by required conditions
Live Provider Tests / Compaction Tests (push) Blocked by required conditions
Publish Docker Image / docker (push) Waiting to run
Scorecard supply-chain security / Scorecard analysis (push) Waiting to run
Deploy Documentation / deploy (push) Has been cancelled
Publish Ask AI Bot Docker Image / docker (push) Has been cancelled
2026-07-02 00:31:15 +00:00
Kyle E DeFreitas
d2090b1de5
feat(gym): make Open Model Gym output dir configurable ( #9789 )
...
Signed-off-by: Kyle De Freitas <kdefreitas@squareup.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Douwe M Osinga <douwe@sidewalklabs.com>
2026-06-17 17:20:56 +00:00
Kyle E DeFreitas
d95df17615
feat(gym): make per-agent timeout configurable (GYM_AGENT_TIMEOUT) ( #9791 )
...
Signed-off-by: Kyle De Freitas <kdefreitas@squareup.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 02:24:29 +00:00
Douwe Osinga
59521f455f
Improve benchmarking ( #9637 )
...
Signed-off-by: Douwe Osinga <douwe@squareup.com>
Co-authored-by: Douwe Osinga <douwe@squareup.com>
2026-06-16 19:13:16 +00:00
dependabot[bot]
1472e7ee89
chore(deps): bump hono from 4.12.18 to 4.12.23 in /evals/open-model-gym/mcp-harness ( #9609 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-15 14:06:57 +00:00
dependabot[bot]
64f9d61647
chore(deps): bump esbuild and tsx in /evals/open-model-gym/suite ( #9775 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-15 14:06:37 +00:00
Douwe Osinga
cd12199604
Bench marking ( #9465 )
...
Signed-off-by: Douwe Osinga <douwe@squareup.com>
Co-authored-by: Douwe Osinga <douwe@squareup.com>
Co-authored-by: Douwe M Osinga <douwe@sidewalklabs.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-02 20:13:48 +00:00
dependabot[bot]
6d544e7b55
chore(deps): bump qs from 6.14.2 to 6.15.2 in /evals/open-model-gym/mcp-harness ( #9395 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-26 15:52:38 +00:00
dependabot[bot]
b3f335b2cd
chore(deps): bump ip-address and express-rate-limit in /evals/open-model-gym/mcp-harness ( #9065 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-22 13:44:29 +00:00
dependabot[bot]
db9f0ee488
chore(deps): bump fast-uri from 3.1.0 to 3.1.2 in /evals/open-model-gym/mcp-harness ( #9117 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-22 13:43:51 +00:00
dependabot[bot]
6787b4eafa
chore(deps): bump brace-expansion from 5.0.5 to 5.0.6 in /evals/open-model-gym/suite ( #9311 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-22 13:43:44 +00:00
Lucas Kim
49f1adedd3
feat: add Harbor eval runner ( #9138 )
2026-05-20 20:03:01 +00:00
dependabot[bot]
960a02fb9b
chore(deps): bump hono from 4.12.14 to 4.12.18 in /evals/open-model-gym/mcp-harness ( #9073 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-13 16:12:40 +00:00
dependabot[bot]
77542db432
chore(deps): bump hono from 4.12.12 to 4.12.14 in /evals/open-model-gym/mcp-harness ( #8579 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-20 10:51:26 +00:00
Kyle E DeFreitas
5b43b5f2db
fix(gym): isolate scenario Cargo projects from parent workspace ( #8640 )
...
Signed-off-by: Kyle De Freitas <kdefreitas@squareup.com>
2026-04-20 08:59:29 +00:00
dependabot[bot]
be690218a9
chore(deps): bump @hono/node-server from 1.19.11 to 1.19.13 in /evals/open-model-gym/mcp-harness ( #8443 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-09 19:21:57 +00:00
dependabot[bot]
c0ea427fcb
chore(deps): bump hono from 4.12.5 to 4.12.12 in /evals/open-model-gym/mcp-harness ( #8433 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-09 14:50:03 +00:00
Jack Amadeo
583acd4335
chore(aaif): rename a bunch of repository references ( #8152 )
...
Signed-off-by: Michael Neale <michael.neale@gmail.com>
Co-authored-by: Michael Neale <michael.neale@gmail.com>
continuing migration to aaif
2026-04-07 15:34:48 +10:00
dependabot[bot]
d9b69ef735
chore(deps): bump path-to-regexp from 8.3.0 to 8.4.0 in /evals/open-model-gym/mcp-harness ( #8178 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-30 14:02:55 +00:00
dependabot[bot]
53712be6ca
chore(deps): bump yaml from 2.8.2 to 2.8.3 in /evals/open-model-gym/suite ( #8124 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Douwe Osinga <douwe@squareup.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Douwe Osinga <douwe@squareup.com>
2026-03-26 20:05:41 +00:00
dependabot[bot]
4a02379731
chore(deps): bump brace-expansion from 5.0.3 to 5.0.5 in /evals/open-model-gym/suite ( #8139 )
...
CI / Check Rust Code Format (push) Blocked by required conditions
CI / Build and Test Rust Project (push) Blocked by required conditions
CI / Build Rust Project on Windows (push) Blocked by required conditions
CI / Lint Rust Code (push) Blocked by required conditions
CI / Check OpenAPI Schema is Up-to-Date (push) Blocked by required conditions
CI / Test and Lint Electron Desktop App (push) Blocked by required conditions
Deploy Documentation / deploy (push) Waiting to run
Live Provider Tests / check-fork (push) Waiting to run
Live Provider Tests / changes (push) Blocked by required conditions
Live Provider Tests / Build Binary (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (Code Execution) (push) Blocked by required conditions
Live Provider Tests / Compaction Tests (push) Blocked by required conditions
Live Provider Tests / goose server HTTP integration tests (push) Blocked by required conditions
Publish Ask AI Bot Docker Image / docker (push) Waiting to run
Publish Docker Image / docker (push) Waiting to run
Scorecard supply-chain security / Scorecard analysis (push) Waiting to run
Build Native Packages / Build darwin-arm64 (push) Has been cancelled
Build Native Packages / Build darwin-x64 (push) Has been cancelled
Build Native Packages / Build linux-arm64 (push) Has been cancelled
Build Native Packages / Build linux-x64 (push) Has been cancelled
Build Native Packages / Build win32-x64 (push) Has been cancelled
Publish to npm / Generate ACP Schema (push) Has been cancelled
Publish to npm / Build goose (darwin-arm64) (push) Has been cancelled
Publish to npm / Build goose (darwin-x64) (push) Has been cancelled
Publish to npm / Build goose (linux-arm64) (push) Has been cancelled
Publish to npm / Build goose (linux-x64) (push) Has been cancelled
Publish to npm / Build goose (win32-x64) (push) Has been cancelled
Build Native Packages / Collect all binaries (push) Has been cancelled
Publish to npm / Release to npm (push) Has been cancelled
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Douwe Osinga <douwe@squareup.com>
2026-03-26 15:43:25 -04:00
dependabot[bot]
7a14aa380d
chore(deps): bump @hono/node-server from 1.19.9 to 1.19.11 in /evals/open-model-gym/mcp-harness ( #7687 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-11 19:38:58 +00:00
dependabot[bot]
a64269c845
chore(deps): bump express-rate-limit from 8.2.1 to 8.3.0 in /evals/open-model-gym/mcp-harness ( #7703 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-10 21:38:25 +00:00
dependabot[bot]
f5e72df904
chore(deps): bump hono from 4.12.3 to 4.12.5 in /evals/open-model-gym/mcp-harness ( #7661 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-04 22:13:56 +00:00
dependabot[bot]
1350542aae
chore(deps): bump hono from 4.12.1 to 4.12.3 in /evals/open-model-gym/mcp-harness ( #7585 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-02 14:43:28 +00:00
dependabot[bot]
495855dc74
chore(deps): bump minimatch from 10.1.1 to 10.2.3 in /evals/open-model-gym/suite ( #7498 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-02 13:48:23 +00:00
dependabot[bot]
72bdc9598d
chore(deps): bump hono from 4.11.7 to 4.12.1 in /evals/open-model-gym/mcp-harness ( #7417 )
...
Canary / Prepare Version (push) Waiting to run
Canary / build-cli (push) Blocked by required conditions
Canary / Upload Install Script (push) Blocked by required conditions
Canary / bundle-desktop (push) Blocked by required conditions
Canary / bundle-desktop-linux (push) Blocked by required conditions
Canary / bundle-desktop-windows (push) Blocked by required conditions
Canary / Release (push) Blocked by required conditions
CI / changes (push) Waiting to run
CI / Check Rust Code Format (push) Blocked by required conditions
CI / Build and Test Rust Project (push) Blocked by required conditions
CI / Lint Rust Code (push) Blocked by required conditions
CI / Check OpenAPI Schema is Up-to-Date (push) Blocked by required conditions
CI / Test and Lint Electron Desktop App (push) Blocked by required conditions
Live Provider Tests / check-fork (push) Waiting to run
Live Provider Tests / changes (push) Blocked by required conditions
Live Provider Tests / Build Binary (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (Code Execution) (push) Blocked by required conditions
Live Provider Tests / Compaction Tests (push) Blocked by required conditions
Live Provider Tests / goose server HTTP integration tests (push) Blocked by required conditions
Publish Docker Image / docker (push) Waiting to run
Scorecard supply-chain security / Scorecard analysis (push) Waiting to run
Cargo Deny / deny (push) Has been cancelled
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-28 01:15:25 +00:00
dependabot[bot]
952562bd5f
chore(deps): bump ajv from 8.17.1 to 8.18.0 in /evals/open-model-gym/mcp-harness ( #7491 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-28 01:15:17 +00:00
Michael Neale
629108d0fc
fix: detect low balance and prompt for top up ( #7166 )
...
Signed-off-by: raj-subhankar <subhankar.rj@gmail.com>
Co-authored-by: Douwe Osinga <douwe@squareup.com>
Co-authored-by: raj-subhankar <subhankar.rj@gmail.com>
2026-02-19 02:20:16 +00:00
dependabot[bot]
d4835cfb1e
chore(deps): bump qs from 6.14.1 to 6.14.2 in /evals/open-model-gym/mcp-harness ( #7184 )
...
Canary / Prepare Version (push) Waiting to run
Canary / build-cli (push) Blocked by required conditions
Canary / Upload Install Script (push) Blocked by required conditions
Canary / bundle-desktop (push) Blocked by required conditions
Canary / bundle-desktop-linux (push) Blocked by required conditions
Canary / bundle-desktop-windows (push) Blocked by required conditions
Canary / Release (push) Blocked by required conditions
Cargo Deny / deny (push) Waiting to run
CI / changes (push) Waiting to run
CI / Check Rust Code Format (push) Blocked by required conditions
CI / Build and Test Rust Project (push) Blocked by required conditions
CI / Lint Rust Code (push) Blocked by required conditions
CI / Check OpenAPI Schema is Up-to-Date (push) Blocked by required conditions
CI / Test and Lint Electron Desktop App (push) Blocked by required conditions
Deploy Documentation / deploy (push) Waiting to run
Live Provider Tests / Smoke Tests (Code Execution) (push) Blocked by required conditions
Live Provider Tests / check-fork (push) Waiting to run
Live Provider Tests / changes (push) Blocked by required conditions
Live Provider Tests / Build Binary (push) Blocked by required conditions
Live Provider Tests / Smoke Tests (push) Blocked by required conditions
Live Provider Tests / Compaction Tests (push) Blocked by required conditions
Publish Ask AI Bot Docker Image / docker (push) Waiting to run
Publish Docker Image / docker (push) Waiting to run
Scorecard supply-chain security / Scorecard analysis (push) Waiting to run
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-12 19:48:07 +00:00
Michael Neale
a484eee8b4
chore: add a new scenario ( #7107 )
2026-02-10 01:49:07 +00:00
dependabot[bot]
3d562f8000
chore(deps): bump @isaacs/brace-expansion from 5.0.0 to 5.0.1 in /evals/open-model-gym/suite ( #7085 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-09 15:18:43 +00:00
dependabot[bot]
5e2fb991df
chore(deps): bump @modelcontextprotocol/sdk from 1.25.3 to 1.26.0 in /evals/open-model-gym/mcp-harness ( #7086 )
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-02-09 15:17:31 +00:00
Michael Neale
a3ba124178
tidy: clean up old benchmark and add gym ( #7081 )
2026-02-09 06:08:46 +00:00
