mirror of
https://github.com/block/goose.git
synced 2026-07-09 16:09:22 +00:00
Update release testing instructions (#10122)
This commit is contained in:
Jack Amadeo
GitHub
parent
f150672e23
commit
363b309861
4 changed files with 20 additions and 125 deletions
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
## Test before Release
|
||||
|
||||
1. Close and reopen this PR to trigger CI and produce the Desktop bundle for testing. Reason: workflows don't run on PRs opened by `GITHUB_TOKEN` ([docs](https://docs.github.com/en/actions/using-workflows/triggering-a-workflow#triggering-a-workflow-from-a-workflow)).
|
||||
1. Approve the workflows for this PR (scroll to bottom) to trigger CI and produce the Desktop bundle for testing.
|
||||
2. Make sure all check workflows pass.
|
||||
3. Install the Desktop bundle from the download links (posted as a comment below when it is ready).
|
||||
4. Complete the goose Release Manual Testing Checklist (posted as a comment below).
|
||||
|
|
|
|||
|
|
@ -85,7 +85,7 @@ jobs:
|
|||
|
||||
**Please follow these steps:**
|
||||
|
||||
1. Close and reopen this PR to trigger CI checks. Reason: workflows don't run on PRs opened by `GITHUB_TOKEN` ([docs](https://docs.github.com/en/actions/using-workflows/triggering-a-workflow#triggering-a-workflow-from-a-workflow)).
|
||||
1. Approve workflows for this PR to trigger CI checks.
|
||||
2. Review and resolve any merge conflicts.
|
||||
3. Approve and merge this PR.
|
||||
4. The `release/${{ env.version }}` PR will be created automatically.
|
||||
|
|
|
|||
|
|
@ -1,134 +1,23 @@
|
|||
# goose Release Manual Testing Checklist
|
||||
|
||||
## Version: {{VERSION}}
|
||||
|
||||
### Identify the high risk changes in this Release
|
||||
Download the release builds from this PR. Once a build is ready, the actions bot will post a comment on this PR
|
||||
with instructions on how to download and sign.
|
||||
|
||||
## Use the following script to create a risk assessment and testing plan:
|
||||
```
|
||||
./workflow_recipes/release_risk_check/run.sh {{VERSION}}
|
||||
```
|
||||
|
||||
It will generate an analysis report in `/tmp/release_report_final.md` and perform testing is necessary for high risk pr changes.
|
||||
It will generate an analysis report in `/tmp/release_report_final.md` and perform testing is necessary for high risk pr changes.
|
||||
|
||||
## Run the goose self-test recipe
|
||||
|
||||
## Regression Testing
|
||||
goose run --recipe goose-self-test.yaml
|
||||
|
||||
Make a copy of this document for each version and check off as steps are verified.
|
||||
## Have goose produce a test plan
|
||||
|
||||
### Provider Testing
|
||||
Open the release candidate desktop app and have goose produce a test plan by pointing it at this PR. Use a prompt like
|
||||
|
||||
- [ ] Run `cd ui/desktop && pnpm run test:integration:providers` locally from the release branch and verify all providers/models work
|
||||
- [ ] Launch goose, click reset providers, choose databricks and a model
|
||||
> Look at the notes in PR <release PR> and the report at `/tmp/release_report_final.md` and investigate potential risks in this release. After familiarizing yourself with the scope of each change, produce a suggested test plan that I should follow before publishing the release.
|
||||
|
||||
### Starting Conversations
|
||||
|
||||
Test various ways to start a conversation:
|
||||
|
||||
- [ ] Open home and start a new conversation with "Hello"
|
||||
- [ ] Agent responds
|
||||
- [ ] Token count is updated after agent finishes
|
||||
- [ ] Go to history and see there is a new entry
|
||||
- [ ] Go back to the main screen, start a new conversation from the hub and see that it opens a new conversation
|
||||
- [ ] Open history and click the Hello conversation - verify it loads
|
||||
- [ ] Add a new message to this conversation and see that it is added
|
||||
- [ ] Change the working directory of an existing conversation
|
||||
- [ ] Ask "what is your working directory?"
|
||||
- [ ] Response should match the new directory
|
||||
- [ ] Open a new window, click chat in left side for new chat
|
||||
- [ ] Type "create a tamagotchi game" in the chat input to test developer extension
|
||||
|
||||
### Recipes
|
||||
|
||||
#### Create Recipe from Session
|
||||
|
||||
- [ ] Start a simple chat conversation like "hi"
|
||||
- [ ] Click "create a recipe from this session" in the bottom chat bar
|
||||
- [ ] Recipe title, description and instructions should be filled in with details from the chat
|
||||
- [ ] Add a few activities and params (params unused indicator should update if added to instructions/prompts or activities)
|
||||
- [ ] Can launch create and run recipe - launches in a new window showing as a recipe agent chat with parameters filled in and interact with it
|
||||
- [ ] Recipe should be saved in recipe library
|
||||
|
||||
#### Use Existing Recipe
|
||||
|
||||
- [ ] Pick trip planner from recipe hub (go/gooserecipes)
|
||||
- [ ] See the warning whether to trust this recipe (only on fresh install)
|
||||
- [ ] See the form pop up
|
||||
- [ ] Fill in the form with "Africa" and "14 days"
|
||||
- [ ] Check results are reasonable
|
||||
- [ ] Ask how many days the trip is for - should say 14
|
||||
|
||||
#### Recipe Management
|
||||
|
||||
- [ ] Go to recipe manager and enter a new recipe to generate a joke
|
||||
- [ ] See that it works if you run it
|
||||
- [ ] Edit the recipe by bottom bar and click "View/Edit Recipe"
|
||||
- [ ] Make it generate a limerick instead
|
||||
- [ ] Check that the updated recipe works
|
||||
- [ ] Delete the recipe from the recipe manager
|
||||
- [ ] Verify recipe is actually deleted
|
||||
|
||||
#### Recipe from File
|
||||
|
||||
- [ ] Create a file `~/.config/goose/recipes/test-recipe.yaml` with the following content:
|
||||
|
||||
```yaml
|
||||
recipe:
|
||||
title: test recipe again
|
||||
description: testing recipe again
|
||||
instructions: The value of test_param is {{test_param}}
|
||||
prompt: What is the value of test_param?
|
||||
parameters:
|
||||
- key: test_param
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Enter value for test_param
|
||||
```
|
||||
|
||||
- [ ] See that it shows up in the list of installed recipes
|
||||
- [ ] Launch the recipe, see that it asks for test_param
|
||||
- [ ] Enter a number, see that it pre-fills the prompt and tells you the value after you hit submit
|
||||
- [ ] Go to hub and enter "what is the value of test_param"
|
||||
- [ ] See a new chat that says it has no idea (recipe is no longer active)
|
||||
|
||||
### Extensions
|
||||
|
||||
#### Manual Extension Addition
|
||||
|
||||
- [ ] Can manually add an extension using random quotes from project
|
||||
- [ ] Add new custom stdio extension with the following command and save:
|
||||
- [ ] `node /ABSOLUTE/PATH/TO/goose/ui/desktop/tests/e2e/basic-mcp.ts` (use your actual project path)
|
||||
- [ ] Should add and can chat to ask for a random quote
|
||||
|
||||
#### Playwright Extension
|
||||
|
||||
- [ ] Install the playwright extension from the extensions hub
|
||||
- [ ] Tell it to open a browser and search on Google for cats
|
||||
- [ ] Verify that the browser opens and navigates
|
||||
|
||||
#### Extension with Environment Variables
|
||||
|
||||
- [ ] Install an extension from deeplink that needs env variables:
|
||||
- [ ] Use: `goose://extension?cmd=npx&arg=-y&arg=%40upstash%2Fcontext7-mcp&id=context7&name=Context7&description=Use%20up-to-date%20code%20and%20docs&env=TEST_ACCESS_TOKEN`
|
||||
- [ ] Extension page should load with env variables modal showing
|
||||
- [ ] Allow form input and saving extension
|
||||
|
||||
### Speech-to-Text (Local Model)
|
||||
|
||||
- [ ] Go to Settings > Chat > Voice dictation provider and select the small model
|
||||
- [ ] Run a quick test that speech-to-text is working (click the mic button, speak, verify transcription)
|
||||
- [ ] Also try OpenAI using your OpenAI key
|
||||
|
||||
### Settings
|
||||
|
||||
- [ ] Settings page loads and all tabs load
|
||||
- [ ] Can change dark mode setting
|
||||
|
||||
### Follow-up Issues
|
||||
|
||||
Link any GitHub issues filed during testing:
|
||||
|
||||
---
|
||||
|
||||
**Tested by:** _____
|
||||
**Date:** _____
|
||||
**Notes:** _____
|
||||
goose will produce a plan. Follow this plan to finish testing.
|
||||
|
|
|
|||
|
|
@ -114,6 +114,12 @@ extensions:
|
|||
timeout: 600
|
||||
bundled: true
|
||||
description: Core tool for file operations, shell commands, and code analysis
|
||||
- type: builtin
|
||||
name: todo
|
||||
- type: builtin
|
||||
name: summon
|
||||
- type: builtin
|
||||
name: extensionmanager
|
||||
|
||||
prompt: |
|
||||
Execute the Goose Self-Testing Integration Suite in {{ workspace_dir }}.
|
||||
|
|
@ -171,7 +177,7 @@ prompt: |
|
|||
{% endif %}
|
||||
|
||||
{% if test_phases == "all" or "delegation" in test_phases %}
|
||||
## 🤖 PHASE 3: Delegate & Load Testing
|
||||
## 🤖 PHASE 3: Summon Testing
|
||||
|
||||
### Load Tool - Discovery Mode
|
||||
Call `load()` with no arguments to discover all available sources:
|
||||
|
|
@ -326,7 +332,7 @@ prompt: |
|
|||
### Vision Smoke Test
|
||||
1. Create a small test image:
|
||||
```
|
||||
python3 -c "import struct, zlib; raw=b'\x00\xff\x00\x00'; d=zlib.compress(raw); ihdr=b'\x00\x00\x00\x01\x00\x00\x00\x01\x08\x02\x00\x00\x00'; print('Created test.png')"
|
||||
python3 -c "import struct, zlib; raw=b'\x00\xff\x00\x00'; d=zlib.compress(raw); ihdr=b'\x00\x00\x00\x01\x00\x00\x00\x01\x08\x02\x00\x00\x00'; print('Created test.png')"
|
||||
```
|
||||
Or simply create a 1-pixel PNG test image using available tools.
|
||||
2. Verify the test image file exists and is valid.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue