red40maxxer
c8a254b825
feat: generic plugin system ( #53 )
...
CI / Check and build (Python 3.10) (push) Has been cancelled
CI / Check and build (Python 3.11) (push) Has been cancelled
CI / Check and build (Python 3.12) (push) Has been cancelled
CI / Check and build (Python 3.13) (push) Has been cancelled
* style: ruff
* feat(wip): populate metadata fields and allow plugins to declare what they need
* refactor: extract metadata logic to separate module
* style: placate ruff
* chore: use eos token for inferring finish reason with fallback
* fix: handle empty responses better
* style: ruff
* refactor: combine response text and metadata into single object
* refactor: clean up tagger and scorer usage
* style: ruff
* chore: remove is_refusal
* style: ruff import ordering
* feat: remove embeddings and generation traces
* feat: return all hidden states instead of just last ones
* chore: remove testing changes
* style: ruff format
* fix: mismatching stop reason identifier
* chore: update default config ordering
* chore: fix merge
* feat: allow external plugin imports
* feat: add good_residuals and bad_residuals to context metadata
* style: ruff
* chore: remove unnecessary allow extra
* chore: remove unnecessary system prompt and model name
* style: ruff
* perf: clear residuals from memory if plugin doesn't need them
* feat: support external filepaths and clean up import logic
* style: ruff
* refactor: consolidate tagger and scorer functionality into a single scorer plugin
* refactor: parent Plugin class for all plugins
* feat: support multiple scorer plugins
* refactor: type fixes
* style: satisfy ruff
* refactor: centralize scorer dataclasses
* refactor: rename MetricResult to Score
* feat: simplify plugin loading
* feat: split response metadata objects and access in evaluationContext
* style: ruff
* style: ruff
* chore: remove old tagger code
* refactor: scorer settings inherit directly from Pydantic
* refactor: move eval prompts and settings to CountRefusals and KLDivergence
* feat: move scorer config to top level and add support for scale factor
* fix: missing config for scorers
* style: ruff
* fix: scale type error
* docs: fix misleading docstring
* fix: clean up old fields
* refactor: use BaseModel for scorer settings
* chore: make scale default to 1 for safety
* refactor: get metadata dynamically through EvaluationContext
* refactor: rename CountRefusals to RefusalRate
* chore: remove unused kl_divergence config fields
* docs: restore missing comment
* refactor: remove unused code
* chore: specify settings and model field types
* refactor: rename to prompts
* refactor: move load_plugin to plugin
* style: ruff
* refactor: update optimization direction config to use StudyDirection directly
* fix: missing TypeVar
* fix: missing imports
* fix: use OptimizationDirection peoperly
* chore: remove names
* chore: remove unecessary future import
* chore: remove unused scorer imports
* refactor: objective should only return tuple of floats
* refactor: use dataclass for scorer config
* feat: support multiple instances of the same scorer
* style: ruff
* fix: nonexistent name attribute in scorer
* refactor: clear residuals and analyser
* docs: MetricResult -> Score
* fix: clean up default toml
* fix: missed renaming to RefusalRate
* chore: missing return ModuleType
* docs: add SPDX header
* docs: add SPDX header
* docs: add SPDX header
* chore: fix misleading field description leftover from old code
* chore: add newline
* chore: unused settings class
* fix: bad import
* refactor: rename ResponseText -> TextCompletion
* feat: simplify api
* refactor: rename to get_score
* feat: namespace scorer configs
* style: ruff
* fix: genericize readme intro
* chore: move init to scorer base class
* refactor: handle direction and scale outside scorer
* chore: use underscore for instance names
* fix: add scorer instance name to scores
* refactor: create structured api for scorers to access model
* refactor: rename plugin-specific Settings to PluginSettings
* feat: add instance name to plugin load logging
* style: ruff
* chore: allow extra fields for plugins
* fix: improve plugin loading logic
* chore: undo change fixed in master
* chore: remove old code
* docs: adjust docstring
* chore: cleanup import
* refactor: unnest plugin settings class and detect from type annotation
* refactor: use plain str instead of Response object with metadata
* refactor: move non evaluator-specific methods out
* refactor: use enum for StudyDirection
* refactor: no strings as type annotations
* chore: let evaluator blow up on error
* refactor: rename metrics to scores globally
* feat: separate cli and hf score displays and clean up readme logic
* fix: direction serialization ValidationError when restoring from save
* refactor: rename scorer start() to setup()
* style: ruff
* fix: remove external plugin test
* refactor: rename setup to init
* docs: formatting
* refactor: move scorers location in config
* docs: add comment describing return tensor shape
* style: ruff
* refactor: simplify scorer setting logic
* refactor: clarify plugin loading logic
* refactor: remove unnecessary hashing and inline import_module
* style: ruff
* fix: don't use classnames for readme
* refactor: don't expose heretic settings to scorer
* fix: adjust print responses logic and move to scorer config level
* refactor: separate baseline score computation
* refactor: rename hf_display to md_display
* style: ruff
* Update src/heretic/scorer.py
Co-authored-by: Philipp Emanuel Weidmann <pew@worldwidemann.com>
* Update src/heretic/scorer.py
Co-authored-by: Philipp Emanuel Weidmann <pew@worldwidemann.com>
* style: ruff
* fix: ty error
* refactor: bind Score names to parent Scorers as class property
* docs: fix doc
* docs: update comment
* style: remove changes
* chore: define default refusal markers
* style: ruff
* style: remove whitespace changes
* docs: tweak docs
* chore: cleanup from merge
* style: ruff
* fix: handle negative floating point kld
* style: formatting
* chore: remove unused code
* chore: ruff
* style: undo line removal
* style: update formatting and remove old comment
* docs: undo style change
* docs: update field description
* docs: tweak docstring
* chore: revert kld absolute value forcing
* style: ruff
* chore: cleanup
* docs: update header
* docs: update header
* refactor: remove unnecessary conditional imports
* fix: apply review omments on refusalrate
* refactor: move contract validation to plugin
* refactor: move Context to Plugin
* refactor: move init to plugin level
* refactor: move init() to plugin
* style: ruff
* docs: update SPDX header
* refactor: derive score name from scorer.score_name
* chore: no None option for baseline_score_displays
* fix: show CLI formatted metrics in trial selection
* fix: sort trials by scores
* chore: remove unnecessary from future import
* chore: remove scorer scale field
* refactor: import Context from plugin
* docs: add quote to direction
* refactor: move model_config to the end of the class
* refactor: use dataclass for consistency
* refactor: use BaseModel and store study direction as str
* docs: move docstring location
* refactor: combine scorer load and init
* refactor: use best_trials for single and multi-objective
* refactor: remove all .get()
* refactor: remove unused dataclass
* refactor: use ScorerEntry dataclass for improved code quality
* style: ruff
* chore: adapt reproducibility to plugin architecture
* chore: address PR comments
* chore: make `ScorerConfig` fields full `Field()`
* chore: address pr comments
* feat: bump to version 3 of reproduce json
* refactor: rename direction to optimization
* refactor: rename loop var
* feat: pin to dataset commit sha for reproducibility
* style: ruff
* feat: show metric as list instead of table
* chore: remove stale comment
* chore: resync with upstream
* fix: trial title formatting
* chore: single source of truth for optimization objective ordering
* feat: fail-fast when there are no optimization objectives
* chore: remove dead `verify_hashes`
* refactor: pair scores with baselines everywhere
* fix: bug
* chore: add recommendation to install heretic 1.4 for older reproduce files
* chore: adapt nohumor and noslop config files to new format
* refactor: rename refusals to residuals everywhere
* fix: merge issues
* fix: fix test configs
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Apply suggestion from @gemini-code-assist[bot]
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* style: ruff
* chore: validate `instance_name` early
* chore: add return type for `load_prompts`
* docs: comment typo
* docs: comments
* docs: comments
* chore: comments and spacing
* docs: comments
* Update src/heretic/evaluator.py
Co-authored-by: Vinay Umrethe <umrethevinay@gmail.com>
* refactor: rename `cli_display` to `rich_display`
* style: ruff
* fix: don't repro external plugins or local datasets
* test: adapt minicpm5 to scorer-based format
* test: adapt qwen2.5 to scorer based format
* chore: restore comment
* chore: address pr comments
* chore: remove stale `keyword_markers`
* chore: string
* style: ruff
* refactor: make KLD and keyword rate scorers default
---------
Co-authored-by: mad-cat-lon <113548315+mad-cat-lon@users.noreply.github.com>
Co-authored-by: Philipp Emanuel Weidmann <pew@worldwidemann.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Vinay Umrethe <umrethevinay@gmail.com>
2026-07-07 14:34:33 +05:30
Philipp Emanuel Weidmann
0146b2760f
feat: headless operation + end-to-end tests ( #392 )
...
CI / Check and build (Python 3.10) (push) Waiting to run
CI / Check and build (Python 3.11) (push) Waiting to run
CI / Check and build (Python 3.12) (push) Waiting to run
CI / Check and build (Python 3.13) (push) Waiting to run
* fix: remove notebook input shims
Closes #280
* feat: support headless operation (no interactive input)
* fix: prevent infinite loops
* feat: add end-to-end tests
* ci: run tests in CI
* ci: fix test output ordering
* fix: replace home-cooked `set_seed` function with Transformers builtin
* feat: print PyTorch config when running tests
* feat: print additional information
* experiment: try to standardize test environment
* fix: revert environment changes
* feat: support multiple valid hashes for each output file
* feat: add test output hashes for CI
* feat: add test output hashes for CI (alternative environment)
* feat: add hashes for Windows (#394 )
* fix: Hash on windows
* trigger ci
* fix: prefer .yaml (used widely than .toml for model configs)
* use removeprefix
* docs: restore commet
* use removeprefix again
* tests: Add windows hash files for all test models
* trigger ci
* fix: minor cleanup
* clean merge mismatch
* remove unnecessary CRLF replace, now that we support more SUMS files
* fix: use binary mode for hashes everywhere
---------
Co-authored-by: Vinay Umrethe <umrethevinay@gmail.com>
2026-06-27 13:41:48 +05:30
Philipp Emanuel Weidmann
6757ada999
fix: minor cleanups and improvements
CI / Check and build (Python 3.10) (push) Waiting to run
CI / Check and build (Python 3.11) (push) Waiting to run
CI / Check and build (Python 3.12) (push) Waiting to run
CI / Check and build (Python 3.13) (push) Waiting to run
2026-06-13 19:48:38 +05:30
Vinay-Umrethe
c9ce36ddde
style: remove annoying gray bg from logo ( #359 )
2026-06-07 08:33:40 +05:30
Philipp Emanuel Weidmann
4e3a3a78a3
docs: update README
2026-05-22 14:51:24 +05:30
Philipp Emanuel Weidmann
79ea9ce905
docs: update README
CI / Check and build (Python 3.10) (push) Waiting to run
CI / Check and build (Python 3.11) (push) Waiting to run
CI / Check and build (Python 3.12) (push) Waiting to run
CI / Check and build (Python 3.13) (push) Waiting to run
2026-05-03 09:08:57 +05:30
Philipp Emanuel Weidmann
cb4ef3fdfc
docs: add Trendshift badge to README
CI / Check and build (Python 3.10) (push) Has been cancelled
CI / Check and build (Python 3.11) (push) Has been cancelled
CI / Check and build (Python 3.12) (push) Has been cancelled
CI / Check and build (Python 3.13) (push) Has been cancelled
2026-02-20 13:00:19 +05:30
Philipp Emanuel Weidmann
1179013999
docs: update README
2026-02-14 16:32:08 +05:30
Philipp Emanuel Weidmann
fe7bc1bae3
docs: update README
CI / Check and build (Python 3.10) (push) Waiting to run
CI / Check and build (Python 3.11) (push) Waiting to run
CI / Check and build (Python 3.12) (push) Waiting to run
CI / Check and build (Python 3.13) (push) Waiting to run
2026-02-14 10:47:28 +05:30
Philipp Emanuel Weidmann
dd0b3a2f69
docs: update README
CI / Check and build (Python 3.10) (push) Has been cancelled
CI / Check and build (Python 3.11) (push) Has been cancelled
CI / Check and build (Python 3.12) (push) Has been cancelled
CI / Check and build (Python 3.13) (push) Has been cancelled
2026-02-11 11:09:17 +05:30
Philipp Emanuel Weidmann
10ceb3098e
chore: update copyright notice
2026-02-11 09:46:36 +05:30
Philipp Emanuel Weidmann
ca783db6c9
docs: update README
2025-12-10 16:30:35 +05:30
Philipp Emanuel Weidmann
22a4a5b5b5
Add citation information to README
CI / Check and build (Python 3.10) (push) Waiting to run
CI / Check and build (Python 3.11) (push) Waiting to run
CI / Check and build (Python 3.12) (push) Waiting to run
CI / Check and build (Python 3.13) (push) Waiting to run
2025-11-19 12:14:17 +05:30
Philipp Emanuel Weidmann
3f346b6150
Change package name
2025-11-16 17:01:50 +05:30
Philipp Emanuel Weidmann
1a59d226c1
Fix spacing after images in README
2025-11-16 16:06:08 +05:30
Philipp Emanuel Weidmann
12ecf50033
Add README
2025-11-16 15:19:27 +05:30
Philipp Emanuel Weidmann
af19fbd254
Initial commit
2025-09-21 11:10:30 +05:30
