heretic/tests
Vinay Umrethe aecd4a845a
Some checks failed
CI / Check and build (Python 3.10) (push) Has been cancelled
CI / Check and build (Python 3.11) (push) Has been cancelled
CI / Check and build (Python 3.12) (push) Has been cancelled
CI / Check and build (Python 3.13) (push) Has been cancelled
CI / Check and build (Python 3.14) (push) Has been cancelled
CI / Check and build (Python 3.15) (push) Has been cancelled
fix: Update model commit for the tiny classifier model (#584)
2026-11-20 00:25:17 +05:30
..
deepseek-v4 fix: Improve response prefix logic, handle default behaviors, and clarify documentation (#523) 2026-10-26 23:43:56 +05:30
deepseek-v4-ara feat: add config.nosyco.toml (#349) 2026-10-30 20:28:40 +05:30
gemma-3 fix: Update model commit for the tiny classifier model (#584) 2026-11-20 00:25:17 +05:30
gemma-4e chore: add new Windows test hashes (#470) 2026-10-02 15:00:57 +05:30
gpt-oss perf(ara): compute closure outputs via low-rank update instead of dense weight (#499) 2026-10-29 20:39:42 +05:30
minicpm5 chore: add new Windows test hashes (#470) 2026-10-02 15:00:57 +05:30
mistral-3 fix: Improve response prefix logic, handle default behaviors, and clarify documentation (#523) 2026-10-26 23:43:56 +05:30
qwen2.5 chore: add new Windows test hashes (#470) 2026-10-02 15:00:57 +05:30
qwen3.5-moe fix: support flattened hyper-connection residual streams (#525) 2026-10-28 22:13:40 +05:30
seed-oss perf(ara): compute closure outputs via low-rank update instead of dense weight (#499) 2026-10-29 20:39:42 +05:30
README.md feat: Generic BERT Classifier scorer plugin (#500) 2026-10-31 22:07:54 +05:30
run_tests.py chore: update dependencies (#469) 2026-10-02 14:49:47 +05:30
test_ara_lowrank_equivalence.py perf(ara): compute closure outputs via low-rank update instead of dense weight (#499) 2026-10-29 20:39:42 +05:30
test_chat_template.py fix: support models whose chat template rejects the system role (#498) 2026-10-20 21:47:51 +05:30
test_config.py feat: allow specifying a custom config file path (#557) 2026-11-03 17:53:18 +05:30
test_keyword_rate.py fix: reject blank refusal keyword markers (#512) 2026-10-23 18:55:58 +05:30
test_plugin.py fix: export plugin settings correctly (#570) 2026-11-07 22:12:32 +05:30
test_residual_streams.py fix: support flattened hyper-connection residual streams (#525) 2026-10-28 22:13:40 +05:30

Test Suite Guide

Whenever we change any code-logic related to src/heretic/model.py or config.toml (e.g. row_normalization, full_normalization_lora_rank, winsorization_quantile, etc) which can affect a model's reproduciblity; Use these tests which are designed to verify that those changes does not affect reproducibility, unless they are meant to (like when we'll integrate ARA branch in future).

How to test

  1. Choose any model from tiny-random org which provides tiny models useful for debugging.

Example: tiny-random/minicpm5.

Note

It is highly recommended to use a model which does not have a special_tokens_map.json file in the repo. Because those files are almost always wrong in tiny-random/* models compared to the original model.

  1. Clone that model repository using Git and generate the SHA256 hashes using sha256sum:

On Linux:

sha256sum -b * > ../SHA256SUMS.LABEL

On Windows:

sha256sum * | Out-File -Encoding utf8NoBOM ../SHA256SUMS.LABEL

Tip

On windows, sha256sum is generally pre-installed by Git for windows.

Verify with:

Get-Command sha256sum

Expected:

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     sha256sum.exe                                      0.0.0.0    C:\Program Files\Git\usr\bin\sha256sum...

Note

You must use Windows Powershell v7.X not the core which is v5.1. This is required for -Encoding utf8NoBOM to work.

See Differences between Windows PowerShell 5.1 and PowerShell 7.x documentation.

Where LABEL describes the type of system you are running the tests on.

Example:

  • SHA256SUMS.windows (For windows)
  • SHA256SUMS.ci (For GitHub CI)
  • SHA256SUMS.linux (For linux)
  1. Run the tests with:
uv run run_tests.py

The output hashes should FAIL against the Valid hashes in SHA256SUMS file of the test model you added. This is expected since Heretic changes the model. Without Step 2, the test model's folder will simply be ignored because it will not have a hash SUMS file to compare against.

  1. After that go to the output TEST_MODEL_DIR/model folder and re-generate the Actual hashes based on the system you are using.
cd TEST_MODEL_DIR/model
sha256sum -b * > ../SHA256SUMS.LABEL # or use windows command.
  1. Re-run the tests with:
uv run run_tests.py

This time the tests should PASS because we added the new hashes which are expected to be reproduced on the same system.

  1. After that push the SHA256SUMS.LABEL files and wait for GitHub CI actions to run those tests.

Since PyTorch does not guarantee exact cross-system reproducibility regardless of configuration, multiple valid hashes can be provided for each output file. The above update must be performed for each TEST_MODEL_DIR and on each type of system.

For this, copy the Actual hash value for each mismatched unidentical file into a SHA256SUMS.ci file.

  1. After that push the SHA256SUMS.ci files and wait for GitHub CI actions to re-run those tests.

This time the tests should PASS because we added the new hashes which are expected to be reproduced on CI.