vrr/heretic
2
0
Fork 0
mirror of https://github.com/p-e-w/heretic.git synced 2026-07-09 17:28:29 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Support Phi-3.5-MoE

Browse source
This commit is contained in:
Philipp Emanuel Weidmann 2025-10-14 11:23:53 +05:30
parent 8b827ee386
commit 7ed0cb1ffb
No known key found for this signature in database
GPG key ID: CE6A7756AD520E0F
1 changed files with 6 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

6
src/heretic/model.py
View file

@ -106,6 +106,12 @@ class Model:
for expert in layer.mlp.experts:
try_add(expert.down_proj.weight)
# Phi-3.5-MoE (and possibly others).
if not matrices:
with suppress(Exception):
for expert in layer.block_sparse_moe.experts:
try_add(expert.w2.weight)
# We need at least one MLP down-projection.
assert matrices