mirror of
https://github.com/facebookresearch/blt.git
synced 2025-02-23 13:32:14 +00:00
b2058fb0f6
Summary: - Make arrow iterator able to read from jsonl files, the entropies are omitted in this case - Make the data/checkpoint code fsspec compatible - Fix issues with all reduce with non-bf16 in dist_sum and norm computation. - Minimal fixes to get eval to run, it is slow currently - Add bpb numbers during training Test Plan: Run unit tests and the commands below ``` python -m bytelatent.train config=internal/configs/s3_debug.yaml eval=null checkpoint.dump.every=100 ``` ``` torchrun --nproc-per-node 8 -m bytelatent.train config=internal/configs/s3_debug.yaml eval=null checkpoint.dump.every=100 ``` ``` python -m bytelatent.train config=internal/configs/s3_debug.yaml eval=null checkpoint.dump.every=100 dump_dir=s3://blt/scratch/checkpoint-test/ ``` ``` torchrun --nproc-per-node 8 -m bytelatent.train config=internal/configs/s3_debug.yaml eval=null checkpoint.dump.every=100 dump_dir=s3://blt/scratch/checkpoint-test/ ``` |
||
|---|---|---|
| .. | ||
| configs | ||
| data | ||
| model | ||
| plotting | ||
| preprocess | ||
| tokenizers | ||
| .DS_Store | ||
| __init__.py | ||
| args.py | ||
| base_transformer.py | ||
| checkpoint.py | ||
| constants.py | ||
| distributed.py | ||
| entropy_model.py | ||
| eval.py | ||
| float8.py | ||
| generate.py | ||
| logger.py | ||
| metrics.py | ||
| optim.py | ||
| probe.py | ||
| profiling.py | ||
| stool.py | ||
| test_blt.py | ||
| test_entropy_model.py | ||
| train.py | ||
| transformer.py | ||