vikarti.anatra/blt
1
0
Fork 0
mirror of https://github.com/facebookresearch/blt.git synced 2025-02-23 05:22:16 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
blt/bytelatent/preprocess
History
Pedro Rodriguez 9c3c997cae
Some checks failed
Lint with Black / lint (push) Has been cancelled
Details
Lint with isort / lint (push) Has been cancelled
Details
Allow ArrowIterator to read from json 
Summary:

Currently, arrow iterator can only read arrow files. However, the pyarrow library can read
other formats, including jsonlines. This allows the same ArrowIterator to read from jsonlines,
so we can read from the original source data, and simply omit the entropy column when doing so

Test Plan:

Run train script until dataloader starts
2025-02-06 17:44:36 +00:00
..
__init__.py Initial commit 2024-12-12 15:32:30 -08:00
data_pipeline.py Initial commit 2024-12-12 15:32:30 -08:00
fsspec_target.py Changes for training entropy model and correcting attention in local models (#25) 2025-01-17 14:23:01 -08:00
parallel_entropies.py Initial commit 2024-12-12 15:32:30 -08:00
preprocess_entropies.py Allow ArrowIterator to read from json 2025-02-06 17:44:36 +00:00