vikarti.anatra/blt
1
0
Fork 0
mirror of https://github.com/facebookresearch/blt.git synced 2025-04-17 07:09:09 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
blt/bytelatent/data
History
Pedro Rodriguez 936d9437be
Allow ArrowIterator to read from json () 
Summary:

Currently, arrow iterator can only read arrow files. However, the pyarrow library can read
other formats, including jsonlines. This allows the same ArrowIterator to read from jsonlines,
so we can read from the original source data, and simply omit the entropy column when doing so

Test Plan:

Run train script until dataloader starts
2025-02-06 09:57:22 -08:00
..
iterators Allow ArrowIterator to read from json () 2025-02-06 09:57:22 -08:00
__init__.py Initial commit 2024-12-12 15:32:30 -08:00
data_types.py This includes fixes that make checkpointing and reloading work correctly. () 2025-01-27 16:56:42 -08:00
file_util.py Update file check script to check sizes () 2025-01-22 13:06:46 -08:00
ngram_processor.py Initial commit 2024-12-12 15:32:30 -08:00
patcher.py Initial codes and scripts for training entropy model () 2025-01-27 09:46:44 -08:00