Pedro Rodriguez fc3399ef40 Update iterator inheritance, pass file format args, limit iterator (#63) ... - Create a common class to use in all inheritance for states - Add a limit iterator that we can use in evals - Modify ArrowFileIterator behavior to not do arrow path inference if file_format='json' - Make EvalArgs valid - Move testing iterators to a common directory to allow usage in multiple test files - Make it so that SequenceIterator can take a None rng_state, to disable all rng ops (for eval mainly) Test Plan: - `pytest bytelatent` - `python -m bytelatent.train config=../internal-blt/configs/entropy_model.yaml logging.wandb=null eval=null`		2025-02-21 16:21:07 -08:00
..
iterators	Update iterator inheritance, pass file format args, limit iterator (#63)	2025-02-21 16:21:07 -08:00
__init__.py	Initial commit	2024-12-12 15:32:30 -08:00
data_types.py	This includes fixes that make checkpointing and reloading work correctly. (#35)	2025-01-27 16:56:42 -08:00
file_util.py	Update file check script to check sizes (#32)	2025-01-22 13:06:46 -08:00
ngram_processor.py	Initial commit	2024-12-12 15:32:30 -08:00
patcher.py	Initial codes and scripts for training entropy model (#34)	2025-01-27 09:46:44 -08:00
test_data.py	Test first batch matches (#53)	2025-02-13 10:05:08 -08:00