Pedro Rodriguez 0ffe2ab685 Update iterator inheritance, pass file format args, limit iterator ... - Create a common class to use in all inheritance for states - Add a limit iterator that we can use in evals - Modify ArrowFileIterator behavior to not do arrow path inference if file_format='json' - Make EvalArgs valid - Move testing iterators to a common directory to allow usage in multiple test files - Make it so that SequenceIterator can take a None rng_state, to disable all rng ops (for eval mainly) Test Plan: - `pytest bytelatent` - `python -m bytelatent.train config=../internal-blt/configs/entropy_model.yaml logging.wandb=null eval=null`		2025-02-20 00:57:17 +00:00
..
__init__.py	Initial commit	2024-12-12 15:32:30 -08:00
abstract_iterator.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
arrow_iterator.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
dev_iterators.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
limit_iterator.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
looping_iterator.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
multiprocess_iterator.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
packing_iterator.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
preprocess_iterator.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
sampling_iterator.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
sequence_iterator.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
test_arrow_iterator.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
test_iters.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00
test_limit_iterator.py	Update iterator inheritance, pass file format args, limit iterator	2025-02-20 00:57:17 +00:00