vikarti.anatra/blt
1
0
Fork 0
mirror of https://github.com/facebookresearch/blt.git synced 2025-02-23 13:32:14 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
blt/bytelatent/data/iterators
History
Pedro Rodriguez fc3399ef40
Some checks failed
Lint with Black / lint (push) Has been cancelled
Details
Lint with isort / lint (push) Has been cancelled
Details
Update iterator inheritance, pass file format args, limit iterator (#63) 
- Create a common class to use in all inheritance for states
- Add a limit iterator that we can use in evals
- Modify ArrowFileIterator behavior to not do arrow path inference if file_format='json'
- Make EvalArgs valid
- Move testing iterators to a common directory to allow usage in multiple test files
- Make it so that SequenceIterator can take a None rng_state, to disable all rng ops (for eval mainly)

Test Plan:

- `pytest bytelatent`
- `python -m bytelatent.train config=../internal-blt/configs/entropy_model.yaml logging.wandb=null eval=null`
2025-02-21 16:21:07 -08:00
..
__init__.py Initial commit 2024-12-12 15:32:30 -08:00
abstract_iterator.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
arrow_iterator.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
dev_iterators.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
limit_iterator.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
looping_iterator.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
multiprocess_iterator.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
packing_iterator.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
preprocess_iterator.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
sampling_iterator.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
sequence_iterator.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
test_arrow_iterator.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
test_iters.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00
test_limit_iterator.py Update iterator inheritance, pass file format args, limit iterator (#63) 2025-02-21 16:21:07 -08:00