Pedro Rodriguez 203bff3696 Pass mask in packing_iterator, correctly handle last batch, fix masking ... This commit does/fixes the following: 1. Adds unit tests for byte and patch packing to ensure it works correctly 2. Fixes a bug where for batches that end up with <max_length number of bytes (e.g., short patches), the mask was including elements that had value pad_id. This fixes the mask by setting it to be !=pad_id, if its not specified. 3. Correctly handles the last batch, where previously it would crash. This didn't affect training since we had enough data and/or looped iterators, but for evaluation perplexity, it comes up if we validation on an entire file. 4. Correctly forward the mask if it exists for byte packing Test Plan: ``` pytest bytelatent/ ``` Testing these changes more thoroughly in a stacked PR that fixes evals		2025-02-22 01:27:13 +00:00
..
iterators	Pass mask in packing_iterator, correctly handle last batch, fix masking	2025-02-22 01:27:13 +00:00
__init__.py	Initial commit	2024-12-12 15:32:30 -08:00
data_types.py	This includes fixes that make checkpointing and reloading work correctly. (#35)	2025-01-27 16:56:42 -08:00
file_util.py	Update file check script to check sizes (#32)	2025-01-22 13:06:46 -08:00
ngram_processor.py	Initial commit	2024-12-12 15:32:30 -08:00
patcher.py	Initial codes and scripts for training entropy model (#34)	2025-01-27 09:46:44 -08:00
test_data.py	Test first batch matches (#53)	2025-02-13 10:05:08 -08:00