Pedro Rodriguez a1d05403b4 Replace regular filesystem calls with fsspec + add s3 support ... Summary: For compatibility with either local/nfs or S3 datasets, swap to fsspec. Add a tool to compare local and remote filesystems Test Plan: - Ran regular train script - Ran with config with data in S3		2025-01-10 01:04:18 +00:00
..
create_env.sh	Initial commit	2024-12-12 15:32:30 -08:00
download_prepare_hf_data.py	Replace regular filesystem calls with fsspec + add s3 support	2025-01-10 01:04:18 +00:00
download_tokenizer.py	Initial commit	2024-12-12 15:32:30 -08:00