blt/requirements.txt

numpy
omegaconf
msgspec
rouge-score
sacrebleu
sentencepiece
tiktoken
fsspec
blobfile
wandb
viztracer
lm-eval
scipy
pynvml
datatrove
orjson
luigi
pydantic
altair
submitit
typer
rich
fsspec[full]
Initial commit 2024-12-12 23:32:30 +00:00			`numpy`
			`omegaconf`
			`msgspec`
			`rouge-score`
			`sacrebleu`
			`sentencepiece`
			`tiktoken`
			`fsspec`
			`blobfile`
			`wandb`
			`viztracer`
			`lm-eval`
			`scipy`
			`pynvml`
			`datatrove`
			`orjson`
			`luigi`
			`pydantic`
			`altair`
			`submitit`
			`typer`
			`rich`
Replace regular filesystem calls with fsspec + add s3 support Summary: For compatibility with either local/nfs or S3 datasets, swap to fsspec. Add a tool to compare local and remote filesystems Test Plan: - Ran regular train script - Ran with config with data in S3 2025-01-10 01:02:25 +00:00			`fsspec[full]`