mirror of
https://github.com/facebookresearch/blt.git
synced 2025-01-18 16:37:46 +00:00
a1d05403b4
Summary: For compatibility with either local/nfs or S3 datasets, swap to fsspec. Add a tool to compare local and remote filesystems Test Plan: - Ran regular train script - Ran with config with data in S3
24 lines
194 B
Plaintext
24 lines
194 B
Plaintext
numpy
|
|
omegaconf
|
|
msgspec
|
|
rouge-score
|
|
sacrebleu
|
|
sentencepiece
|
|
tiktoken
|
|
fsspec
|
|
blobfile
|
|
wandb
|
|
viztracer
|
|
lm-eval
|
|
scipy
|
|
pynvml
|
|
datatrove
|
|
orjson
|
|
luigi
|
|
pydantic
|
|
altair
|
|
submitit
|
|
typer
|
|
rich
|
|
fsspec[full]
|