Bringing Elastically Deterministic Data Loading to TorchTitan with Zephon - PyTorch
Zephon integrates with TorchTitan to preprocess training data on the fly and preserve identical global batches across GPU counts when the data recipe stays fixed.
PyTorch · bsterling · https://www.facebook.com/pytorch