bk99.de entertain the web since 1997

bk99.de
HF · TrainingCurated

Switching between DeepSpeed and FSDP

Summary

Accelerate unifies configurations for DeepSpeed and PyTorch FSDP so that training code stays largely stable. The article compares the concepts and migration paths of both sharding systems. DeepSpeed ZeRO and FSDP distribute model states across several devices.

Ideas

  • Interchangeable backends prevent training logic from being tied to one distributed runtime provider.
  • The same API does not mean the same memory, checkpoint or performance semantics.

Insights

  • Scalable training needs reproducible transitions between code, data, devices and checkpoints.

Facts

  • Accelerate can choose either backend through configuration instead of extensive code changes.

Critique

  • Abstraction can only partly hide differences in state formats, offloading and troubleshooting.

Recommendations

  • Test checkpoint restore and numerical convergence with every backend switch.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2024

Merging LoRA adapters

PEFT adds methods for combining several trained adapters. This allows specialisations to be mixed without training a complete model from scratch each time. PEFT supports several merge methods for LoRA adapters.

HF · 2023

Training LLaMA with RLHF

StackLLaMA combines supervised fine-tuning, a reward model and PPO into a complete RLHF pipeline. The guide shows how PEFT and quantisation lower the hardware requirements. The pipeline first uses instruction data for supervised fine-tuning.

HF · 2023

Parameter-efficient fine-tuning with PEFT

PEFT bundles LoRA and related methods that train only a small number of additional parameters. This makes large base models adaptable on limited hardware. PEFT supports several parameter-efficient adaptation methods.