bk99.de entertain the web since 1997

bk99.de
HF · TrainingCurated

Training LLaMA with RLHF

Summary

StackLLaMA combines supervised fine-tuning, a reward model and PPO into a complete RLHF pipeline. The guide shows how PEFT and quantisation lower the hardware requirements. The pipeline first uses instruction data for supervised fine-tuning.

Ideas

  • RLHF consists of several interdependent models and data phases.
  • A faulty reward model can encourage strategies that look convincing but are unwanted.

Insights

  • Scalable training needs reproducible transitions between code, data, devices and checkpoints.

Facts

  • PPO then optimises answers against a learned reward model.

Critique

  • Preference data reflects the values and blind spots of the people annotating it.

Recommendations

  • Evaluate each training stage separately and look specifically for reward hacking.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2023

Parameter-efficient fine-tuning with PEFT

PEFT bundles LoRA and related methods that train only a small number of additional parameters. This makes large base models adaptable on limited hardware. PEFT supports several parameter-efficient adaptation methods.

HF · 2021

Simplifying distributed training with Accelerate

Accelerate encapsulates device selection, mixed precision and distributed training behind a small interface. Existing PyTorch code is meant to scale without a complete framework rebuild. Accelerate supports CPU, single and multiple GPUs as well as TPU configurations.

HF · 2024

Switching between DeepSpeed and FSDP

Accelerate unifies configurations for DeepSpeed and PyTorch FSDP so that training code stays largely stable. The article compares the concepts and migration paths of both sharding systems. DeepSpeed ZeRO and FSDP distribute model states across several devices.