bk99.de entertain the web since 1997

bk99.de
HF · EvaluationCurated

Comparing RoBERTa, Llama 2 and Mistral with LoRA

Summary

The article compares three model families on classifying disaster tweets with LoRA. It shows that a larger generative model is not automatically the best or most economical classification solution. The study uses the Disaster Tweets data set.

Ideas

  • Task-specific encoders remain strong and efficient baselines for classification.
  • Parameter-efficient training makes model comparisons cheaper, but not automatically fair.

Insights

  • A fair model comparison needs identical data, metrics, budgets and troubleshooting.

Facts

  • RoBERTa, Llama 2 and Mistral are adapted for sequence classification with LoRA.

Critique

  • A single English data set is not enough for general statements about model families.

Recommendations

  • Compare accuracy, calibration, runtime and memory against a simple encoder baseline.

References

Read the original article on Hugging Face

Search the Web Archive

HF · 2024

Serving thirty LoRA models with one server

Text Generation Inference loads many LoRA adapters on top of a shared base model and serves them together. This avoids complete model copies and improves utilisation. The demonstration serves 30 LoRA models through one TGI instance.

HF · 2023

Training LLaMA with RLHF

StackLLaMA combines supervised fine-tuning, a reward model and PPO into a complete RLHF pipeline. The guide shows how PEFT and quantisation lower the hardware requirements. The pipeline first uses instruction data for supervised fine-tuning.

HF · 2023

Adapting Stable Diffusion efficiently with LoRA

LoRA trains small additional matrices instead of all the weights of a diffusion model. This lowers memory requirements and produces compact adapters that can be distributed separately. LoRA freezes the original model weights.