Llama 2

Title and Authors:

• Title: "Llama 2: Open Foundation and Fine-Tuned Chat Models"
• Authors: Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom.

Abstract Summary:

• The paper introduces Llama 2, a collection of pretrained and fine-tuned large language models ranging from 7 billion to 70 billion parameters, optimized for dialogue use cases. Llama 2-Chat models outperform existing open-source models on most benchmarks and offer a viable alternative to some closed-source models, with detailed fine-tuning and safety improvements shared to encourage responsible development of large language models.

Key Concepts:

• Large Language Models (LLMs)
• Pretrained and fine-tuned models
• Llama 2-Chat
• Dialogue optimization
• Safety and helpfulness benchmarks
• Reinforcement Learning with Human Feedback (RLHF)
• Supervised Fine-Tuning (SFT)
• Model safety and evaluation

Problem Statement:

• The main problem addressed by the paper is the development of open, high-performing language models that can serve as viable alternatives to closed-source models, particularly optimized for dialogue use cases while ensuring safety and helpfulness.

Methods and Techniques:

1. Pretraining:
   • Utilized a large corpus of publicly available data, excluding data from Meta’s products and services, with robust data cleaning and increased context length.
   • Applied the AdamW optimizer, cosine learning rate schedule, and techniques like grouped-query attention for better scalability.
2. Supervised Fine-Tuning (SFT):
   • Collected high-quality instruction tuning data focusing on dialogue-style instructions.
   • Fine-tuned using a special token to separate prompt and answer segments, with a cosine learning rate schedule and weight decay.
3. Reinforcement Learning with Human Feedback (RLHF):
   • Collected human preference data using a binary comparison protocol.
   • Trained two separate reward models for helpfulness and safety.
   • Employed Proximal Policy Optimization (PPO) and Rejection Sampling for iterative fine-tuning, with safety checks and margin components in the loss function to improve alignment.
4. Ghost Attention (GAtt):
   • Introduced a technique to maintain instruction consistency over multiple turns in dialogue by synthetically concatenating instructions to user messages.

Key Results:

• Llama 2-Chat models generally perform better than existing open-source models and are on par with some closed-source models on various benchmarks.
• The models demonstrated significant improvements in safety and helpfulness through iterative fine-tuning and reward modeling.
• Human evaluations showed that Llama 2-Chat outperformed ChatGPT in both safety and helpfulness.

Contributions and Innovations:

• Development and open release of Llama 2, a family of pretrained and fine-tuned LLMs, with detailed methodology to improve model safety and performance.
• Introduction of Ghost Attention (GAtt) to enhance multi-turn dialogue consistency.
• Iterative application of RLHF combining PPO and Rejection Sampling for better alignment with human preferences.
• Comprehensive evaluation framework and release of models for research and commercial use, fostering community collaboration.

Future Work:

• The authors suggest further research on fine-tuning techniques, handling instruction consistency in dialogues, and exploring additional methods to enhance model safety and alignment with human preferences.

Applications:

• Customer service chatbots
• Virtual assistants
• Interactive educational tools
• Content generation and summarization
• Automated technical support

Relevant Links:

• Meta AI resources for Llama
• Llama responsible use guide
• Llama GitHub repository
• MAX end to end llama pipelines example