In general, a fine-tuning pipeline consists of an input model, fine-tuned on a new dataset, to produce an output model. Here are the details of our pipeline.

Input model

Creating an input model class requires static model weights as well as a model definition — also known as a model architecture. To create our input model class, which we call LLaMA LoRA 30B, we loaded the 30B weights from Meta’s LLaMA model into a LoRA-adapted model architecture that uses HuggingFace transformers and the bitsandbytes library.

Dataset

The English and Chinese datasets we used for fine-tuning came from the paper Instruction Tuning with GPT-4. The paper authors generated synthetic datasets in English and Chinese using GPT-4. Then they used the two datasets individually to fine-tune a model. We combined the two datasets into a single one.

As you can see in the datasets on Github, each example is formatted as: an instruction, an input, and an output. Note that in the dataset, coding questions include explanations as part of the output. Here’s an excerpt:

[	
  {
    "instruction": "Extract the positive integers from an array?",
    "input": "arr = [5,-2,8,-7,3,-1,0,5]",
    "output": "To extract the positive integers from the given array, you can use the filter() function which will return the values that evaluate to True based on the condition provided in the lambda function. \\n\\nHere's the Python code snippet:\\n\\n```\\narr = [5,-2,8,-7,3,-1,0,5]\\n\\nresult = list(filter(lambda x: x > 0, arr))\\n\\nprint(result)\\n```\\n\\nOutput: [5, 8, 3, 5]\\n\\nThe filter function iterates through each element in the array and checks if the element is greater than 0, which means it's a positive integer. If it's true, the element is included in the result list. Next, we display the result list which contains only the positive integers."
  },

  [...]

  {
    "instruction": "分析给定的法律文件并解释关键点。",
    "input": "",
    "output": "非常抱歉，你没有提供法律文件，所以我无法为你进行分析。 请提供具体的法律文件，我将尽力为你解释其中的关键点。"
  }
]

You can see the impact of these training examples on our fine-tuned model.

Fine-tuning loop with LoRA

Our fine-tuning process leverages LoRA using the same adapter as alpaca-lora.

LoRA is a more efficient fine-tuning technique. In traditional fine-tuning, the weights of the original model are unfrozen and updated. In LoRA, instead of unfreezing the original model, a new layer of weights — called adapter weights — is added.

To quantify the efficiency gain: Meta’s LLaMA model has 30B weights. Our resulting adapter was about 50k weights.

Output model

As explained above, we produced a set of static adapter weights. We call this Alpaca LoRA 30B.

To use this output model to do inference, all you need to do is load the original LLaMA weights along with these adapter weights into the same model architecture. Then you can start generating text completions.