微信扫码
与创始人交个朋友
我要投稿
目录概览:
开始微调Llama-3 8B
pip install huggingface_hub ipython
:这将安装两个库:用于从 Hugging Face Hub 访问模型和用于交互式编码。huggingface_hub
ipython
"unsloth[colab] @ git+https://github.com/unslothai/unsloth.git" "unsloth[conda] @git+https://github.com/unslothai/unsloth.git"
:这将从 GitHub 安装 Unsloth 库,为 Google Colab() 和 conda 环境()指定不同的选项。[colab]
[conda]
export HF_TOKEN=xxxxxxxxxxxxx
:可能用于为 Hugging Face Hub 设置身份验证令牌,但出于安全原因,实际令牌值是隐藏的。
pip install huggingface_hub ipython "unsloth[colab] @ git+https://github.com/unslothai/unsloth.git" "unsloth[conda] @ git+https://github.com/unslothai/unsloth.git"export HF_TOKEN=xxxxxxxxxxxxx
安装 Wandb 库:安装与 Wandb 交互所需的库。pip install wandb
wandb login
pip install wandbwandb logio
import osfrom unsloth import FastLanguageModelimport torchfrom trl import SFTTrainerfrom transformers import TrainingArgumentsfrom datasets import load_dataset
设置最大序列长度:定义每个训练示例中允许的最大标记数。这有助于在训练期间管理内存和计算资源。max_seq_length = 2048
定义数据 URL:以 JSONL 格式存储数据集的 Web 地址,可能包含文本数据。url
dataset = load_dataset("json", data_files = {"train" : url}, split = "train")
datasets
load_dataset("json")
将数据格式指定为 JSON。
data_files
dictionary 使用键“train”和 URL 作为其值定义训练数据位置。
split="train"
表示我们正在加载数据集的训练部分。
max_seq_length = 2048url = "https://huggingface.co/datasets/laion/OIG/resolve/main/unified_chip2.jsonl"dataset = load_dataset("json", data_files = {"train" : url}, split = "train")
# 2. Load Llama3 modelmodel, tokenizer = FastLanguageModel.from_pretrained(model_name = "unsloth/llama-3-8b-bnb-4bit", # 指定 Unsloth 库中的确切模型。“Llama3”可能是型号名称,“8b”表示 80 亿个参数,“bnb”可能是指特定的架构,“4bit”表示使用内存效率高的格式。max_seq_length = max_seq_length, # 设置最大序列长度(前面定义)以限制模型可以处理的输入长度。dtype = None, # (假设它设置为 None)允许库选择最合适的数据类型load_in_4bit = True, # 允许以内存高效的 4 位格式加载模型(如果模型和硬件支持))
def generate_text(text):inputs = tokenizer([text], return_tensors="pt").to("cuda")outputs = model.generate(**inputs, max_new_tokens=20, use_cache=True)tokenizer.batch_decode(outputs)print("Before training\n")
model = FastLanguageModel.get_peft_model(model,r = 16,target_modules = ["q_proj", "k_proj", "v_proj", "o_proj","gate_proj", "up_proj", "down_proj",],lora_alpha = 16,lora_dropout = 0, # Supports any, but = 0 is optimizedbias = "none",# Supports any, but = "none" is optimizeduse_gradient_checkpointing = True,random_state = 3407,max_seq_length = max_seq_length,use_rslora = False,# Rank stabilized LoRAloftq_config = None, # LoftQ)
trainer = SFTTrainer(model = model,train_dataset = dataset,dataset_text_field = "text",max_seq_length = max_seq_length,tokenizer = tokenizer,args = TrainingArguments(per_device_train_batch_size = 2,gradient_accumulation_steps = 4,warmup_steps = 10,max_steps = 60,fp16 = not torch.cuda.is_bf16_supported(),bf16 = torch.cuda.is_bf16_supported(),logging_steps = 1,output_dir = "outputs",optim = "adamw_8bit",weight_decay = 0.01,lr_scheduler_type = "linear",seed = 3407,),)trainer.train()
print("\n ######## \nAfter training\n")generate_text("<human>: List the top 5 most popular movies of all time.\n<bot>: ")
model.save_pretrained("lora_model")model.save_pretrained_merged("outputs", tokenizer, save_method = "merged_16bit",)model.push_to_hub_merged("YOURUSERNAME/llama3-8b-oig-unsloth-merged", tokenizer, save_method = "merged_16bit", token = os.environ.get("HF_TOKEN"))model.push_to_hub("YOURUSERNAME/llama3-8b-oig-unsloth", tokenizer, save_method = "lora", token = os.environ.get("HF_TOKEN"))
53AI,企业落地应用大模型首选服务商
产品:大模型应用平台+智能体定制开发+落地咨询服务
承诺:先做场景POC验证,看到效果再签署服务协议。零风险落地应用大模型,已交付160+中大型企业
2024-05-14
2024-04-26
2024-03-30
2024-04-12
2024-05-10
2024-07-18
2024-05-22
2024-05-28
2024-04-25
2024-04-26
2024-11-14
2024-11-13
2024-11-13
2024-11-13
2024-11-12
2024-11-11
2024-11-08
2024-11-07