lora的训练使用的文件是https://github.com/Akegarasu/lora-scripts

lora训练是需要成对的文本图像对的，需要准备相应的训练数据。

1.训练数据准备

使用deepbooru/blip生成训练数据，建筑类建议使用blip来生成。

2.lora在linux上环境

cuda 10.1 p40 python3.7

accelerate==0.15.0 应该只能在虚拟环境中，在train.sh中把accelerate launch --num_cpu_threads_per_process=8换成python，这么改accelerate多卡训练有问题

albumentations==0.2.0

scikit-image==0.14 版本高了会报错

numpy==1.17

这里面有个skimage的版本问题，会报错

[bug解决] cannot import name ‘_validate_lengths‘ from ‘numpy.lib.arraypad‘_arrycrop.py在那_Harry嗷的博客-CSDN博客

safetensors==0.3.0

voluptuous==0.12.1

huggingface-hub==0.12.0

transformers==4.20.0

tokenizers==0.11.6

opencv-python==4.0.0.21

einops==0.3.0

ftfy==6.0

pytorch-lightning==1.2.8

xformers==0.0.9(torch可以支持torch==1.8.1)

diffusers==0.10.0

pyre-extensions==0.3.0

regex==2021.4.4

升级glic

3.sh train.sh训练

openai的clip权重要配置一下

library/train_util/ 1900多行中load_tokenizer函数中的tokenizer=CLIPTokenizer.from_pretrained()

使用的是openai的clip-vit-large-patch14参数

4.lora-scripts的核心代码解析

train_network.py->train->
train_util.load_tokenizer->
BuleprintGenerator->
config_util.generate_dreambooth_subsets_config_by_subdirs->
blueprint_generator.generate->
config_util.generate_dataset_group_by_blueprint 加载数据->
train_util.prepare_accelerator->
train_util.prepare_dtype->
train_util.load_target_model 加载sd模型->
train_util.replace_unet_modules-> 
vae.to(accelerator.device)->
vae.requires_grad_(False)->
vae.eval()->
train_dataset_group.cache_lantents->
network_module(LoRANetwork)->
network.apply_to(text_encoder,unet,train_text_encoder,train_unet)->
network.prepare_optimizer_params->
train_util.get_optimizer->
train_dataloader=torch.utils.data.DataLoader(train_dataset_group)->
lr_scheduler=train_util.get_scheduler_fix->
unet,text_encoder,network,optimizer,train_dataloader,lr_scheduler=accelerator.prepare(unet,text_encoder,network,optimizer,train_dataloader,lr_scheduler)->
unet.requires_grad_(False)->
unet.to(accelerator.device)->
text_encoder.requires_grad_->
text_encoder.to(accelerator-device)->
unet.eval()->
text_encoder.eval()->
network.prepare_grad_etc(text_encoder,unet)->
dataset=train_dataset_group.dataset[0]->
noise_scheduler=DDPMScheduler(beta_start=0.00085,beta_end=0.012,beta_schedule='scaled_linear',num_train_timesteps=1000,clip_sample=False)->
accelerator.init_trackers('netwoek_train')->
network.on_epoch_start(text_encoder,unet)->
latents=batch['latents'].to(accelerator.device)->
latents=latents*0.18215->
encoder_hidden_states=train_util.get_hidden_states->
noise = torch.randn(latents,device=latent.device)->
timesteps=torch.randint(0,noise_scheduler.config.num_train_timesteps,(b_size,),device=latents.device)->
noise_latents=noise_scheduler.add_noise(lantents,noise,timesteps) [1,4,64,64]->
noise_pred=unet(noisy_latents,timesteps,encoder_hidden_states).sample [1,4,64,64]->
target=noise_scheduler.get_velocity(latnets,noise,timesteps) [1,4,64,64]->
loss=torch.nn.functional.mse_loss(noise_pred.float(),target.float(),reduction='none') [1,4,64,64]->
loss=loss.mean([1,2,3])->
loss_weights=batch['loss_weights']->
loss=loss*loss_weights->
accelerator.backward(loss)->
param_to_clip=network.get_trainable_params()->
accelerator.clip_grad_norm_()->
optimizer.grad()->
lr_scheduler.step()->
optimizer.zero_grad()->
train_util.sample_images(accelerator,args,None,global_step,accelerator.device,vae,tokenizer,text_encoder,unet)

5.入参

args = Namespace(
bucket_no_upscale=False, 
bucket_reso_steps=64, 
cache_latents=True, 
caption_dropout_every_n_epochs=0, 
caption_dropout_rate=0.0,
caption_extension='.txt',
caption_extention=None, 
caption_tag_dropout_rate=0.0, 
clip_skip=2, 
color_aug=False, 
dataset_config=None, 
dataset_repeats=1, 
debug_dataset=False, 
enable_bucket=True, 
face_crop_aug_range=None, 
flip_aug=False, 
full_fp16=False, 
gradient_accumulation_steps=1, 
gradient_checkpointing=False, 
in_json=None, 
keep_tokens=0, 
learning_rate=0.0001, 
log_prefix=None, 
logging_dir='./logs', 
lowram=False, 
lr_scheduler='cosine_with_restarts', 
lr_scheduler_num_cycles=1, 
lr_scheduler_power=1, 
lr_warmup_steps=0, 
max_bucket_reso=1024, 
max_data_loader_n_workers=8, 
max_grad_norm=1.0, 
max_token_length=225, 
max_train_epochs=10, 
max_train_steps=1600, 
mem_eff_attn=False, 
min_bucket_reso=256, 
mixed_precision='fp16', 
network_alpha=32.0, 
network_args=None, 
network_dim=32, 
network_module='networks.lora', 
network_train_text_encoder_only=False, 
network_train_unet_only=False, 
network_weights=None, 
no_metadata=False, 
noise_offset=0.0, 
optimizer_args=None, 
optimizer_type='', 
output_dir='./output', 
output_name='/home/sniss/local_disk/lora-scripts/output', 
persistent_data_loader_workers=False, 
pretrained_model_name_or_path='/home/sniss/local_disk/stable-diffusion-webui_23-02-17/models/Stable-diffusion/sd-v1.5.ckpt', 
prior_loss_weight=1.0, 
random_crop=False, 
reg_data_dir=None, 
resolution=(512, 512), 
resume=None, 
sample_every_n_epochs=None,
sample_every_n_steps=None, 
sample_prompts=None, 
sample_sampler='ddim', 
save_every_n_epochs=2,
save_last_n_epochs=None, 
save_last_n_epochs_state=None, 
save_model_as='ckpt', 
save_n_epoch_ratio=None, 
save_precision='fp16', 
save_state=False, 
seed=1337, 
shuffle_caption=True, 
text_encoder_lr=1e-05,
tokenizer_cache_dir=None, 
train_batch_size=1, 
train_data_dir='/home/sniss/local_disk/lora-scripts/data', 
training_comment=None, 
unet_lr=0.0001, 
use_8bit_adam=False, 
use_lion_optimizer=False, 
v2=False, 
v_parameterization=False, 
vae=None, 
xformers=True)

参数的设置有好几块：

1.输入的train.sh

2.train_network.py中的main部分

3.train_util.py中的1536行附近的

add_sd_models_args/add_optimizer_args/add_training_args/add_dataset_args

6.注意事项

a.数据集名称第一个是数字20_arch，这个数字和训练轮数epoch有关

b.train_network中

PyTorch 训练时中遇到的卡住停住等问题_训练时候卡住_yyywxk的博客-CSDN博客- 问题描述使用 PyTorch 框架训练模型，训练第一个 epoch 时，在最后一个 batch 处卡死，卡了一天都没有动弹，而 CPU 和 GPU 都处于正常运行的状态，程序也没有报错，并且之前训练一直都是正常的。最终，只能通过 Ctrl+C 强制性暂停。如下图所示。- 可能的原因搜索文章发现，有人出现这种问题是和 cv2.imread 有关，用 OpenCV 的接口进行数据读取，而没有用 PIL，导致出现 OpenCV与Pytorch互锁的问题，关闭OpenCV的多线程即可解决问题1 2。但https://blog.csdn.net/yyywxk/article/details/106323049在args.max_data_loader_n_workers改为0

c.max_token_length=75,150,225,使用225会报错？RuntimeError: The size of tensor a (227) must match the size of tensor b (77) at non-singleton dimension 1,这块是个巨坑，主要是clip初始化的时候，忘了加tokenizer_config.json这个文件。

d.bash: accelerate: command not found，没搞定

多卡训练

python -m torch.distributed.launch --nproc_per_node=4 --nnodes=1 --node_rank=0 --master_addr=localhost   --master_port=22222 --use_env "./sd-scripts/train_network.py" \

import torch.distributed as dist 
dist.init_process_group(backend='gloo', init_method='env://')