[docs]: Add RL-DPO Tutorial (#1733)

doc/en/DPO_tutorial.md

@@ -0,0 +1,147 @@
+# DPO Training with LLaMA-Factory
+
+This tutorial demonstrates how to use Direct Preference Optimization (DPO) to fine-tune a language model using the LLaMA-Factory framework. DPO is a method for training models based on human preferences, allowing for more aligned and user-centric outputs.
+
+## Installation
+
+### Step 1: Create a conda environment and suit it for KTransformers
+
+```Bash
+conda create -n Kllama python=3.12 # choose from : [3.10, 3.11, 3.12, 3.13]
+conda install -y -c conda-forge libstdcxx-ng gcc_impl_linux-64
+conda install -y -c nvidia/label/cuda-12.8.0 cuda-runtime
+```
+
+### Step 2: Install the LLaMA-Factory environment
+
+```Bash
+git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
+cd LLaMA-Factory
+pip install -e ".[torch,metrics]" --no-build-isolation
+```
+
+
+### Step 3: Install KTransformers
+#### Option 1: Install the KTransformers wheel that matches your Torch and Python versions, from https://github.com/kvcache-ai/ktransformers/releases/tag/v0.4.4
+
+(Note: The CUDA version can differ from that in the wheel filename.)
+
+```Bash
+pip install ktransformers-0.4.4+cu128torch28fancy-cp312-cp312-linux_x86_64.whl
+```
+
+#### Option 2: Install KTransformers from source
+
+```Bash
+git clone --depth 1 https://github.com/kvcache-ai/ktransformers.git
+cd ktransformers/kt-sft
+export TORCH_CUDA_ARCH_LIST="8.0;8.9;9.0" # set according to your GPU
+
+pip install -r "requirements-sft.txt"
+KTRANSFORMERS_FORCE_BUILD=TRUE pip install -v . --no-build-isolation
+
+```
+
+### Step 4: Install the Flash-attention wheel that matches your Torch and Python versions, from: https://github.com/Dao-AILab/flash-attention/releases
+
+```Bash
+# abi=True/False can find from below
+# import torch
+# print(torch._C._GLIBCXX_USE_CXX11_ABI)
+
+pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.8cxx11abiTRUE-cp312-cp312-linux_x86_64.whl
+```
+
+### Step 5: (Optional) If you want to use flash_infer (otherwise it defaults to triton)
+
+```Bash
+git clone https://github.com/kvcache-ai/custom_flashinfer.git
+pip install custom_flashinfer/
+```
+
+## Prepare Models
+
+We uses `DeepSeek-V2-Lite-Chat` as an example here. You can replace it with other models such as Kimi K2.
+
+## How to start
+
+```Python
+# For LoRA SFT
+USE_KT=1 llamafactory-cli train examples/train_lora/deepseek2_lora_dpo_kt.yaml
+# For Chat with model after LoRA SFT
+llamafactory-cli chat examples/inference/deepseek2_lora_dpo_kt.yaml
+# For API with model after LoRA SFT
+llamafactory-cli api examples/inference/deepseek2_lora_dpo_kt.yaml
+```
+
+For example, we provide the YAML file as follows: 
+
+（1）examples/train_lora/deepseek2_lora_dpo_kt.yaml
+
+```YAML
+### model
+model_name_or_path: DeepSeek-V2-Lite-Chat
+trust_remote_code: true
+
+### method
+stage: dpo
+do_train: true
+finetuning_type: lora
+lora_rank: 8
+lora_target: all
+pref_beta: 0.1
+pref_loss: sigmoid  # choices: [sigmoid (dpo), orpo, simpo]
+
+### dataset
+dataset: dpo_en_demo
+template: llama3
+cutoff_len: 2048
+max_samples: 1000
+overwrite_cache: true
+preprocessing_num_workers: 16
+dataloader_num_workers: 4
+
+### output
+output_dir: saves/Kllama_deepseekV2_DPO
+logging_steps: 10
+save_steps: 500
+plot_loss: true
+overwrite_output_dir: true
+save_only_model: false
+report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]
+
+### train
+per_device_train_batch_size: 1
+gradient_accumulation_steps: 8
+learning_rate: 5.0e-6
+num_train_epochs: 0.1
+lr_scheduler_type: cosine
+warmup_ratio: 0.1
+bf16: true
+ddp_timeout: 180000000
+resume_from_checkpoint: null
+
+### ktransformers
+use_kt: true # use KTransformers as LoRA sft backend
+kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V2-Lite-Chat-sft-amx.yaml
+cpu_infer: 64
+chunk_size: 8192
+```
+
+For more details about --kt_optimize_rule, please refer to https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/KTransformers-Fine-Tuning_User-Guide.md 
+
+（2）examples/inference/deepseek2_lora_dpo_kt.yaml
+
+```YAML
+model_name_or_path: DeepSeek-V2-Lite-Chat 
+adapter_name_or_path: saves/Kllama_deepseekV2_DPO
+template: deepseek
+infer_backend: ktransformers  # choices: [huggingface, vllm, sglang, ktransformers]
+trust_remote_code: true
+
+use_kt: true # use KTransformers as LoRA sft backend to inference
+kt_optimize_rule: examples/kt_optimize_rules/DeepSeek-V2-Lite-Chat-sft-amx.yaml
+cpu_infer: 32
+chunk_size: 8192
+
+```