implemention of lqlora

[Liebele]

PR types

PR changes

Description

[paddle-bot]

Thanks for your contribution!

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Liebele]

使用paddle中LoRA支持的量化算法进行混合量化。
使用LQ-LoRA，需要先获取整数线性规划的准备数据，接着使用整数线性规划求解每个矩阵对应的量化算法，然后进行迭代初始化获取修改后的原模型及LoRA模块的参数。
在使用LQ-LoRA进行微调时，将“weight_quantize_algo”设置为“lqlora”，同时提供“qconfig_path”以及“lqlora_state_dict_path”（分别对应每个矩阵对应的量化算法、迭代初始化后的原模型及LoRA模块的参数）。

[Liebele]

使用该方法在LLaMA-7B上的实验结果：