Fix multi-model training with deepspeed nvme-offloading

[xu-song]

What does this PR do?

This PR fix "multi-model training with deepspeed nvme offloading"

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

To Reproduce

It gets error /home/nvme/param/zero_stage_3/float16params/rank0/0_param.tensor.swp: buffer nbytes != file bytes when multi-model training with deepspeed nvme offloading

Here is an example deepspeed config

    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "nvme",
            "nvme_path": "/home/nvme/optim",
        },
        "offload_param": {
            "device": "nvme",
            "nvme_path": "/home/nvme/param",
        },
    }

If you are training with multiple models (such as rlhf, dpo, multimodal) like this

model_1 = AutoModelForCausalLM.from_pretrained(MODEL_1_DIR)   # it calls deepspeed.zero.Init, then swap_out param to nvme_path 
model_2 = AutoModelForCausalLM.from_pretrained(MODEL_2_DIR)   # it calls deepspeed.zero.Init again

deepspeed.zero.Init first remove all files in nvme_path, which causes the swap files of model_1 be overwritten by model_2.

details of deepspeed.zero.Init

# https://github.com/microsoft/DeepSpeed/blob/v0.14.4/deepspeed/runtime/zero/partition_parameters.py#L1010
class Init:   
    def __init__(
        ....
        if self.remote_device == OffloadDeviceEnum.nvme:
            self.param_swapper = param_swapper or AsyncPartitionedParameterSwapper(_ds_config, self.dtype)

# https://github.com/microsoft/DeepSpeed/blob/v0.14.4/deepspeed/runtime/swap_tensor/partitioned_param_swapper.py#L89
class AsyncPartitionedParameterSwapper(object):
    def __init__(self, ds_config, model_dtype):
         ..
        shutil.rmtree(self.swap_folder, ignore_errors=True)    

After this PR

You can initialize two models with different nvme_path

model_1 = AutoModelForCausalLM.from_pretrained(MODEL_1_DIR)   # with default nvme_path
model_2 = AutoModelForCausalLM.from_pretrained(MODEL_2_DIR, disable_zero_init=True)
model_2_engine, *_ = deepspeed.initialize(model=model_2,
                                      config=another_ds_config)  # with a different nvme_path

Who can review?

• deepspeed: HF Trainer/Accelerate: @muellerzr
• trainer: @muellerzr and @SunMarc