[BUG] VITS微调时报错

[SnowFox4004]

在vits微调5个epoch后报错，重新开始微调立即报错
似乎是validation时的问题， 给embedding层传了一个float类型的tensor
命令是

python fish_speech/train.py --config-name vits_decoder_finetune

修改了yml中的数据集路径以及batch_size
pytorch版本2.3.1 cuda12.1

以下是日志

Error executing job with overrides: []
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/snowfox/fish_audio/fish_speech/train.py", line 139, in
[rank0]: main()
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main
[rank0]: _run_hydra(
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
[rank0]: _run_app(
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app
[rank0]: run_and_report(
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
[rank0]: raise ex
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
[rank0]: return func()
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in
[rank0]: lambda: hydra.run(
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
[rank0]: _ = ret.return_value
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
[rank0]: raise self._return_value
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
[rank0]: ret.return_value = task_function(task_cfg)
[rank0]: File "/home/snowfox/fish_audio/fish_speech/train.py", line 135, in main
[rank0]: train(cfg)
[rank0]: File "/home/snowfox/fish_audio/fish_speech/utils/utils.py", line 77, in wrap
[rank0]: raise ex
[rank0]: File "/home/snowfox/fish_audio/fish_speech/utils/utils.py", line 66, in wrap
[rank0]: metric_dict, object_dict = task_func(cfg=cfg)
[rank0]: File "/home/snowfox/fish_audio/fish_speech/train.py", line 108, in train
[rank0]: trainer.fit(model=model, datamodule=datamodule, ckpt_path=ckpt_path)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 543, in fit
[rank0]: call._call_and_handle_interrupt(
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
[rank0]: return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
[rank0]: return function(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 579, in _fit_impl
[rank0]: self._run(model, ckpt_path=ckpt_path)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 986, in _run
[rank0]: results = self._run_stage()
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1030, in _run_stage
[rank0]: self.fit_loop.run()
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 205, in run
[rank0]: self.advance()
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 363, in advance
[rank0]: self.epoch_loop.run(self._data_fetcher)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 141, in run
[rank0]: self.on_advance_end(data_fetcher)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 295, in on_advance_end
[rank0]: self.val_loop.run()
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 182, in _decorator
[rank0]: return loop_run(self, *args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 135, in run
[rank0]: self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 396, in _evaluation_step
[rank0]: output = call._call_strategy_hook(trainer, hook_name, *step_args)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 311, in _call_strategy_hook
[rank0]: output = fn(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 410, in validation_step
[rank0]: return self._forward_redirection(self.model, self.lightning_module, "validation_step", *args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 640, in call
[rank0]: wrapper_output = wrapper_module(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1593, in forward
[rank0]: else self._run_ddp_forward(*inputs, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1411, in _run_ddp_forward
[rank0]: return self.module(*inputs, **kwargs) # type: ignore[index]
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]: return forward_call(args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 633, in wrapped_forward
[rank0]: out = method(_args, **_kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_speech/models/vits_decoder/lit_module.py", line 266, in validation_step
[rank0]: prior_audios = self.generator.infer(
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_speech/models/vits_decoder/modules/models.py", line 549, in infer
[rank0]: return self.decode(
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_speech/models/vits_decoder/modules/models.py", line 583, in decode
[rank0]: x, m_p, logs_p, y_mask = self.enc_p(
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_speech/models/vits_decoder/modules/models.py", line 78, in forward
[rank0]: text = self.text_embedding(text).transpose(1, 2)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 163, in forward
[rank0]: return F.embedding(
[rank0]: File "/home/snowfox/fish_audio/fish_env/lib/python3.10/site-packages/torch/nn/functional.py", line 2265, in embedding
[rank0]: return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
[rank0]: RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)
Epoch 5: 0%| | 0/17 [00:05<?, ?it/s]

在torch库里调用embedding前打印了一下发现前几个正常，中间有一个是Float类型的数据但是小数位为0