New option called "best"
for args.save_strategy
.
#31817
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Addresses #31626.
Adds a new option called
"best"
forTrainingArguments.save_strategy
which saves the model checkpoint each time a new best performance is achieved.Details
_save_checkpoint
method was in charge of not only saving the model checkpoint but also determining the best metric and best checkpoint. The logic for determining a new best metric was separated out into the_determine_best_metric
method._determine_best_metric
is called after every evaluation inside of_maybe_log_save_evaluate
. The return valuenew_best_metric
is used to determine whether or not a new best metric has been achieved, and if the save strategy is"best"
then the TrainerControl'sshould_save
flag is switched on.best_metric
does not seem to be tracked by default. Rather, it's only tracked whenargs.metric_for_best_model
is provided. I believe that a best metric of some sort should always be tracked, and therefore if a value is not provided then the validation loss is used to determine a new best.SaveStrategy
was created intrainer_utils
that adds a new attribute calledBEST
to the previousIntervalStrategy
.I'm not sure if I like the rather "hack-y" way that I implemented this by manually switching the TrainerControl's
should_save
flag rather than delegating it to the callback handler like the other flags are dealt with. The problem is that the flags are normally updated before calling_maybe_log_save_evaluate
inside of the inner training loop, which means there's no way for us to determine whether or not a new best metric has been achieved with the current logic. I'm not sure if I'm making sense, but I'm open to any other suggestions.Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@muellerzr @SunMarc