Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logging with Fabric using steps #19994

Open
liambsmith opened this issue Jun 18, 2024 · 2 comments
Open

Logging with Fabric using steps #19994

liambsmith opened this issue Jun 18, 2024 · 2 comments
Labels
feature Is an improvement or enhancement needs triage Waiting to be triaged by maintainers

Comments

@liambsmith
Copy link
Contributor

liambsmith commented Jun 18, 2024

Description & Motivation

Logging using Fabric does not consider any steps during training, unlike when using the Lightning Trainer. A LightningModule calling self.log simply passes the logged dictionary and nothing else to the Fabric logging code when using Fabric but when using the Trainer it is handled by grouping/frequency adjustments (such as aggregating during multi-gpu training or logging every X steps [default 50]).

Pitch

An option to enable similar logging in Fabric as the Lightning Trainer. This could be off by default but could track steps that are submitted with fabric hooks/calls, such as:

fabric.call('on_train_step')

This would allow for logged values to be aggregated during the same step, which makes logs more readable.

Alternatives

No response

Additional context

No response

cc @Borda

@liambsmith liambsmith added feature Is an improvement or enhancement needs triage Waiting to be triaged by maintainers labels Jun 18, 2024
@liambsmith
Copy link
Contributor Author

It might be better to actually have a function that accepts a log handler that by default logs how it works now but can be customized by the user.

@liambsmith
Copy link
Contributor Author

It might be better to actually have a function that accepts a log handler that by default logs how it works now but can be customized by the user.

This could be done during setup, passing an optional function that is called when fabric receives logs from LightningModules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Is an improvement or enhancement needs triage Waiting to be triaged by maintainers
Projects
None yet
Development

No branches or pull requests

1 participant