[RLlib] Make most RLlib components implementers of the new Checkpointable API for unified checkpointing experience.

[sven1977]

Make most RLlib components implementers of the new Checkpointable API for unified checkpointing experience.
In particular, these components are:
Learner, LearnerGroup, RLModule (and MARLModule), EnvRunners (single- and multi-agent), ConnectorV2 (and pipelines).

This now allows for maximum flexibility when creating checkpoints of Algorithms and any possible subsets of component(s) of this Algorithm. Each component can define its own subcomponents, each of which will create its own subdirectory in the component's checkpoint dir.

For example, when doing ...

[LearnerGroup].save_to_path([dir])

... the resulting directory tree in [dir] will be:

dir/
    learner/
        rl_module/
            default_policy/
            metadata.json
            state.pkl
            class_and_ctor_args.pkl
        metadata.json
        state.pkl
        class_and_ctor_args.pkl
    metadata.json
    state.pkl
    class_and_ctor_args.pkl

Where

• metadata.json contains convenience user-readable meta information of the (sub)component saved. This information is NOT part of the state and not required to restore the respective component from disk.
• state.pkl contains all state information that is NOT part of any subcomponent (which has its own state.pkl file). For example, the optimizer state of a Learner.
• class_and_ctor_args.pkl stores the class of the (sub)component and its required args and kwargs to create an initial instance. This file is only needed when using the Checkpointable.from_checkpoint([dir]) method to create a new instance (and restore its state from the checkpoint).

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(