Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Improve pyarrow-free remote-IO performance #16166

Draft
wants to merge 23 commits into
base: branch-24.08
Choose a base branch
from

Conversation

rjzamora
Copy link
Member

@rjzamora rjzamora commented Jul 2, 2024

Description

Improves fsspec-only behavior for multi-file and partial reads from remote storage. Host-memory usage will be suboptimal compared to NativeFile (for now), but performance will be comparable in most cases.

This PR also includes deprecations for pyarrow-based IO. However, #16132 should take priority for the deprecations (and this PR should be modified accordingly).

TODO

  • Needs cleanup
  • Must be revised to depend on Deprecate Arrow support in I/O #16132. Also, dask-cudf open_file_options deprecation should probably be pulled out and merged before both 16132 and this PR.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@rjzamora rjzamora added 2 - In Progress Currently a work in progress Performance Performance related issue breaking Breaking change labels Jul 2, 2024
@rjzamora rjzamora self-assigned this Jul 2, 2024
@github-actions github-actions bot added the Python Affects Python cuDF API. label Jul 2, 2024
@rjzamora rjzamora added dask Dask issue improvement Improvement / enhancement to an existing function cuDF (Python) and removed Python Affects Python cuDF API. labels Jul 2, 2024
@github-actions github-actions bot added the Python Affects Python cuDF API. label Jul 2, 2024
@rjzamora rjzamora changed the title [DNM][WIP] Improve pyarrow-free remote-IO performance [WIP] Improve pyarrow-free remote-IO performance Jul 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2 - In Progress Currently a work in progress breaking Breaking change cuDF (Python) dask Dask issue improvement Improvement / enhancement to an existing function Performance Performance related issue Python Affects Python cuDF API.
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

None yet

1 participant