[FEA] Adjust read_json
to allow reading byte ranges from source files >2 GB
#16138
Labels
Milestone
read_json
to allow reading byte ranges from source files >2 GB
#16138
Is your feature request related to a problem? Please describe.
When reading a byte range from >2GB large source file, cudf
read_json
throws:CUDF failure at: /opt/conda/conda-bld/work/cpp/src/io/json/read_json.cu:311: The size of each source file must be less than INT_MAX bytes
Is it possible to adjust this exception to allow for byte range reading from large source files?
Describe the solution you'd like
Hopefully we can adjust the batching in
read_json
and allow <2 GB byte range reads from source files >2 GB to succeed.The text was updated successfully, but these errors were encountered: