You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We write data to delta table using delta-rs with PyArrow engine with DayHour as partition column. However when we run the optimize.compact() on the table, it creates partitions with spaces and does not properly encode the partition urls as shown in the below image i.e; it creates new partitions url with spaces (.zstd.parquet).
Can you please let me know how we can run the optimize.compact without having partitions with spaces?
The text was updated successfully, but these errors were encountered:
Confirmed that this also seems to happen in 0.18.0.
This breaks compatibility for our Trino connector to enable visualization on dashboards. Ideally, Trino wouldn't care about spaces (it has a lot of other issues like not being compatible with timezone-aware timestamps).
ion-elgreco
changed the title
Optimize compact not encoding correct URL for partitions in delta table
Rust writer not encoding correct URL for partitions in delta table
Jul 1, 2024
@ion-elgreco the issue we see here is with the Pyarrow engine
gprashmi
changed the title
Rust writer not encoding correct URL for partitions in delta table
Pyarrow writer not encoding correct URL for partitions in delta table
Jul 8, 2024
Environment
Delta-rs version: 0.17.4
Bug
We write data to delta table using delta-rs with PyArrow engine with DayHour as partition column. However when we run the optimize.compact() on the table, it creates partitions with spaces and does not properly encode the partition urls as shown in the below image i.e; it creates new partitions url with spaces (.zstd.parquet).
Can you please let me know how we can run the optimize.compact without having partitions with spaces?
The text was updated successfully, but these errors were encountered: