You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Encountered errors when trying to write to a Delta Lake table stored in S3 with DynamoDB used for transaction logs. The errors indicate failures to write to DynamoDB for transaction entries.
Error log:
[2024-07-05T13:42:07Z ERROR deltalake_aws::logstore] retry #0 on log entry CommitEntry { version: 1, temp_path: Path { raw: "_delta_log/_commit_9da23947-e5ef-4ed2-a95a-fe3c14f4e66c.json.tmp" }, complete: false, expire_time: None } failed to update lock db: 'Transaction failed: unable to complete entry for '1': failure to write to DynamoDb'
[2024-07-05T13:42:07Z ERROR deltalake_aws::logstore] retry #1 on log entry CommitEntry { version: 1, temp_path: Path { raw: "_delta_log/_commit_9da23947-e5ef-4ed2-a95a-fe3c14f4e66c.json.tmp" }, complete: false, expire_time: None } failed to update lock db: 'Transaction failed: unable to complete entry for '1': failure to write to DynamoDb'
[2024-07-05T13:42:07Z ERROR deltalake_aws::logstore] retry #2 on log entry CommitEntry { version: 1, temp_path: Path { raw: "_delta_log/_commit_9da23947-e5ef-4ed2-a95a-fe3c14f4e66c.json.tmp" }, complete: false, expire_time: None } failed to update lock db: 'Transaction failed: unable to complete entry for '1': failure to write to DynamoDb'
Traceback (most recent call last):
File "/Users/marcosmartinez/coding/market-updates-module/local.py", line 22, in<module>
write_deltalake(
File "/Users/marcosmartinez/coding/market-updates-module/venv/lib/python3.12/site-packages/deltalake/writer.py", line 556, in write_deltalake
table._table.create_write_transaction(
_internal.CommitFailedError: Transaction failed: dynamodb client failed to delete log entry
What you expected to happen:
Expected the Delta Lake write operation to complete successfully, with transaction entries properly written to DynamoDB.
How to reproduce it:
Ensure AWS credentials and region are configured.
Create a script that writes to a Delta Lake table stored in S3, using DynamoDB for transaction logs.
Ensure that the necessary IAM permissions are in place for both S3 and DynamoDB.
Verify that the DynamoDB table used for transaction logs exists and is correctly configured.
I have confirmed that I have admin permissions on my AWS account, so permissions should not be an issue. This problem persists even with the correct configurations in place. Funny enough the operation append the item to the table so the error is not affecting the write apparently.
The text was updated successfully, but these errors were encountered:
Do you have the script or terraform handy used to create the DynamoDB table? I've not seen this error in use of this code, so I'm curious if tit's possible to configure DynamoDB in a way that triggers this error
Environment
Delta-rs version: 0.18.2
Binding: Python
Environment:
Bug
What happened:
Encountered errors when trying to write to a Delta Lake table stored in S3 with DynamoDB used for transaction logs. The errors indicate failures to write to DynamoDB for transaction entries.
Error log:
What you expected to happen:
Expected the Delta Lake write operation to complete successfully, with transaction entries properly written to DynamoDB.
How to reproduce it:
Example script to reproduce the issue:
More details:
I have confirmed that I have admin permissions on my AWS account, so permissions should not be an issue. This problem persists even with the correct configurations in place. Funny enough the operation append the item to the table so the error is not affecting the write apparently.
The text was updated successfully, but these errors were encountered: