How to read delta lake table via an s3 bucket

[finlaydotb]

As seen from the documentation here it is possible to use the deltalake::open_table method to open the delta lake table located on the file system. This I have been able to do.

But I have not been able to figure out how to open delta lake table on s3.

The documentation for deltalake::open_table which can be found here says

Creates and loads a DeltaTable from the given path with current metadata. Infers the storage back-end to use from the scheme in the given table path.

I tried to pass in a url to s3 bucket since the documentation says the storage back-end will be inferred. But doing this open_table("s3://127.0.0.1:9000/deltalake/delta-table") leads to this panic

thread 'main' panicked at 'called Result::unwrap() on an Err value: LoadCheckpoint { source: Storage { source: Generic { store: "S3", source: Error { retries: 0, message: "request error", source: Some(reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(169.254.169.254)), port: None, path: "/latest/api/token", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 60, kind: TimedOut, message: "Operation timed out" })) }) } } } }', src/main.rs:71:10
stack backtrace:
0: rust_begin_unwind

I had a feeling this should be trivial but unfortunately it has not been :(

So anyone knows how to get open_table to open a table located on an s3 bucket?

[roeap]

Hi @finlaydotb,

Working against an emulated service requires a bit more configuration, and we should update the docs to reflect that. Specifically, you can either set some configuration in the environment, or pass it explitily by using open_table_with_storage_options.

Looking at your example code, it would likely look something like this.

storage_options = HashMap::from([
    ("allow_http", "true"),
    ("endpoint_url", "http://127.0.0.1:9000"),
    ("access_key_id", "some id"),
    ("access_key_secret", "some secret")
]);

The actual error you are seeing is a different one though. Essentially the underlying object store tries to use the the metadata endpoint to fetch an access token, which is the default, if no credential is provided. If the local service allows anonymous access, an empty key id / secret should still be provided, to not query the metadata endpoint.

If you want to set the config in the environment, you can use the same kieys as above, just capitalized and prefixed with AWS_. A list of all supported config for S3 stores can be found here.

Last but not least, the URL should be in the form of s3://<bucket>/<path>, which in your case would be s3://deltalake/detla-table.

[roeap]

hmm ... I would have to investigate that a bit more. Could you try patching the dynamodb_lock dependency to point at the repo and see if that compiles?

Also, since you are local, you may also set AWS_S3_ALLOW_UNSAFE_RENAME to "true" in your environment. This is really only needed in multi-.writer scenarios, and allows you to avoid setting up a local dynamodb.

[finlaydotb]

Could you try patching the dynamodb_lock dependency to point at the repo and see if that compiles?

So first time doing this so I had to look up how to...so I checked out the repo https://github.com/delta-io/delta-rs, then modified my Cargo.toml to

[patch.crates-io]
dynamodb_lock = {path = "./delta-rs/dynamodb_lock"}

This compiled fine. (curious is there a shorter way to patch dependencies?)

I tried your second suggestion by running AWS_S3_ALLOW_UNSAFE_RENAME="true" cargo build It seems not to remove the need for dynamodb as the compilation still failed with it.

[roeap]

curious is there a shorter way to patch dependencies?

You can directly reference a git repo

[patch.crates-io]
dynamodb_lock = {git = "https://github.com/delta-io/delta-rs", rev="..."}

for the second part I was a little ambiguous :). Compilation is still needed and the parameter will not help during build. Just when using the S3 there is also a Dynamo DB required, since S3 does not support atomic renames. If there is a single writer though, we can live without atomic renames, and the option AWS_S3_ALLOW_UNSAFE_RENAME allows you to avoid instantiating the dynamo client at runtime...

[finlaydotb]

Thanks! I was not sure about using the git route for the patch, as the url points to the root of the project not specifically to dynamodb_lock, but with the path route, the path given actually points to dynamodb_lock's directory and where it's Cargo.toml file is located.

So I guess having this path is a workable workaround? Because it seems the dynamodb_lock on crates.io is stuck at version 0.4.2 - is the plan to stop publishing dynamodb_lock to crates.io?

[roeap]

We definitely want to keep publishing dynamodb_lock! There was a larger recent refactor concerning building the repository, where we needed to pin the rusoto dependencies, which is likely the root cause. @houqp - can we do another release for dynamodb_lock?

[vegarsti]

In case anyone needs this: We need to also call deltalake::aws::register_handlers(None)