Hey all - does anybody have experience with running delta-rs from a lambda (using Rust) in AWS? I'm struggling to work out how to open the Delta table with the IAM permissions I've given to the lambda. When testing locally I've used my temporary credentials (session token etc) alongside

open_table_with_storage_options

, but not sure how to do this from the lambda itself Thanks

Maybe try this:

let mut backend_config: HashMap<String, String> = HashMap::new();
    backend_config.insert("AWS_PROFILE".to_string(), profile);
    backend_config.insert("AWS_REGION".to_string(), region);
    backend_config.insert("AWS_ACCESS_KEY_ID".to_string(), key.to_string());
    backend_config.insert("AWS_SECRET_ACCESS_KEY".to_string(), secret.to_string());
    backend_config.insert("AWS_S3_LOCKING_PROVIDER".to_string(), "dynamodb".to_string());
    backend_config.insert("DYNAMO_LOCK_TABLE_NAME".to_string(), lock_table_name.to_string());


    let mut table = DeltaTableBuilder::from_uri(path)
        .with_storage_options(backend_config)
        .build()
        .unwrap();

@Nick Karpov is going to publish a blog post on how to use delta-rs from an AWS Lambda function using Python soon. I don’t think we’ve tried out the Rust API in a Lambda yet.

@Ben Temple i didn't run into any issues using an iamrole with access to S3 (and dynamo to support the locking)... i'm using the python deltalake and can successfully read/write, which should mean vanilla rust works too.... what is the issue you're running into? a 403?

I was able to write using vanilla Rust. Maybe I am missing stuff here or there but I had to dig into delta-rs code to find out what are the config values for AWS. Also not too much detail about locking so I had to go create DynamoDb table and tinker with names of columns, etc. I am supposed to do a part 2 of this post for AWS but busy with client work 🙂

I have also been able to do writes with vanilla Rust. Does the execution role ARN for the lambda have access to S3?

@Yousry Mohamed - sweet blog post 😎 Let me know if you’d ever like to collab on a Delta Lake blog post on the delta.io/blog site 😉

@Matthew Powers I’d be keen to help.

@Yousry Mohamed that was my experience too, I don’t know if I’m just missing some stupid obvious documentation but figuring out the schema for the lock table was not obvious and eventually I found that kafka ingest repo was using “key”, I didn’t investigate whether we could configure that more granularly - would be nice to make it the same as the Spark connector so we can support safe multi engine concurrency

Yes, it was “key” and just hard coded in source code as far as I recall. Regarding permissions and because I am not so good with AWS IAM, I did a shortcut and granted the user account I have full S3 permissions on all buckets. Sure not something for production but I wanted to focus on certain delta-rs bits.

The Spark connector is now supporting DynamoDB-based locking for S3 writers? That would be news to me, but if so, they should use our "key" since we implemented it first 😆

Here is part two of the post including a bit of concurrency tinkering https://www.cuusoo.com.au/guide/navigating-the-data-lake-using-rust-part-two