@Nick Karpov and I are working on a blog post for how to use Delta Lake and aws-sdk-pandas. Thanks for the awesome contribution from @Florian Valeye, delta-rs is now an optional dependency for that project. Since it’s an optional dependency, it’s not going to be included in the layers that are attached to the releases. Hopefully we can just add a layer with delta-rs + the official layer and that’ll work (multiple layers can be added to an AWS Lambda function). In any case, we need to spell this out clearly for the community, so they know exactly how to start using Delta Lake in AWS Lambda functions.

has anyone successfully used this in lambda? I'm stepping through it right now and my lamdba function barfs that it can't find

_internal.abi3.so

even though i'm definitely packaging it in the layer:

creating: python/deltalake/
   creating: python/deltalake/__pycache__/
  inflating: python/deltalake/__pycache__/schema.cpython-37.pyc
  inflating: python/deltalake/__pycache__/table.cpython-37.pyc
  inflating: python/deltalake/__pycache__/writer.cpython-37.pyc
  inflating: python/deltalake/__pycache__/data_catalog.cpython-37.pyc
  inflating: python/deltalake/__pycache__/fs.cpython-37.pyc
  inflating: python/deltalake/__pycache__/__init__.cpython-37.pyc
  inflating: python/deltalake/__init__.py
  inflating: python/deltalake/table.py
  inflating: python/deltalake/_internal.pyi
  inflating: python/deltalake/schema.py
  inflating: python/deltalake/fs.py
  inflating: python/deltalake/data_catalog.py
  inflating: python/deltalake/writer.py
  inflating: python/deltalake/_internal.abi3.so
   creating: python/deltalake-0.6.4.dist-info/
  inflating: python/deltalake-0.6.4.dist-info/WHEEL
   creating: python/deltalake-0.6.4.dist-info/license_files/
  inflating: python/deltalake-0.6.4.dist-info/license_files/LICENSE.txt
  inflating: python/deltalake-0.6.4.dist-info/RECORD
 extracting: python/deltalake-0.6.4.dist-info/REQUESTED
  inflating: python/deltalake-0.6.4.dist-info/METADATA
 extracting: python/deltalake-0.6.4.dist-info/INSTALLER

then when running the lambda with the layer attached (no import even in code yet):

{
  "errorMessage": "Unable to import module 'lambda_function': /opt/python/deltalake/_internal.abi3.so: cannot open shared object file: No such file or directory",
  "errorType": "Runtime.ImportModuleError",
  "stackTrace": []
}

cc: @Florian Valeye

hey @rtyler we're trying to have

aws-sdk-pandas

provide the correct pyarrow dependencies for

deltalake

https://github.com/aws/aws-sdk-pandas/issues/1969 - they seem to be open to doing it if the size doesn't blow up... any chance you know exactly which optional arrow modules are needed? I didn't have luck searching in the repo and trying to see if there's a quick answer before diving in