Paweł Romanowski

09/06/2023, 8:30 AM
Hello people, I have a question regarding authenticating to Azure using the Python library. In particular, I'm trying to use the managed identity authentication. While it works just fine using pure Python code (the fsspec library) and the Azure Function runtime sets all the necessary environment variables, deltalake fails to obtain the access token. I know this logic is technically a part of the arrow / object_store library in Rust, but maybe someone here would be able to help me out here. Any pointers are much appreciated! Thanks
For what it's worth, I found a workaround: I'm using the official azure Python library with DefaultAzureCredential, and retrieve the access token there. I pass it to deltalake via
key in storage options. It works fine, but won't rotate the token automatically if it expires. Clearly something wrong in the object_store logic for managed identity authentication -- it has access to all the environment variables that the Python library has, and fails to authenticate/parse the token response. So its logic is most likely wrong. I will raise this with the arrow project.