Hello. I was trying to test delta-rs

0.8.1

with a local

k8s

MinIO

. I get

thread '<unnamed>' panicked at 'not stream', /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/object_store-0.5.5/src/aws/credential.rs:173:27

File "/Users/simon/Library/Caches/pypoetry/virtualenvs/heureka-MBcEyvac-py3.9/lib/python3.9/site-packages/deltalake/table.py", line 122, in __init__
    self._table = RawDeltaTable(
pyo3_runtime.PanicException: not stream

storage_options = {"AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
"AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
"AWS_REGION": aws_region,
"AWS_ENDPOINT_URL": "localhost:30000"  # I also tried, trino-minio-svc.trino:9000, and host.docker.internal:30000
}

I know my delta table in MinIO works fine because I also use it for Trino, I was keen on testing with MinIO here instead of AWS S3.

DeltaTable(
    table_uri=table_uri,
    storage_options=self.storage_options,
    version=version,
)

Anyone encountered this issue? I found this open issue: https://github.com/delta-io/delta-rs/issues/809 Seems like an open issue perhaps?

Is your MinIO server running over HTTPS? You might have to specify

AWS_STORAGE_ALLOW_HTTP=1

if you are just using HTTP over local host

I run over

HTTP

in this case. I get the same issue. Currently tried this:

os.environ["AWS_S3_ALLOW_UNSAFE_RENAME"] = "true"
        os.environ["AWS_STORAGE_ALLOW_HTTP"] = "1"

        return {
            "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
            "AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
            "AWS_REGION": aws_region,
            "AWS_ENDPOINT_URL": "host.docker.internal:30000",
        }

Have you tried prefixing your

AWS_ENDPOINT_URL

with

http://

?

"AWS_ENDPOINT_URL": "<http://host.docker.internal:30000>"

Yes I tried that as well. However I got it working now @Cole MacKenzie. I defined my service like this:

apiVersion: v1
kind: Service
metadata:
  name: trino-minio-svc
  namespace: trino
spec:
  type: NodePort
  ports:
    - name: "9000"
      port: 9000
      targetPort: 9000
      nodePort: 30000
    - name: "9001"
      port: 9001
      targetPort: 9001
      nodePort: 30001
  selector:
    app: minio

I then configured it like:

@staticmethod
def _setup_storage_options(aws_region: str) -> Dict[str, str]:
    os.environ["AWS_S3_ALLOW_UNSAFE_RENAME"] = "true"
    os.environ["AWS_STORAGE_ALLOW_HTTP"] = "1"

    return {
        "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
        "AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
        "AWS_REGION": aws_region,
        "AWS_ENDPOINT_URL": "<http://localhost:30000>",
    }

This worked. When I browse in my local browser,

30001

is what I use to access the

UI

. But to make the calls I have to use port

9000 -> 30000

I then also tried with:

"trino-minio-svc.trino:9001"

but that did not work to my surprise. I would have imagined it would be able to pick this up, but yeah there is something to it here with the

http

and

https

. And if you add

http://

to the

service.namespace

it freaks out. If I configure an ingress I am sure that would work as well.