This is just a modified version of #1273 with concerns from @Will Jones addressed. I will really appreciate it if we can get this reviewed and merged soon..

what does anyone think on https://blog.rust-lang.org/inside-rust/2023/04/12/trademark-policy-draft-feedback.html?

@Will Jones For some more Delta related context on this question you got mentioned in, I basically cannot run our delta-rs optimize on Spark created parquet files with a

primitive("timestamp")

column in the schema because the ParquetReader gives

Timestamp(nanosecond)

from the int96 column, but the Arrow schema delta-rs thinks it should be using is

Timestamp(microsecond)

, which I believe to be the correct interpretation here. My gut feel is that this will be the case for parquet files written by Delta/Spark, but I'm not sure where the best place to introduce the necessary conversion will be. Putting it in our writer feels correct but I think we would have other readers not doing the right thing on this type either

Hi people, how can i handle PyDeltaTableError because it keeps throwing 500 internal server error when the control reaches there only workaround was to raise an exception manually

try:
            DeltaTable(str(delta_path))
            return True
        except PyDeltaTableError:
            raise NotFoundException(f'Delta Table not found: delta_table_path={delta_path}')
            # app_logger.error(f'Delta Table not found: delta_table_path={delta_path}')
            # return False
        except Exception as e:
            app_logger.error(e)
            return False

@Robert nice work! I just merged

main

into the branch, there's one minor thing that's broken from my local build and I think we might be close to being mergable

It is release time again! Here is the PR that updates the changelog: https://github.com/delta-io/delta-rs/pull/1290 I'm glad that we got several old issues taken care of! Please review the changelog and let me know if it is inaccurate.

I am quite annoyed with some of the API changes in Arrow 37 which require further work from the upgrade to 37 to incorporate, but 37 is where the fix for https://github.com/delta-io/delta-rs/pull/1226 is 😒

@rtyler I see…Do we need that in for 0.9.0, may I ask?

need is a strong word 😆 @Ian Joiner are you trying to cut 0.9.0 today?

@rtyler Ideally haha. I can do tomorrow or Sunday as well.

I can help you speed it up a bit over the weekend if necessary

let's just cut 0.9.0 then with

main

, based on some comments from @Robert I think there may be some datafusion incompatibilities to fix with arrow 37 and we'd be waiting for datafusion 23 there anyways 😒

releases are cheap and easy IMO

Sure!

next up 0.10.0 😆

We can always do an extra release if necessary

@rtyler Really thanks for reviewing the PR! Could you please trigger the release by creating the

rust-v0.9.0

tag?

@Will Jones @Robert @Florian Valeye @QP Hou @Christian Williams can I get at least one or two other 👍 to cut a 0.9.0 release?

oh bollocks, we have an old token for crates.io

I am pleased to announce the release of deltalake 0.9.0

A really nice blog by Marko Grujic on delta-rs: https://www.splitgraph.com/blog/seafowl-delta-storage-layer with a really nice quote:

In particular, there are three open and competing table formats that bring an ACID layer to data lakes, one of which will likely become the de-facto industry standard eventually. Besides Delta Lake, there are also Apache Iceberg and Apache Hudi, and while they all offer a variety of advanced features and open-source implementations, we've found that the most mature Rust implementation is that of Delta via the delta-rs project. It also has an active and welcoming community of maintainers, so we finally decided to replace our custom storage layer with delta-rs instead.

I just pushed an

arrow-37

topic branch to collaborate on those API changes. I'll spend more time with that over the weekend if I'm able (FYI @Robert)

Hey all! I'm new to the community but it's been cool to see all of the chat here! I've been working with delta lake / databricks over the past two years and stumbled across delta-rs about a year ago. For someone with no Rust experience (but a capable Python, R, SQL programmer), how would you recommend they get involved? I'm trying to wrap my mind around how all of this is put together and what ways I could contribute. For example, I've seen Arrow and DataFusion mentioned, is data fusion the key to to accomplishing a lot of what delta-rs wants to do? I would love to contribute to breaking delta free of being a spark only file format. Final note, I work in public health and the majority of our people use R as their primary language. I would love to help contribute to making it a reality to bring delta lake to our R users (whether that be through delta-rs or through duck db in the future).

Hey folks! I was super eager to start using

delta-rs@0.9.0

and I think I found a regression. https://github.com/delta-io/delta-rs/issues/1291

It's interesting because I see the same issue when using

string

as the column type as well. The resulting parquet file also doesn't happen to include that column.

There is also this issue i opened with regards datafusion and partition cols in the schema which existed prior to 0.9.0: https://github.com/delta-io/delta-rs/issues/1168

Hello. I was trying to test delta-rs

0.8.1

with a local

k8s

MinIO

. I get

thread '<unnamed>' panicked at 'not stream', /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/object_store-0.5.5/src/aws/credential.rs:173:27

File "/Users/simon/Library/Caches/pypoetry/virtualenvs/heureka-MBcEyvac-py3.9/lib/python3.9/site-packages/deltalake/table.py", line 122, in __init__
    self._table = RawDeltaTable(
pyo3_runtime.PanicException: not stream

storage_options = {"AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
"AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
"AWS_REGION": aws_region,
"AWS_ENDPOINT_URL": "localhost:30000"  # I also tried, trino-minio-svc.trino:9000, and host.docker.internal:30000
}

I know my delta table in MinIO works fine because I also use it for Trino, I was keen on testing with MinIO here instead of AWS S3.

DeltaTable(
    table_uri=table_uri,
    storage_options=self.storage_options,
    version=version,
)

Anyone encountered this issue? I found this open issue: https://github.com/delta-io/delta-rs/issues/809 Seems like an open issue perhaps?

I ported the existing Delta Rust docs to mdBook and created a POC website. Here is the code (just POC, will commit to delta-rs repo is folks like this approach). mdBook provides a lot of advantages over Sphinx documentation in my opinion: • Markdown instead of RST • mdBook makes it easy to properly structure URLs (sphinx creates URLs like

delta-rs/python/usage.html#loading-a-delta-table

that should be

delta-rs/python/usage/loading-a-delta-table

). This seems minor, but it’s actually a major SEO issue. • mdBook provides some nice formatting, navigation, and search options out of the box Interested in what folks think about this option. After we decide on the framework, we can brainstorm the optimal structure of the docs. I have some ideas that should work well from a readability & SEO perspective.

999 stars 🤩