Hi, Is there any slack group for Databricks? (not just Delta Lake)

Hi All, I have few

.SQL

in my repo. Is there any way to execute those SQLs on Databrivks via Azure DevOps CO/CD pipeline. Please help I'm very new to CI/CD. Thanks!

Hi All, I'm new to Delta Lake and considering to use Delta lake for one of the project with S3 or GCS as file storage. I would like to understand how the data cataloging works. Does the open source delta lake automatically creates and maintains data catalog when we create delta tables or do we need call any APIs to register the table metadata with catalog? Any pointers to documentation would be helpful. thanks.

Hello I have 1 question. Is there any application or open source can query data in Delta lake-house ? And return json data to client application, not using delta-sharing

https://www.databricks.com/company/newsroom/press-releases/announcing-delta-lake-30-new-universal-format-offers-automatic 🤯

Hi... I'm Albert, the developer advocate @ StarRocks. If you have questions about how StarRocks can be used to query delta lake, let me know. BTW, here's a screenshot of StarRocks in the DataBrick's CEO keynote. What is unique about us is if you use StarRocks to query instead of Spark, you'll get 2x to 5x more faster query responses.

Delta Lake 3.0 Out of nowhere, yesterday Databricks announced the preview release of Delta Lake 3.0.0. I was wondering: was this a secret ninja project of Databricks? How does this fit together with Delta being officially open-sourced since 2022? Was OSS community part of this release or was it implemented Databricks interally exclusively? I did not hear about it until a day ago. Don't get me wrong. I like all these new features but I don't understand the governance model of Delta Lake OSS project if Databricks is pulling all the strings in the background. And if DB is serious about OSS I don't understand why features like

identity columns

still not have been donated to OSS; contrary to promises made a year ago:

NEW Delta Lake blog by @Nick Karpov 🦀 This blog introduces the new Deletion Vectors table feature for Delta Lake tables, and explains how Deletion Vectors speed up operations that modify existing data in your lakehouse! ➡️ https://delta.io/blog/2023-07-05-deletion-vectors/

Hi folks. Recently wrote a piece on experimenting with the new Liquid Clustering feature (presented during the Data + AI Summit). You can read it on LinkedIn or Medium if interested.

@rtyler and the team graciously accepted my offer to donate kotosiro-sharing, the Rust implementation of the Delta sharing server! It's easy to read because the implementation follows the typical DDD design pattern, so I believe it's a good starting point for beginners in Rust programming (as I began it for my own learning purposes). I am planning to implement its GUI using TypeScript/Next.js, making it a great opportunity for frontend enthusiasts who are also curious about data engineering. The repository still requires some adjustments due to the move, but I will address them in the near future. https://github.com/delta-incubator/delta-sharing-rs

Feel free to bore yourself stupid watching me present integrating data from Unity Catalog into Microsoft Purview from my presentation at SQLBits this year https://www.linkedin.com/feed/update/urn:li:activity:7082723016526176256/

Hey everyone 👋 Anyone here working with LLMs or Langchain in particular? Have been working on this open source gateway to enable LLM agents to securely perform read and write operations in tools like Airtable, Notion, Hubspot and Slack (GitHub: https://github.com/felixbrock/lemonai-py-client). This makes it easy to go beyond information retrieval use cases and build powerful AI assistants that can reliably perform workflow automations. Happy to answer any questions and please do reach out should you have any feedback on this - I’d really appreciate to hear your thoughts 🙏

Just published a piece on UniForm with Delta lake. presented at the Databricks Data + AI Summit. You can read it on LinkedIn or Medium, if interested.

Check out the latest edition of 𝙻𝚊𝚜𝚝 𝚆𝚎𝚎𝚔 𝚒𝚗 𝚊 𝙱𝚢𝚝𝚎! We recap all of the great Delta Lake sessions at this year's Data+AI Summit held in San Francisco, CA in this 2-part series. 🦀 Special shoutout to @will for hosting this week's 𝙻𝚆𝙸𝙰𝙱! Check it out here: https://www.linkedin.com/pulse/last-week-byte-2023-07-18-deltalake/

A good read: https://lakefs.io/blog/is-rust-the-future-of-analytics/

Hi every one I'm using delta-kernel to read from a table using JAVA. I'm able to get table files but i'm not able to read data.

Configuration conf = new Configuration(); 

        TableClient myTableClient = DefaultTableClient.create(conf);  

        Table myTable = Table.forPath("Table/Path");  
        Snapshot mySnapshot = myTable.getLatestSnapshot(myTableClient);  


        Scan myScan = mySnapshot.getScanBuilder(myTableClient).build();
        Row state = myScan.getScanState(myTableClient);
        
        CloseableIterator<ColumnarBatch> FilesBatchIter = myScan.getScanFiles(myTableClient);
        Optional<Expression> exp = Optional.empty();

this is the configuration that seems right.(I dont know about Optional<Expression>) Not i want to use them to fetch data from table

while (FilesBatchIter.hasNext()) {
            CloseableIterator<Row> files = FilesBatchIter.next().getRows();
            CloseableIterator<DataReadResult> dataResult = Scan.readData(myTableClient, state, files, exp);
            
            // prints the file names
            while (files.hasNext()) {
                Row file = files.next();
                String column1Value = file.getString(0);
                System.out.println(column1Value);

            }

            // Checking if any result is fetched by Scan.readData
            while (dataResult.hasNext()) {
                System.out.println("Data Found!");
            }


        }

It will prints file names but it seems there would be nothing in dataResult. Any Ideas? or any sample code?

Hi All, Is there any way to copy Databricks Notebook from Workspace to DBFS. Please help.

Anyone aware of why this issue occurrs with delta https://github.com/delta-io/delta/issues/1971

Hi All, Can anyone help me resolve the below issue: I have a scenario where I'm converting dataframe to dictionary and iterating over every value to get the absolute value. But getting error. Refer SS-1 & SS-2. However, when I'm running the same code directly on hard coded dictionary same code is working fine. Refer SS-3 & SS-4. Please help. Thanks!

Hi guys. I am trying to convert tables columns names upper case to lower case . For that i am using below code. for table in table_list:     spark.sql(f'''ALTER TABLE {table}         SET TBLPROPERTIES (         'delta.columnMapping.mode' = 'name',         'delta.minReaderVersion' = '1',         'delta.minWriterVersion' = '2')''')     print(table)     col_list=spark.table(table).columns     if len(col_list) > 0 :         for col in col_list:             spark.sql(f'''ALTER TABLE {table} RENAME COLUMN {col} to {col+'_tmp'} ''')             spark.sql(f'''ALTER TABLE {table} RENAME COLUMN {col+'_tmp'} to {col.lower()} ''') I am getting below error in sometimes if i rerun its working fine. AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: at least one column must be specified for the table Can anyone help me out here why i am getting above error. Thanks

Attribute Lineage for PySpark DataFrame My question is not Delta specific, sorry for that. I was wondering if it would be possible to extract attribute lineage from the execution plan of a PySpark DataFrame. Pseudo-Example:

df = spark.table("myTable").withColumn("C", col("A") + col("B")).withColumnRenamed("A", "Z")

magicLineageAnalyzer(df)
>> df.Z <-- myTable.A
>> df.B <-- myTable.B
>> df.C <-- myTable.A, myTable.B

what's in your datalake? 😆 https://infosec.exchange/@0x58/110814917022499333

Hi Everyone, I am trying to dispaly dataframe which have 20 rows.but i am getting below error. com.databricks.rpc.RPCResponseTooLarge: rpc response (of 20979958 bytes) exceeds limit of 20971520 bytes Can anyone please help here to fix this issue. Thanks

Hello. I am building a low code ELT platform (as a hobby project) which currently is powered by delta-rs and duckdb right now. Please have a look at my tool and if there is any interest please get in touch! The tool is oriented for business analysts/data scientists working in data teams wanting to automate ELT pipelines faster. https://wrapbytes.com/ https://wrapbytes.com/2023/09/09/demo-update/ If you happen to know anyone in the space that might be interested, please get in touch! Thanks

Hey all, happy to share this article about Event-Driven Architecture with Serverless Functions. Combining EDAs and serverless functions will enable you to build a scalable and efficient solution for many microservices-based apps. So what are the differences, and similarities, and why combining both? https://medium.com/@memphis-dev/event-driven-architecture-with-serverless-functions-part-1-d2b54babc193

Shoutout to @Florian Valeye for being at Open Source Summit EU this week representing Delta Lake! 🦀🎉 Check him out interviewing with The New Stack today. You can tune into his session tomorrow, Sept 20 virtually! See here: https://osseu2023.sched.com/event/1OIhf

Hey all! I've been exploring Rust for a few months and am excited to introduce "spark-connect-rs," an experimental Spark connect client for Rust. 🦀 🎉 You can check out the docs here! https://docs.rs/spark-connect-rs/0.0.1-beta/spark_connect_rs/ While Rust isn't my day-to-day language, blending it with my daily use of Spark/Delta seemed like a good way to learn it. There’s gonna be some areas of improvement in that rust code for sure 🙂 Be sure to checkout other Spark connect clients like the one for golang from “apache/spark-connect-go” or another rust one from “edmondop/spark-connect-rust”. Cheers!