delta-lake #deltalake-questions

Chetan Joshi

03/30/2023, 1:10 AM

Hi everyone! We are using Deltalake 2.0.1 and trying to get value of

logRetentionDuration

for our delta table. Could someone advise on fetching this property using Python or Java? Thanks! We tried the following but “detail” method is not available in 2.0.1 it seem:

Copy code

DeltaTable deltaTable = DeltaTable.forPath(spark, pathToTable);
DataFrame detailDF = deltaTable.detail();

Martin Beaussart

03/30/2023, 1:41 PM

Hello Everyone ! I have a question regarding how PySpark works with Spark Clusters when loading Delta Table : I have a 6 TB Delta table, i am loading it in a dataframe for analysis works How is it possible for Spark to load a 6TB Delta table in a dataframe , it is impossible to load 6TB in the living memory isn't it ? So i guess it partition somehow the data and load it in memory in a 'queue' Can you help me to understand how clearly Spark can manage doing this ? Big thanks in advance !

Alber Tadrous

03/30/2023, 2:55 PM

Hi Team, I want to ingest a csv files to delta tables and I am facing a weird behavior , So I can read the file using spark context

spark.read.format("csv").load("s3://<file-path>")

but when I use Pandas it throws an error

Alber Tadrous

03/30/2023, 2:57 PM

image.png

Yan Zhao

03/30/2023, 3:55 PM

Could load the table due to the hadoop checksum error. Anyone can help to check it? https://github.com/delta-io/delta/issues/1675

rtyler

03/30/2023, 4:46 PM

is this an exam question? 😆

😂 5

Louis Frolio

03/30/2023, 7:35 PM

I just ran a simple test. 1. Create Delta table (myTable), load it up approx 65k rows 2. Three parquet files were created 3. One of the columns is called InvoiceID 4. Ran query: select max(InvoiceID) from myTable 5. I took a look at the query plan and noticed that the table was scanned to get the max(InvoiceID). I would have thought that Delta could leverage file stats to determine this? It seems that it would be a much more efficient process. Any and all feedback would be appreciated. Cheers, Louis.

Josué Castañeda Landa

03/31/2023, 6:32 AM

Good evening everyone, I have a question, I'm doing a merge, in delta lake, but when I look at the data, I can see that the condition doesn't work: whenNotMatchedInsertAll ,here is a sample of the data

Josué Castañeda Landa

03/31/2023, 6:32 AM

Copy code

Apple	iPhone14,7	Luis Alejandro	-vcbc6bf9169a54d92a4ae7783581cf7c0	16.3.1	iOS	Direct	2023-03-30 15:39:50.000	<http://hotmail.com|hotmail.com>
Apple	iPhone14,7	Luis Alejandro	-vcbc6bf9169a54d92a4ae7783581cf7c0	16.3.1	iOS	Direct	2023-03-30 15:39:50.000	<http://hotmail.com|hotmail.com>

Josué Castañeda Landa

03/31/2023, 6:33 AM

Has anyone solved this?

Christina

03/31/2023, 4:56 PM

Are versioned buckets advised when we use the delta format? Concerned it will blow up s3 costs

Yan Zhao

04/01/2023, 3:50 PM

Hello, I have a question about the ConflictChecker. I see the code:

Copy code

def checkConflicts(): Unit = {
    checkProtocolCompatibility()
    checkNoMetadataUpdates()
    checkForAddedFilesThatShouldHaveBeenReadByCurrentTxn()
    checkForDeletedFilesAgainstCurrentTxnReadFiles()
    checkForDeletedFilesAgainstCurrentTxnDeletedFiles()
    checkForUpdatedApplicationTransactionIdsThatCurrentTxnDependsOn()
    reportMetrics()
  }

I want to know how dose

checkNoMetadataUpdates()

work. Here I give a case to describe it. I have two writers, both of them will to update the metadata. Before update, the snapshot version is 1. Both of them commit the updateSchema action to the table, and the target metadata structure is same. The first writer commit 2.json succeed, the second writer commit 2.json failed, then the second writer will do the retry commit, and check the conflicting. In this case, dose the second writer

checkNoMetadataUpdates()

will throw

MetadataChangedException

Iwan Aucamp

04/03/2023, 1:31 PM

Probably not the best place to ask about Parquet, but what would be the “canonical” way of writing Parquet from go or python? is it with Apache arrow or using purpose built libraries?

Matt Dalesio

04/03/2023, 8:24 PM

Is it possible to share a view with Delta Sharing?

Gnanasoundari Soundarajan

04/04/2023, 7:56 AM

How to load a set of added files between the versions in deltalake and process?

swest

04/04/2023, 3:28 PM

Is there any way to write a field metadata with Python delta-rs? Looking at the API reference for

write_deltalake()

, it seems like we can only specify schemas with pyarrow, which unfortunately doesn't preserve the metadata dictionary in the conversion to a

deltalake.schema

Robert Thompson

04/04/2023, 4:01 PM

curious is anyone thinking of the ability to mask a column in delta? i know you can do it at the compute level using unity and databricks. But that doesn't apply if you read the table from a rust endpoint.

Nermin Yehia

04/04/2023, 10:46 PM

Hey, I'm using a dlt pipeline and I'm getting that error in initializing satage after long time of running.

Copy code

org.apache.spark.sql.AnalysisException: No query found for dataset 'dlt_table_123' in class 'com.databricks.pipelines.execution.core.languages.PythonPipeline'

Starkadur Hrobjartsson

04/05/2023, 11:47 AM

Hi, I'm trying to create a delta table locally containing a generated column as follows:

CREATE TABLE IF NOT EXISTS test_schema.test_table

timestamp TIMESTAMP,

year INT GENERATED ALWAYS AS (YEAR(timestamp))

USING DELTA LOCATION '/tmp/delta-lake/delta-table'

TBLPROPERTIES(delta.minWriterVersion = '4')

I get the following error:

Syntax error at or near 'GENERATED'

The same table I can create in Databricks with the appropriate location, catalog and schema. Is this feature exclusive to Databricks or am I missing something?

junhui huang

04/06/2023, 2:54 AM

Hi, I'm a new user of delta. How to use delta lake manage machine learning massive datasets, images (belong to a dataset), metadata (dataset metadata, images metadata)? Are there any suggestions or similar user cases？ Does the number of delta lake tables have an impact on performance, especially when the number of tables reaches hundreds of millions. Thanks.

Grainne B

04/06/2023, 4:00 AM

Hi all, I'm having an issue registering a delta table to the aws glue catalog I am using aws' glue docker image amazon/aws-glue-libs:glue_libs_4.0.0_image_01 When I try to create a delta table, I get this error

Copy code

23/04/06 01:55:48 WARN HiveExternalCatalog: Couldn't find corresponding Hive SerDe for data source provider delta. Persisting data source table `spark_docker_testing`.`aws_glue_docker` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/glue_user/spark/python/pyspark/sql/session.py", line 1034, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self)
  File "/home/glue_user/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
  File "/home/glue_user/spark/python/pyspark/sql/utils.py", line 196, in deco
    raise converted from None
pyspark.sql.utils.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: <s3://grainne-data-testingaws_glue_docker-__PLACEHOLDER__>
>>> spark.sql("create table spark_docker_testing.aws_glue_docker using delta location '<s3a://grainne-data-testing/workforce_good>'")
23/04/06 01:55:59 WARN HiveExternalCatalog: Couldn't find corresponding Hive SerDe for data source provider delta. Persisting data source table `spark_docker_testing`.`aws_glue_docker` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/glue_user/spark/python/pyspark/sql/session.py", line 1034, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self)
  File "/home/glue_user/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
  File "/home/glue_user/spark/python/pyspark/sql/utils.py", line 196, in deco
    raise converted from None
pyspark.sql.utils.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: <s3://data-testingaws_glue_docker-__PLACEHOLDER__>

Even when I use EMR (which works) I notice that when writing a delta file to s3 there is always a PLACEHOLDER appended to the s3 path. I can't find any relevant documentation for this online ? Is there a way to possibly create the delta table pointing to the generated manifest path ? Code to replicate

Copy code

# Run docker image 
docker run -it -v ~/.aws:/home/glue_user/.aws -e AWS_PROFILE=saml -e DISABLE_SSL=true --rm -p 4040:4040 -p 18080:18080 --name glue_pyspark amazon/aws-glue-libs:glue_libs_4.0.0_image_01

# Start pyspark shell 
pyspark --packages org.apache.hadoop:hadoop-aws:3.2.2,io.delta:delta-core_2.12:1.2.1 \

# Create table 
spark.sql("create table spark_docker_testing.aws_glue_docker using delta location '<s3://data-testing/test_data>'")

Nagendra Darla

04/06/2023, 5:44 AM

Hi, I am following this document to verify multi cluster writes on Amazon EMR. https://delta.io/blog/2022-05-18-multi-cluster-writes-to-delta-lake-storage-in-s3/ I do not see either the table or record getting created in DynamoDB. Am I missing anything? Any inputs are much appreciated. Below are my configurations

Copy code

SparkSession spark = SparkSession.builder()
        .config("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
        .config("fs.AbstractFileSystem.s3.impl", "org.apache.hadoop.fs.s3a.S3A")
        .config("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.DefaultAWSCredentialsProviderChain")
        .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
        .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
        .config("spark.delta.logStore.s3.impl", "io.delta.storage.S3DynamoDBLogStore")
        .config("spark.io.delta.storage.S3DynamoDBLogStore.ddb.tableName", "delta_log")
        .config("spark.io.delta.storage.S3DynamoDBLogStore.ddb.region", "us-east-1")
        .config("spark.io.delta.storage.S3DynamoDBLogStore.credentials.provider",
            "com.amazonaws.auth.DefaultAWSCredentialsProviderChain")
        .getOrCreate();

➕ 2

Ian

04/06/2023, 1:13 PM

Hi Team, I was creating a delta table with empty dataframe since yesterday its giving me the below error, let me know if anyone of you have come across this before.

Copy code

{
  "message": "Error while creating the delta table=<s3a://test/15f2a198-6302-410f-afc7-6f6aa5dd0f27/description_field_table/> Error=Failed to load checkpoint: Failed to read checkpoint content: Generic S3 error: response error \"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"\"<http://www.w3.org/TR/html4/strict.dtd>\">\r\n<HTML><HEAD><TITLE>Length Required</TITLE>\r\n<META HTTP-EQUIV=\"Content-Type\" Content=\"text/html; charset=us-ascii\"></HEAD>\r\n<BODY><h2>Length Required</h2>\r\n<hr><p>HTTP Error 411. The request must be chunked or have a content length.</p>\r\n</BODY></HTML>\r\n\", after 0 retries: HTTP status client error (411 Length Required) for url (<http://169.254.169.254/latest/api/token>)."
}

Satyam Singh

04/06/2023, 4:27 PM

I have created a delta table with columns . e.g. ColA, ColB, ColC, ColD. Scenario - First one notebook is writing to this delta table where dataframe has columns ColA, ColB (i.e. first 2 columns only) . Then a second notebook is writing to the same delta table where dataframe has columns ColA, ColC, ColD. When this second notebook is trying to write to delta table it is through error - "schema mismatch" (my investigation says because first notebook only wrote 2 columns). I do not want to enable schema merge option. Looking for suggestion.

Lucas Zago

04/06/2023, 5:27 PM

Hi guys, z-ordering a delta table always it is considered a good practice? There is some article speaking about it?

sabari dass

04/06/2023, 8:16 PM

Hi All, I have a use case where I need to write the data into same delta-lake location Parallely. When I try to do this using pyspark,I got the concurrency error is there any way to handle this issue?

Kenny Ma

04/07/2023, 12:00 AM

Hi - I am using the Delta connector for Flink to streaming Delta logs. When starting from an old version (>50 versions from the latest), it doesn’t consume any data. It works if the starting version is couple of versions from the latest. Has anyone seen this behavior or suggestions to troubleshoot/fix the issue? cc @Krzysztof Chmielewski

junhui huang

04/07/2023, 1:20 AM

Hi, Can Delta Lake be the backend of the online web system, providing the function of filtering and querying datasets? how much latency can generally achieve?

Ian

04/07/2023, 3:04 AM

anyone aware of this ?

Jalil Alchy

04/07/2023, 8:45 PM

Maybe a simple question, but when using the Delta Connector for Flink, if I set the RowKind to Update or Delete, will this be written or does the connector strictly support inserts?