https://delta.io logo
h

Harun

03/08/2023, 6:25 PM
Hi - I'm trying to create a delta table with
write_deltalake
It raises `deltalake.PyDeltaTableError: Schema error: Invalid data type for Delta Lake: Null`error if there is a column with all none values. Is there a way to solve this? E.g. define the schema for table or define column type
Copy code
import pandas as pd
from deltalake.writer import write_deltalake

data = [['tom', None], ['nick', None], ['juli', None]]
data = pd.DataFrame(data, columns=['Name', 'City'])
write_deltalake("s3a://.....", data, mode='overwrite')
j

JosephK (exDatabricks)

03/08/2023, 6:27 PM
I think doing a create table statement and then appending those records should work
h

Harun

03/09/2023, 9:05 AM
Create table with spark? Isn't there a solution with pure python?
j

JosephK (exDatabricks)

03/09/2023, 12:03 PM
you can create an empty dataframe in python and write that.
29 Views