Anmol Rastogi

05/05/2023, 7:55 AM
hi #deltalake-questions I am trying to read a deltalake table using pure python and trying to create a pipeline using pandas :
Copy code
This Python module is used to load data from RDS to data lake storage (i.e. S3 with respect to AWS)

import os
import deltalake

env = 'dev'
region = 'ap-south-1'

os.environ["ENV"] = env
os.environ["REGION"] = region

bucket = 'analytics-pipeline-ap-south-1'

def main():
    data_read = deltalake.table.DeltaTable(f's3://{bucket}/DeltaLake_convert/Anmol', storage_options={'region': region})

# entry point for ELT application
if __name__ == '__main__':
while doing this i am getting following error: deltalake.PyDeltaTableError: Not a Delta table: No snapshot or version 0 found, perhaps s3://{bucket}/DeltaLake_convert/Anmol is an empty dir? while if you see below image the deltalog directory and parquet exists there. am i doing something wrong? Note: I am not looking for something with spark. 🙂

Gerhard Brueckl

05/05/2023, 8:32 AM
can you open the _delta_Log folder?_ do you see a
file in there?

Anmol Rastogi

05/05/2023, 9:47 AM
hi @Gerhard Brueckl
This is what i am seeing
i am writing the delta with the following snippet
df_read = pd.read_csv(StringIO(v_file['Body'].read().decode('utf-8'))) print(df_read) try: write_deltalake(f's3://{bucket}/flask_poc/dev/DeltaLake_convert/Anmol', df_read, storage_options={'region': region}) except Exception:"Wait for sometime")

Gerhard Brueckl

05/05/2023, 9:53 AM
according to the _delta _log folder no data was written yet

Anmol Rastogi

05/05/2023, 9:55 AM
I have tried the documented way to write deltalake but it isn’t working. Do you have any references that i can follow?

Gerhard Brueckl

05/05/2023, 10:05 AM
sorry, I am not familiar with S3 or using pure python for this
❤️ 1

Will Jones

05/05/2023, 2:52 PM
If all you see is
in the
folder, that means the transaction failed to commit. I notice in your snippet you have a
try: … except:
that captures all exceptions. That’s probably hiding the error that’s causing it to fail.