hello Team i am getting below error while reading delta tabl

Question

hello Team i am getting below error while reading delta table data using athena external table from manifast loc

Jon Stockham · Answer

don t use athena table in spark

Rahul Sharma · Answer

Then how I read external table in spark

Rahul Sharma · Answer

Earlier it was running

Jon Stockham · Answer

why do you need to just use the delta table itself

Rahul Sharma · Answer

Bcz my end user want to see the schema of the table but delta doesn t show columns in Athena

Jon Stockham · Answer

ok so keep the external table for them but why do you need to use the external table in your spark code

Rahul Sharma · Answer

my existing etl was using the athena table so i was changed the pointing

Jon Stockham · Answer

Well that s reason for the error

Rahul Sharma · Answer

but earlier i was able to read the data

Jon Stockham · Answer

yes because the catalog was referencing a delta table now it is referencing an athena compatible external table

Rahul Sharma · Answer

so is there any way to see delta column metadta in athena

Jon Stockham · Answer

yes with an external table they are not mutually exclusive you can have both

Jon Stockham · Answer

use the native delta table catalog entry for your spark code have another catalog table which will be your external table

Rahul Sharma · Answer

``` sql CREATE EXTERNAL TABLE udp cash `id` int `status` string `value` int `dt` date ROW FORMAT SERDE org apache hadoop hive ql io parquet serde ParquetHiveSerDe LOCATION <s3 udp lakehouse dev jwr cash symlink format manifest > ```

Jon Stockham · Answer

and just remember to re generate manifest whenever you write

Rahul Sharma · Answer

if i create native table from glue crawler then also i have to re generate manifest file

Rahul Sharma · Answer

can you give me command how to create native delta table catalog

Jon Stockham · Answer

you just said this was how you had it before you changed it to an external table

Rahul Sharma · Answer

i have created by below command ``` sql CREATE EXTERNAL TABLE udp cash `id` int `status` string `value` int `dt` date ROW FORMAT SERDE org apache hadoop hive ql io parquet serde ParquetHiveSerDe LOCATION <s3 udp lakehouse dev jwr cash symlink format manifest > ```

Rahul Sharma · Answer

is this ok

Rahul Sharma · Answer

i have one more doubt if my streaming continuously writing the data into refine zone then how i re generate manifast file

Jon Stockham · Answer

I just seem to confuse you more when I try to help perhaps someone else can explain clearer

Rahul Sharma · Answer

i have below doubts 1 if my streaming running continuously then how to regenerate my mani fast file 2 what is the best way to create external delta table

Jon Stockham · Answer

with code read <https docs delta io latest presto integration html> or you can perform both tasks with glue crawlers