Hi All,
I am getting below error when trying to get the value of timestamp field. Can anyone plz help me on this to fix it?
Elastic Search data in my index- index1:
{
“id”: “123”,
“__ts”: 1649289672103
}
Pyspark code:
es_df = spark.read.format(“es”).options(**es_conn_details)
.options(“es.read.metadata”,”true”)
.options(“es.read.metadata.version”,”true”)
.options(“es.mapping.date.rich”,”false”)
.load(“index1”)
.select(“id”, “__ts”)
es_xfm_df = es_df.selectExpr(“id”, “from_unixtime(__ts/1000, ‘yyyy-MM-dd HHmmss’) as __ts”)
Error when try to execute display(es_xfm_df):
RuntimeException: Error while encoding: Java.lang.RuntimeException: Java.lang.Long is not a valid external type for schema of string Caused by: RuntimeException: Java.lang.Long is not a valid external type for schema of string
Thanks in Advance