`spark read format xml option rowTag Documents load data pat

Question

`spark read format xml option rowTag Documents load data pat 66086B XML write format delta mode append save <s3a delta lake demo1> ` ```org apache spark sql AnalysisException Failed to merge fields PatentDocument and PatentDocument Failed to merge fields BibliographicData and BibliographicData Failed to merge fields Parties and Parties Failed to merge fields Agents and Agents Failed to merge fields Agent and Agent Failed to merge incompatible data types StructType StructField Name StructType StructField VALUE StringType true StructField lang StringType true true StructField format StringType true StructField sequence LongType true and ArrayType StructType StructField Name StructType StructField VALUE StringType true StructField lang StringType true true StructField format StringType true StructField sequence LongType true true ``` Why do the above errors occur

JosephK (exDatabricks) · Answer

The XML shockingly didn t parse correctly

Jim Hibbard · Answer

Hi Abel It looks like your fields change data types part way through your xml parse You ll have to resolve that to avoid the error Either casting the incoming values to have the same dtype or merging the schemas so that the dtypes are flexible enough to accommodate all incoming values Hope that helps

吕 · Answer

I am sure that the XML is parsed properly

吕 · Answer

< Jim Hibbard> I ll give it a try

Jim Hibbard · Answer

Sounds good It looks like initially the field is parsing as an ArrayType and later as a StructType If you look at the schema of your destination table and compare it to the schema of the parsed and unmerged records what do you get

吕 · Answer

< Jim Hibbard> As you said changing the field format to String is OK tks

Jim Hibbard · Answer

Excellent Glad that worked for you Let me know if you need anything else raised hands

Kees Duvekot · Answer

< Jim Hibbard> if the XML has pointers to the XSD or even multiple It would be very handy when the XSDs are used to determine the actual schema instead of just the contents which is what it looks to be doing now

Jim Hibbard · Answer

I m not very familiar with XSD would you be up for creating a PR describing this approach If there s already schemas built into a document it d be great to take advantage of that I could see some complexity cropping up if XSD schemas don t map well onto Delta Lake s data types or the schemas are especially difficult to parse in an inconvenient part of the document But parsing XML is a common use case so definitely interested in any enhancements possible Thanks Kees

Jim Hibbard · Answer

Also if not up for writing that PR I can definitely submit something on your behalf It d be a huge help if you could jot some bullets down on what you d like to see though

Kees Duvekot · Answer

I know how to write Programmer but I am not one But a writeup on what I am thinking about is definitely possible

Kees Duvekot · Answer

Let me plan something for later this week

Jim Hibbard · Answer

Thank you so much A write up is exactly what we need You definitely don t need to contribute the whole feature your insights are super valuable though For example I didn t realize XSD existed until you mentioned it Thanks Kees raised hands