The more time I spend with the arrow crate the more convince

Question

The more time I spend with the arrow crate the more convinced that I am we should switch to arrow2 by default as soon is it has the necessary support we need It s insanely complicated and has a fair bit of `unsafe` floating around I am running into more and more face palm scream cat

rtyler · Answer

hah found <https github com apache arrow rs blob master arrow array src delta rs|delta rs> in the arrow repo Not what I expected though laughing

Robert · Answer

I think arrow rs and arrow2 are in the process of merging or at least becoming compatible From what i understand is the main idea is to for now make arrow2 and arrays compatible by arrow rs adopting the lo level safe implementations from arrow2

rtyler · Answer

ohno laughing

Will Jones · Answer

There s definitely lots of APIs that need improving but in my experience the maintainers in arrow rs are very responsive to improvement ideas So it s probably worth creating issues on the repo if you find friction especially if there is a solution prototyped in arrow2 slightly smiling face

rtyler · Answer

Unfortunately because there s no writer support as far as I can tell in arrow2 there is no parallel The underlying problems that I am running into is that the RecordBatch API and underlying arrow array stuff is nightmarishly complex if you need to do any data manipulation

rtyler · Answer

I was just sitting down this evening to work on implementing what < Will Jones> already implemented partying face