So just a quick question why don t we have anything on the m

Question

So just a quick question why don t we have anything on the main documentation page for how to write to delta tables Like I m trying to figure it out and just been scattered across the repo From what I gather I build a `DeltaWriter` write then close it to get the `Add` actions which I then have to commit myself

John Darrington · Answer

as it states here <https docs rs deltalake latest deltalake writer record batch index html> just feels like maybe having it also on the front page makes sense

Robert · Answer

We actually have two implementaions of the RecordBatchWriter or rather we have `operations writer DeltaWriter` as well Its been on my todo list for a while re consolidate those This is an artifact from introducing conflict resolution without breaking downstream applications

Robert · Answer

100 agree that this needs to be better discoverable and we should finally converge on one slightly smiling face

Will Jones · Answer

I d add that we probably have two different write use cases one is users who have streams of rows like the kafka ingest project and the other who have columnar batches from polars or datafusion or elsewhere So we should be able to guide each user type to the right interface

Will Jones · Answer

As an aside a while ago I started to look at improving our rust docs but found that our module structure was such a mess that I felt like we should fix that up first Created <https github com delta io delta rs issues 1136> I think that s nearly done

John Darrington · Answer

Lol I feel like each time I ask a question I open a can of worms in a good way First the locking store and now documentation D

Will Jones · Answer

ha sorry if I make it feel more complicated than it is I just like to provide additional context when I can slightly smiling face

John Darrington · Answer

I do the exact same thing so I get it

Will Jones · Answer

Any incremental improvement is very welcome

Matthew Powers · Answer

< John Darrington> thanks for the great questions and horray for the culture of constant incremental improvements

John Darrington · Answer

Of course I ll see what I can do to help with documentation

Robert · Answer

we also still have <https github com delta io delta rs pull 1434> open which implements most of what Wills issue mentions Specifically it tries to isolate future kernel stuff a bit more Would be happy to re open it or cherry pick some re orgs base on your feedback

rtyler · Answer

< Robert> I was actually looking at that before I cut the last rust release I wasn t certain whether I should try to resolve a bunch of those conflicts or not I feel really bad you did that work right before summit and everything got stale really quick confused

Robert · Answer

No worries slightly smiling face just did some merging with main and that was quite quickly resolved

Robert · Answer

gt I d add that we probably have two different write use cases one is users who have streams of rows like the kafka ingest project and the other who have columnar batches from polars or datafusion or elsewhere So we should be able to guide each user type to the right interface Fully agree I think Kafka ingest only uses the JSON writer right < rtyler> Right now its still not quite a streaming API for writing the json batches AFAIK tyler recently moved to the new json decoder APIs already which can quite readily be made to process streams just did something similar in kernel slightly smiling face