Matthew Powers

04/12/2023, 6:07 PM
Just got off the phone with @Michael Shtelma who is working on the torchdelta connector. I think a PyTorch connector is a wonderful idea! Suggested next steps (feel free to chime in with thoughts): • possibly rename the project to deltatorch consistent with deltaray & deltadask • switch to poetry consistent with the other Delta Lake Python ecosystem projects • Publish to PyPi • I can then make a Jupyter notebook & write a blog post. We can work on the story for why Delta Lake is great for PyTorch analyses. I am guessing that the file skipping and versioned data would be huge benefits for the PyTorch community. Anyone else interested in getting involved with this project?
🤔 1

Jim Hibbard

04/12/2023, 6:59 PM
Count me in, there's probably some consideration with how this will dovetail with existing tooling in the space. But I think we can definitely figure that out and help contribute 🙂