Just got off the phone with
@Michael Shtelma who is working on
the torchdelta connector. I think a PyTorch connector is a wonderful idea! Suggested next steps (feel free to chime in with thoughts):
• possibly rename the project to deltatorch consistent with deltaray & deltadask
• switch to poetry consistent with the other Delta Lake Python ecosystem projects
• Publish to PyPi
• I can then make a Jupyter notebook & write a blog post. We can work on the story for why Delta Lake is great for PyTorch analyses.
I am guessing that the file skipping and versioned data would be huge benefits for the PyTorch community. Anyone else interested in getting involved with this project?