DVC
DVC is a command line tool and VS Code plugin used to manage data sets, models and experiments in machine learning projects. While keeping data and model files in cloud storage areas, version information of these files follows through Git repositories. It defines dependencies within the project, allowing light data lines to be created where only the changing steps are executed.
DVC
84iterative · MLOps & AI Infrastructure
Common commercial products in this category: Weights & Biases, Amazon SageMaker, Databricks
- You can version it with Git by keeping your data sets and models on cloud storage.
- With the data lines you define, you can only run the steps that have been changed automatically.
- You can follow different experiments through your Local Git repository, compare code, data, and performance metrics.
- • NeutralHacker News
DVC – Open Source Machine Learning Version Control System
“We had troubles with setups on different OS, hard to understand error messages, and in general had to use multiple repos, some with Git LFS and some with "standard" Git and combine each, which was a mess. We ended up deleting the local clones and re-cloning quite often, and…”
- • NeutralHacker News
DVC – Open Source Machine Learning Version Control System
“Good points, I'd appreciate if you could elaborate since it seems you've thought a lot about this. > Docker can help only if there is a single step in your project. In ML projects you usually have many steps - Preprocess, Train. Each of the steps can be divided: extract Evaluate…”