Data Version Control

by shcheklein • 2 years ago

213 52 2 years ago

jerednel• 2 years ago

It's not super clear to me how this interacts with data. If I have am using ADLS to store delta tables, and I cannot pull prod to my local can I still use this? Is there a point if I can just look at delta log to switch between past versions?

riedel2• 2 years ago

DVC is (at least as I use it) pretty much just git LFS with multiple backends (guess actually a more simple git annex). It further has some rather MLOps specific stuff. Is handy if you do versions model training with changing data on S3.

starkparker• 2 years ago

I've used it for storing rasters alongside georeferencing data in small GIS projects, as an alternative to git LFS. It not only works like git but can integrate with git repos through commit and push/pull hooks, storing DVC pointers and managing .gitignore files while retaining directory structure of the DVC-managed files. It's neat, even if the initial learning curve was a little steep.

We used Google Drive as a storage backend and had to grow out of it to a WebDAV backend, and it was nearly trivial to swap them out and migrate.

haensi• 2 years ago

There’s another thread from October 2022 on that topic.

https://news.ycombinator.com/item?id=33047634

What makes DVC especially useful for MLOps? Aren’t MLFlow or W&B solving that in a way that’s open source (the former) or just increases the speed and scale massively ( the latter)?

Disclaimer: I work at W&B.

matrss• 2 years ago

Speaking of git-annex, there is another project called DataLad (https://www.datalad.org/), which has some overlap with DVC. It uses git-annex under the hood and is domain-agnostic, compared to the ML focus that DVC has.

shicholas2• 2 years ago

What are the benefits of DVC over Apache Iceberg? If anyone used both, I'd be curious about your take. Thanks!

andrew_lettuce• 2 years ago

I don't see any real benefits, as it feels like using the tool you already know even though it's not quite right. Iceberg is maybe geared towards slower changing models than this approach?

foobarbecue• 2 years ago

username checks out

dijksterhuis• 2 years ago

head is a bit discombobulated today, but i’ll give this a shot

when i say ‘blob’ data, a good example to think of is a set of really long 1080p video files.

tl;dr version

* throw data into dvc when unstructured ‘blob’ data.

* throw it into iceberg when you’ve got structured data.

benefits of dvc over iceberg:

* not forcing ‘blob’ data into a tabular format and all the “fun” (read: annoying) processing steps that come with doing that

* don’t have to have to run some processing step to extract ‘blob’ data out of what is basically a parquet file, dvc pull (?) will just download each file as is.

* edit files locally then run three-ish (?) commands to commit changes, without needing to run a data ingestion pipeline to force ‘blob’ data into a table

* completely schema less, so don’t have to worry about ‘blob’ data being the wrong type, just shove it in the repo and commit it

* roll back throughout all of commit history, not just to the last vacuum/checkpoint

basically, tabular data formats and ‘blob’ data shoved into them is a recipe for pain.

shoving ‘blobs’ into a git like repo is much faster and easier.

especially if you need full version history, branches for different outcomes etc.

trying to have different branches in Iceberg for your set of really long 1080p video files where you have applied different ffmpeg filters in different branches and want people to be able to access all of them, and the history of them sounds nightmare-ish.

in dvc, that’s ^ easy.

basically, it’s like creating a data lake which won’t turn into a data swamp because everything is version controlled.

shicholas2• 2 years ago

This is great thank you!

tomnicholas12• 2 years ago

If you're wondering this you should look at Icechunk too, which was open-sourced just this week. It's Apache Iceberg but for multidimensional data (e.g. Zarr).

https://earthmover.io/blog/icechunk

https://news.ycombinator.com/item?id=41850352

bramathon• 2 years ago

I've used DVC for most of my projects for the past five years. The good things is that it works a lot like git. If your scientists understand branches, commits and diffs, they should be able to understand DVC. The bad thing is that it works like git. Scientists often do not, in fact, understand or use branches, commits and diffs. The best thing is that it essentially forces you to follow Ten Simple Rules for Reproducible Computational Research [1]. Reproducibility has been a huge challenge on teams I've worked on.

[1] https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...

bach4ants2• 2 years ago

I have noticed this as well. There is a huge resistance to learning Git, and I think it's partly warranted. Researchers know what it is, and know that it's valuable, but think it will take too long to learn and they want to move fast. I recently started building a tool called Calkit (https://github.com/calkit/calkit) in an attempt to simply and unify Git and DVC for these types of researchers. Hoping to convince folks that working reproducibly is actually faster in the long run, never mind the fact that it makes their work more directly usable for pushing the field forward more quickly overall.

dmpetrov5• 2 years ago

hi there! Maintainer and author here. Excited to see DVC on the front page!

Happy to answer any questions about DVC and our sister project DataChain https://github.com/iterative/datachain that does data versioning with a bit different assumptions: no file copy and built-in data transformations.

ajoseps• 2 years ago

if the data files are all just text files, what are the differences between DVC and using plain git?

dmpetrov5• 2 years ago

In this cases, you need DVC if:

1. File are too large for Git and Git LFS.

2. You prefer using S3/GCS/Azure as a storage.

3. You need to track transformations/piplines on the file - clean up text file, train mode, etc.

Otherwise, vanilla Git may be sufficient.

miki123211• 2 years ago

DVC does a lot more than git.

It essentially makes sure that your results can reproducibly be generated from your original data. If any script or data file is changed, the parts of your pipeline that depend on it, possibly recursively, get re-run and the relevant results get updated automatically.

There's no chance of e.g. changing the structure of your original dataset slightly, forgetting to regenerate one of the intermediate models by accident, not noticing that the script to regenerate it doesn't work any more due to the new dataset structure, and then getting reminded a year later when moving to a new computer and trying to regen everything from scratch.

It's a lot like Unix make, but with the ability to keep track of different git branches and the data / intermediates they need, which saves you from needing to regen everything every time you make a new checkout, lets you easily exchange large datasets with teammates etc.

In theory, you could store everything in git, but then every time you made a small change to your scripts that e.g. changed the way some model works and slightly adjusted a score for each of ten million rows, your diff would be 10m LOC, and all versions of that dataset would be stored in your repo, forever, making it unbelievably large.

agile-gift0262• 2 years ago

It's not just to manage file versioning. Yo can define a pipeline with different stages, the dependencies and outputs of each stage and DVC will figure out which stages need running depending on what dependencies have changed. Stages can also output metrics and plots, and DVC has utilities to expose, explore and compare those.

johanneskanybal• 2 years ago

Mostly consult as a data engineer not ML ops but I’m interested in some aspects of this. We have 10 years of parquet files from 300+ different kafka topic and we’re currently migrating to apache iceberg. We’ll back fill on a need only basis and it would be nice to track that with git. Would this be a good fit for that?

Another potential aspect would be tracking schema evolution in a nicer way than we currently do.

thx in advance, huge fan of anything-as-code and think it’s a great fit for data (20+ years in this area).

stochastastic2• 2 years ago

Thanks for making and sharing DVC! It’s been a big help.

Is there any support that would be helpful? I’ll look at the project page too.

dmpetrov5• 2 years ago

Thank you!

Just shoot an email to support and mention HN. I’ll read and reply.

causal• 2 years ago

This useful for large binaries?

dmpetrov5• 2 years ago

Yes. And if you track transformations of the binaries or ml training

natsucks• 2 years ago

Would appreciate a good answer to this question. I deal with large medical imaging data (DICOM) and i cannot tell whether it's worth it and/or feasible.

thangngoc892• 2 years ago

It's very much feasible. I'm currently using DVC for DICOM, the repo has growth to about 5TB of small dcm files (less than < 100KB each). We use a NFS mounted NAS for development but the DVC's cache needs to be on the NVMe, otherwise performance would be terrible.

tomnicholas12• 2 years ago

You should look at Icechunk. Your imaging data is structured (it's a multidimensional array), so it should be possible be to represent it as "Virtual Zarr". Then you could commit it to an Icechunk store.

https://earthmover.io/blog/icechunk

mkbehbehani• 2 years ago

Yes, I’ve been using it for about a year to populate databases with a reference DB dump. The current file is about 18GB. I use cloudflare R2 as the backing store so even though it’s being pulled very frequently the cloudflare bill is a few bucks per month.

kbumsik• 2 years ago

Large files are good, but it may have performance issues with many (millions) small files.

notrealyme1232• 2 years ago

I had a lot of problems when using it with a dataset of many jpg Files.

The indexing for every dvc status took many minutes to check every file. Caching did not work.

Sadly I had to let go of it.

woodson• 2 years ago

Yes, its performance is rather poor and there can be a lot of headaches with caching (especially if you're using a file system that doesn't support reflinks). For large sharded datasets (e.g. WebDataset), you're better off with other solutions, especially when your ML pipeline can stream them directly from object storage.

dmpetrov5• 2 years ago

Right, DVC caches data for consistency and reproducibility.

If caching is not needed and streaming required, we've created a sister tool DataChain. It's even supports WebDataset and can stream from tar archives and filter images by metadata.

WebDataset example: https://github.com/iterative/datachain/blob/main/examples/mu...

jiangplus• 2 years ago

How does it compare to Oxen?

https://github.com/Oxen-AI/Oxen

jFriedensreich• 2 years ago

never heard of oxen but it looks like a super interesting alternative. would love to hear from someone who has experience with both.

my first impression: dvc is made to use with git where there are arbitrary folders handled by dvc INSIDE your git repo, where oxen is an alternative for a separate data repo. also oxen has lots of integration with dataframes and tabular, ai training and infernece data that dvc is missing. on the other hand dvc has a full DAG pipeline engine integrated as well as import/ export and pluggable backends.

gregschoeninger• 2 years ago

Maintainer of Oxen here, we initially built Oxen because DVC was pretty painfully slow to work with, and had a lot of extra bells and whistles that we didn’t need. Under the hood we optimized the merkle tree structure, hashing algorithms, network protocols, etc to make it speedy when it came to large datasets. We have a pretty nice front end at https://oxen.ai for viewing and querying the data as well.

Happy to answer any thoughts or questions!

bagavi• 2 years ago

Can this be used with GitHub? If yes, I would shift from dvc immediately

ulnarkressty• 2 years ago

We actually were considering DVC, however for our particular use case (huge video files which don't change much) the git paradigm was not that useful - you need at least one copy of the data on the origin and another one on each system that's doing the training. So in the end we just went with files and folders on a NAS, seemed to work good enough.

A hybrid solution of keeping dataset metadata under DVC and then versioning that could work. This was many years ago though and I would be curious if there are any other on-prem data versioning solutions, when I last searched all of them seem geared towards the cloud.

tomtom1337• 2 years ago

The animated ripple across the «what’s new» button is infuriating. It keeps drawing my attention from reading what this is.

FergusArgyll• 2 years ago

I bound Ctrl-Shift-Z to Ublock Origin's Zapper mode. It's really helpful

jendefig• 2 years ago

Thank you for your feedback! We can see how this could be distracting. I'm the Community Manager, this has been fixed!

sohooo• 2 years ago

I also heart about lakeFS for data versioning on S3 object stores. Is DVC a contender in this area?

dpleban• 2 years ago

Great to see DVC being discussed here! As a tool, it’s done a lot to simplify version control for data and models, and it’s been a game-changer for many in the MLOps space.

Specifically, it's a genius way to store large files in git repos directly on any object storage without custom application servers like git-lfs or rewriting git from scratch...

At DagsHub [0], we've integrated directly with DVC for a looong time, so teams can use it with added features like visualizing and labeling datasets managing and models, running experiments collaboratively, and tracking everything (code, data, models, etc.) all in one place.

Just wanted to share that for those already using or considering DVC—there are some options to use it as a building block in a more end-to-end toolchain.

[0] https://dagshub.com

Data Version Control

Related Stories