Change remote globally in Git history

[dmpetrov]

Changing buckets and cloud easily for a single project is a very compelling feature.

But in fact, when I need to transfer a data-remote from one bucket to another (or another cloud) I can do that properly only on HEAD of the repo. All the old commit will have old remote (in .dvc/config). As a result, when I checkout back in Git history an old remote will be used.

So, I need to keep old data remote (bucket) or I'll have troubles using my old Git commits.

Is it possible to make remote settings "global"? A single remote change should change it everywhere in the Git history. How Git does that and can it work for DVC? Are there other options?

All ideas are welcome!

[efiop]

@dmpetrov This is mostly about properly maintaining your project. E.g. using tags or branches for released versions, which can be easilly updated to point to the new bucket.

[efiop]

E.g. say some project died and closed its remote, but someone has the cache somwhere. In that case, you fork it and put your own remote and rely on it. Or, you could specify -r remote to specific commands, like import/get(which we don't have support for right now, but we do have a ticket opened).

[dmpetrov]

@efiop could you please elaborate? How should the team properly maintain the project to minimize this amount of work?

Use case: A team has a long-running project (year+), hundreds of commits, dozens of releases (with tags). All Git history is important. Some clients might still use old ML models, all changes in data sources have to be fully tracked from the inception. Not released commits (without tags) might be also important (we decided to release an old but a promising experiment).

Suddenly the team decided to change a cloud provider. It looks like a tremendous amount of work has to be done to make it happen. It would be great if DVC can make it in a few commands.

[efiop]

Oops, sorry for the delay.

Not released commits (without tags) might be also important (we decided to release an old but a promising experiment).

Well, giving that to your customers was a bad idea from beginning 🙂

So say you have some tag like v1 that was using s3. Now you are migrating to gs, so what you do is you go to v1, create a branch from it, adjust the remote to point to the new gs location, commit, move v1 to this new commit. Then you'll have to make your users update.

But if you, as a maintainer, would take it more seriously earlier, you would create some kind of proxy remote. E.g. http remote (like we do in dvc core project) that you will be able to trivially switch from s3 to gs without needing to adjust anything in the projects themselves.

[dmpetrov]

@efiop these are good workarounds. I urge everyone to think about a holistic solution instead.

Ideally, we need the same solution as Git has - I can easily move a repo with the entire history from, for example, GitHub to GitLab by a couple of commands just by changing remote, pushing and removing the old one. Is there a way to implement something similar in DVC?

@dmpetrov , @efiop , what about using external tools like the AWS CLI or the Google Cloud Platform CLI to sync the cache?

For example, if you are migrating to GCP, it would be gsutil cp -r .dvc/cache gs://my-bucket

If you don't care about modifying your Git history, git rebase -i -root and modifying the commit who added the remote, pointing to the new one.

[dmpetrov]

@MrOutis sure, you can use the tools. But it is not clear how to change the links in Git history - you will still have .dvc/config files in history pointing to the old repo. This is the major question.

Modifying the commit - yes, it is a possibility. Is there any better way to define and change data remotes? (most likely it should not be committed to config)

[shcheklein]

👍 on my end, it feels that there should be a better solution to this.

[efiop]

Ideally, we need the same solution as Git has - I can easily move a repo with the entire history from, for example, GitHub to GitLab by a couple of commands just by changing remote, pushing and removing the old one. Is there a way to implement something similar in DVC?

Sure:

dvc pull (maybe with --all-commits or --all-tags or --all-branches)
dvc remote add gs gs://bucket/path
dvc push -r gs (maybe with --all-commits or --all-tags or --all-branches)
dvc remote remove s3
dvc remote default gs
git add .dvc/config
git commit
git push

Maybe I don't quite understand how you propose to "modify git history". The workarounds I've provided initially update the branches/tags that are used or propose to use some type of proxy to route your pulls. Currently I can't think of any other way to achieve this.

[shcheklein]

How about we always rely on the latest commit in a branch to determine the actual remote? No matter what is committed in the history.

[efiop]

@shcheklein sounds fragile and non-obvious. Plus it again won't work until the user dvc updates or something, to update the git repo cache. You might also want to move to another remote, while dropping the old one, which would also break this.