CVS outage (and how we're going to get independent of SF)

What do you want to see in Armagetron soon? Any new feature ideas? Let's ponder these ground breaking ideas...
Post Reply
User avatar
ivantis
Round Winner
Posts: 269
Joined: Mon Mar 03, 2008 2:33 pm
Contact:

Post by ivantis »

aw, does bzr have to be upgraded to get it now?
Image
Image
Image
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

No; the different versions of bzr are compatible, and what is important to you is the version running on launchpad, and that is unchanged.

What could go wrong was only the bzr <-> svn syncing my server is doing. And indeed, the new bzr-svn doesn't like anything happening outside of configured branches in svn, so I had to revert to an earlier version. It's now bzr 1.2 and bzr-svn 0.4.8.
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

Ok, after having now spent some time with both bzr and svn, time for some thoughts.

Good things about both:
- day-to-day operation is typically flawless. Both to their job of keeping revision history, both show reasonable performance.

Good things about bzr beyond things implied by its distributed nature:
- It does merging absolutely right. 100%. Perfectly. You can merge back and forth between branches without getting into trouble, and merges are tracked in history, which means that however complicated the merges have been, you're always able to fetch the source tree state as it was when each revision was committed. Even those revisions that now only appear in second level merges. We don't know yet how it would handle merging hell situations like we have on 0.2.8 -> trunk now, but it can't be worse there than svn.
- Therefore, the trunk always contains the whole project history in all detail, as far as it has been merged into it.

Bad things:
- you can't get a part of a branch, neither in space nor time. Thus, branches need to be atomic (as small as possible to still be useful)
- therefore, management of branches and whole project merges is cumbersome

Good things about svn:
- a nice repository tree. Work anywhere, get any part of it, move stuff around freely at zero cost.

Bad things:
- branching and tagging needs you to fiddle manually with paths.
- merge history is not represented in the repository.
- as does merging, plus manual fiddling with revision numbers. We have a script for that, luckily.

We can sort of get the best of both worlds if we continue to use bzr-svn. The problem is that we won't be able to do that forever. At one point, the bzr version bazaarmagetron is using is going to get incompatible with the one on launchpad, and later versions produce incompatible bzr branches; we'll then have the choice between starting all bzr branches from scratch, which leaves the hack branches stranded and in need of medical care (it's possible to bring them over to the new branches, but they probably lose their own history in the process), or to ditch one of the two systems.

My vote would therefore go to use bzr exclusively ASAP. Tagging and merging is easier in bzr. Branching of the whole project is easier in svn. We're (and by that I mean "I'm") doing far more merging than tagging and branching together.The little complication of having to do simultaneous branches of four bzr branches is nothing when weighted against the ease of merging.

And by ASAP I mean soon after 0.2.8.3 is out. We should block write access to svn to everyone except bazaarmagetron (and maybe the release manager of 0.2.8.3.x, which would probably be me, for tagging purposes) and all work from bzr from then on. We can keep bazaarmagetron running for as long as it's feasible; there's no harm in keeping svn a bit up to date.
Luke-Jr
Dr Z Level
Posts: 2246
Joined: Sun Mar 20, 2005 4:03 pm
Location: IM: [email protected]

Post by Luke-Jr »

Little reminder that I'm still stranded on Subversion as far as commit access.
User avatar
Lucifer
Project Developer
Posts: 8640
Joined: Sun Aug 15, 2004 3:32 pm
Location: Republic of Texas
Contact:

Post by Lucifer »

Make your own bzr branches, eh? When you have something you want to merge, get someone who has write access to merge it. :)
Image

Be the devil's own, Lucifer's my name.
- Iron Maiden
User avatar
Tank Program
Forum & Project Admin, PhD
Posts: 6711
Joined: Thu Dec 18, 2003 7:03 pm

Post by Tank Program »

What happens to the private folders in bzr?
Image
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

Ah, the private stuff. I can't have bazaarmagetron turn that into bzr branches right now; it's a limitation of bzr-svn and its configuration. I'll see what I can do about that when I'm home at a stronger PC, maybe a "anything in svn can be a branch" configuration can get that stuff over. That also may be able to get us a new trunk with our full trunk history over to bzr. We'll have to see.

Also, of course, our web pages and aabeta are on svn exclusively currently. Personally, I'm not in a big hurry to change them over. So maybe we should delay blocking svn writes for a bit and instead just have bazaarmagetron overwrite renegade changes in svn on the trunk.

Regarding luke-jr's write access, that is of course tied to your status on the project, to be discussed elsewhere. Officially, you're still an ex-member.
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

Ok, by telling bzr-svn that everything can be a branch, checking out the trunk gives a bzr branch with full version history as it is in svn. It's bigger than our current branch (about 50 Mb as opposed to 30 Mb), but that was to be expected.

Two things are lost in the process:
1. the fancy bzr stuff in the current trunk bzr branch (merge metainfo). There may be a chance to get it back with some magic merges, but so far, all attempts got me a bag of conflicts on top of a history break at the current bzr trunk starting point. There may be another chance to get it back by letting bzr-svn store bzr metadata in revprops in svn and having that restored when the new trunk gets created, but that requires more recent versions of bzr and svn than I currently can reasonably install here, so experiments will have to wait.
2. file moves in svn. Although supported in bzr, they get mapped to file deletions and recreations. Bummer. File copies aren't tracked in bzr at the moment, so file copies in svn can't be mapped, either. File copies are mostly used by luke-jr in the gentoo overlay to copy manifests, I don't think they're used much in our main modules. But the moves are a problem.

That creates a dilemma. If we restart the trunk early, before file moves are properly handled on the way from svn to bzr, we lose them. However, the longer we wait, the more fancy metadata from bzr gets lost if the metadata rescue plans fail.

Edit: the stitching option is out. The problem is that bzr treats files and directories as different if they've been added in different revisions even if they have the same name. So the sources added in our current trunk and those in the new trunk are different files, and changes in both are not mergeable. History could be preserved, possibly, but not to the point that 'bzr blame' works correctly, because we'd have to visibly delete and recreate files in the process.
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

The svn-setrevprops path to store bzr metadata in svn also is out for now. It simply doesn't work. After the data is in svn, bzr-svn can't branch from it any more.
epsy
Adjust Outside Corner Grinder
Posts: 2003
Joined: Tue Nov 07, 2006 6:02 pm
Location: paris
Contact:

Post by epsy »

What do we do about the trunk-utf8 branch ?
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

We can either merge it with svn or convert it to bzr and merge it there. After we got everything working, of course. Umm. Sorry. I guess that was my job.
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

Found a way to get all bzr revisions into the new trunk :) It's rather involved, though:

You make a local copy of the svn repository. You pick up the full bzr branch "full". You bzr-branch the svn repository into "new".

You find the latest revision in "full" that is not in "new". You branch that into "fix". Into "fix", you merge from svn, then you push. That gets the missing revision into svn, but hides all revisions after it (that's why we made a copy :) ). Then, you pull from svn to "new", that only adds the formerly missing revision without removing the other revisions (they get marked as 'merged', though). You repeat these steps until there are no more missing revisions.

Lastly, you restore order in "new" by branching the revision that was last originally into a new branch. That removes all the extra merges, too.

Of course, every bzr-svn operation has to be done with the correct branching scheme set.

Getting merged revisions that come after that into svn would be easier; you just instruct the bzr-svn push to go revision for revision. This does some trickery with svn cp; it's only visible in the svn log if you check the parent directory.

Oh, and since we last checked, bzr and svn have gotten new features. svn now tracks merges (I don't know what it does with them, how much it improves its behavior on wild criss-cross merges).

bzr allows "stacked" branches, that are regular branches where only local changes are stored and everything that is in the remote branch is omitted and fetched on demand. Kind of what lightweight checkouts are, only for regular branches. The stack target doesn't even need to be a push/pull target, as far as I can see. That's new since 1.6.1.

If that sounds like too much work, there's also bzr repositories. You can create one with

Code: Select all

bzr init-repo <repository name> --rich-root-pack
Then, whatever you do with bzr in a subdirectory of <repository name> will not store revision info in individual .bzr directories, but in the repository .bzr directory. Common data can so be shared between branches. This speeds up operations immensely and saves space. It sounds to me like it basically solves the "bzr will use more and more space over time" problem. It still does that, but not for each branch individually, so the space usage increase is given by the amount of work we do only, and not the number of branches. I don't know when this was added.
epsy
Adjust Outside Corner Grinder
Posts: 2003
Joined: Tue Nov 07, 2006 6:02 pm
Location: paris
Contact:

Post by epsy »

also, to convert a lonely branch to use a repository:

Code: Select all

bzr reconfigure --use-shared [LOCATION]
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

That's really useful, epsy :) I had already restarted half my branches the hard way.

Bah. Of course, my "get everything into svn and back" method fails in practice because bzr-svn borks up on the second iteration. Oh well. New method: Get copy of svn repository. Revert it to a state before the first commit by bzr-svn. Get a log of the bzr branch. Identify revisions without children in bzr (easier than it sounds). Push them to svn in the order they appear in the log (lowest first, of course). Then make your new bzr branch with the new branching scheme from the repository.
User avatar
Z-Man
God & Project Admin
Posts: 11585
Joined: Sun Jan 23, 2005 6:01 pm
Location: Cologne
Contact:

Post by Z-Man »

That, or simply use 'bzr svn-push --merged'. D'oh :)
Post Reply