Crash with "broken pipe" #10
Comments
|
This seems to be related to a problem I keep bumping into sporadically. Check the fix from #4. |
|
I got The new crash report reads: |
|
Glad the summary helped. Here's a direct link for future reference: #4 (comment). |
This doesn't solve the actual issues, but helps mitigate them in many cases. I am not completely sure what the "actual" issues are, but one seems to be that it is apparently relatively easy to end up with a corrupt marks file (and indeed, the code loading / saving marks does not attempt to be particular robust). The other problem hinted at in issue #4 seems to be when tips in the remote hg repository change, then this seems to also cause troubles. See also: felipec#4 felipec#10
|
I think I just triggered this issue by doing a "git gc" (possibly with the --aggressive option). |
|
As far as I can tell, git-remote-hg does not perform any locking on the marks files, nor are save atomic. This makes me wonder what happens if one runs two git commands involving git-remote-hg simultaneously. I hope the locking git performs will already take care of git-remote-hg not being run twice in parallel -- for if that ever happens, troubles are likely to ensue. More worrisome is that writing the marks files is not atomic, thus if this is interrupted for whatever reason, you'd end up with a broken marks file. I do not know if this ever happens, but it might be one cause for the broken marks files people keep running into... Perhaps one can somehow use more of the git (or mercurial?) infrastructure to make this safer, and attain some resemblance of ACID guarantees. Say by storing the marks in git objects instead of files... But perhaps I am just overcomplicating things.... In any case, as long as there is no reliable way to reproduce this issue, it's very had to do anything about it :-( |
|
git gc triggers this by calling git prune and killing loose objects. |
|
Since git does gc automatically once in a while it is almost inevitable that a repository is going to break and need the git-marks-check dance eventually. Can't this just be built into git-remote-hg itself? |
|
@ghostganz +1 |
|
Today this issue hit me again. Very annoying. |
|
@g2p: You're right that removing the contents of the "tips" dictionary doesn't seem to be necessary. I updated #4 (comment) to reflect this. @ghostganz: If this could be excited via git garbage collection, we would be seeing issues all the time and the frequency for me can be measured in weeks/months. I tend to modify the repo history regularly on the git end and this hasn't caused any issues I'm aware of. @fingolfin: +1 for atomic saves. One reliable way I found to reproduce the issue is to synchronize the git and hg repos, then proceed to strip the tip commit from the hg repo. Fetching from git fails with the broken pipe issue because "last-note" points into the ether. In fact, if you make "last-note" point to a non-existent mark you can generate a broken pipe error at will and just reverting the change fixes it. |
|
Using stripping history in hg will very likely confuse git-remote-hg, but crashing git-remote-hg is not a great solution either. Since I have no way to reproduce this, I just made a wrapper around #!/bin/sh
remote=$1
remotedir="$GIT_DIR/hg/$remote/"
suffix=.$$
for f in \
"$remotedir/marks-git" \
"$remotedir/marks-hg" \
; do
[ ! -e "$f" ] || cp -a "$f" "$f$suffix"
done
git-remote-hg.bin "$@"
ret=$?
if [ $ret -eq 0 ]; then
rm -f "$remotedir/marks-git$suffix" "$remotedir/marks-hg$suffix"
else
echo "Stored a marks copy at $remotedir/marks-git$suffix"
fi
exit $ret |
This doesn't solve the actual issues, but helps mitigate them in many cases. I am not completely sure what the "actual" issues are, but one seems to be that it is apparently relatively easy to end up with a corrupt marks file (and indeed, the code loading / saving marks does not attempt to be particular robust). The other problem hinted at in issue #4 seems to be when tips in the remote hg repository change, then this seems to also cause troubles. See also: felipec#4 felipec#10
|
@fgb: I think the issue may be git prune. It's not the normal garbage collection cycle, but the few that remove long unreferenced objects. You won't see them often -- you'll only see them when you force a prune, or when the prunable objects finally expire, which means this should only be seen on the order of months |
|
Today encountered the same kind of error as @larsmans :
Traceback (most recent call last):
File "/usr/bin/git-remote-hg", line 1322, in
sys.exit(main(sys.argv))
File "/usr/bin/git-remote-hg", line 1306, in main
do_import(parser)
File "/usr/bin/git-remote-hg", line 721, in do_import
export_branch(repo, branch)
File "/usr/bin/git-remote-hg", line 613, in export_branch
export_ref(repo, branch, 'branches', head)
File "/usr/bin/git-remote-hg", line 557, in export_ref
print "from :%s" % (rev_to_mark(parents[0]))
File "/usr/bin/git-remote-hg", line 440, in rev_to_mark
return marks.from_rev(rev.hex())
File "/usr/bin/git-remote-hg", line 173, in from_rev
return self.marks[rev]
KeyError: 'dc419a7899b22cf06cd1a60ee0ed3d0021fb45fb'
fatal: stream ends early
fast-import: dumping crash report to .git/fast_import_crash_7609
fatal: Error while running fast-import
|
|
@Lekensteyn: What do you mean there's no way to reproduce the issue? I detail just a way to do so in my #10 (comment) to @fingolfin. There are different triggers for the broken pipe issue that I don't quite understand yet, though, but that can be fixed following the same steps. @apeyser: Except that this is not on the git side. The only way I found to systematically trigger a broken pipe is to strip the tip commit in hg, as I detail above, which immediately causes the broken pipe. One reason I bump into this issue only sporadically is that I strip a commit from hg on the order of weeks / months. @nengxu: I bumped into the same issue a month or so ago and ended up adding back the removal of the "tips" contents at the end of step 2 in #4 (comment). |
|
@fgb AFAIK my affected repo did not have stripped commits (or else I need to ask my team members), so I thought there is some other issue. If a reproducer for actions other than stripping could be found, it could give some insight. |
|
There are different issues at hand. For the one regarding the missing commits, that's an issue of Git's core. I have the patches that fix the problem. I also found a way to reproduce the issues that happen after that: missing marks, specially for the last_note: issue #56. |
While trying to pull changes into a clone of the CPython repo with
git fetch origin, I got the following crash:Crash report
fast_import_crash_17220reads:git-remote-hgversion is 185852e.The text was updated successfully, but these errors were encountered: