Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tarball with tar_scm service fetch from github ends up almost empty #228

Closed
frispete opened this issue Apr 6, 2018 · 9 comments · May be fixed by #229
Closed

tarball with tar_scm service fetch from github ends up almost empty #228

frispete opened this issue Apr 6, 2018 · 9 comments · May be fixed by #229

Comments

@frispete
Copy link

frispete commented Apr 6, 2018

[Migrated this issue from osc]

Hi,

today, I tried to update my FreeRDP build as always in a checked out copy:

$ osc service disabledrun
Cloning into bare repository '/var/cache/obs/tar_scm/56b4eb933fa2e77b484bad0ce4fb93a994aa647426be674d3f66f69a6b033de0/FreeRDP'...
a9ecd6a6ccc71130805a6bf3214c169fc42746ba
Cloning into '/home/hp/src/obs/home:frispete:RemoteDesktop/freerdp2/FreeRDP'...
a9ecd6a6ccc71130805a6bf3214c169fc42746ba
Compressed freerdp-2.0.0~git.1522916443.a9ecd6a6c.tar to freerdp-2.0.0~git.1522916443.a9ecd6a6c.tar.xz

I already removed the cached repo from /var/cache/obs/tar_scm, just in case, but the resulting tarball is almost empty (second tar.xz file):

$ l
drwxrwxr-x  3 hp lisa      18  5. Apr 20:21 FreeRDP/
-rw-rw-r--  1 hp lisa 4335868 14. Feb 11:55 freerdp-2.0.0~git.1518601944.ed4f54ab4.tar.xz
-rw-rw-r--  1 hp lisa     200  5. Apr 20:21 freerdp-2.0.0~git.1522916443.a9ecd6a6c.tar.xz
-rw-r--r--  1 hp lisa   66170  5. Apr 20:21 freerdp.changes
-rw-rw-r--  1 hp lisa    3943  5. Apr 20:21 freerdp.spec
drwxrwxr-x  2 hp lisa    4096 14. Feb 11:58 .osc/
-rw-rw-r--  1 hp lisa     562 26. Jan 2017  _service
-rw-rw-r--  1 hp lisa     236  5. Apr 20:21 _servicedata
$ tar tvf freerdp-2.0.0~git.1522916443.a9ecd6a6c.tar.xz
drwxrwxr-x root/root         0 2018-04-05 10:20 freerdp-2.0.0~git.1522916443.a9ecd6a6c/

The checked out FreeRDP dir just contains the .git folder:

$ l FreeRDP/
insgesamt 0
drwxrwxr-x 3 hp lisa  18  5. Apr 20:46 ./
drwxrwxr-x 4 hp lisa 275  5. Apr 20:47 ../
drwxrwxr-x 8 hp lisa 180  5. Apr 20:46 .git/

My versions are:

$ git --version
git version 2.16.3
$ osc --version
0.162.1

Content of _service:

<services>
  <service name="tar_scm" mode="disabled">
    <param name="scm">git</param>
    <param name="url">https://github.com/FreeRDP/FreeRDP.git</param>
    <param name="filename">freerdp</param>
    <param name="versionprefix">2.0.0~git</param>
    <param name="changesgenerate">enable</param>
    <param name="revision">refs/heads/master</param>
  </service>
  <service name="recompress" mode="disabled">
    <param name="file">*.tar</param>
    <param name="compression">xz</param>
  </service>
  <service name="set_version" mode="disabled"/>
</services>

The branch names of the FreeRDP github project didn't change to my knowledge. A normal git clone succeeds as usual.

This happens with obs-service-tar_scm-0.8.0.1520581079.e26b0ae-198.1.noarch here.
Reverting to obs-service-tar_scm-0.8.0.1507129410.0cb2d44-198.1 fixes this behaviour here.

If you compare the git commands of Git.prepare_working_copy in both versions:

ok:
['git', 'clone', '--reference', u'/var/cache/obs/tar_scm/56b4eb933fa2e77b484bad0ce4fb93a994aa647426be674d3f66f69a6b033de0/FreeRDP', 'https://github.com/FreeRDP/FreeRDP.git', '/home/hp/src/obs/home:frispete:RemoteDesktop/freerdp2/FreeRDP']
fail:
['git', 'clone', '--no-checkout', '--reference', u'/var/cache/obs/tar_scm/56b4eb933fa2e77b484bad0ce4fb93a994aa647426be674d3f66f69a6b033de0/FreeRDP', 'https://github.com/FreeRDP/FreeRDP.git', '/home/hp/src/obs/home:frispete:RemoteDesktop/freerdp2/FreeRDP']

hence, due to --no-checkout, the result is that the checked out directory is mostly empty for the "with reference" case.

Question is, why does this work for you, guys, does it?

@M0ses
Copy link
Collaborator

M0ses commented Jun 8, 2018

I cannot reproduce the problem, even if I install e26b0ae.

But I´m wondering about your settings - you have set the CACHEDIRECTORY, but maybe you might want to disable this. When running in osc, the local osc working copy will include the directory of the clone and also be used for caching. But I also have to admit, that this is no explaination for this problem and why I cannot reproduce it.

Maybe you could disable the CACHEDIRECTORY in your setting and try if the problem still exists.
This settings can be done in

/etc/obs/services/tar_scm
or
~/.obs/tar_scm

@cmurphy
Copy link

cmurphy commented Jun 12, 2018

I am running into this too and I have some more information. This is reproduceable if CACHEDIRECTORY is set to anything but the empty string in ~/.obs/tar_scm and if the env var OSC_VERSION is set to anything: http://paste.opensuse.org/5156bb0a

When CACHEDIRECTORY is set, a bare clone is made:

if self.repocachedir:
command.insert(2, '--mirror')

When it prepares a working copy from the CACHEDIRECTORY, it uses --no-checkout, which causes the working copy to start empty:

command = self._get_scm_cmd() + ['clone', '--no-checkout']

Then it is supposed to check out the given revision which would make it no longer empty, but if OSC_VERSION is set (which happens when invoked by osc) there is this funny behavior with git-stash on an un-checked-out tree, where restoring the stash causes the tree to be empty again even after the hard reset:

if os.getenv('OSC_VERSION'):
stash_text = self.helpers.safe_run(
self._get_scm_cmd() +
['stash'],
cwd=self.clone_dir)[1]
text = self.helpers.safe_run(
self._get_scm_cmd() + ['reset', '--hard', rev],
cwd=self.clone_dir
)[1]
if stash_text != "No local changes to save\n":
logging.debug("[switch_revision] GIT STASHING")
text += self.helpers.safe_run(
self._get_scm_cmd() +
['stash', 'pop'],
cwd=self.clone_dir)[1]

The result is that when this is invoked by osc service disabledrun the obscpio file ends up basically empty and the package won't build.

@M0ses
Copy link
Collaborator

M0ses commented Jun 13, 2018

@cmurphy Thanks for your detailed explaination. Now I see the problem, but as already said:

I don't see any use case in using a CACHEDIRECTORY within "osc" as in the osc case the git repo gets already cached in the osc working copy, but I would like to hear your comments.
@adrianschroeter, @aspiers, @frispete ,@cmurphy - Is there any use case for this, that I don`t see?

I only see, that you have to remove the local checkout in the osc working manually before triggering the next service run - thats an real disadvantage for me.

Maybe we should throw an exception (or at least print a warning) if you try to use a CACHEDIRECTORY while calling from 'osc'?

@cmurphy
Copy link

cmurphy commented Jun 13, 2018

In my case I am pretty sure I didn't set CACHEDIRECTORY myself, something else must have generated ~/.obs/tar_scm but I'm not sure what. An error message warning about it would be an improvement, though maybe frustrating if either tar_scm or osc or some other tool is creating it without the user's knowledge.

@M0ses
Copy link
Collaborator

M0ses commented Jun 13, 2018

Nor osc neither obs-service-tar_scm should have created a file in your home. IMHO it is common sense, that rpms do not create files in users home directories (for good reason).

@aspiers
Copy link
Member

aspiers commented Jun 19, 2018

Thanks @cmurphy for the great analysis!

@aspiers
Copy link
Member

aspiers commented Jun 22, 2018

I reproduced this by accident just now ...

@M0ses commented on 13 Jun 2018, 10:03 BST:

I don't see any use case in using a CACHEDIRECTORY within "osc" as in the osc case the git repo gets already cached in the osc working copy, but I would like to hear your comments.
@adrianschroeter, @aspiers, @frispete ,@cmurphy - Is there any use case for this, that I don`t see?

Sure - the cache is supposed to be global (or at least per-user) not per working copy. This is in case you have multiple osc branches of the same package, because then it means you only need one clone into the global cache, rather than one per package.

I only see, that you have to remove the local checkout in the osc working manually before triggering the next service run - thats an real disadvantage for me.

If that's true (why would it be?) then IMHO it's a bug.

Maybe we should throw an exception (or at least print a warning) if you try to use a CACHEDIRECTORY while calling from 'osc'?

Definitely not - IIRC I originally designed the cache for client-side usage. Server-side efficiency was just a bonus :-)

I did some archaeology and the stash code comes from 4e4b2d66.

@aspiers
Copy link
Member

aspiers commented Jun 22, 2018

Here is the debug I obtained by manually hacking a workaround to #240:

POST https://api.opensuse.org/source/home:aspiers:branches:server:monitoring/lnav?cmd=getprojectservices
Run source service: /usr/lib/obs/service/tar_scm --url git://github.com/aspiers/lnav.git --scm git --exclude .git --versionformat 0.8.3b+git.%ct.%h --revision sp3 --changesgenerate enable --outdir /home/adam/SUSE/OBS/home/aspiers/branches/server/monitoring/lnav/tmpDmDG_t.tar_scm.service
Ignoring config files: test-mode detected
Using fakeheader for file '/etc/obs/services/tar_scm'
Using fakeheader for file '/home/adam/.obs/tar_scm'
SECTION: tar_scm
REPOCACHE: /home/adam/.obs/cache/tar_scm
CACHEDIR: '/home/adam/.obs/cache/tar_scm/66c4d41370d7046aa194a4c9616f4a599cf50023b54d88182a4d50f66348747c'
SCM: 'git'
Stripping 'git' extension from '/aspiers/lnav.git'
 - New  url_path: '/aspiers/lnav'
[_calc_dir_to_clone_to] CLONE_DIR: /home/adam/.obs/cache/tar_scm/66c4d41370d7046aa194a4c9616f4a599cf50023b54d88182a4d50f66348747c/lnav
REPOCACHEDIR = '/home/adam/.obs/cache/tar_scm/66c4d41370d7046aa194a4c9616f4a599cf50023b54d88182a4d50f66348747c'
[fetch_upstream] Initial checkout/clone to directory: '/home/adam/.obs/cache/tar_scm/66c4d41370d7046aa194a4c9616f4a599cf50023b54d88182a4d50f66348747c/lnav'
COMMAND: ['git', 'clone', '--mirror', 'git://github.com/aspiers/lnav.git', u'/home/adam/.obs/cache/tar_scm/66c4d41370d7046aa194a4c9616f4a599cf50023b54d88182a4d50f66348747c/lnav']
RESULT(0): "Cloning into bare repository '/home/adam/.obs/cache/tar_scm/66c4d41370d7046aa194a4c9616f4a599cf50023b54d88182a4d50f66348747c/lnav'...\n"
COMMAND: ['git', 'rev-parse', '--verify', '--quiet', 'sp3']
RESULT(0): '3d19d0865693847c1e1b3b8133faecae155946a9\n'
COMMAND: ['git', 'clone', '--no-checkout', '--reference', u'/home/adam/.obs/cache/tar_scm/66c4d41370d7046aa194a4c9616f4a599cf50023b54d88182a4d50f6
6348747c/lnav', 'git://github.com/aspiers/lnav.git', '/home/adam/SUSE/OBS/home/aspiers/branches/server/monitoring/lnav/lnav']
RESULT(0): "Cloning into '/home/adam/SUSE/OBS/home/aspiers/branches/server/monitoring/lnav/lnav'...\n"
[switch_revision] Starting ...
COMMAND: ['git', 'rev-parse', '--verify', '--quiet', 'origin/sp3']
RESULT(0): '3d19d0865693847c1e1b3b8133faecae155946a9\n'
COMMAND: ['git', 'stash']
RESULT(0): 'Saved working directory and index state WIP on master: 4712bb6 update test for json logs with objects/arrays\n'
COMMAND: ['git', 'reset', '--hard', 'origin/sp3']
RESULT(0): 'HEAD is now at 3d19d08 update to latest ax_cxx_compile_stdcxx.m4\n'
[switch_revision] GIT STASHING
COMMAND: ['git', 'stash', 'pop']

At this point, popping the stash which removed the whole tree failed with conflicts.

@aspiers
Copy link
Member

aspiers commented Jun 22, 2018

OK, so this breakage comes from e822379 in #209.

aspiers added a commit to aspiers/obs-service-tar_scm that referenced this issue Jun 22, 2018
…USE#228)

Thanks to openSUSE#209 sometimes our working tree will be empty.  In
that case don't try to stash a bunch of changes which effectively
delete the whole working tree and then re-pop them after switching
version.

Fixes openSUSE#228.
aspiers added a commit to aspiers/obs-service-tar_scm that referenced this issue Jun 22, 2018
…USE#228)

Thanks to openSUSE#209 sometimes our working tree will be empty.  In
that case don't try to stash a bunch of changes which effectively
delete the whole working tree and then re-pop them after switching
version.

Fixes openSUSE#228.
aspiers added a commit to aspiers/obs-service-tar_scm that referenced this issue Jun 22, 2018
…USE#228)

Thanks to openSUSE#209 sometimes our working tree will be empty.  In
that case don't try to stash a bunch of changes which effectively
delete the whole working tree and then re-pop them after switching
version.

Fixes openSUSE#228.
M0ses added a commit that referenced this issue Jun 28, 2018
only do git stash save/pop if we have a non-empty working tree (#228)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants