Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make freeze/thaw scripts "smarter" #27

Open
yarikoptic opened this issue Oct 16, 2024 · 1 comment
Open

Make freeze/thaw scripts "smarter" #27

yarikoptic opened this issue Oct 16, 2024 · 1 comment

Comments

@yarikoptic
Copy link
Member

ATM we just specify them user-wide and then actual git repository which makes use of them does not even know which version of the script it used etc. We already got a problem that we are changing the way we freeze and thus current thaw script is

[f006rq8@discovery-01 ds000003-qc]$ cat /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/thaw-content | grep -v '^#[^!]'
#!/bin/bash

set -eu

{
        # TODO: avoid double invocation for getfacl and ideally may be record
        # somewhere the way how it was all frozen
        # TODO: make scripts smart and "deploy" themselves into a repository
        # TODO: check "better" protections I have tried in
        #  https://git-annex.branchable.com/projects/datalad/bugs-done/can__39__t_make_annex_happy_in_freeze__47__thaw/
        # i.e. D::EVERYONE@:wadTN
        if nfs4_getfacl "$@" | grep -q "D::EVERYONE@:.*w.*D"; then
                # new way which seems to work
                nfs4_setfacl -x D::EVERYONE@:wD -R "$@"
        else
                # it must have been frozen using prior ways
                nfs4_getfacl "$@" | sed -e 's,:rxtn,:rwadxtTnN,g' | nfs4_setfacl -S- -R "$@"
        fi
#
} >&2

so we could account for older way where we explicitly modified permissions to remove "write" ones.

Also these scripts do not care/know if location is with ACL or POSIX, such as e.g. under /scratch (just regular XFS). As a result they fail, and we get git-annex working correctly (just using posix permissions I guess) as a side effect of ignoring them those failing commands: https://git-annex.branchable.com/bugs/git-annex_proceeds_forward_if_freeze_script_fails/?updated .

So, overall the script should somehow annotate any given git repository's config with information on either some "version" of freeze/thaw procedure or just "instantiate" and configure that repository to use specific versions of those scripts, by placing them e.g. under .git/git-annex/freeze-content and .git/git-annex/thaw-content scripts, and configure git-annex to use them:

(datalad) [f006rq8@discovery-01 subdir]$ pwd
/dartfs/rc/lab/D/DBIC/DBIC/CON/asmacdo/tmp/test-local-thaw/subdir

(datalad) [f006rq8@discovery-01 subdir]$ git config -l | grep content
annex.thawcontent-command=/dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/thaw-content %path
annex.freezecontent-command=/dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content %path
annex.thawcontent-command=.git/annex/thaw-content %path
annex.freezecontent-command=.git/annex/freeze-content %path

(datalad) [f006rq8@discovery-01 subdir]$ git config get annex.thawcontent-command
.git/annex/thaw-content %path

(datalad) [f006rq8@discovery-01 subdir]$ git config get annex.freezecontent-command
.git/annex/freeze-content %path

but that doesn't work since git-annex operates from current directory and thus relative path does not work when calling from subfolder:

[2024-10-16 14:47:08.941720897] (Annex.Perms) freezing content ../.git/annex/objects/6k/VJ/MD5E-s115--9a295e3f5f148380d74c3ff3ebdaa173/MD5E-s115--9a295e3f5f148380d74c3ff3ebdaa173            
[2024-10-16 14:47:08.948171243] (Utility.Process) process [2572997] call: sh ["-c",".git/annex/freeze-content '../.git/annex/objects/6k/VJ/MD5E-s115--9a295e3f5f148380d74c3ff3ebdaa173/MD5E-s1
15--9a295e3f5f148380d74c3ff3ebdaa173'"]                                                                                                                                                       
sh: .git/annex/freeze-content: No such file or directory  

filed https://git-annex.branchable.com/todo/specify_freeze__47__thaw_scripts_relative_to_topdir/?updated

So, until we hear back on that, it would only be the case of possibly specifying/storing version of freeze/thaw procedure inside local config, or may be to speed up (since per file) -- just a file, e.g. .git/annex/freezethaw-version with 0 to avoid any freezing in case of POSIX system.

@yarikoptic
Copy link
Member Author

unfortunately it is impossible to specify relative path there ATM. @joeyh suggested to specify some scripts which are in the PATH which would actually make it only trickier for transfer of those into containerized execution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant