Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removing hard dependency on SCION lab packages, SCION build config #3

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

divinepaul
Copy link
Collaborator

@divinepaul divinepaul commented Jan 5, 2025

Ability to set a build config on the scion layer which can,

  1. Clone any SCION git repository with mentioned tag, commit or branch and build SCION binaries.
scion.setBuildConfiguration({
    "mode": "build",
    "gitRepoUrl": "https://github.com/scionproto/scion.git",
    "checkout": "v0.11.0" # could be tag, branch or commit (ex "efbbd5835f33ab52389976d4b69d68fa7c087230")
})
  1. Use a local directory ( absolute path ) to import the SCION binaries
scion.setBuildConfiguration({
    "mode": "release",
    "releaseLocation": "/absolute/path",   
    "version": "v0.12.0" # does nothing except suffix the directory containing the binaries
})
  1. Use any HTTP URL that returns a .tar.gz file containing the binaries
scion.setBuildConfiguration({
    "mode": "release",
    "releaseLocation": "https://github.com/scionproto/scion/releases/download/v0.12.0/scion_0.12.0_amd64_linux.tar.gz"
    "version": "v0.12.0" # does nothing except suffix the directory containing the binaries
})

Things to note,
All configurations write the binaries to a directory called .scion_build_output in the current working directory.
the directory will contain sudirectories with the binaries named with a suffix to either the version or the checkout.
the build is not repeated for the same checkout or version.
any scion output will now have shared directories with docker volumes, so deleting the binaries or the shared folder will break the produced docker compose configuration.
I did not make another file for the ScionBuildConfig class since I was unsure where it could go in the project.
And I need help nailing down how changing the build config for individual nodes could look like.


This change is Reviewable

Copy link

@martenwallewein martenwallewein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, great work!

@martenwallewein
Copy link

While testing this with the example S02-scion-bgp-mixed I encountered the following issue:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/marten/go/src/github.com/netsys-lab/seed-emulator/examples/scion/S02_scion_bgp_mixed/.scion_build_output/scion_binaries_v0.12.0" to rootfs at "/bin/scion": mount /home/marten/go/src/github.com/netsys-lab/seed-emulator/examples/scion/S02_scion_bgp_mixed/.scion_build_output/scion_binaries_v0.12.0:/bin/scion (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Can you replicate this? On my machine this leads to the containers not starting in this example.

Copy link
Member

@lschulz lschulz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add some documentation on what the new build options are and what they do, e.g., as a longer Doxygen docstring.

It would also be good to give examples of the possible configuration options in the SCION example. I'd add a paragraph to the README in examples/S01_scion.

I was not able to replicate the issue @martenwallewein was facing, but I only tried the default build configuration.

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @divinepaul and @martenwallewein)


seedemu/layers/ScionRouting.py line 108 at r1 (raw file):

    def __install_scion(self, emulator: Emulator, node: Node):
        """Install SCION packages on the node."""

Please change the comment as well, as the method is no longer installing SCION from the ETH packages.


seedemu/layers/Scion.py line 130 at r1 (raw file):

        try:
            result = urlparse(url)
            return result.scheme in ("http", "https") and bool(result.netloc)

Casting to bool looks kind of ugly to me. Maybe do a is not None'or != ""check instead. Whichever is actually intended.


seedemu/layers/Scion.py line 141 at r1 (raw file):

        # Ensure the URL ends with .git for Git repositories
        if not url.endswith(".git"):
            raise ValueError("URL does not look like a Git repository (missing .git)")

Requiring the URL to end in .git seems overly restrictive. Git repos don't have to end in .git. I'd remove the check.


seedemu/layers/Scion.py line 182 at r1 (raw file):

    def generateBuild(self) -> str :
        """
        method to build all scion binaries and ouput to .scion_build_output based on the configuration mode

ouput -> output


seedemu/layers/Scion.py line 246 at r1 (raw file):

        self.__links = {}
        self.__ix_links = {}
        self.__default_build_config = ScionBuildConfig()

I'd pass the build config in as keyword args for the Scion layer constructor instead of using a separate method.


seedemu/layers/Scion.py line 252 at r1 (raw file):

        return "Scion"

    def getBuildConfiguration(self) -> ScionBuildConfig:

There is an asymmetry between setBuildConfiguration and getBuildConfiguration. The getter is returning a different type the the setter is setting.
Should getBuildConfiguration even by a public method? Only the ScionRouting layer is calling it, and only to call generateBuild() on the build config. Maybe make generateBuild a method on the SCION layer?


seedemu/layers/Scion.py line 255 at r1 (raw file):

        return self.__default_build_config

    def setBuildConfiguration(self, buildConfig: Dict[str,str]):

Keyword arguments would be more Pythonic than passing a dict. Make the signature setBuildConfiguration(**buildConfig). Or move to constructor as noted above.

@lschulz
Copy link
Member

lschulz commented Jan 23, 2025

I just noticed that if you log in to one of the SCION containers, the SCION binaries scion etc. are not in PATH.

root@f722caa71ff7 / # scion
zsh: permission denied: scion

@amdfxlucas
Copy link
Collaborator

I can also stumbled over this. Node::addBuildCommand( export PATH=xyz) gets translated to a RUN directive in the dockerfile, which sets the variable only in the shell it is run and the changes do not persist to after the image build.
You could instead add an ENV directive to the dockerfile like here: https://github.com/amdfxlucas/seed-emulator/blob/3ebe5add0a692184a54927ac6813e3922d5395ea/seedemu/layers/ScionRouting.py#L881

But from a design perspective it would be best to set all variables in the docker-compose.yml file where they are clearly visible and in one and the same place with all other configurations i.e. IP addresses. This also has the benefit that variables can be tweaked without having to rebuild any images.
https://github.com/amdfxlucas/seed-emulator/blob/3ebe5add0a692184a54927ac6813e3922d5395ea/seedemu/core/Node.py#L472

We should open an issue for 'Node custom ENV variables' and defer this one until it is merged.. so it can resort to this functionality

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pin SCION Version and remove hard dependency on scionlab packages
4 participants