Skip to content

Latest commit

 

History

History
197 lines (157 loc) · 9.35 KB

setup-and-run-a-contest.md

File metadata and controls

197 lines (157 loc) · 9.35 KB

Setting up and running a contest

The following is a guide on how to run a contest with this framework using the available services. There are 3 key components:

  1. Gameserver: Drives the game.
  2. Container registry: Stores all the container images.
  3. Container server: Runs all the containers.

This document is not comprehensive - not all optional settings are covered.

Setting up gameserver machine

  1. During testing, we ran the gameserver on an Ubuntu 14.04 machine.
  2. Install Docker as described in https://docs.docker.com.
  3. Install these packages: MySQL server, redis server, python-mysqldb, python-redis, python-tornado, python-flask, Flask-HTTPAuth, honcho and docker-py. The last 3 are available from PyPI only while other python libraries are available from Ubuntu's package repositories as well.

Building service container images

  1. Download the contest framework repository.

  2. Download the 32 bit root archive from https://partner-images.canonical.com/core/. This is used as base image for building service containers. We used Ubuntu 14.04. Images from https://cloud-images.ubuntu.com do not seem to work correctly with Docker as of writing.

  3. Configure your contest by modifying container-creator/example.json. You will want to set the following:

    • Service count.
    • Services to use: See services directory for available services.
    • Team details: name and namespace for the team on the container registry. Currently, the gameserver requires that the namespace have same name as the team name. This also simplifies management in PORTUS since every user is assigned a private namespace with same name as the user account.
    • Containers host: IP address of machine where all containers will execute.
    • Top level flag storage folder: Sub-folders of this on the container host will be mapped into the service containers. We currently do not support full fledged DBMSs such as MySQL and PostgreSQL since they require provisioning a data-only container.
    • Starting port number on containers host for containers. Ports are assigned incrementally so highest port number can be computed using number of teams and services.

  4. Build containers using container-creator/create_containers.py. It expects following options:

    • -sl: Location of the services directory.
    • -i: Location of the cloud image downloaded above.
    • -c: Location of the configuration file of the contest created above.
    • -od: Output directory to write all files of the contest to. Optional - defaults to $PWD/output.
    • IP and port of proxy for apt-get to use. Optional - defaults to None.

    Note down the location of the initial DB configuration file generated - it is displayed at the end of the process. The entire process of building service containers image will take a while depending on processor speed and number of services. The output of build process is not displayed unless there is an error in building an image.

Setting up gameserver

Setting up the database API service

  1. Setup the game database in MySQL and specify the game database name and user credentials to access the game database in database/settings.py.
  2. Reset database using database/reset_db.py. This expects path to the DB config file generated by create_containers.py.
  3. Change the API secret in database/database_service.py. Ensure it cannot be easily guessed since it provides direct access to the game database.
  4. Set listening IP address in database/database_tornado.py so that it can receive notifications from Docker distribution.

The remaining configuration needed is specified in later sections.

Setting up the gamebot

The gamebot drives the game by initiating new rounds at end of the old round. The duration of a round(round tick time) can be set in database/gamebot.py. By default, a round lasts 5 minutes. The gamebot also uses settings from database/settings.py.

Setting up the scorebot

Modify the API location and secret specified in scorebot/settings.py. The remaining configuration required is specified in following sections.

Setting up team dashboard

Modify dashboard/config.json to set the following information:

  1. Team credentials for logging into the dashboard. The team name must match same name as specified in the contest configuration file. See example in dashboard/config.json.
  2. The API base URL and API secret.
  3. The name of the contest.

Setting up the container registry.

There are two possible choices we've tried out:

  1. Docker distribution + PORTUS on OpenSUSE.
  2. Gitlab CE with container registry support.

We have tested Distribution + PORTUS more but Gitlab is easier to setup and works almost similarly. We believe as long as the callback notifications are configured properly, any Docker container registry can be used. We document both methods below.

Setting up PORTUS and Docker distribution

  1. During testing, we used OpenSUSE Leap 42.1 to host the Docker distribution and PORTUS. See http://port.us.org/documentation.html for installation and configuration instructions. It is recommended to setup TLS with a proper certificate but self-signed also works fine. If you are using PORTUS generated self-signed certificates, they have to be trusted by the Docker daemons on the gameserver and container server. See https://docs.docker.com/registry/insecure/ for how to configure the daemons.

  2. Test Docker distribution and PORTUS by creating an unprivileged users and a namespace and ensure both administrator and normal user can push and pull container images from the namespace. Use any image available from Dockerhub for testing.

  3. After setting up and testing Docker distribution and PORTUS as required, configure Docker distribution to send notifications to the gameserver whenever an operation is performed. An example configuration is specified below:

     - name: Gameserver
   url: http://gameserver.domain.or.ip:port/container_changed
   headers:
    secret: [YOUKNOWSOMETHINGYOUSUCK]
   timeout: 1s
   threshold: 5
   backoff: 2s

    The secret value is the database API secret. Restart registry service after modifying the configuration.

  4. Navigate to the container image registry using a web browser to create first admin user in PORTUS and configure the Docker registry.

  5. Create user accounts and namespaces for all teams with values specified in the contest configuration. See PORTUS documentation for more on this.

  6. Set the location of Docker distribution and credentials of the PORTUS administrator user created when configuring PORTUS in database/settings.py and scorebot/invoke_container.py.

Setting up Gitlab CE

Note: These steps could be incomplete/incorrect. Corrections and approvals welcome.

Gitlab CE is simpler to setup than PORTUS + Distribution since Distribution is well integrated into Gitlab. It also provides lots of monitoring tools, easily scales to multiple machines and more features.

  1. Install Gitlab CE from https://about.gitlab.com/downloads/ and configure it.
  2. Enable container registry Gitlab as specified in http://docs.gitlab.com/ce/administration/container_registry.html.
  3. Create groups for different teams and member users in each team. The Gitlab API can be useful to automate the process.
  4. Configure callback notification just like in case of PORTUS + Distribution. Be sure to restart Gitlab after doing so.
  5. Set the location of Docker distribution and credentials of Gitlab administrator in database/settings.py and scorebot/invoke_container.py

Pushing container images to the registry

We have written a small utility script, container-creator/push_containers.py, for pushing containers to the registry. This script accepts the following options:

  • -sl: Location of the services directory.
  • -c: The config file of the contest.
  • -ds: Domain name of the Docker distribution server.
  • -dpo: Port number on which the Docker distribution server is listening.
  • -du: Name of the PORTUS administrator user account who has push access to namespaces of all teams.
  • -dpass: Password of the administrator user.

Setting up the container server

  1. During testing, we ran all containers on an Ubuntu 14.04 machine.
  2. Install Docker as described in https://docs.docker.com.
  3. Configure the Docker daemon to listen to a TCP port in addition to the standard Unix socket. The gameserver does not support TLS connections to the daemon. However, adding support for this is fairly trivial.
  4. Set the TCP port specified above in database/settings.py and scorebot/invoke_container.py as the remote Docker daemon port.
  5. Create folders for every service of every team with 777 permissions inside the top level flag storage folder specified in the contest configuration. This is necessary because the Docker daemon automatically creates folders with wrong permissions leading to the services not functioning correctly. Additionally, Docker daemon does not respect the default settings specified via umask or ACLs.

Running the game

  1. Run database_tornado.py, gamebot.py, scorebot.py and dashboard/Procfile using honcho to start the database API, game, scoring and dashboard respectively.
  2. Alternatively, copy ctf-database.conf, gamebot.conf, scorebot.conf and dashboard.conf to /etc/init and start them using the service command, the System V init script runner.