Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WDQS Updater issue fix correction, add Wikibase Port, etc #722

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

WDQS Updater issue fix correction, add Wikibase Port, etc #722

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5d0c01c
WDQS Updater issue fix correction, add Wikibase Port, etc
lorenjohnson Jul 15, 2024
23b092e
Merge branch 'main' into wdqs-image-clean-up
lorenjohnson Jul 15, 2024
a1e2d62
Merge branch 'main' into wdqs-image-clean-up
lorenjohnson Jul 17, 2024
aa24c8f
Update README
lorenjohnson Jul 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions build/WDQS/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,18 +61,20 @@ RUN mkdir /var/log/wdqs && chown blazegraph /var/log/wdqs
ENV MEMORY=""\
HEAP_SIZE="1g"\
HOST="0.0.0.0"\
WDQS_ENTITY_NAMESPACES="120,122"\
WDQS_SCHEME="http"\
WDQS_HOST="wdqs"\
WDQS_PORT="9999"\
WDQS_ENTITY_NAMESPACES="120,122"\
WIKIBASE_SCHEME="http"\
WIKIBASE_HOST="wikibase"\
WIKIBASE_PORT="80"\
WIKIBASE_MAX_DAYS_BACK="90"\
BLAZEGRAPH_EXTRA_OPTS=""\
LOG_CONFIG="/wdqs/logback.xml"

WORKDIR /wdqs

COPY --chown=blazegraph:blazegraph wait-for-it.sh entrypoint.sh runBlazegraph.sh runUpdate.sh /
COPY --chown=blazegraph:blazegraph entrypoint.sh runBlazegraph.sh runUpdate.sh /
COPY --chown=blazegraph:blazegraph mwservices.json /templates/mwservices.json
COPY --chown=blazegraph:blazegraph RWStore.properties allowlist.txt logback.xml /wdqs/

Expand Down
13 changes: 8 additions & 5 deletions build/WDQS/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,13 @@ Variables in **bold** are required.

| Variable | Default | Description |
| ------------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`WIKIBASE_HOST`** | "wikibase" | Hostname of the Wikibase service |
| **`WDQS_HOST`** | "wdqs" | WDQS hostname (this service) |
| **`WDQS_PORT`** | "9999" | WDQS port (this service) |
| `WIKIBASE_SCHEME` | "http" | URL scheme of the Wikibase service |
| `WDQS_ENTITY_NAMESPACES` | "120,122" | Wikibase namespaces to load data from |
| `WDQS_SCHEME` | "http" | WDQS scheme (this service) |
| `WDQS_HOST` | "wdqs" | WDQS hostname (this service) |
| `WDQS_PORT`. | "9999" | WDQS port (this service) |
| `WIKIBASE_SCHEME` | "http" | URL scheme of the Wikibase service |
| `WIKIBASE_HOST`. | "wikibase" | Hostname of the Wikibase service |
| `WIKIBASE_PORT` | "80" | Port of the Wikibase service |
| `WIKIBASE_MAX_DAYS_BACK` | "90" | Maximum number of days updater can reach back in time from now |
| `MEMORY` | "" | Memory limit for Blazegraph |
| `HEAP_SIZE` | "1g" | Heap size for Blazegraph |
Expand Down Expand Up @@ -206,7 +208,8 @@ In the Docker Compose example provided above, you might run:

```sh
docker compose stop wdqs-updater
docker compose run --rm wdqs-updater bash '/wdqs/runUpdate.sh -h http://"$WDQS_HOST":"$WDQS_PORT" -- --wikibaseUrl "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --conceptUri "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --entityNamespaces "$WDQS_ENTITY_NAMESPACES" --init --start $(date +%Y%m%d000000)'
docker compose run --rm wdqs-updater bash '/wdqs/runUpdate.sh -h http://"$WDQS_HOST":"$WDQS_PORT" -- --wikibaseUrl "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST":"$WIKIBASE_PORT" --conceptUri "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST":"$WIKIBASE_PORT" --entityNamespaces "$WDQS_ENTITY_NAMESPACES" --init --start $(date +%Y%m%d000000)'
# Ctrl-C to cancel once "Sleeping for 10 secs" is seen
docker compose start wdqs-updater
```

Expand Down
10 changes: 0 additions & 10 deletions build/WDQS/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,6 @@
#!/usr/bin/env bash
# This file is provided by the wikibase/wdqs docker image.

# Test if required environment variables have been set
REQUIRED_VARIABLES=(WIKIBASE_HOST WDQS_HOST WDQS_PORT)
for i in "${REQUIRED_VARIABLES[@]}"; do
eval THISSHOULDBESET=\$"$i"
if [ -z "$THISSHOULDBESET" ]; then
echo "$i is required but isn't set. You should pass it to docker. See: https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file";
exit 1;
fi
done

set -eu

export BLAZEGRAPH_OPTS="${BLAZEGRAPH_EXTRA_OPTS} -DwikibaseHost=${WIKIBASE_HOST}"
Expand Down
5 changes: 1 addition & 4 deletions build/WDQS/runUpdate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,4 @@

cd /wdqs || exit

# TODO env vars for entity namespaces, scheme and other settings
/wait-for-it.sh "$WIKIBASE_HOST:80" -t 300 -- \
/wait-for-it.sh "$WDQS_HOST:$WDQS_PORT" -t 300 -- \
./runUpdate.sh -h http://"$WDQS_HOST":"$WDQS_PORT" -- --wikibaseUrl "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --conceptUri "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --entityNamespaces "$WDQS_ENTITY_NAMESPACES"
./runUpdate.sh -h "$WDQS_SCHEME"://"$WDQS_HOST":"$WDQS_PORT" -- --wikibaseUrl "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST":"$WIKIBASE_PORT" --conceptUri "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST":"$WIKIBASE_PORT" --entityNamespaces "$WDQS_ENTITY_NAMESPACES"
176 changes: 0 additions & 176 deletions build/WDQS/wait-for-it.sh
View file
Open in desktop

This file was deleted.

Loading