-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix/ci cache #319
Fix/ci cache #319
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
name: create-requirements | ||
|
||
on: | ||
workflow_dispatch: | ||
push: | ||
branches: [ master ] | ||
paths: | ||
- poetry.lock | ||
|
||
jobs: | ||
|
||
create-reqs: | ||
runs-on: ubuntu-latest | ||
env: | ||
TF_CPP_MIN_LOG_LEVEL: 3 | ||
|
||
steps: | ||
- name: Setup Mambaforge | ||
uses: conda-incubator/setup-miniconda@v2 | ||
with: | ||
python-version: 3.9 | ||
miniforge-variant: Mambaforge | ||
miniforge-version: latest | ||
activate-environment: qmlcourse.ai | ||
environment-file: tools/environment.yml | ||
use-mamba: true | ||
- name: install the main dependencies | ||
run: | | ||
poetry install | ||
# todo: replace with a more sane solution | ||
- name: install tensorflow_quantum # https://www.tensorflow.org/quantum/install | ||
run: | | ||
poetry run pip install --upgrade pip | ||
poetry run pip install -U tensorflow==2.5.1 | ||
poetry run pip install -U tensorflow_quantum | ||
poetry run pip install -U tfq-nightly | ||
- name: create requirements.yml | ||
run: conda env export > environment.yml | ||
- name: Upload output file | ||
uses: actions/upload-artifact@v2 | ||
with: | ||
name: environment-yml-file | ||
path: environment.yml |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
name: create-requirements | ||
|
||
on: | ||
workflow_dispatch: | ||
push: | ||
branches: [ master ] | ||
paths: | ||
- poetry.lock | ||
|
||
jobs: | ||
|
||
create-reqs: | ||
runs-on: ubuntu-latest | ||
env: | ||
TF_CPP_MIN_LOG_LEVEL: 3 | ||
|
||
steps: | ||
- name: Setup Mambaforge | ||
uses: conda-incubator/setup-miniconda@v2 | ||
with: | ||
python-version: 3.9 | ||
miniforge-variant: Mambaforge | ||
miniforge-version: latest | ||
activate-environment: qmlcourse.ai | ||
environment-file: tools/environment.yml | ||
use-mamba: true | ||
- name: install the main dependencies | ||
run: | | ||
poetry install | ||
# todo: replace with a more sane solution | ||
- name: install tensorflow_quantum # https://www.tensorflow.org/quantum/install | ||
run: | | ||
poetry run pip install --upgrade pip | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Может сразу попробуем затащить tfq в poetry? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Да это тоже самое будет, что и это. Можно попробовать, конечно. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Нет, в том то и фишка, что это будет на 2-3 минуты быстрее. Сейчас он сначала долго ставит зависимости из поетри, а потом долго ставит их еще раз при установке tfq. Там прямо в логах видно, как он удаляет одни версии и скачивает + ставит другие. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Это один раз делается для создания requirements.yml, вообще возможно там ничего не соберётся и придется вручную править зависимости 😔 |
||
poetry run pip install -U tensorflow==2.5.1 | ||
poetry run pip install -U tensorflow_quantum | ||
poetry run pip install -U tfq-nightly | ||
- name: create requirements.yml | ||
run: conda env export > environment.yml | ||
- name: Upload output file | ||
uses: actions/upload-artifact@v2 | ||
with: | ||
name: environment-yml-file | ||
path: environment.yml | ||
# TODO: commit this file |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,100 @@ | ||
name: new-deploy-book | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. зачем There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. добавить зависимость что это этот файл должен запускаться при условии успешного There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ещё не оттестировано, заренеймить не долго, мы же потом мерджим сквошем, будет как будто изначально меняли те файлы. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. почему, ты ведь создал новый файл? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Потому что сквош переписывает историю. Если будет нужно, я сделаю rebase и все будет как будто мы изначально их правили, если потребуется тестирование в мастере, я так понял, в ветках actions не подтягиваются? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ты меня уже запутал этими файлами и не понятными дублями |
||
|
||
on: | ||
workflow_dispatch: | ||
inputs: | ||
DisableCache: | ||
description: 'Disable cache' | ||
required: false | ||
default: 'false' | ||
push: | ||
branches: [ master ] | ||
strategy: | ||
matrix: | ||
include: | ||
- os: ubuntu-latest | ||
label: linux-64 | ||
prefix: /usr/share/miniconda3/envs/qmlcourse.ai | ||
|
||
# - os: macos-latest | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Надо это удалить. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Это нельзя удалять, с помощью этого префикса сделано кэширование у них:-( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Я про то, что удалить тестирование сборки для других осей There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Пока не узнаю, что можно задать матрицу из одного элемента удалять нельзя, просто придется иначе колхозить как-то отключение сборки для других платформ внутри кода, типа if вставить лишний. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
# label: osx-64 | ||
# prefix: /Users/runner/miniconda3/envs/qmlcourse.ai | ||
|
||
# - os: windows-latest | ||
# label: win-64 | ||
# prefix: C:\Miniconda3\envs\qmlcourse.ai | ||
jobs: | ||
|
||
deploy_book: | ||
runs-on: ubuntu-latest | ||
timeout-minutes: 10 | ||
env: | ||
TF_CPP_MIN_LOG_LEVEL: 3 | ||
steps: | ||
- name: Checkout branch for build | ||
uses: actions/checkout@v2 | ||
|
||
- name: Checkout branch with cache | ||
uses: actions/checkout@v2 | ||
with: | ||
ref: "gh-pages" | ||
path: ./gh-pages | ||
|
||
- name: Copy cache for build book | ||
if: ${{ github.event.inputs.DisableCache != 'true' }} | ||
run: | | ||
cp -r ./gh-pages/_build/ ./qmlcourseRU/ || exit 0 | ||
- name: Download a requirements.yml artifact | ||
uses: actions/download-artifact@v2 | ||
with: | ||
name: requirements-yml-file | ||
|
||
# Install dependencies | ||
- name: install qmlcourse.ai mamba env | ||
uses: conda-incubator/setup-miniconda@v2 | ||
with: | ||
miniforge-variant: Mambaforge | ||
miniforge-version: latest | ||
use-mamba: true | ||
python-version: 3.9 | ||
mamba-version: "*" | ||
channels: conda-forge, defaults | ||
channel-priority: true | ||
activate-environment: qmlcourse.ai | ||
environment-file: environment.yml | ||
|
||
# Use cache | ||
- uses: actions/cache@v2 | ||
with: | ||
path: ${{ matrix.prefix }} | ||
key: ${{ matrix.label }}-conda-${{ hashFiles('environment.yml') }} | ||
id: mamba-cache | ||
|
||
- name: export some tokens | ||
env: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Задание этого env надо перенести в сборку книги, а сам шаг удалить. |
||
DWAVE_TOKEN: ${{ secrets.DWAVE_TOKEN }} | ||
run: | | ||
export DWAVE_TOKEN="$DWAVE_TOKEN" | ||
|
||
- name: Build the book | ||
run: | | ||
poetry run jupyter-book build ./qmlcourseRU | ||
|
||
# Push the book's HTML to github-pages | ||
- name: GitHub Pages action | ||
uses: peaceiris/[email protected] | ||
with: | ||
github_token: ${{ secrets.GITHUB_TOKEN }} | ||
publish_dir: ./qmlcourseRU | ||
|
||
# - name: Build PDF file | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Пока удалить There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Да починим pdf совсем скоро, если все собирается There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Его нет смысла чинить в текущем виде. Надо собирать только через ковертацию в LaTeX, а там сразу будут проблемы (как минимум, с 99% вероятностью не будет русского языка из коробки и т.д.) |
||
# run: | | ||
# poetry run jupyter-book build ./qmlcourseRU --builder pdfhtml | ||
# mv ./qmlcourseRU/_build/pdf/book.pdf ./qmlcourseRU/_build/pdf/all_book.pdf | ||
|
||
# - name: Deploy only pdf files | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Покаудалить |
||
# uses: peaceiris/actions-gh-pages@v3 | ||
# with: | ||
# github_token: ${{ secrets.GITHUB_TOKEN }} | ||
# publish_dir: ./qmlcourseRU/_build/pdf/ | ||
# destination_dir: pdf |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
name: deploy-branch | ||
|
||
on: | ||
workflow_dispatch: | ||
|
||
jobs: | ||
|
||
deploy_branch: | ||
runs-on: ubuntu-latest | ||
timeout-minutes: 15 | ||
|
||
steps: | ||
- name: checkout code | ||
uses: actions/checkout@v2 | ||
|
||
- name: get branch name | ||
uses: nelonoel/[email protected] | ||
|
||
- name: install mamba and poetry | ||
- uses: conda-incubator/setup-miniconda@v2 | ||
with: | ||
python-version: 3.9 | ||
mamba-version: "*" | ||
channels: conda-forge, psi4, defaults | ||
channel-priority: true | ||
activate-environment: qmlcourse.ai | ||
environment-file: tools/environment.yml | ||
|
||
- name: install poetry dependencies | ||
- shell: bash -l {0} | ||
run: poetry install | ||
|
||
# todo: replace with a more sane solution | ||
- name: install tensorflow_quantum | ||
- shell: bash -l {0} | ||
run: | | ||
poetry run pip install -U tensorflow==2.5.1 | ||
poetry run pip install -U tensorflow_quantum | ||
poetry run pip install -U tfq-nightly | ||
|
||
- name: add SSH key-file | ||
run: | | ||
mkdir -p /home/runner/.ssh | ||
echo "${{ secrets.AWS_SSH_KEY }}" > /home/runner/.ssh/key.pem | ||
chmod 600 /home/runner/.ssh/key.pem | ||
|
||
- name: build the book | ||
run: | | ||
export DWAVE_TOKEN="${{ secrets.DWAVE_TOKEN }}" | ||
poetry run jupyter-book build ./qmlcourseRU | ||
|
||
- name: deploy | ||
run: | | ||
export CURRENT_BRANCH="${BRANCH_NAME}" | ||
|
||
export AWS_HOST="54.89.156.216" | ||
export AWS_USER="ubuntu" | ||
export SSH_KEY="/home/runner/.ssh/key.pem" | ||
|
||
poetry run python ./tools/deploy2aws.py ./ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
а как этот файл будет запускаться автоматом? и он вроде как дубль что в workflows
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Я так понимаю, что это правильный путь как надо сделать, и во все воркфлоу добавить if про то, что если *.lock или *.toml изменён, то запустить этот экшен. Я посмотрю как в докер билдах делают, там такая же проблема
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
почему «изменен»? нужно чтобы выполнялось только если оно зелёное
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
про дубль этого файл в этом pr так и не понял
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@vtrokhymenko один в actions, совсем правильно будет сделать через него, то есть вставить if файлы с библиотеками поменялись, тогда запустить сначала этот action. Второй файл в workflows нужен чтобы просто пока тестировать, но в целом и с его помощью можно тоже сделать, типа придется добавить тайм-аут и такое решение не очень.
Сам пока не разобрался как это делается правильно, но тут https://github.community/t/is-it-possible-to-run-the-job-only-when-a-specific-file-changes/115484 есть разные варианты
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
а как именно и часто будет запускаться файл что в
actions
? и ты уверен что он там будет так работать, а то по твоей ссылки сходу не нашел что именно в той директории оно должно быть