Implement manual initiation of noble migration #7416
Comments
|
The manual workflow should look something like:
|
|
I'm going to start working on the playbook for this and see how far I get. |
|
I successfully got the playbook to migrate a mon server, work is in the stg-noble-playbook branch. I'll plumb it into |
legoktm
added a commit
that referenced
this issue
Jan 27, 2025
Admins can run `./securedrop-admin noble_migration` to trigger a manual noble migration. At a high level the playbook: * disables OSSEC notifications * triggers the app upgrade, waiting through two reboots * triggers the mon upgrade, again waiting through reboots * re-enables OSSEC notifications The most complicated part is how we for the reboots. We first have a `wait_for` that looks for a specific stage in the state file. Because the upgrade script writes the state file and then immediately reboots, it should never actually succeed and fail because the connection is interrupted. So we set `ignore_unreachable` and `ignore_errors`, and the next block is `wait_for_connection` for the server to come back up. There is a delay before we begin checking just in case the wait_for did succeed and we need to wait for the reboot to happen. Because of this sequencing, there isn't any support for the playbook failing mid-host and restarting it. It is probably unnecessary since, once started, the upgrade will automatically finish by itself. The script does support one host already being upgraded and the other still needing migration. So if e.g. app migration fails, you can manually fix the host, let it auto finish the upgrade, and then re-run the playbook to migrate mon. Fixes #7416.
5 tasks
legoktm
added a commit
that referenced
this issue
Jan 27, 2025
Admins can run `./securedrop-admin noble_migration` to trigger a manual noble migration. At a high level the playbook: * disables OSSEC notifications * triggers the app upgrade, waiting through two reboots * triggers the mon upgrade, again waiting through reboots * re-enables OSSEC notifications The most complicated part is how we for the reboots. We first have a `wait_for` that looks for a specific stage in the state file. Because the upgrade script writes the state file and then immediately reboots, it should never actually succeed and fail because the connection is interrupted. So we set `ignore_unreachable` and `ignore_errors`, and the next block is `wait_for_connection` for the server to come back up. There is a delay before we begin checking just in case the wait_for did succeed and we need to wait for the reboot to happen. Because of this sequencing, there isn't any support for the playbook failing mid-host and restarting it. It is probably unnecessary since, once started, the upgrade will automatically finish by itself. The script does support one host already being upgraded and the other still needing migration. So if e.g. app migration fails, you can manually fix the host, let it auto finish the upgrade, and then re-run the playbook to migrate mon. Fixes #7416.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There should be some kind of script on the admin workstation that triggers starting the noble migration on the app and then mon servers.
The text was updated successfully, but these errors were encountered: