SegmentationFault on startup in Alpine based docker image

[laurent-bohr]

Bug summary

Build a docker image with Prefect server inside, if I make the image based on Alpine, the server startup raised a Segmentation Fault.
Using a Debian based image, it is working fine.

Not working Dockerfile

ARG PYTHON_IMAGE=python:3.12-alpine
FROM ${PYTHON_IMAGE} AS build
RUN apk update && apk add curl
RUN adduser --u 4242 -D -h /user user
USER user
WORKDIR /user
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
RUN . /user/.local/bin/env
COPY --chown=user:user . .
RUN /user/.local/bin/uv sync

FROM ${PYTHON_IMAGE}
RUN adduser --u 4242 -D -h /user user
USER user
WORKDIR /user
COPY --from=build --chown=user:user /user/.venv /user/.venv
COPY --chown=user:user .docker/*.sh /user/
CMD ["/container/run-server.sh"]

run-server.sh is :

#!/bin/sh
. /user/.venv/bin/activate
prefect server start

Running the container crash:

docker run --rm -p 4200:4200 my-prefect
Segmentation fault

Doing the same with a slim image, it is working fine.

ARG PYTHON_IMAGE=python:3.12-slim
FROM ${PYTHON_IMAGE} AS build
RUN apt-get update && apt-get install -y curl
RUN useradd --uid 6016 -m -d /container container
USER user
WORKDIR /user
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
RUN . /user/.local/bin/env
COPY --chown=user:user . .
RUN /user/.local/bin/uv sync

FROM ${PYTHON_IMAGE}
RUN useradd --uid 6016 -m -d /container container
USER user
WORKDIR /user
COPY --from=build --chown=user:user /user/.venv /user/.venv
COPY --chown=user:user .docker/*.sh /user/
CMD ["/container/run-server.sh"]

Running this one :

docker run --rm -p 4200:4200 my-prefect

 ___ ___ ___ ___ ___ ___ _____ 
| _ \ _ \ __| __| __/ __|_   _| 
|  _/   / _|| _|| _| (__  | |  
|_| |_|_\___|_| |___\___| |_|  

Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api

View the API reference documentation at http://127.0.0.1:4200/docs

Check out the dashboard at http://127.0.0.1:4200

Version info

Python 3.12
Prefect 3.1.11

Additional context

No response

[zzstoatzz]

hi @laurent-bohr - i have reproduced this. I think the issue is (once again) related to pendulum, and perhaps in the same vein as this issue. I haven't figured out what exactly is missing between alpine and slim but I've taken some notes based on the repro so far.

Attempts to fix Alpine segfault:

1. Initial setup:

   • Created Dockerfile with Python 3.12 Alpine base
   • Used uv to install dependencies
   • Server segfaults on startup

2. Investigated library differences between Alpine and slim:
   Alpine:

   • /lib/ld-musl-aarch64.so.1
   • libpython3.12.so.1.0
   • libc.musl-aarch64.so.1

   Slim:

   • linux-vdso.so.1
   • libpython3.12.so.1.0
   • libc.so.6 (glibc)
   • libm.so.6
   • /lib/ld-linux-aarch64.so.1

3. Tried adding libm-dev to Alpine:

   • Failed because package doesn't exist
   • Alpine uses musl's math functions instead of glibc's

4. Confirmed the issue:

   • Build stage works fine (all packages install correctly)
   • Segfault happens at runtime when starting the server
   • Same issue occurs with both Python 3.11 and 3.12
   • Same issue whether using uv sync or direct package install

5. Identified specific segfault location:

   • Added faulthandler to get crash info
   • Segfault occurs in pendulum during import
   • Specifically in pendulum/parsing/init.py line 27
   • This happens during prefect.server.start()
   • All other imports work fine before this point

Key observations:

• Segfault occurs in pendulum's C extensions
• Issue is specific to musl libc vs glibc
• Happens during runtime, not build time
• Basic Python and prefect imports work
• Only fails when starting the server

Have you previously used prefect with an alpine image? We have historically used slim for our base images