Extract Python components into their own packages

The following packages were introduced:

* "indexify-python-utils" with common functionality for components written in Python.
  Not published to PyPi. It currently includes:
  ** Logging configuration for services written in Python (used by Function Executor and Executor).
  ** HTTP client utils (used by Python SDK and Executor).
  ** Common build and testing scripts for all other Python packages.
* "indexify-python-sdk" for Python SDK library. Not published to PyPi.
* "indexify-function-executor" for Function Executor. Not published to PyPi.
* "indexify-executor" for Executor. Not published to PyPi.
* "indexify" (depends on all the 4 packages above ^ and includes the CLI).
  Published to PyPi. This is the same "all in one" package we were using right now.
  It should be compatible with all existing customer UX.
  The idea is that "indexify" package combines all Open Source Indexify components
  in one package and customers can use the package to develop their workflows and to
  run local or distributed Indexify clusters using the CLI and also to build their
  function images.

Also added README.md for all the packages and their tests to clarify what functionality
belongs where and how the packages relate to each other.

Versioning:

"indexify-python-sdk" package has the current latest version 0.2.48.
"indexify" package has the same current version 0.2.48.
All other packages have version 0.1.0 as it's not as important to keep their versions
the same as the current 0.2.48.

The versioning policy from now on is the following:
* Versions of all the packages are independent except "indexify" package.
* When a package version gets incremented then "indexify" package version
  gets incremented too.
  ** This ensures that all important enough updates in any Indexify component
     results in user visible version change.

Also update github actions to:
* Work with "indexify" package folder instead of
python-sdk package folder.
* Run all the linter checks and tests in all the packages.

Testing:

make check
make test

.github/pull_request_template.md

@@ -27,17 +27,17 @@ Consider providing screenshots, logs, code or other visual aids to help the revi
 
 ## Contribution Checklist
 
-- [ ] If the python-sdk was changed, please run `make fmt` in `python-sdk/`.
+- [ ] If a Python package was changed, please run `make fmt` in the package directory.
 - [ ] If the server was changed, please run `make fmt` in `server/`.
 - [ ] Make sure all PR Checks are passing.
 <!--
-You can run the tests manually:
-
 Notes:
 
-- Tests can be run manually: start the server and executor, `cd python-sdk`,
-  run `make test`.
-- To test if changes to the server are backward compatible with the latest
-  release, label the PR with `ci_compat_test`. This might report failures
-  unrelated to your change if previous incompatible changes were pushed without
-  being released yet -->
+Tests of a Python package can be run manually. Start a Server and an Executor then
+run `make test` in the Python package directory.
+
+To test if changes to the server are backward compatible with the latest
+release, label the PR with `ci_compat_test`. This might report failures
+unrelated to your change if previous incompatible changes were pushed without
+being released yet
+-->

.github/workflows/publish_indexify_pypi.yaml

@@ -21,7 +21,7 @@ permissions:
 
 defaults:
   run:
-    working-directory: ./python-sdk
+    working-directory: ./indexify
 
 jobs:
   extract-version:
@@ -38,28 +38,10 @@ jobs:
         run: pip install toml
       - id: version_extraction
         run: echo "version=$(python -c 'import toml; print(toml.load("pyproject.toml")["tool"]["poetry"]["version"])')" >> $GITHUB_OUTPUT
-
-  # create-release:
-  #   name: Create GitHub Release
-  #   runs-on: ubuntu-latest
-  #   needs:
-  #     - extract-version
-  #   steps:
-  #     - name: Create GitHub Release
-  #       id: create_release
-  #       uses: actions/create-release@v1
-  #       with:
-  #         tag_name: "v${{ needs.extract-version.outputs.version }}"
-  #         prerelease: ${{ github.event.inputs.prerelease }}
-  #         body: ${{ github.event.inputs.release_message }}
-  #       env:
-  #         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
   publish-indexify-to-pypi:
-    name: Publish Indexify Python Client
+    name: Publish Indexify package
     runs-on: ubuntu-latest
-    # needs:
-    #   - create-release
     environment:
       name: pypi
       url: https://pypi.org/p/indexify
@@ -74,7 +56,7 @@ jobs:
         run: pipx install poetry
       - name: Build python-sdk
         run: make build
-      - name: Publish Indexify Client to PyPI
+      - name: Publish Indexify to PyPI
         uses: pypa/gh-action-pypi-publish@release/v1
         with:
           packages-dir: python-sdk/dist/

.github/workflows/tests.yaml

@@ -10,11 +10,16 @@ on:
       - 'main'
     paths:
       - 'server/**'
+      - 'executor/**'
+      - 'function_executor/**'
+      - 'indexify/**'
+      - 'python_utils/**'
       - 'python-sdk/**'
       - '.github/workflows/tests.yaml'
 
 env:
   CARGO_TERM_COLOR: always
+  INDEXIFY_URL: http://localhost:8900
 
 jobs:
   lint_server:
@@ -63,8 +68,8 @@ jobs:
           name: indexify-server
           path: server/target/debug/indexify-server
 
-  lint_python_sdk:
-    name: Lint Indexify Python SDK
+  lint_python_packages:
+    name: Lint Python packages
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
@@ -75,61 +80,101 @@ jobs:
         with:
           python-version: 3.11
           cache: 'poetry'
+      - name: Build executor
+        run: cd executor && make build
+      - name: Lint executor
+        run: cd executor && make check
+      - name: Build function_executor
+        run: cd function_executor && make build
+      - name: Lint function_executor
+        run: cd function_executor && make check
+      - name: Build indexify
+        run: cd indexify && make build
+      - name: Lint indexify
+        run: cd indexify && make check
+      - name: Build python_utils
+        run: cd python_utils && make build
+      - name: Lint python_utils
+        run: cd python_utils && make check
       - name: Build python-sdk
-        run: |
-          cd python-sdk
-          make build
+        run: cd python-sdk && make build
       - name: Lint python-sdk
-        run: |
-          cd python-sdk
-          make check
+        run: cd python-sdk && make check
 
   acceptance_tests:
     name: Run Acceptance Tests
-    needs: [build_server, lint_python_sdk]
+    # needs: [build_server, lint_python_packages]
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
-      - name: Download indexify-server
-        uses: actions/download-artifact@v4
-        with:
-          name: indexify-server
+      # - name: Download indexify-server
+      #   uses: actions/download-artifact@v4
+      #   with:
+      #     name: indexify-server
       - name: Install poetry
         run: pipx install poetry
       - name: Setup Python
         uses: actions/setup-python@v5
         with:
           python-version: 3.11
           cache: 'poetry'
-      - name: Build python-sdk
+      - name: Build indexify
         run: |
-          cd python-sdk
+          cd indexify
           make build
-      - name: Lint python-sdk
+      - name: Lint indexify
         run: |
-          cd python-sdk
+          cd indexify
           make check
-      - name: Start Background Indexify Server
-        uses: JarvusInnovations/background-action@v1
-        with:
-          run: |
-            chmod u+x ./indexify-server
-            RUST_LOG=debug ./indexify-server &
-            echo $! > /tmp/indexify-server.pid &
+      # - name: Start Background Indexify Server
+      #   uses: JarvusInnovations/background-action@v1
+      #   with:
+      #     run: |
+      #       chmod u+x ./indexify-server
+      #       RUST_LOG=debug ./indexify-server &
+      #       echo $! > /tmp/indexify-server.pid &
 
-          wait-on: |
-            tcp:localhost:8900
+      #     wait-on: |
+      #       tcp:localhost:8900
 
-          tail: true
-          wait-for: 30s
-          log-output: true
-          # always logging the output to debug test failures.
-          log-output-if: true
+      #     tail: true
+      #     wait-for: 30s
+      #     log-output: true
+      #     # always logging the output to debug test failures.
+      #     log-output-if: true
+      - name: Start Indexify Executor
+        run: |
+          ls
+          cd indexify
+          `poetry env list --full-path | awk '{ print $1 }'`/bin/indexify-cli
+          strace -f poetry run indexify-cli
+          poetry run indexify-cli --help
+          poetry run indexify-cli executor
+      # ls
+      # pwd
+      # poetry debug info
+      # make check
+      # poetry run python -c "print('python version:', __import__('sys').version)"
+      # poetry run python -c "import python_utils"
+      # echo "python_utils imported"
+      # poetry run python -c "import function_executor"
+      # echo "function_executor imported"
+      # poetry run python -c "import executor"
+      # echo "executor imported"
+      # poetry run python -c "import indexify"
+      # echo "indexify imported"
+      # poetry run python -c "import cli"
+      # echo "cli imported"
+      # ls -l `poetry env list --full-path | awk '{ print $1 }'`/bin/indexify-cli
+      # file `poetry env list --full-path | awk '{ print $1 }'`/bin/indexify-cli
+      # cat `poetry env list --full-path | awk '{ print $1 }'`/bin/indexify-cli
+      # ls -l `poetry env list --full-path | awk '{ print $1 }'`/bin/
+      # ls -l `poetry env list --full-path | awk '{ print $1 }'`/lib/python3.11/site-packages/
       - name: Start Background Indexify Executor
         uses: JarvusInnovations/background-action@v1
         with:
           run: |
-            cd python-sdk
+            cd indexify
             poetry run indexify-cli executor &
             echo $! > /tmp/indexify-executor.pid &
 
@@ -160,11 +205,16 @@ jobs:
                 sleep 5
             fi
           done
-      - name: Run All Tests
-        run: |
-          cd python-sdk
-          export INDEXIFY_URL=http://localhost:8900
-          make test
+      - name: Run executor tests
+        run: cd executor && make test
+      - name: Run function_executor tests
+        run: cd function_executor && make test
+      - name: Run indexify tests
+        run: cd indexify && make test
+      - name: Run python_utils tests
+        run: cd python_utils && make test
+      - name: Run python-sdk tests
+        run: cd python-sdk && make test
       - name: Terminate processes
         # We want to test clean termination of processes.
         run: |
@@ -189,7 +239,7 @@ jobs:
   last_release_acceptance_tests:
     name: 'Last Release Acceptance Tests (trigger with label: ci_compat_test)'
     if: contains(github.event.pull_request.labels.*.name, 'ci_compat_test')
-    needs: [build_server, lint_python_sdk]
+    needs: [build_server, lint_python_packages]
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
@@ -202,17 +252,6 @@ jobs:
         uses: actions/download-artifact@v4
         with:
           name: indexify-server
-      - name: Install poetry
-        run: pipx install poetry
-      - name: Setup Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: 3.11
-          cache: 'poetry'
-      - name: Build python-sdk
-        run: |
-          cd python-sdk
-          make build
       - name: Start Background Indexify Server
         uses: JarvusInnovations/background-action@v1
         with:
@@ -229,11 +268,20 @@ jobs:
           log-output: true
           # always logging the output to debug test failures.
           log-output-if: true
+      - name: Install poetry
+        run: pipx install poetry
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.11
+          cache: 'poetry'
+      - name: Build indexify
+        run: cd indexify && make build
       - name: Start Background Indexify Executor
         uses: JarvusInnovations/background-action@v1
         with:
           run: |
-            cd python-sdk
+            cd indexify
             poetry run indexify-cli executor &
             echo $! > /tmp/indexify-executor.pid &
 
@@ -264,11 +312,16 @@ jobs:
                 sleep 5
             fi
           done
-      - name: Run All Tests
-        run: |
-          cd python-sdk
-          export INDEXIFY_URL=http://localhost:8900
-          make test
+      - name: Run executor tests
+        run: cd executor && make test
+      - name: Run function_executor tests
+        run: cd function_executor && make test
+      - name: Run indexify tests
+        run: cd indexify && make test
+      - name: Run python_utils tests
+        run: cd python_utils && make test
+      - name: Run python-sdk tests
+        run: cd python-sdk && make test
       - name: Terminate processes
         # We want to test clean termination of processes.
         run: |

.vscode/settings.json

@@ -0,0 +1,9 @@
+{
+    "python.analysis.extraPaths": [
+        "./python-sdk/src",
+        "./function_executor/src",
+        "./executor/src",
+        "./indexify/src",
+        "./python_utils/src"
+    ]
+}

executor/Makefile

@@ -0,0 +1 @@
+include ../python_utils/common.mk

executor/README.md

@@ -0,0 +1,29 @@
+## Overview
+
+Executor registers at Indexify Server and continiously pulls tasks assigned to it from the Indexify Server
+and executes them. While registering it shares its capabilities like available hardware with the Indexify
+Server and periodically updates the Server about its current state. Executor spins up Function Executors
+to run customer functions. Executor should never link with Indexify Python-SDK. It should not know anything
+about programming languages and runtime environments used by Indexify Functions. Function Executor is
+responsible for this.
+
+## Deployment
+
+An Executor can run in a Virtual Machine, container or a in bare metal host. Each vm/container/bare metal
+host in an Indexify cluster deployment runs a single Executor daemon process.
+Open Source users manage the Executors fleet themself e.g. using Kubernetes, any other cluster
+orchestrator or even manually.
+
+An Executor can be configured to only run particular functions. This allows to route certain Indexify
+functions to only particular subsets of Executors to e.g. implement a load balancing strategy.
+
+This package doesn't provide an executable entry point that runs an Executor. This is intentional
+as Executor has many configurable subcomponents. `indexify` package provides a cli with `executor`
+command that runs Executor with functionality available in Open Source offering.
+
+## Threat model
+
+A VM/container/bare metal host where an Executor is running is fully trusted. This works well for single
+tenant deployments where customer functions' code is fully trusted. If this is not the case then Function
+Executors that run customer functions need to get isolated from Executor using e.g. Virtual Machines.
+This functionality is not included into the Open Source offering.