feat: replace scan paths options to single one

It has been found that previous system didn't allow to configure
specific cases (e.g. a whitelisted directory inside /usr/lib/cmake). The
current solution replaces two options to single one, which also allows a
regular expressions

Also PackageArchive class has been moved to core package, because it is
more about service rather than model

CONTRIBUTING.md

@@ -92,7 +92,7 @@ Again, the most checks can be performed by `tox` command, though some additional
     ```
 
 * Type annotations are the must, even for local functions. For the function argument `self` (for instance methods) and `cls` (for class methods) should not be annotated.
-* For collection types built-in classes must be used if possible (e.g. `dict` instead of `typing.Dict`, `tuple` instead of `typing.Tuple`). In case if built-in type is not available, but `collections.abc` provides interface, it must be used (e.g. `collections.abc.Awaitable` instead of `typing.Awaitable`, `collections.abc.Iterable` instead of `typing.Iterable`). For union classes, the bar operator (`|`) must be used (e.g. `float | int` instead of `typing.Union[float, int]`), which also includes `typinng.Optional` (e.g. `str | None` instead of `Optional[str]`).
+* For collection types built-in classes must be used if possible (e.g. `dict` instead of `typing.Dict`, `tuple` instead of `typing.Tuple`). In case if built-in type is not available, but `collections.abc` provides interface, it must be used (e.g. `collections.abc.Awaitable` instead of `typing.Awaitable`, `collections.abc.Iterable` instead of `typing.Iterable`). For union classes, the bar operator (`|`) must be used (e.g. `float | int` instead of `typing.Union[float, int]`), which also includes `typing.Optional` (e.g. `str | None` instead of `Optional[str]`).
 * `classmethod` should (almost) always return `Self`. In case of mypy warning (e.g. if there is a branch in which function doesn't return the instance of `cls`) consider using `staticmethod` instead.
 * Recommended order of function definitions in class:
 

README.md

@@ -2,7 +2,7 @@
 
 [![tests status](https://github.com/arcan1s/ahriman/actions/workflows/tests.yml/badge.svg)](https://github.com/arcan1s/ahriman/actions/workflows/tests.yml)
 [![setup status](https://github.com/arcan1s/ahriman/actions/workflows/setup.yml/badge.svg)](https://github.com/arcan1s/ahriman/actions/workflows/setup.yml)
-[![Docker Image Version (latest semver)](https://img.shields.io/docker/v/arcan1s/ahriman?label=Docker%20image)](https://hub.docker.com/r/arcan1s/ahriman)
+[![Docker Image Version (latest semver)](https://img.shields.io/docker/v/arcan1s/ahriman?label=Docker%20image&sort=semver)](https://hub.docker.com/r/arcan1s/ahriman)
 [![CodeFactor](https://www.codefactor.io/repository/github/arcan1s/ahriman/badge)](https://www.codefactor.io/repository/github/arcan1s/ahriman)
 [![Documentation Status](https://readthedocs.org/projects/ahriman/badge/?version=latest)](https://ahriman.readthedocs.io)
 

docs/ahriman.core.build_tools.rst

@@ -4,6 +4,14 @@ ahriman.core.build\_tools package
 Submodules
 ----------
 
+ahriman.core.build\_tools.package\_archive module
+-------------------------------------------------
+
+.. automodule:: ahriman.core.build_tools.package_archive
+   :members:
+   :no-undoc-members:
+   :show-inheritance:
+
 ahriman.core.build\_tools.sources module
 ----------------------------------------
 

docs/ahriman.models.rst

@@ -124,14 +124,6 @@ ahriman.models.package module
    :no-undoc-members:
    :show-inheritance:
 
-ahriman.models.package\_archive module
---------------------------------------
-
-.. automodule:: ahriman.models.package_archive
-   :members:
-   :no-undoc-members:
-   :show-inheritance:
-
 ahriman.models.package\_description module
 ------------------------------------------
 

docs/configuration.rst

@@ -81,14 +81,13 @@ Authorized users are stored inside internal database, if any of external provide
 
 Build related configuration. Group name can refer to architecture, e.g. ``build:x86_64`` can be used for x86_64 architecture specific settings.
 
-* ``allowed_scan_paths`` - paths to be used for implicit dependencies scan, scape separated list of paths, optional.
 * ``archbuild_flags`` - additional flags passed to ``archbuild`` command, space separated list of strings, optional.
-* ``blacklisted_scan_paths`` - paths to be excluded for implicit dependencies scan, scape separated list of paths, optional. Normally all elements of this option must be child paths of any of ``allowed_scan_paths`` element.
 * ``build_command`` - default build command, string, required.
 * ``ignore_packages`` - list packages to ignore during a regular update (manual update will still work), space separated list of strings, optional.
 * ``include_debug_packages`` - distribute debug packages, boolean, optional, default ``yes``.
 * ``makepkg_flags`` - additional flags passed to ``makepkg`` command, space separated list of strings, optional.
 * ``makechrootpkg_flags`` - additional flags passed to ``makechrootpkg`` command, space separated list of strings, optional.
+* ``scan_paths`` - paths to be used for implicit dependencies scan, space separated list of strings, optional. If any of those paths is matched against the path, it will be added to the allowed list.
 * ``triggers`` - list of ``ahriman.core.triggers.Trigger`` class implementation (e.g. ``ahriman.core.report.ReportTrigger ahriman.core.upload.UploadTrigger``) which will be loaded and run at the end of processing, space separated list of strings, optional. You can also specify triggers by their paths, e.g. ``/usr/lib/python3.10/site-packages/ahriman/core/report/report.py.ReportTrigger``. Triggers are run in the order of definition.
 * ``triggers_known`` - optional list of ``ahriman.core.triggers.Trigger`` class implementations which are not run automatically and used only for trigger discovery and configuration validation.
 * ``vcs_allowed_age`` - maximal age in seconds of the VCS packages before their version will be updated with its remote source, integer, optional, default is 7 days.

docs/faq/general.rst

@@ -379,7 +379,7 @@ After the success build the application extracts all linked libraries and used d
 
 In order to disable this check completely, the ``--no-check-files`` flag can be used.
 
-In addition, there is possibility to control paths which will be used for checking, by using options ``build.allowed_scan_paths`` and ``build.blacklisted_scan_paths``. Leaving ``build.allowed_scan_paths`` blank will effectively disable any check too.
+In addition, there is possibility to control paths which will be used for checking, by using option ``build.scan_paths``, which supports regular expressions. Leaving ``build.scan_paths`` blank will effectively disable any check too.
 
 How to install built packages
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

package/share/ahriman/settings/ahriman.ini

@@ -50,12 +50,8 @@ allow_read_only = yes
 ;salt =
 
 [build]
-; List of paths to be used for implicit dependency scan
-allowed_scan_paths = /usr/lib
 ; List of additional flags passed to archbuild command.
 ;archbuild_flags =
-; List of paths to be excluded for implicit dependency scan. Usually they should be subpaths of allowed_scan_paths
-blacklisted_scan_paths = /usr/lib/cmake
 ; Path to build command
 ;build_command =
 ; List of packages to be ignored during automatic updates.
@@ -66,6 +62,8 @@ blacklisted_scan_paths = /usr/lib/cmake
 ;makechrootpkg_flags =
 ; List of additional flags passed to makepkg command.
 makepkg_flags = --nocolor --ignorearch
+; List of paths to be used for implicit dependency scan. Regular expressions are supported
+scan_paths = ^usr/lib(?!/cmake).*$
 ; List of enabled triggers in the order of calls.
 triggers = ahriman.core.gitremote.RemotePullTrigger ahriman.core.report.ReportTrigger ahriman.core.upload.UploadTrigger ahriman.core.gitremote.RemotePushTrigger
 ; List of well-known triggers. Used only for configuration purposes.

src/ahriman/models/package_archive.py → src/ahriman/core/build_tools/package_archive.py

@@ -17,7 +17,6 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
-from dataclasses import dataclass
 from elftools.elf.dynamic import DynamicSection
 from elftools.elf.elffile import ELFFile
 from pathlib import Path
@@ -33,7 +32,6 @@
 from ahriman.models.scan_paths import ScanPaths
 
 
-@dataclass
 class PackageArchive:
     """
     helper for package archives
@@ -45,10 +43,20 @@ class PackageArchive:
         scan_paths(ScanPaths): scan paths holder
     """
 
-    root: Path
-    package: Package
-    pacman: Pacman
-    scan_paths: ScanPaths
+    def __init__(self, root: Path, package: Package, pacman: Pacman, scan_paths: ScanPaths) -> None:
+        """
+        default constructor
+
+        Args:
+            root(Path): path to root filesystem
+            package(Package): package descriptor
+            pacman(Pacman): alpm wrapper instance
+            scan_paths(ScanPaths): scan paths holder
+        """
+        self.root = root
+        self.package = package
+        self.pacman = pacman
+        self.scan_paths = scan_paths
 
     @staticmethod
     def dynamic_needed(binary_path: Path) -> list[str]:

src/ahriman/core/configuration/schema.py

@@ -169,14 +169,6 @@
     "build": {
         "type": "dict",
         "schema": {
-            "allowed_scan_paths": {
-                "type": "list",
-                "coerce": "list",
-                "schema": {
-                    "type": "path",
-                    "coerce": "absolute_path",
-                },
-            },
             "archbuild_flags": {
                 "type": "list",
                 "coerce": "list",
@@ -185,14 +177,6 @@
                     "empty": False,
                 },
             },
-            "blacklisted_scan_paths": {
-                "type": "list",
-                "coerce": "list",
-                "schema": {
-                    "type": "path",
-                    "coerce": "absolute_path",
-                },
-            },
             "build_command": {
                 "type": "string",
                 "required": True,
@@ -226,6 +210,14 @@
                     "empty": False,
                 },
             },
+            "scan_paths": {
+                "type": "list",
+                "coerce": "list",
+                "schema": {
+                    "type": "string",
+                    "empty": False,
+                },
+            },
             "triggers": {
                 "type": "list",
                 "coerce": "list",

src/ahriman/core/repository/executor.py

@@ -23,13 +23,13 @@
 from pathlib import Path
 from tempfile import TemporaryDirectory
 
+from ahriman.core.build_tools.package_archive import PackageArchive
 from ahriman.core.build_tools.task import Task
 from ahriman.core.repository.cleaner import Cleaner
 from ahriman.core.repository.package_info import PackageInfo
 from ahriman.core.utils import safe_filename
 from ahriman.models.changes import Changes
 from ahriman.models.package import Package
-from ahriman.models.package_archive import PackageArchive
 from ahriman.models.package_description import PackageDescription
 from ahriman.models.packagers import Packagers
 from ahriman.models.result import Result

src/ahriman/core/repository/repository_properties.py

@@ -80,10 +80,7 @@ def __init__(self, repository_id: RepositoryId, configuration: Configuration, da
         self.reporter = Client.load(repository_id, configuration, database, report=report)
         self.triggers = TriggerLoader.load(repository_id, configuration)
 
-        self.scan_paths = ScanPaths(
-            allowed_paths=configuration.getpathlist("build", "allowed_scan_paths", fallback=[]),
-            blacklisted_paths=configuration.getpathlist("build", "blacklisted_scan_paths", fallback=[]),
-        )
+        self.scan_paths = ScanPaths(configuration.getlist("build", "scan_paths", fallback=[]))
 
     @property
     def architecture(self) -> str:

src/ahriman/models/scan_paths.py

@@ -17,29 +17,33 @@
 # You should have received a copy of the GNU General Public License
 # along with this program. If not, see <http://www.gnu.org/licenses/>.
 #
+import re
+
 from dataclasses import dataclass
+from functools import cached_property
 from pathlib import Path
 
 
-@dataclass(frozen=True, kw_only=True)
+@dataclass(frozen=True)
 class ScanPaths:
     """
     paths used for scan filesystem
 
     Attributes:
-        allowed_paths(list[Path]): list of whitelisted paths
-        blacklisted_paths(list[Path]): list of paths to be skipped from scan
+        paths(list[str]): list of regular expressions to be used to match paths
     """
 
-    allowed_paths: list[Path]
-    blacklisted_paths: list[Path]
+    paths: list[str]
 
-    def __post_init__(self) -> None:
+    @cached_property
+    def patterns(self) -> list[re.Pattern[str]]:
         """
-        compute relative to / paths
+        compiled regular expressions
+
+        Returns:
+            list[re.Pattern]: a list of compiled regular expressions
         """
-        object.__setattr__(self, "allowed_paths", [path.relative_to("/") for path in self.allowed_paths])
-        object.__setattr__(self, "blacklisted_paths", [path.relative_to("/") for path in self.blacklisted_paths])
+        return [re.compile(path) for path in self.paths]
 
     def is_allowed(self, path: Path) -> bool:
         """
@@ -49,10 +53,7 @@ def is_allowed(self, path: Path) -> bool:
             path(Path): path to be checked
 
         Returns:
-            bool: ``True`` in case if :attr:`allowed_paths` contains element which is parent for the path and
-            :attr:`blacklisted_paths` doesn't and ``False`` otherwise
+            bool: ``True`` in case if :attr:`paths` contains at least one element to which the path is matched
+            and ``False`` otherwise
         """
-        if any(path.is_relative_to(blacklisted) for blacklisted in self.blacklisted_paths):
-            return False  # path is blacklisted
-        # check if we actually have to check this path
-        return any(path.is_relative_to(allowed) for allowed in self.allowed_paths)
+        return any(pattern.match(str(path)) for pattern in self.patterns)

tests/ahriman/conftest.py

@@ -25,6 +25,7 @@
 from ahriman.models.repository_id import RepositoryId
 from ahriman.models.repository_paths import RepositoryPaths
 from ahriman.models.result import Result
+from ahriman.models.scan_paths import ScanPaths
 from ahriman.models.user import User
 from ahriman.models.user_access import UserAccess
 
@@ -587,6 +588,20 @@ def result(package_ahriman: Package) -> Result:
     return result
 
 
+@pytest.fixture
+def scan_paths(configuration: Configuration) -> ScanPaths:
+    """
+    scan paths fixture
+
+    Args:
+        configuration(Configuration): configuration test instance
+
+    Returns:
+        ScanPaths: scan paths test instance
+    """
+    return ScanPaths(configuration.getlist("build", "scan_paths", fallback=[]))
+
+
 @pytest.fixture
 def spawner(configuration: Configuration) -> Spawn:
     """

tests/ahriman/core/build_tools/conftest.py

@@ -1,6 +1,35 @@
 import pytest
 
+from pytest_mock import MockerFixture
+from typing import Any
+
+from ahriman.core.alpm.pacman import Pacman
+from ahriman.core.build_tools.package_archive import PackageArchive
 from ahriman.core.build_tools.sources import Sources
+from ahriman.models.package import Package
+from ahriman.models.repository_paths import RepositoryPaths
+from ahriman.models.scan_paths import ScanPaths
+
+
+@pytest.fixture
+def package_archive_ahriman(package_ahriman: Package, repository_paths: RepositoryPaths, pacman: Pacman,
+                            scan_paths: ScanPaths, passwd: Any, mocker: MockerFixture) -> PackageArchive:
+    """
+    package archive fixture
+
+    Args:
+        package_ahriman(Package): package test instance
+        repository_paths(RepositoryPaths): repository paths test instance
+        pacman(Pacman): pacman test instance
+        scan_paths(ScanPaths): scan paths test instance
+        passwd(Any): passwd structure test instance
+        mocker(MockerFixture): mocker object
+
+    Returns:
+        PackageArchive: package archive test instance
+    """
+    mocker.patch("ahriman.models.repository_paths.getpwuid", return_value=passwd)
+    return PackageArchive(repository_paths.build_directory, package_ahriman, pacman, scan_paths)
 
 
 @pytest.fixture