feat: provide access to arbitrary interpreters #2507

philsc · 2024-12-16T03:09:57Z

There are some use cases that folks want to cover here. They are
discussed in this Slack thread. The high-level summary is:

Users want to run the exact same interpreter that Bazel is running
to minimize environmental issues.
It is useful to pass a target label to third-party tools like mypy
so that they can use the correct interpreter.

This patch adds to @rickeylev's work from #2359 by adding docs
and a few integration tests.

Run a specific interpreter: * `bazel run @rules_python//tools/run --@rules_python//python/config_settings:python_version=3.12` Run interpreter from a binary: * `bazel run @rules_python//tools/run --@rules_python//tools/run:bin=//my:binary`

$ bazel run //python/bin:repl $ bazel run //python/bin:repl --//python/bin:repl_dep=//python/runfiles

docs/toolchains.md

aignas

I know this is still a draft, so take my suggestions with a grain of salt :)

docs/toolchains.md

python/bin/interpreter.bzl

docs/toolchains.md

aignas · 2024-12-16T03:21:37Z

tests/integration/interpreter_test.py

+        self._run_module_test("3.11")
+
+    def test_run_module_3_12(self):
+        self._run_module_test("3.12")


It would be great to add a test that imports:

The code that is in the //python/bin:interpreter_src

The code that is in the transitive deps of the //python/bin:interpreter_src. Maybe depending on twine could be an option in the tests and then we could attempt importing one of its deps?

Unless I'm misunderstanding something, that's currently not possible with the current implementation. The current implementation is purely there to get the interpreter, nothing else.

I'd like to follow this PR up with another PR that enables the use of the REPL with a specific py_* target (and its dependencies) importable.

docs/toolchains.md

tests/integration/BUILD.bazel

I noticed that my `$(location //path/to:target)` wasn't getting expanded when writing a test. This patch fixes the issue by forwarding the already-expanded environment from the inner target to the outer target.

tests/support/sh_py_run_test.bzl

rickeylev

re: the change the the env attribute in sh_py_run_test: Please partially undo this change.

The key thing is that I don't want the inner py binary to contaminate the outer sh binary's environment. One of the purpose's of the sh-side of sh_py_run_test is to run a py_binary with a particular environment.

So I think it's fine if sh_py_run_test.env is passed to only the inner py_binary and not the outer sh_binary. Similarly, the outer sh_binary shouldn't inherit TestEnvironment from the inner py binary. If there are separate env vars that should be set only in the sh-side of things, then either (a) create a sh_env arg to pass to sh, or (b) create a .sh file that sets the desired env prior to invoking the inner py binary.

Almost LGTM. Just that and the visibility comment.

rickeylev · 2025-01-27T16:40:24Z

docs/toolchains.md

+
+To run the interpreter that Bazel will use, you can use the
+`@rules_python//python/bin:python` target. This is a binary target with
+the executable pointing at the `python3` binary plus the relevent runfiles.


Suggested change

the executable pointing at the `python3` binary plus the relevent runfiles.

the executable pointing at the `python3` binary plus its relevant runfiles.

rickeylev · 2025-01-27T16:42:06Z

docs/toolchains.md

+
+:::{note}
+The `python` target does not provide access to any modules from `py_*`
+targets on its own. Work is ongoing to support that.


Suggested change

targets on its own. Work is ongoing to support that.

targets on its own. Please file a feature request if this is desired.

Just encouraging users to give feedback

rickeylev · 2025-01-27T16:43:12Z

python/bin/BUILD.bazel

@@ -0,0 +1,21 @@
+load(":interpreter.bzl", "interpreter")
+
+package(default_visibility = ["//visibility:public"])


Suggested change

package(default_visibility = ["//visibility:public"])

Leave visibility as private (default). Explicitly add visbility=public where necessary. This just helps make it clear about what is actually public.

rickeylev · 2025-01-27T16:44:30Z

python/bin/BUILD.bazel

+filegroup(
+    name = "distribution",
+    srcs = glob(["**"]),
+    visibility = ["//python:__pkg__"],


Suggested change

visibility = ["//python:__pkg__"],

visibility = ["//:__subpackages__"],

For simplicity, we generally just grant visibility to our whole repo or public. It just makes it easier to reuse other parts within our codebase without having to go edit visibility stuff frequently.

I think only distribution was following this pedantic visibility setting.

rickeylev · 2025-01-27T16:44:49Z

python/bin/BUILD.bazel

+
+interpreter(
+    name = "python",
+    binary = ":python_src",


Suggested change

binary = ":python_src",

binary = ":python_src",

visibility = ["//visibility:public"],

rickeylev · 2025-01-27T16:46:30Z

python/bin/interpreter.bzl

+
+load("//python/private:interpreter.bzl", _interpeter = "interpreter")
+
+interpreter = _interpeter


Hm. Should this be public?

What use case would users have for defining their own interpreter target?

rickeylev · 2025-01-27T16:46:59Z

python/bin/interpreter.bzl

+
+load("//python/private:interpreter.bzl", _interpeter = "interpreter")
+
+interpreter = _interpeter


Please name this interpreter_binary. This is to match the Bazel convention that runnable things are suffixed with _binary.

rickeylev · 2025-01-27T16:50:23Z

python/private/interpreter.bzl

+        ),
+    ]
+
+interpreter = rule(


If this is public, then we'll need to add docs to the rule impl. Eventually, anyways -- I'm fine with docs being done in a separate PR.

rickeylev · 2025-01-27T16:54:10Z

tests/interpreter/interpreter_test.py

+
+
+class InterpreterTest(unittest.TestCase):
+    def setUp(self):


Suggested change

def setUp(self):

def setUp(self):

super().setUp()

Best practice: call the superclass unless there's a specific reason the parent functionality shouldn't be invoked.

tests/support/sh_py_run_test.bzl

rickeylev · 2025-01-29T06:19:01Z

Doh, actually, I was mistaken. The RunEnvironmentInfo propagation is fine.

I misread the diff and thought the ctx.attr.target[RunEnvironmentInfo], line was in the wiring between the sh_test and the py target -- it's not. It's between the py_reconfig rule and the inner py target. Propagating that makes sense.

philsc · 2025-01-31T17:01:31Z

Doh, actually, I was mistaken. The RunEnvironmentInfo propagation is fine.

I misread the diff and thought the ctx.attr.target[RunEnvironmentInfo], line was in the wiring between the sh_test and the py target -- it's not. It's between the py_reconfig rule and the inner py target. Propagating that makes sense.

Whew. Okay. I was not sure how I was going to accomplish that otherwise 🙂 Will clean up the rest this weekend (and also figure out how to fix the RBE failures).

rickeylev and others added 10 commits October 29, 2024 21:05

Basic REPL support

7507626

$ bazel run //python/bin:repl $ bazel run //python/bin:repl --//python/bin:repl_dep=//python/runfiles

experiment with site packages

9730855

Merge remote-tracking branch 'upstream/main' into HEAD

2c3fc1b

Merge remote-tracking branch 'rickeylev/feat.tools.run' into HEAD

f819bb0

consolidate files a bit

286b4f1

revert REPL-related things

314447d

Merge remote-tracking branch 'upstream/main' into HEAD

3fb3236

add integration test

e07a528

add some docs

91d66df

philsc commented Dec 16, 2024

View reviewed changes

docs/toolchains.md Outdated Show resolved Hide resolved

aignas reviewed Dec 16, 2024

View reviewed changes

rickeylev requested changes Dec 27, 2024

View reviewed changes

docs/toolchains.md Outdated Show resolved Hide resolved

tests/integration/BUILD.bazel Outdated Show resolved Hide resolved

philsc added 17 commits January 5, 2025 19:31

rename to :python and incorporate readme feedback

c71629f

move implementation to //python/private

2ee2311

Add a test using transitions instead of an integration test

620bf31

Add python_src tests

d4b7978

Merge remote-tracking branch 'upstream/main' into HEAD

c2f927f

black

7ede58a

delete integration test

8cd6de0

delete unused files

8c2495b

delete unused files

d7bce25

delete unused files

d0c8304

revert unused changes

63924fe

pre-commit

94d5ee3

Fix Windows hopefully

8e935f4

fixup

d39a0a1

fix: Enable location expansion for sh_py_run_test

abcc5e9

I noticed that my `$(location //path/to:target)` wasn't getting expanded when writing a test. This patch fixes the issue by forwarding the already-expanded environment from the inner target to the outer target.

Merge commit 'abcc5e9d' into HEAD

0c365e9

buildifier fixes hopefully

81b26c6

philsc commented Jan 26, 2025

View reviewed changes

tests/support/sh_py_run_test.bzl Show resolved Hide resolved

delete superfluous runfiles

cfd96c8

rickeylev marked this pull request as ready for review January 27, 2025 16:41

rickeylev requested a review from groodt as a code owner January 27, 2025 16:41

rickeylev requested changes Jan 27, 2025

View reviewed changes

rickeylev mentioned this pull request Jan 29, 2025

tests: don't propagage inner py_binary env to outer sh_test #2591

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: provide access to arbitrary interpreters #2507

feat: provide access to arbitrary interpreters #2507

philsc commented Dec 16, 2024

aignas left a comment

aignas Dec 16, 2024

philsc Jan 6, 2025

rickeylev left a comment

rickeylev Jan 27, 2025

rickeylev Jan 27, 2025

rickeylev Jan 27, 2025

rickeylev Jan 27, 2025

aignas Jan 28, 2025

rickeylev Jan 27, 2025

rickeylev Jan 27, 2025

rickeylev Jan 27, 2025

rickeylev Jan 27, 2025

rickeylev Jan 27, 2025

rickeylev commented Jan 29, 2025

philsc commented Jan 31, 2025

	the executable pointing at the `python3` binary plus the relevent runfiles.
	the executable pointing at the `python3` binary plus its relevant runfiles.

	targets on its own. Work is ongoing to support that.
	targets on its own. Please file a feature request if this is desired.

		@@ -0,0 +1,21 @@
		load(":interpreter.bzl", "interpreter")

		package(default_visibility = ["//visibility:public"])

	visibility = ["//python:__pkg__"],
	visibility = ["//:__subpackages__"],

	binary = ":python_src",
	binary = ":python_src",
	visibility = ["//visibility:public"],


		load("//python/private:interpreter.bzl", _interpeter = "interpreter")

		interpreter = _interpeter

feat: provide access to arbitrary interpreters #2507

Are you sure you want to change the base?

feat: provide access to arbitrary interpreters #2507

Conversation

philsc commented Dec 16, 2024

aignas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rickeylev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rickeylev commented Jan 29, 2025

philsc commented Jan 31, 2025