Add Python 3.12 and Python 3.13 as probe-able versions

[huonw]

This (theoretically) adds support for running future Pants PEXes that requires Python 3.12 or 3.13 by adding them as versions that the fallback "probing" loop will query.

This will hopefully reduce how often we need to force users to upgrade their scie-pants runner, because current versions of scie-pants will support more Python versions, giving us a bit more flexibility with doing internal upgrades.

(Worst case: there's a problem in this PR and it doesn't work for the actual future 3.12/3.13-using versions of Pants... but, that's seemingly no worse than we were before, without any support at all.)

Background

Scie-pants downloads and runs PEXes from Pants releases. These PEXes require a specific Python version, that's indicated in the URL, e.g. https://github.com/pantsbuild/pants/releases/download/release_2.25.0.dev4/pants.2.25.0.dev4-cp311-darwin_arm64.pex (NB. cp311). There's two aspects of scie-pants working with these:

• hard-coding the different Python versions, to have lift/ptex be able to download the right interpreter (in scie-pants.toml)
• logic to find the right artifact to download:

  scie-pants/tools/src/scie_pants/pants_version.py

  Lines 239 to 253 in 9d73b15

  	ptex_urls = get_bootstrap_urls(bootstrap_urls_path)
  	py = get_python_id_for_pants_version(version)
  	error: str | None = None
  	if py:
  	pex_url, error = get_download_url(version, platform, py, ptex_urls)
  	if pex_url:
  	return pex_url, py
  	# Else, try all known Pythons...
  	for maybe_py in PYTHON_IDS.keys():
  	pex_url, err = get_download_url(version, platform, maybe_py, ptex_urls)
  	if pex_url:
  	return pex_url, maybe_py
  	elif not error:
  	error = err

  • first guess the URL based on the requested version, with hard-coded known cut-overs:

    scie-pants/tools/src/scie_pants/pants_version.py

    Lines 27 to 33 in 9d73b15

    	PANTS_PYTHON_VERSIONS = [
    	# Sorted on pants version in descending order. Add a new entry when the python version for a
    	# particular pants version changes.
    	{"pants": "2.25.0.dev0", "python": "cp311"},
    	{"pants": "2.5.0.dev0", "python": "cp39"},
    	{"pants": "2.0.0.dev0", "python": "cp38"},
    	]

  • if that fails, then fallback to guessing successive URLs based on the known-versions (i.e. several requests to probe the possible request artifacts)

The fallback feature allows us to have current scie-pants support potential future versions of Pants, to reduce how often users are forced to upgrade their scie-pants runner installation.

Example

Imagine we release 2.34.0 with cp312 support and had made no other changes, this gives an artifact like pants.2.34.0-cp312-darwin_arm64.pex:

1. Scie-pants' initial guess tries to download pants.2.34.0-cp311-darwin_arm64.pex (because 2.34.0 >= 2.25.0.dev0, which is the hard-coded cp311 threshold in PANTS_PYTHON_VERSIONS), but this would fail.
2. The fallback kicks in to go through the various PYTHON_IDS. It eventually hits pants.2.34.0-cp312-darwin_arm64.pex and succeeds, getting Pants 2.34 running for that user!

Without this PR, step 2 would only run through 3.8 - 3.11, not find the artifact, and thus the overall installation would fail.

(Hard-coding the version threshold once we know it will make this bootstrapping (slightly) more efficient/send fewer requests, but the fallback + extra versions means scie-pants is at least functional.)

[benjyw]

This is good as far as it goes, but I wonder if we should be creating pants scies instead of pexes (via pex tool's ability to build scies) and modify scie-pants to pick the right scie based on the platform and the requested Pants version?

[huonw]

Yeah, I'm thinking this PR is an incremental win that gives us more time/more flexibility immediately, and then a bigger project like that can happen in parallel/eventually replace this.

(I imagine the build-side of publishing scies likely isn't so much bigger with the recent-ish https://docs.pex-tool.org/scie.html features)

[huonw]

It'd be good to get this merged before 2.24 goes out, since 2.24 will encourage people to update their runners on "long-lived" machines. If we've released this PR before there, the users updating in response to 2.24 will likely grab this version, and thus be using a 3.12/3.13-supporting scie-pants runner, reducing the total 'pain' of any future python version upgrades.

(The use-a-scie approach would be even better, obviously, but that's far more effort that this.)

What do you think @benjyw and @cburroughs ? Thanks

[benjyw]

It'd be good to get this merged before 2.24 goes out, since 2.24 will encourage people to update their runners on "long-lived" machines. If we've released this PR before there, the users updating in response to 2.24 will likely grab this version, and thus be using a 3.12/3.13-supporting scie-pants runner, reducing the total 'pain' of any future python version upgrades.

(The use-a-scie approach would be even better, obviously, but that's far more effort that this.)

What do you think @benjyw and @cburroughs ? Thanks

Oh yes definitely. Sorry, thought I had approved this.

[cburroughs]

It'd be good to get this merged before 2.24 goes out

To get this merged, or to get this merged and bump the mini scie-pants version in the 2.24.x branch?

[huonw]

I wasn't going to bother with bumping that version: when someone installs 2.24 and sees the warnings, I imagine they'll usually do brew upgrade ... or SCIE_BOOT=update scie-pants (i.e. default settings) and thus get the latest version 0.12.3, not the minimal-valid-version 0.12.2.

For those who've already installed 0.12.2, I think it's fine to prompt an upgrade when actually required.