feat: support recursive stubgen

[cemlyn007]

To support producing stubs for submodules.

#43

[nicholasjng]

Thanks for taking this up! Here are a few comments.

If I may ask: In which setup do you use recursive stubgen? I'm wondering if for multiple extensions, you need to make all extension labels available to the stubgen binary as data, cf. L198 in the diff.

[cemlyn007]

Thanks for taking this up! Here are a few comments.

If I may ask: In which setup do you use recursive stubgen? I'm wondering if for multiple extensions, you need to make all extension labels available to the stubgen binary as data, cf. L198 in the diff.

I started a project based off the Nanobind examples bazel branch, but my code was getting a bit chunky so I was looking to modularise the code but I still only wanted a single Nanobind extension. I saw in the Nanobind documentation they have support for sub-modules.

I was hoping ultimately have source code looking something like:

src/packageA/moduleA_0/<some_python_files>
src/packageA/ext_moduleA_0/ext_moduleA_0.cpp <- The Nanobind extension
src/packageA/ext_moduleA_0/sub_module_A_0_a/<some_cpp_files_included_by_ext_moduleA_0.cpp>
src/BUILD.bazel

And produce a wheel that looks something like:

packageA/moduleA_0/<some_python_files>
packageA/moduleA_0/ext_moduleA_0.so
packageA/moduleA_0/__init__.pyi
packageA/moduleA_0/sub_module_A_0_a/__init__.py

But my MR is not entirely correct with naming the stubs correctly, at the moment packageA/moduleA_0/__init__.pyi looks like packageA/moduleA_0/_main.src.__init__.pyi (I think but AWOL from my home desktop), and they're not being picked into the wheel, although that might be resolved when I try your comment with using the -O flag in the stubgen script 🤷🏻‍♂️

[cemlyn007]

I had another go, I also have modified the Bazel Nanobind example which can be found here for a minimal example. Unfortunately the root stub is named like _main.src.pyi which led me to do some awful hackery, would appreciate any advise on how to better tackle it!

If you use this branch with the this example branch, you should get a working (brittle) example.

[nicholasjng]

Hi @cemlyn007, how is the situation?
Since a new nanobind release was just made, I'm going to need to put one out as well on the BCR. I'd still love to see this go through. Could you address the remaining comments? Then I'd take care of your PR in nanobind_example.

(I can also take over from here if you like.)

[cemlyn007]

Hey @nicholasjng, sorry for the delay, I've made some changes based on your comments, I do still feel like the top-level stub renaming is hacky but I'm not sure how we can do it in a better way.

FYI for testing I was pip installing and inspecting the file structure in site packages, but this wasn't great because for reasons unknown to me that file structure looked dirty (had old files being seen in). I thought a simple bazel clean --expunge would work but at least for me it didn't.

I think for testing this PR, one working way is to use the python build module (pip install build), but I also needed to create MANIFEST.in and paste the following to get python -m build to run successfully:

include MODULE.bazel
include MODULE.bazel.lock
include .bazelrc
recursive-include src *

If you have a better strategy for testing I am all ears, also let me know if you would like any further changes

[nicholasjng]

Hey! I merged your additions in the nanobind_example. Could you rebase on current master, so that the CI changes are reflected here?

[nicholasjng]

Thanks! The main outstanding point (i.e. path fixups after stub generation) likely has to be fixed in upstream nanobind, so I'll go ahead and merge this.

Thank you for your persistence and for the contribution!