It would be cool to compile lisp core file into .so

[svetlyak40wt]

This will make it easier to distribute lisp libraries.

Here I found an article showing how to link a binary file into the library and access it's data:
https://balau82.wordpress.com/2012/02/19/linking-a-binary-blob-with-gcc/

[svetlyak40wt]

Guys, if you think it is a good idea, I'm ready to experiment with that.

[stylewarning]

It's definitely of interest. We definitely don't want a separate core file (by default) in the long term. We'd need to ensure it works for Win, Mac, and Linux.

[karlosz]

This is a good idea. We've thought about the issue a bit, and I'll dump some thoughts here:

It would be good to not force a specific distribution model for users. Rather, we should provide/document different ways to do distribution, which will fit downstream users use-cases as needed. (Each have their associated trade-offs). These different models include:

1. Bundling everything, including the lisp core, libsbcl.so, and all generated binding code into one .so/.dylib,.dll or static .a file. Good for people who want to ship one monolithic artifact, not dealing with paths, etc.
2. Bundling libsbcl.so with generated binding code. This has the advantage of allowing developers to dump new cores and load them in during development with very little cost without needing to recompile or relink on the C side.
3. Bundling the lisp core with generated binding code. This has the advantage of not duplicating the runtime provided by libsbcl.so, making it a true shared library.
4. Keeping all artifacts separate (perhaps just glomming together any binding code). This combines the advantages of 2. and 3.

Currently, we don't know of any good way to provide the ability to do 1, i.e. embedding the core into a shared library file. It is easy for users to currently choose between distribution models 2. 3. and 4. by playing with the linker, potentially after building libsbcl.a in the sbcl runtime directory, depending on the operating system and toolchain.

The main issue with embedding the core into a shared library is that even given an operating system or toolchain agnostic way to convert the core into a shared library (one portable method being by dumping the core into an array textually and compiling that into static data), we'd probably need to dump that back out into disk on-load somehow before initialization since initialize_lisp takes a pathname, rather than a pointer or file descriptor even, or use unportable /proc hackery. I don't know, but this would be something good to explore further. We might be able to leverage the file system somehow to accomplish.

The other thing is that currently, users explicitly initialize the Lisp runtime with a core file pathname, again because of the nature of initialize_lisp. Figuring out how to deal with the explicit initialization aspect with an embedded core is another decision we will want to give the user some control over. There are clearly some use cases in which users want fine grained control over when the Lisp runtime is initialized, e.g. when sharing process resources such as file descriptors (think *standard-input*, *standard-output*) or signals.

[braised-babbage]

I think that is a nice summary, @karlosz

Regarding the specific case of (1), i.e. stuffing a core into a .so, .dll, or .dylib, it does not feel so different from the existing mechanism for save-lisp-and-die with :executable t. We would of course need to arrange fo initialize_lisp to be called after the shared library is loaded, but but then as in https://github.com/sbcl/sbcl/blob/9e1c52f5d2ca2d4cccb0f2534d442201227e43b3/src/runtime/runtime.c#L645 we would have

• os_get_runtime_executable_path() identifies the current .so, .dylib, .dll via OS-specific system calls
• search_for_embedded_core(sbcl_runtime, &memsize_options) recovers the offset to the core file
  and then (I think) the rest should work as usual.

At present, search_for_embedded_core just looks for some magic words at the end of the file (cf. https://github.com/sbcl/sbcl/blob/eef10d2e21243c273a1882069b9c4c187c0194ed/src/runtime/coreparse.c#L99). Linking in a binary blob would give you an extra entry or section in the ELF or PE file, which is not what we want. Instead, we want to safely concatenated the .so,.dylib,.dll with the .core + metadata. I don't understand ELF or PE formats well enough to know what's allowed here, but https://www.strchr.com/creating_self-extracting_executables is an interesting link that popped up when googling.