Javascript C++ binding and Lomse future

[cecilios]

Hi all!

I'm going to start using the GitHub Discussions feature by opening this brainstorming thread with my current ideas about lomse future. Please feel free to participate in this brainstorming!

I would like to simplify lomse use, as well as to enable lomse to be used in Android and web applications.

Lomse API is complex because it forces to use a very specific architecture for your application, based on bitmaps and in very specific ways to interact with the score. Also lomse provides, perhaps, too much functionality that should be on the user application.

As modern html and web applications require svg and a different approach to playback and interaction (add/modify/delete music elements), I'm working in the creation of a JavaScript binding with a very simple interface.

For creating the JavaScript binding I'm testing Emscripten (Embind).

To reduce maintenance costs and the JavaScript module size, probably it would be good to split Lomse in two libraries:

1. The basic library, with the internal model, the MusicXML importer and SVG render.
2. A second library with all code related to bitmap rendition, playback, mouse handling, etc. as all this code is oriented to traditional applications.

Or, perhaps, better: just remove all the code that it is not required in the basic library.

I'm starting to work on all these ideas, and this is my tentative roadmap:

• Create an script to build from sources not only Lomse but also its dependencies (zlib, libpng, freetype, etc.). This is required for cross-compilation and for using Emscripten.
• Freeze Lomse repo, clone it and rename it (e.g. lomsejs, lomse.js, other?).
• Start developing the JavaScript binding.
• Test it with a simple web application: render svg, playback, interaction with the score, etc.
• At this point, based on gained knowledge during the previous tests, feedback and other ideas, take decisions about splitting Lomse and/or modifications and removals in Lomse code.

Feedback, opinions, ideas and collaboration are greatly welcome!

[dmitrio95]

Actually Lomse can already be used for Android and iOS applications, although it is true that JavaScript bindings would probably simplify Lomse usage for some apps, especially those using React Native or other JavaScript-based mobile frameworks. Also a possibility to use Lomse in a web context would indeed be a great feature for the library.

I don't fully understand the idea of moving to the SVG rendering though. From what I see web applications don't really require SVG. An application can use something like the Canvas API or WebGL for rendering without a need to use SVG. It may perhaps be convenient to use SVG for some applications, but forcefully moving everything to SVG sounds like a large performance loss to me: now the application would have to generate an SVG document, parse this document back and draw the parsed content instead of just drawing the necessary content directly. From my point of view it would be good to have several options for rendering, and the current scheme of having built-in bitmap and SVG renderers and allowing a user application to define a custom renderer if necessary is already good, at least for the C++ interface.

Also I agree that Lomse sometimes tries to solve problems that are better to be handled on the user application side, but to me the issue is not in the amount of different functionality but in forcing a specific way to use that functionality. For example, it is definitely good to have the playback functionality, but a thread management logic for playback is probably something that may be left to a user application to handle in a way that would fit the application's needs. It is good to have a basic functionality for mouse handling (like locating staves or score objects) but it might be not necessary to implement a mouse events handling system in Lomse. If some of these parts can be easily split off the core library it might make sense to do so (but it would be good to consider other use cases too). But I am not sure that splitting off entire parts of functionality would simplify things for developing and using the library.

Other than that, the idea of creating JavaScript bindings seems indeed an interesting direction for improving the project!

[cecilios]

Many thanks Dmitri. Your feedback is of great help to me.

Lomse was developed in parallel with my application, and I just moved to Lomse all the functionality that I needed for my application and that I though could be general enough to be useful for other Lomse users. But there are very few Lomse users and thus, I never receive direct feedback about the real needs of others and the problems to use Lomse.

I'm considering to rewrite my application. I did some tests using Electron with JavaScript and a local REST server for Lomse to read the MusicXML files and serve the bitmaps or the SVG code. And the idea of using JavaScript is gaining strength. That is the reason for JavaScript bindings.

The next step is to decide the JavaScript API and this is an opportunity to also review the Lomse API. I have the feeling that it is costly for tentative users to quickly build a simple test to play with Lomse and that Lomse learning curve is steep. Also, for simple use cases, Lomse could be complex to use. I hope that these issues can be avoided with the simple SVG rendition approach. But it is true that the bitmap approach is very useful and efficient.

I need to identify library parts that are not useful and can be safely removed. And the limitations that should be corrected. For instance, rendering full documents containing scores is not needed and the manageable documents are too simple and barely support styles. The same is applicable to controls (buttons, links, etc.). Lomse should only deal with scores and texts embedded in the score.

As to the code you mention:

• Mouse handling to identify the clicked staff or score object is not needed as it is enough to pass Lomse the point coordinates. But mouse handling is the simplest and most efficient way of drawing selection rectangles as the mouse is moved. So, I would keep mouse handling but the API should be reviewed for ensuring that it is not mandatory to use Lomse mouse handling and that the user application can do it as it prefers.

• It is true that playback managed in a thread and serving MIDI events could be not appropriate for some applications and that other alternatives should be available. Playback for web oriented applications should be done in a different way, perhaps providing a midi file with information for syncing. a cursor on the score, and for this probably SVG rendition is better than bitmap rendition. This is an area to explore and ideas and needs will be of great help.

As to splitting the library, I have the feeling that it is costly to do it and it will be very difficult to define clear boundaries. The only gain is to reduce memory size when this is an issue and only core functionality is needed. But it is difficult to define what is this core functionality! The best approach would be to split the library in several small modules that the user can combine. But in any case, this is probably very difficult and there are better things to do.

So, some tentative conclusions:

• Create JavaScript bindings
• Do not remove bitmap rendition and current features.
• Do not split the library.
• I will work on a playback solution for web oriented applications and this could seed light on improvements for playback support.

[dmitrio95]

For playback I just directly use the existing SoundEventsTable, and this seems to be enough for my application. It is just not something that I was probably supposed to do, according to the library documentation. For me it is very good that Lomse allows accessing its internal structures as I anyway often need something that would be difficult to predict and include into any stable public interface. But for applications with smaller requirements, or to make it easier to start using Lomse, a good simple public interface for most common things would indeed be preferable.

I agree that it would be difficult to clearly define what should be the core functionality and which functionality is less useful for this project. Maybe it would be better to separately discuss a potential usage of parts of functionality that may be considered for removing.

[cecilios]

Really interesting! I never though that someone would like to do playback using the SoundEventsTable and coding a specific playback method and the associated visual highlight. But this approach gives, certainly, a lot of freedom to user applications. Definitely, I have to think on this and document this in the API. I'm not going to work on this now, but I certainly will consider any ideas, based on your experience and needs, to improve the SoundEventsTable and API methods for direct usage in user applications.

No need to discuss this now and better not in this thread. Feel free to open an issue or a discussion when you have time or any needs.

And I totally agree on the difficulty of defining the core functionality and the code, if any, that should be removed. So, as you suggest, it is better to move this to a new future discussion. But in any case, if anyone have ideas at any time, feel free to open the discussion.

I'm going to focus now on the JavaScript bindings and the tests with a web application. This could help to clarify my ideas.

So, the roadmap is now:

• Create an script to build from sources not only Lomse but also its dependencies. This is required for cross-compilation and for using Emscripten.
• Start developing a minimal JavaScript binding.
• Tests with a simple web application: render svg or bitmaps, playback, interaction with the score, etc. This could seed light on improvements for playback support and other needs.

I will share my findings and ideas in this thread.