Partial View for rendering systems individually

[npiegdon]

This is conceptually similar to the Free Flow View, but would allow drawing one system at a time. (I also couldn't think of a very good name for it.) 😅

Motivation: When using hardware accelerated drawing (3D APIs like DirectX, OpenGL, etc.), it is best to upload a rarely-changing texture only once and then use it repeatedly. (This assumes the app will be controlling which part of that texture is drawn each frame, effectively managing the viewport itself.)

If that were the only limitation, Free Flow View would be the right answer: render the whole score for a given screen width, upload it to the GPU, and draw portions as needed. But some mobile platforms still have the unfortunate combination of high DPI screens and rather limited GPU memory. So, a score with many pages, rendered at high DPI, all at once as a huge bitmap would immediately crash the app. Instead, more manual viewport management can be introduced to only render and upload the systems that are currently in the viewport. (An app could build arbitrarily complex caching systems on top of this basic single-system capability to do things like evict the least-recently-used systems as video memory pressure begins to ramp up, solving the problem.)

Because so much is being managed by the app manually, it will be important to be able to get at some information (horizontal begin/end pixels of each measure/beat for hardware-accelerated highlighting, etc.) that appears to normally be handled internally to lomse.

API: There are probably lots of ways this could work, but the API that feels most natural to me is something like the following.

int Paginate(width, DPI)

After creating the view, you could call something like Prepare or Paginate with width and DPI parameters. This function would get everything ready internally (clefs/key signatures at the beginning of each system, measure justification/spacing for the given width, slurs carried across line changes, etc.) the way Free Flow View does today. It returns the number of systems in the score. (My background is in less sophisticated sheet music rendering. Perhaps it isn't "number of systems" but rather "number of block elements in the document"?)

Paginate might be called again when the window size changes or the user requests a change to the dpi/zoom/size of the music, but it's a rare enough event that as much of the work should be done here up-front as possible.

After a Paginate, given an index (between 0 and the number returned by Paginate), at least this information should be available at this point (generally in units of pixels):

• The height of the system.
• The baseline (or some other anchor point) of each staff in the system.
• The beginning/end number of the measures in that system.
• Some way to retrieve a list of associated horizontal pixel coordinates & timepos for each measure and each beat in that measure (for manually controlling the highlight, etc.)

(Again, the presence of non-system block elements like paragraphs might make this a little messier. Using sentinel values like returning a measure range of zero to zero for the song title block could probably work. The measure/beat metrics would simply be an empty vector, etc.)

Using the system height, the set_rendering_buffer call can now be made with an appropriately-sized buffer. Assuming the height of the tallest system is similar to the average, you could get away with setting the buffer only once to the largest required for any system in the score.

Finally, redraw_bitmap would work as-usual but take a system index as a parameter. Only that system would be drawn at the top-left corner of the buffer (the way Free Flow View draws its first system).

As the music is scrolled or repeated or navigated by the user, many calls to redraw_bitmap(systemId) might be made (as systems are evicted from GPU memory and need to be redrawn) for any single call to Paginate. So ideally redraw_bitmap would need to do as little work as possible.

This relatively simple scheme allows for using lomse in a hardware accelerated environment while tackling the primary limitations. It does leave viewport management up to the app developer... but that's the part I already have written, so that works for me. 🤣

[cecilios]

I'm converting this issue into a discussion about the API, until we define some actionable issues.

If I understand your request correctly I think that no changes are needed: Lomse always renders only the viewport and the app has full control of the viewport; in fact the app has only two responsibilities defining the viewport and allocation the bitmap.

Lomse works as follows: It maintains an internal model (IM) with the description of the music (you could think of it as a MusicXML DOM). When the score must be displayed, lomse builds from the IM a graphical model (the GM): a collection of shape objects (GmoShape and GmoBoxobjects) representing the objects in the score (lines, symbols, etc.). From the point of view of the rendering process GmoShape objects behave as vertex sources, providing the coordinates for the polygons that define the shape.

Coordinates can be then converted by applying different transformations, such as scaling, rotation and shift. The converted coordinates creates a collection of polygons that define the final image to be generated. And once the graphical model is created, the rendering process can take place.

The graphical model is only built once (in method Interactor::create_graphic_model()), when a new document is loaded or it is modified. And the rendering process can be executed many times (normally when invoking Interactor::redraw_bitmap()). But rendition is always limited to the current viewport. Therefore, in an app in which the viewport is only half page, the other half page and all remaining pages will not be rendered.

The rendering bitmap must be just the viewport area, normally the portion displayed in the app window. Whenever the viewport is modified (by changing viewport origin, e.g. scrolling, or by changing the bitmap size, e.g. window resize,) the new viewport area is rendered on the bitmap (See https://lenmus.github.io/lomse/page-render-overview.html#page-render-overview-control )

Method set_rendering_buffer() must be invoked rarely, only at app start and whenever the app window is resized. If the app. window does not change, the bitmap is reused by lomse to re-render the new viewport.

Therefore, user application has only two tasks to do: allocate a new bitmap (and destroy the old one) whenever the app window is resized, and manage the viewport origin to display the desired portion of the score.

If you would like to control the viewport using system numbers instead of coordinates then it is just to code a few simple methods to determine coordinates from system number.

As the music is scrolled or repeated or navigated by the user, many calls to redraw_bitmap(systemId) might be made (as systems are evicted from GPU memory and need to be redrawn) for any single call to Paginate. So ideally redraw_bitmap would need to do as little work as possible.

Yes, this is current situation. You proposed Paginate() method is equivalent to building the GM. And redraw_bitmap(systemId)
is equivalent to redraw_bitmap(viewport origin) and does the minimal work to render that portion of the score onto the bitmap.

The only difference is that Lomse does not explicitly use hardware accelerated rendition. The GmoShapeobjects behave as vertex sources that create a collection of polygons that define the final image to be generated. For creating the image, an scanline rendering algorithm is applied, and the result of this process is a bitmap with the image. For this process the CPU is used. Perhaps Lomse could use the GPU but I have no idea of how to do it. Lomse does not use any platform rendition facilities, such as OpenGL or DirectX, but it includes its own rendering software (in fact AGG library, a very compact and fast collection of templated algorithms).

API

the API that feels most natural to me is something like the following.
int Paginate(width, DPI)
After creating the view, you could call something like Prepare or Paginate with width and DPI parameters. This function would get everything ready internally (clefs/key signatures at the beginning of each system, measure justification/spacing for the given width, slurs carried across line changes, etc.) the way Free Flow View does today.

The equivalent to the proposed Paginate() method is building the Graphical Model. This is automatically triggered by some Interactor methods when the document has been modified and the GM has not been rebuilt. Please note that trying to laying out the score in 'chunks' (pages, systems, or other slices) doesn't work well for complex scores as it is necessary to take into account issues such as that system height and number of staves can change from system to system, laying out some notation at system boundaries requires to layout current system and next system, page breaks and systems spacing depend on the total number of resulting pages, etc.

Paginate might be called again when the window size changes or the user requests a change to the dpi/zoom/size of the music, but it's a rare enough event that as much of the work should be done here up-front as possible.

Once the Graphical Model is built it must be rebuilt only if the document changes. Other changes, such as resolution, window size, allocation a new rendering bitmap, do not affect the GM.

After a Paginate, given an index (between 0 and the number returned by Paginate), at least this information should be available at this point (generally in units of pixels):

Once the Graphical Model is built all graphic information is available. But certainly some API methods to get information from the graphical model can be added. In general, user apps. do not need to interact with the GM or, at least, I never feel the need. But there are methods to get information like the one you suggest:

The height of the system.
The baseline (or some other anchor point) of each staff in the system.
The beginning/end number of the measures in that system.
Some way to retrieve a list of associated horizontal pixel coordinates & timepos for each measure and each beat in that measure (for manually controlling the highlight, etc.)

And in any case it should be simple to add any required method as it is just to access the GM objects and take the information.

Using the system height, the set_rendering_buffer call can now be made with an appropriately-sized buffer. Assuming the height of the tallest system is similar to the average, you could get away with setting the buffer only once to the largest required for any system in the score.

This is a misconception of how lomse works. As explained before, the rendering buffer should be fixed size, ideally the window buffer if the OS allows direct writes on it. Otherwise a bitmap of the same size than the window size. Once allocated, there is no need to re-allocate new buffers, only if the window size changes.

I'm noticing that although you referred to Free Flow View perhaps you are thinking on displaying only one system, the one to be played, and to scroll to a new system when playback is near the end of current system. If this is the case, perhaps Single System View would be more appropriate: the score is just one very long system and your application just scrolls right as playback advances (just shift right the viewport origin, perhaps using measure number instead of coordinates). Or a variation of it, Partial View, to be defined.

[cecilios]

A possibility I devise to integrate Lomse with OpenGL in case it would be necessary, will be to write a specific Drawerclass.

Drawer is the lomse abstract base class for any object responsible for transforming drawing commands into something understandable by the underlying rendering engine (e.g.: a bitmap, svg file, paths, etc.), to display the sheet music.

The Drawer class is equivalent to the typical “Device Context” (DC) that is part of the “Graphics Device Interface” (GDI) in the Microsoft Windows API.

As Lomse aims to be platform independent, it avoids using any platform specific graphics interface, such as OpenGL. Instead it was decided to create an abstract interface class: Drawer. And any object in the GM is drawn by invoking methods in a drawer class. Currently, there are two Drawer derived classes: BitmapDrawer that renders on a bitmap, and SvgDrawer that renders on a std:stream.

But as the interface of Drawer class mimics SVG commands, it would be very simple to write a OpenGLDrawer class issuing OpenGL commands.

[npiegdon]

If I understand your request correctly I think that no changes are needed...

This is great news!

The graphical model is only built once (in method Interactor::create_graphic_model()), when a new document is loaded or it is modified.

I'm guessing a DPI change counts as a modification? It's nice to be able to give users a way to adjust the zoom level of the music (while still filling/reflowing-to the complete width of the window/screen), but because it necessarily pushes measures around to different systems, it most-likely has to go through that process.

If you would like to control the viewport using system numbers instead of coordinates then it is just to code a few simple methods to determine coordinates from system number.

This would be perfect. I hadn't considered treating the single-system requirement as just another way to set the viewport. It's an elegant solution that exactly solves the problem and removes the need for a new kind of view.

The only difference is that Lomse does not explicitly use hardware accelerated rendition.

Sorry for any confusion; this was my expectation. The plain, platform-independent bitmap is all I need. The app is in charge of doing the hardware-accelerated part by using the resulting bitmap as a texture resource on the GPU.

Regarding hardware accelerated rendering of fonts/glyphs, it's an interesting subject. The Slug library does it better than anything else I've seen--their downloadable demo is truly impressive--but that type of thing is way outside the scope of the kind of change I was requesting!

Please note that trying to laying out the score in 'chunks' (pages, systems, or other slices) doesn't work well for complex scores...

Agreed. The Paginate() idea was also to lay out the whole score at once for those same reasons, so it sounds like the two ideas continue to be equivalent.

Once the Graphical Model is built it must be rebuilt only if the document changes. Other changes, such as resolution, window size, allocation a new rendering bitmap, do not affect the GM.

Interesting. Like I mentioned above, doesn't changing the resolution (without changing the window size!) necessarily impact the flow of the measures? A measure pushed onto the next line would need to have clefs and key signatures added to it, which changes its width, which may push the measure at the end of that system onto the next line, etc., right? Or did I misunderstand?

This is a misconception of how lomse works. [...] Once allocated, there is no need to re-allocate new buffers, only if the window size changes.

Sorry again for not being as clear as I should have been. Multiple buffers are a requirement on the app side. There is a bit of pipelining involved to keep the amount of work per-frame low. (Today it's being done on the same thread as everything else, but moving it to separate thread would only reinforce the need.) Uploading large textures to the graphics hardware is reasonably slow on older and lower-spec mobile hardware, so it's something the app does in a piecemeal way over time. It's not uncommon to be juggling several buffers at once: the next buffer to be rendered by the sheet drawing component, another buffer that's in the middle of being copied out to graphics memory, etc.

So being able to change the buffer between draw calls is important for external reasons.

I'm noticing that although you referred to Free Flow View perhaps you are thinking on displaying only one system...

There's an option in the app to do either. (Here is a tiny, short animation showing both in use at once. Pardon the awful notation; that's what I'm hoping Lomse will fix!) 😄

It's not visible in that clip, but there is a whole suite of dragging/panning/navigation controls built on top of individual "strips" of single systems being rendered one at a time. (This video starting at 0:40 is better.) Both the single-line view and the equivalent of Free Flow View built into the app use the same "single system bitmap" abstraction by drawing just the pieces of each, only when they're needed.

In both examples, the equivalent of Interactor::redraw_bitmap() is only called once per system, ever, for the entire duration of the video. (There isn't enough scrolling to exceed the size of the cache, so the textures are reused perfectly. The blue highlight is done by drawing portions of the same/existing bitmap while "multiplying" the texture's vertex color by varying shades of blue for those portions. redraw_bitmap is only called when a new system that hasn't been seen yet is about to be scrolled into view.)

As Lomse aims to be platform independent, it avoids using any platform specific graphics interface, such as OpenGL.

If it was an interest of yours and you did eventually want to head down the path of getting things ready for hardware acceleration, a nice way to keep things platform independent might be to follow ImGui's example. It is a GUI toolkit made for hardware accelerated apps that manages to remain platform independent. The How it Works section on their GitHub has a nice explanation but the short version is that instead of calling any particular 3D framework directly, it just emits the vertex buffer and list of draw calls that you'd need in any 3D framework, making it very flexible. Lomse could do something similar along with also emitting the required bitmaps, which would become textures in an app's preferred graphics framework.

Again though, that isn't anything I would need in my situation. (I've effectively already done that work, arduously, across a decade of cross-platform development.)

---

It sounds like all of this boils down to just being able to set a Free Form View's viewport using a system number along with any small helper methods that might be missing to retrieve things like system count and measure/beat positions from the GM.

[cecilios]

I'm guessing a DPI change counts as a modification? It's nice to be able to give users a way to adjust the zoom level of the music (while still filling/reflowing-to the complete width of the window/screen) ...

... doesn't changing the resolution (without changing the window size!) necessarily impact the flow of the measures?

Yes, you are right. The issue here is that the terminology we use is different. For lomse, zooming and DPI are unrelated concepts. Zooming refers to a scaling factor for 'copying' the view content onto the bitmap. And resolution (DPI) is just a parameter for the rasterizer, to determine how many 'dots' is it necessary to generate when rendering something. The rendering bitmap resolution is something that Lomse expects to be fixed and that is determined by the physical device (e.g. screen or printer resolution). The dpi to use can always be determined by asking the OS. In the old days a fixed value of 72 dpi or 96 dpi was enough for any application but nowadays the real screen dpi is different from the logical screen dpi and the OS makes adjustments. I have not tested lomse in very high resolution devices such as iPhones but I would bet that a fixed value lower than 200 dpi is enough for current devices.

If I correctly understand the required behavior in your app, when the user adjust the zoom level two actions will be required in lomse: 1) to apply the new zoom factor so that the music symbols will be displayed bigger/smaller, and 2) adjust the width of the View so that the music fits in the window (this action forces to re-build the GM as, as you said, the width impacts in the flow of measures).

But ... good news! this is just the current behavior of FreeFlowView. Both actions, zooming and View width change, are automatically done when the zoom factor is modified in FreeFlowView. The width of the music is also adjusted when the zoom factor is changed, and as a consequence, the GM is automatically re-build.

Note. While reviewing the code and testing all this I've noticed a bug: when the zooming factor is changed in FreeFlowView the width of the View is not properly updated and this might cause a blank window (blank bitmap). First of all, I will fix this.

So being able to change the buffer between draw calls is important for external reasons.

No problem. The buffer can be re-allocated as many times as required. If your app require several buffers there are no limitations. Minimizing buffer re-allocation is only an optimization issue.

I have been watching the demo videos of your app and the visual effects overlaid on the score are supported in Lomse. But it is better if they are already implemented in the app because then the dependency on lomse is less and you have more control.

As to scrolling, my impression is that the bitmap contains more than the part shown in the window and that you scrolls the bitmap. In Lomse this is not needed (but works); Lomse re-paints the bitmap with the new scrolled content. Therefore, you don't need to code all this. Nevertheless, if you would like to position the viewport at the start of systems, for smooth scrolling you will have to use a bigger bitmap and manage the scrolling, at least between sytems. Perhaps it could be simpler to delegate scrolling in lomse and position the vieport using logical units. But, anyway, this is up to you, as it is more convenient. Both approaches will work.

It sounds like all of this boils down to just being able to set a Free Form View's viewport using a system number along with any small helper methods that might be missing to retrieve things like system count and measure/beat positions from the GM.

So it seems. I will start then with two issues:

• First, to fix the bug in FreeFlowView when changing the zooming factor.
• Next, methods to determine coordinates from system number, to be used to set the viewport.
• any other helper method you could need. Please let me know.

Once this is done you probably could start testing integration, and providing feedback.

Regarding hardware accelerated rendering of fonts/glyphs, it's an interesting subject. The Slug library does it better than anything else I've seen--their downloadable demo is truly impressive--but that type of thing is way outside the scope of the kind of change I was requesting!

Interesting! Thank you. I didn't know about this. I'll take a look at it to see if I can get interesting ideas for the future.

If it was an interest of yours and you did eventually want to head down the path of getting things ready for hardware acceleration, a nice way to keep things platform independent might be to follow ImGui's example.
it just emits the vertex buffer and list of draw calls that you'd need in any 3D framework, making it very flexible.

Thanks also for this reference. Maybe I can learn something interesting and apply it to Lomse.

Lomse could do something similar along with also emitting the required bitmaps, which would become textures in an app's preferred graphics framework.

That was my idea when creating the Drawer class. But it is not clean enough as it is necessary to write specific classes for interfacing each specific graphics framework. Perhaps I can learn something interesting from ImGUI.

[cecilios]

PR #378 fixes the bug in FreeFlowView

[cecilios]

I've started documenting the graphical model and have just updated the API documentation with information about the GraphicModel object (see https://lenmus.github.io/lomse/classGraphicModel.html ). I'm not sure about the need to expose all objects and methods, as user applications in general will not need to access the GM and in any case, only to very few methods. But the added documentation should be enough to understand the GM and access the documented relevant methods. I will continue adding more documentation about the Graphical Model objects and their API, but it will take some time. But first, lets try to solve any necessay issues for your integration tests.

So I have also opened an issue #379 to solve the first two needs that you mentioned:

• Determine coordinates from system number
• Determine the height of a system.

Also I have opened a second issue #380 to talk about the possibility of positioning the viewport at start of a system.

[npiegdon]

As to scrolling, my impression is that the bitmap contains more than the part shown in the window and that you scrolls the bitmap.
I wish I could make a simple visualization showing where the textured rectangles are being drawn, but I haven't found an easy way to do it.

Each system is sent to the GPU as two triangles (forming a rectangle) and told to use the pre-rendered texture for that system. To do smooth scrolling, the y coordinates of the triangles are animated smoothly. To prevent the sheet music from being drawn out of bounds, we use the scissor test/rectangle built into the hardware to discard pixels outside of the area the sheet music should appear.

So there is technically a little bit of overdraw happening, but those pixels are being discarded very early in the graphics pipeline.

To do things like the blue highlight, it's actually the exact same triangles but using a different scissor rectangle instead. It's a neat way to do it that ends up being super fast (I've seen all of the screen elements draw at something like 3000 fps on modern hardware when V-sync is disabled).

I've started documenting the graphical model and have just updated the API documentation with information about the GraphicModel object...

This is very useful. Thank you!

[cecilios]

PR #385 adds more documentation for some objects in the graphical model. I also have created issues #383 and #384 to address two other questions you did at beginning of this discussion. With this, I understand that all requests in the conversation have been addressed. If not, please open specific issues. Thank you.