Reduce memory usage

[thestinger]

Android 13 has greatly increased the required memory for successful flashing. Users have been reporting lots of issues flashing 6th and 7th generation Pixels due to the site process running out of memory. Upgrading zip.js was previously blocked due to a major memory usage increase but there have been substantial changes and it's quite possible that it's now leaner on memory instead of much worse. It's worth trying to see if this problem can be resolved that way.

If upgrading zip.js doesn't resolve this, it would be nice to find another way to reduce memory usage because the portability has suffered a lot due to the memory requirement increase. We're now seeing this as one of the main issues people encounter during flashing. I'm not sure why exactly Android 13 made this so much worse. I wonder if zip.js or the dependencies it has are leaking memory and having additional images / overall more data to flash ended up causing this.

It appears that some API breakage needs to be resolved to update zip.js. I might try this myself soon but I'm unfamiliar with the zip.js API and most of the fastboot.js code.

[kdrag0n]

I agree that this is an issue, but I don't think zip.js is to blame. Extracting entire images in memory is likely already a non-starter for memory-constrained devices. I'm not sure if there are JS zip libraries that support streaming and reading the central directory in advance though.

Potentially relevant API: https://developer.mozilla.org/en-US/docs/Web/API/Blob/stream

[kdrag0n]

To clarify, reading the central directory in advance and jumping to files out-of-order is needed because of the fastboot flashing order, particularly boot-critical images.

[thestinger]

GrapheneOS@e7a9403 mostly resolves this but results in higher storage usage which isn't exactly great but it's much better than the current memory usage overall.

[kdrag0n]

That seems like a good fix for memory usage caused by sparse image splitting. How does memory and storage usage compare to Google's flasher now?

[thestinger]

It has significantly lower memory usage than Google's flashing tool but dramatically higher storage use.

[thestinger]

The storage usage was already quite high before this change but it does increase the peak storage. Worth noting that the browser can and does choose to store the blobs in memory when it has memory available so it's tricky to measure. On a desktop with a lot of memory, I think it only ends up using storage for the largest blob for the system image. Giving the browser a choice is essentially ideal if it's smart about it. Not very much memory or storage is available on Android. This makes it work far better from Android phones since the memory usage had gotten so high that it would crash even from 6GB memory phones, but it probably can't work with less than 32GB free storage. There's a formula for calculating how much storage is allowed to be used which is different on mobile and desktop. I don't have a link to it available.

[thestinger]

Ideally it would partially store the blobs in memory instead of it being all or nothing but Chromium doesn't support that.