[Bug]: Saving world for over 30 seconds causes all players to timeout

[YueLengM]

Possible Fixes

Yes

Modpack Version

2.5 & 2.7

What happened?

Without mods like RandomPatches to modify the timeout value, if the server takes over 30 seconds to save the world, all players will be kicked out of the server.
This issue occurs when a player has explored far enough, and it happens every 5 minutes (auto save).

Tested with one player and looter alone took 22s to save.
Spark log: https://spark.lucko.me/JOgnCzrSwu

Even without any player, the server still takes ~10s to save the world.
releated: #264

latest.log

No response

Developer reports

No response

[YueLengM]

Found that adding Fast Async World Save to the server resolves this issue.
However, why does this mod exist in the client pack but not on the server?

[ghost]

fast async world save was removed recently for crashing issues, the dev has fixed said issues and we will probably get it back into the server pack sooner rather than later

[ArcadeArchie]

How much RAM did you give the server?

[YueLengM]

How much RAM did you give the server?

-Xms4G -Xmx8G

[ArcadeArchie]

i've created an upstream issue on Lootr as it seems to cause those lag spikes when saving

[noobanidus]

After looking at the issue that was reported on my tracker, I'm quite confident that this isn't being caused by Lootr specifically, as digging down through the actual method calls indicates that Minecraft's SavedData and Lootr's SavedData bottleneck at a similar point in the I/O functions.

As noted by the original author, Lootr does take 22 seconds to save, but Minecraft's SavedData also takes 12 seconds to save, which is quite significant.

In theory, regular restarts would reduce the amount of loaded SavedData by Lootr, which would reduce the amount of time it takes to save, but that obviously doesn't address the core issue.

I'm happy to be proven wrong, but I don't believe Lootr is responsible for this issue.

EDIT: Just so this does continue to get looked into, I think removing the "Reported to Mod Dev" tag would be useful. As I said, there really isn't much I can do on Lootr's end.

Further, if the original author is able to provide some more information about their server -- specifically the size of the lootr folder within the data would give some indication of the total amount of information, although most Lootr data files are less than a kilobytee.

[TheBedrockMaster]

@noobanidus I only added the Reported to Mod Dev in case it was caused by the mod, I specifically didn't add the Mod Bug label because I wasn't sure it was caused by the mod, I'll removed the label now.

[YueLengM]

Before creating this issue, I wrote a tool to monitor the time spent saving all files during auto-save. I think this is an I/O bottleneck.

However, I have never encountered such stuttering in the Minecraft servers I’ve hosted before. That said, this is the first time I’ve hosted a 1.21 server, so I suspect there are some changes to the file-saving algorithm.

Each file takes at least 30ms to save, regardless of its size. For example, DIM-1/data/raids.dat, which is only 90B, still took 348ms to save. The 20 seconds Lootr took to save is simply because there are more files, not because the files are particularly large.

[noobanidus]

I've raised this with some of the NeoForge maintainers. There is a patch which ensures the safety of SavedData that may be causing performance issues when it comes to writing those files specifically.

I'm also going to look into potentially unloading SavedData for Lootr; if necessary I can try collating data into one larger file, although this current system is how Lootr has worked since 1.14 and doesn't seem to have caused this kind of issue before now (or at least, if it did, it wasn't reported widely or to me).

[YueLengM]

I've raised this with some of the NeoForge maintainers. There is a patch which ensures the safety of SavedData that may be causing performance issues when it comes to writing those files specifically.

I have did some test on that, actually i
forget to disable the patch so you can see neo's writeNbtCompressed() calls the original writeCompressed() directly from the spark log in this issue

[noobanidus]

I've raised this with some of the NeoForge maintainers. There is a patch which ensures the safety of SavedData that may be causing performance issues when it comes to writing those files specifically.

I have did some test on that, actually i forget to disable the patch so you can see neo's writeNbtCompressed() calls the original writeCompressed() directly from the spark log in this issue

It would be interesting to see a profile with a version of NeoForge that has this patch disabled.

Additionally, I can make a custom build of Lootr for 1.21 that includes some code to discard Lootr's SavedData after 3 minutes of not having been accessed. I'm not sure whether it would improve things or not, but at the very least (unless multiple people are constantly opening Lootr containers), it would cut down on the number of files queued for saves.