Question Unexpected lag spikes on server 1.20.4

[ShnitzelZ]

Hey sorry for posting but today my server got unexpected lags i updated all plugins but they suddenly keep coming back here's the messege i get in the logs:

[22:34:54 ERROR]: Current Thread: Server thread
[22:34:54 ERROR]: PID: 98 | Suspended: false | Native: true | State: RUNNABLE
[22:34:54 ERROR]: Stack:
[22:34:54 ERROR]: [email protected]/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
[22:34:54 ERROR]: [email protected]/sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62)
[22:34:54 ERROR]: [email protected]/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:137)
[22:34:54 ERROR]: [email protected]/sun.nio.ch.IOUtil.write(IOUtil.java:102)
[22:34:54 ERROR]: [email protected]/sun.nio.ch.IOUtil.write(IOUtil.java:72)
[22:34:54 ERROR]: [email protected]/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:288)
[22:34:54 ERROR]: [email protected]/sun.nio.ch.ChannelOutputStream.writeFullyImpl(ChannelOutputStream.java:60)
[22:34:54 ERROR]: [email protected]/sun.nio.ch.ChannelOutputStream.writeFully(ChannelOutputStream.java:82)
[22:34:54 ERROR]: [email protected]/sun.nio.ch.ChannelOutputStream.write(ChannelOutputStream.java:122)
[22:34:54 ERROR]: [email protected]/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
[22:34:54 ERROR]: [email protected]/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
[22:34:54 ERROR]: [email protected]/java.util.zip.DeflaterOutputStream.flush(DeflaterOutputStream.java:290)
[22:34:54 ERROR]: [email protected]/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:143)
[22:34:54 ERROR]: [email protected]/java.io.DataOutputStream.flush(DataOutputStream.java:128)
[22:34:54 ERROR]: [email protected]/java.io.FilterOutputStream.close(FilterOutputStream.java:182)
[22:34:54 ERROR]: net.minecraft.nbt.NbtIo.writeCompressed(NbtIo.java:194)
[22:34:54 ERROR]: net.minecraft.nbt.NbtIo.writeCompressed(NbtIo.java:146)
[22:34:54 ERROR]: net.minecraft.world.level.storage.LevelStorageSource$LevelStorageAccess.saveLevelData(LevelStorageSource.java:616)
[22:34:54 ERROR]: net.minecraft.world.level.storage.LevelStorageSource$LevelStorageAccess.saveDataTag(LevelStorageSource.java:607)
[22:34:54 ERROR]: net.minecraft.server.level.ServerLevel.saveIncrementally(ServerLevel.java:1503)
[22:34:54 ERROR]: net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1536)
[22:34:54 ERROR]: net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1216)
[22:34:54 ERROR]: net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:319)
[22:34:54 ERROR]: net.minecraft.server.MinecraftServer$$Lambda$6328/0x0000000801785f88.run(Unknown Source)
[22:34:54 ERROR]: [email protected]/java.lang.Thread.run(Thread.java:833)

i don't see any plugins in here that could cause the issue so if anyone has any ideas about what could cause this pls reply

 

[stefvanschie]

Your server probably took too long to save the data. Some reasons this can happen are:

• Editing too many blocks at once (e.g., via WorldEdit)
• Having a slow disk (e.g., a HDD instead of an SSD)
• Using a slow file system

 

[Coffee]

Your server probably took too long to save the data. Some reasons this can happen are:

• Editing too many blocks at once (e.g., via WorldEdit)
• Having a slow disk (e.g., a HDD instead of an SSD)
• Using a slow file system

Can confirm this happening on our server too (Paper 1.20.4, git-paper-459).

And to answer stefvanschie's question:
1. WorldEdit or anything similar was not used during the lag spikes
2. We are using a NVMe SSD (128kb Sequential Write : 5000 Mb/s)
3. Filesystem: ext4

And I would like to add that this only started happening after updating to paper 1.20.4, we had no issues at all on 1.20.2 on the same day before updating.

 

[electronicboy]

our only real understanding of this is that mojang might be causing IO to be overloaded with map files, otherwise all we can see is that the thing stalled writing to the disk

 

[Coffee]

our only real understanding of this is that mojang might be causing IO to be overloaded with map files, otherwise all we can see is that the thing stalled writing to the disk

Hmm okay. Is it possible to disable automatic world saves for the time being? They're making the server unplayable at this point.

 

[electronicboy]

Not really, disabling that prevents chunks unloading in general