Question Unexpected lag spikes on server 1.20.4

[ShnitzelZ]

Hey sorry for posting but today my server got unexpected lags i updated all plugins but they suddenly keep coming back here's the messege i get in the logs:

[22:34:54 ERROR]: Current Thread: Server thread
[22:34:54 ERROR]: PID: 98 | Suspended: false | Native: true | State: RUNNABLE
[22:34:54 ERROR]: Stack:
[22:34:54 ERROR]: java.base@18.0.2/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
[22:34:54 ERROR]: java.base@18.0.2/sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62)
[22:34:54 ERROR]: java.base@18.0.2/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:137)
[22:34:54 ERROR]: java.base@18.0.2/sun.nio.ch.IOUtil.write(IOUtil.java:102)
[22:34:54 ERROR]: java.base@18.0.2/sun.nio.ch.IOUtil.write(IOUtil.java:72)
[22:34:54 ERROR]: java.base@18.0.2/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:288)
[22:34:54 ERROR]: java.base@18.0.2/sun.nio.ch.ChannelOutputStream.writeFullyImpl(ChannelOutputStream.java:60)
[22:34:54 ERROR]: java.base@18.0.2/sun.nio.ch.ChannelOutputStream.writeFully(ChannelOutputStream.java:82)
[22:34:54 ERROR]: java.base@18.0.2/sun.nio.ch.ChannelOutputStream.write(ChannelOutputStream.java:122)
[22:34:54 ERROR]: java.base@18.0.2/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
[22:34:54 ERROR]: java.base@18.0.2/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
[22:34:54 ERROR]: java.base@18.0.2/java.util.zip.DeflaterOutputStream.flush(DeflaterOutputStream.java:290)
[22:34:54 ERROR]: java.base@18.0.2/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:143)
[22:34:54 ERROR]: java.base@18.0.2/java.io.DataOutputStream.flush(DataOutputStream.java:128)
[22:34:54 ERROR]: java.base@18.0.2/java.io.FilterOutputStream.close(FilterOutputStream.java:182)
[22:34:54 ERROR]: net.minecraft.nbt.NbtIo.writeCompressed(NbtIo.java:194)
[22:34:54 ERROR]: net.minecraft.nbt.NbtIo.writeCompressed(NbtIo.java:146)
[22:34:54 ERROR]: net.minecraft.world.level.storage.LevelStorageSource$LevelStorageAccess.saveLevelData(LevelStorageSource.java:616)
[22:34:54 ERROR]: net.minecraft.world.level.storage.LevelStorageSource$LevelStorageAccess.saveDataTag(LevelStorageSource.java:607)
[22:34:54 ERROR]: net.minecraft.server.level.ServerLevel.saveIncrementally(ServerLevel.java:1503)
[22:34:54 ERROR]: net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1536)
[22:34:54 ERROR]: net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1216)
[22:34:54 ERROR]: net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:319)
[22:34:54 ERROR]: net.minecraft.server.MinecraftServer$$Lambda$6328/0x0000000801785f88.run(Unknown Source)
[22:34:54 ERROR]: java.base@18.0.2/java.lang.Thread.run(Thread.java:833)

i don't see any plugins in here that could cause the issue so if anyone has any ideas about what could cause this pls reply

 

[stefvanschie]

Your server probably took too long to save the data. Some reasons this can happen are:

• Editing too many blocks at once (e.g., via WorldEdit)
• Having a slow disk (e.g., a HDD instead of an SSD)
• Using a slow file system

 

[Coffee]

Your server probably took too long to save the data. Some reasons this can happen are:

• Editing too many blocks at once (e.g., via WorldEdit)
• Having a slow disk (e.g., a HDD instead of an SSD)
• Using a slow file system

Can confirm this happening on our server too (Paper 1.20.4, git-paper-459).

And to answer stefvanschie's question:
1. WorldEdit or anything similar was not used during the lag spikes
2. We are using a NVMe SSD (128kb Sequential Write : 5000 Mb/s)
3. Filesystem: ext4

And I would like to add that this only started happening after updating to paper 1.20.4, we had no issues at all on 1.20.2 on the same day before updating.

 

[electronicboy]

our only real understanding of this is that mojang might be causing IO to be overloaded with map files, otherwise all we can see is that the thing stalled writing to the disk

 

[Coffee]

our only real understanding of this is that mojang might be causing IO to be overloaded with map files, otherwise all we can see is that the thing stalled writing to the disk

Hmm okay. Is it possible to disable automatic world saves for the time being? They're making the server unplayable at this point.

 

[electronicboy]

Not really, disabling that prevents chunks unloading in general

 

[TheFishNextDoor]

I have also been experiencing this issue on my server since updating to 1.20.4

 

[mbaxter]

I have also been experiencing this issue on my server since updating to 1.20.4

Maybe hop over to our Discord with a spark report from during a lag spike

 

[TheFishNextDoor]

Maybe hop over to our Discord with a spark report from during a lag spike

I am on a purpur server with lots of plugins so I'm not sure my reports would be that useful but I am experiencing the same error in my log. So i'll post my information here and maybe me and the original poster share some of the same plugins or something that could be a clue for us. It appears to be on a data tag save that is taking sometimes upwards of 50 seconds.

[14:31:54] [Server thread/WARN]: Can't keep up! Is the server overloaded? Running 7795ms or 155 ticks behind
[18:32:04] [Watchdog Thread/ERROR]: --- DO NOT REPORT THIS TO PURPUR - THIS IS NOT A BUG OR A CRASH - git-Purpur-2176 (MC: 1.20.4) ---
[18:32:04] [Watchdog Thread/ERROR]: The server has not responded for 10 seconds! Creating thread dump
[18:32:04] [Watchdog Thread/ERROR]: ------------------------------
[18:32:04] [Watchdog Thread/ERROR]: Server thread dump (Look for plugins here before reporting to Purpur!):
[18:32:04] [Watchdog Thread/ERROR]: ------------------------------
[18:32:04] [Watchdog Thread/ERROR]: Current Thread: Server thread
[18:32:04] [Watchdog Thread/ERROR]: PID: 27 | Suspended: false | Native: true | State: RUNNABLE
[18:32:04] [Watchdog Thread/ERROR]: Stack:
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/sun.nio.ch.IOUtil.write(IOUtil.java:97)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/sun.nio.ch.IOUtil.write(IOUtil.java:67)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:288)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/java.nio.channels.Channels.writeFullyImpl(Channels.java:74)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/java.nio.channels.Channels.writeFully(Channels.java:96)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/java.nio.channels.Channels$1.write(Channels.java:171)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/java.util.zip.DeflaterOutputStream.flush(DeflaterOutputStream.java:290)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:143)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/java.io.DataOutputStream.flush(DataOutputStream.java:128)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/java.io.FilterOutputStream.close(FilterOutputStream.java:182)
[18:32:04] [Watchdog Thread/ERROR]: net.minecraft.nbt.NbtIo.writeCompressed(NbtIo.java:194)
[18:32:04] [Watchdog Thread/ERROR]: net.minecraft.nbt.NbtIo.writeCompressed(NbtIo.java:146)
[18:32:04] [Watchdog Thread/ERROR]: net.minecraft.world.level.storage.LevelStorageSource$LevelStorageAccess.saveLevelData(LevelStorageSource.java:616)
[18:32:04] [Watchdog Thread/ERROR]: net.minecraft.world.level.storage.LevelStorageSource$LevelStorageAccess.saveDataTag(LevelStorageSource.java:607)
[18:32:04] [Watchdog Thread/ERROR]: net.minecraft.server.level.ServerLevel.saveIncrementally(ServerLevel.java:1586)
[18:32:04] [Watchdog Thread/ERROR]: net.minecraft.server.MinecraftServer.tickServer(MinecraftServer.java:1579)
[18:32:04] [Watchdog Thread/ERROR]: net.minecraft.server.MinecraftServer.runServer(MinecraftServer.java:1246)
[18:32:04] [Watchdog Thread/ERROR]: net.minecraft.server.MinecraftServer.lambda$spin$0(MinecraftServer.java:323)
[18:32:04] [Watchdog Thread/ERROR]: net.minecraft.server.MinecraftServer$$Lambda$6327/0x00007f68d8bb6350.run(Unknown Source)
[18:32:04] [Watchdog Thread/ERROR]: java.base@17.0.11/java.lang.Thread.run(Thread.java:840)
[18:32:04] [Watchdog Thread/ERROR]: ------------------------------
[18:32:04] [Watchdog Thread/ERROR]: --- DO NOT REPORT THIS TO PURPUR - THIS IS NOT A BUG OR A CRASH ---
[18:32:04] [Watchdog Thread/ERROR]: ------------------------------

The issue freezes the server for 10 to 50 seconds usually and happens a couple times a day sometimes even when no players have been online for hours.

Version: git-Purpur-2176 (MC 1.20.4)

Plugins: ArmorStandAdjustor, AuctionHouse, BestTools, BetterRTP, ChatEmojis, ChatItem, ChestCommands, ChestShop, ChestSort, Citizens,
CrazyEnvoys, DecentHolograms, DrMap, Elevator, Essentials, EssentialsChat, EssentialsSpawn, ExcellentCrates, floodgate, Geyser-Spigot, GPFlags, GriefPrevention, GSit, HeadDatabase, InfiniteJump, ItemEdit, Leashable, LuckPerms, mcMMO, MiniMOTD, Multiverse-Core, Multiverse-NetherPortals, NexEngine, PlayerWarps, ProtocolLib
spark, TAB, Terra, TerraformGenerator, TradeSystem, UnbreakableLeashes, Vault, VehicleTeleportation, ViaVersion, VotifierPlus, VotingPlugin, WorldEdit, WorldGuard

 

[mbaxter]

This is the Paper forums. We support Paper.

Locking this as it was already answered and the necropost wasn't even on Paper.