Allow Zap to Mediate Throughput to Stop Roblox Servers being Unable To Process All Events

[CoIorEvent8]

Describe the bug

Zap has a major vulnerability where exploiters can crash entire game servers (test in a team test or live game) with oversized buffers because of constant long loops without delays trying to handle buffers. This leads to an out of memory crash that is difficult to trace.

Our game received a spike in out of memory crashes while no updates were made. My team initially thought this was a DDOS, but after seeing a spike in errors from network requests, I looked deeper and found a security issue in ZAP which allows servers to crash instantly.

After deploying a fix to the output.rs, rebuilding the EXE, and running zap packets.zap, I've seen no more out of memory crashes even though attempts have been made.

In addition, the logs indicate that this was in fact the method that exploits were using to crash servers.

https://devforum.roblox.com/t/surge-in-out-of-memory-crashes-without-any-updates-added/4528808/3?u=coiorevent

Reproduction

1. Create a Zap funct for anything (not sure if its only for calls)
2. Run zap packets.zap to output the lua code
3. Modify the client source in Zap's output files to get rid of the call error on large buffers e.g. error("Zap has more than 256 calls awaiting a response, and therefore this packet has been dropped")

4. Create a local script that requires the Zap module. In that localscript, call the modified function through Zap several times. I'm not sure if it has to have arguments or not.

game:GetService("RunService").RenderStepped:Connect(function()
    for i = 1, 256, 1 do
        task.spawn(function()
            Zap.castFishingPole.call(1 * 100)
        end)
    end
end)

5. Notice that on a team test (or live server), the game immediately freezes for all clients. Players freeze, remotes no longer go through, etc. Server ping will go up, and eventually the server will crash due to an out of memory crash. Sometimes, this crash will be recorded by Roblox on the create page.

Expected behavior

I expect ZAP to handle large buffers without crashing OR DENY LARGE BUFFERS entirely, so that the server doesn't get put into an infinite, unending loop state that causes out of memory crashes.

Additional context

I've attached the updated server.rs file as a text file that my game is currently using to implement this fix into our game. This was generated with AI, but works. I'd recommend reviewing this and only using it as a glimpse of a final solution as there may be a better way.

With this solution however, my game went from having 30 minute sessions back to having servers as old as several days. I hope that a solution can be implemented into the official Zap codebase, so that I don't have to rely on a custom EXE to get an output with the solutions.

server.txt

[CoIorEvent8]

Directly sending a large buffer to the event itself also works, obviously. I've been able to deploy several methods to mitigate this, but I'm currently testing a third solution to hopefully combine batches, while still managing the queue, rather than having an arbitrary queue that doesn't take into account total batch size. Since Zap tries to handle events almost instantly once an event is called, adding a queue naturally yields that. If another event were to come in, Zap would cause it to go out of order with other batches. To solve this, I also implemented a single loop per player where events will get handled in the exact order they come. I'll share my findings here once it is employed to a live game.

When implementing a solution, one thing to keep in mind is that you need to drop entire batches - not individual packets, or the packet data could be changed and do something unexpected. That's why it's so important to queue these batches properly so players sending data faster than the server can handle don't reach queue and have total packet loss.

The core concept of this vulnerability is that batches can be sent with 0 bandwidth (which is great), but the server takes a lot more resources to go through each packet. If it has to go through each packet, or constantly deal with packets without stops, crashes will occur. The queue solution would effectively act as a ratelimit, that takes into account total size of the batch buffers, rather than the amount of batches sent into the queue at a time. It would also save some memory space by combing buffers instead of appending each new buffer to an additional index in a table.

Some other modules like Packet have implemented packet dropping to try to solve this, but to be honest that is not the greatest solution. By queuing and handling on a size-by-size basis, normal large batches can still be handled without being dropped.

[jackdotink]

The vulnerability here is the server code being unable to properly handle the large amount of throughput that zap enables. Zap will not decrease the amount of throughput it enables, so ratelimits need to be properly implemented in server code. One thing we could do, perhaps, is after each event call check to see if the player is still connected, if they were kicked we could drop the rest of the buffer.

[sasial-dev]

While @jackdotink's comment above discusses the crux of the issue, just summarising a lot of discussion over Discord:

Some proposed solutions to this include:

• Checking before each event deserialisation that the player actually exists.
• Adding an option to remove batching.
• Adding an option to have a max size limit on events.

@CoIorEvent8 also has a fork that has an alternate solution where deserialisation is instead yielded after a large amount of compute. This will not be considerd as a possible solution for zap itself as yielding deserialisation can lead to possible edge cases due to the fact that the way zap handles buffers is designed to be non-yielding and single threaded.