As we’re all too painfully aware, when a few hundred pod pilots get together and exchange ammunition, the dreaded “lag monster” often comes around. There are a number of distinct phenomena that are referred to as lag. It is important to distinguish between them, as they typically have different causes/solutions (even if the symptoms seem the same, like “my guns didn’t work”). This blog is about the type called “module lag,” in which modules start to respond very poorly – sometimes for minutes late. Specifically, this blog is about the bug that causes this for repeats/deactivations, and why it’s not fixed on Tranquility (TQ) yet.

At the Council of Stellar Management (CSM) summit in June, the CSM made an excellent presentation of the issues around fleet-fight lag, specifically about the issues of modules becoming ‘stuck’ and the workarounds used by players. This provided useful insight into a problem which, until recently, we’d never been able to reproduce in a development environment. This made the problem very tangible, and gave us a great symptom to begin digging into.

Like all the best bugs, this one has layers. We’ll start at the first thing we noticed when digging at it: the system responsible for telling the server when modules should be turned off or repeated would get minutes behind in processing when fleet fights happen while other systems remain reasonably responsive. This system, named Dogma, handles module activation/repeat/deactivation, as well as the actual effects of those modules. It also handles the various skill/ship modifiers you see in the game.

This sort of lopsided performance should be easily tweaked. Tasks on the EVE servers use a time-sharing technique called cooperative multitasking which, in short, means that a task has to willingly yield execution to other tasks, otherwise it will run forever. In this case, it would seem the part of Dogma handling module repeat and deactivation was being too nice – yielding execution too much.

Looking at the code some, a theory emerged as to why. There was an error case that stuck out as odd – if an effect was supposed to be stopped or repeated, but the effect system itself didn’t agree that it was time yet, the code would throw up its hands and give up. If that error case gets hit, the processing loop would yield to other systems early. A code comment was very reassuring though – this error was supposedly “rare.”

The “rare” error happened 1.5 million times in the month of June, 2010 on TQ.

More information can be found here.