EVE Online: Tyrannis 1.0.5 has been deployed successfully and Tranquility is now accepting connections. You can review the list of fixes and changes at our patch notes page.

Forum threads are available in the EVE Information Portal for players to give feedback and alert us to any issues they may be experiencing.

EVE has a deadly cobra strike force team alpha of extremely dedicated and proficient developers fighting the Lagmonster tooth and nail as their only mission. Through their work and the work of others, there’s now, at this moment, a perfect storm within the company and we have a great number of fixes in the pipes that will knock your socks off.

Our hope is that very soon our beloved Tranquility will be able to support fleet fights of a scale that far exceeds anything you’ve seen before, hopefully going beyond the roof of roughly one thousand on a dedicated node.

That’s tough talk, and we mean it. We will continue to go into detail in our ongoing series of dev blogs, some of which have previously been outlined in a dev blog by CCP Zulu. As soon as the optimizations are ready they will be pushed out to Tranquility individually and you will be able to gauge the difference yourself.

Character Nodes
In this blog I am going to talk about one of the optimizations we’ve been working on over the past few months: Character Nodes which we have already started deploying to Tranquility with phenomenal success.

The EVE Server is architected such a way that functionality is split into logical load balancing units and these units are statically assigned to a node which is a server process running on a single CPU core (since the server process is single-threaded). As long as any individual load balancing unit does not exceed the capacity of that CPU core we’re fine since we can just add more nodes if the load becomes too high. However, if a single load balancing unit needs to do more work than a single CPU core can handle then we’re in trouble.

A market region (Forge, Lonetrek, etc) is an example of a load balancing unit. We have multiple market regions living on a single node and currently four nodes servicing all the market regions. If the load on the market increases we can just increase the number of nodes dedicated to that task and decrease the number of markets on a given node. What this means is that when you’re in Jita and browsing the market, you’re not talking to the Jita node at all and don’t feel any Jita lag effects (up until the point where you buy or sell something in which case the Jita inventory system gets involved).

Another example of a load balancing unit is a solar system. Typically we will have multiple, even hundreds of solar systems living on a single node. We call these types of nodes Location Nodes. Typically these solar systems have such low load that they can be mapped onto a node with a lot of other systems in the same way as the market is. However, now comes the gotcha: If a single solar system exceeds the capacity of the CPU core we have lost the ability to further balance it.

This is the problem in solar systems like Jita and in systems where fleet fights are occurring. We cannot split the solar system up into more units and spread them out and we cannot spread the work out onto multiple cores. We are effectively stuck between a rock and a hard place (stupid GIL!).

Because of these absolute constraints it’s important that any work that doesn’t need to be done by the location node is taken elsewhere. Therefore things like planets (in Planetary Interaction), markets, corporations and alliances are load balanced separate of the solar system you’re in and if your EVE Client calls these services (such as when you are viewing your corp bulletins) then it’s talking to different node than your location node. Your client is at any one time talking to half a dozen different nodes, depending on the call context.

Because of the very strict request-response model that we employ (you click a button (make a request), the server does work, you get a response back) in our business logic a lot of the game systems lend themselves very well to distributed load balancing (e.g. away from your location). However, historically the location node has been viewed as your ‘primary’ node for a lot of the auxiliary logic that runs on the cluster. If a particular call doesn’t have a specific place to go to it will get routed to your location node. Now, that’s just lazy.

Today, after years of optimizations the only true remaining bottleneck on the tranquility cluster is the solar system location and we need to shave off every single cpu cycle that we can. With this in mind we introduced the concept of the Character Node in Tyrannis and have been moving services over to this paradigm since.

Figure 1: Node configuration

Figure 1 shows what this look like. Calls that would otherwise have gone to ‘Location’ are now routed to ‘Character’, which is a set of nodes that is very easily load balanced according to number of logged-in characters. This fits nicely with the schema that is already in place. We need roughly 8 character nodes (out of the total of 204 sol nodes in the cluster) to handle the load and this is very predictable.

Now that most of these changes have been deployed, when your client makes a call for things that aren’t directly related to your location, that call will typically be routed to your dedicated character node where your character happily ‘lives’ along with tens of thousands of other characters.

Since the work coming from an individual client has a long way to go before it exceeds the capacity of a single CPU core and is easily predictable, we’re in a good place. The client still makes the same number of calls to the cluster as before and the amount of work done on the cluster is unchanged. All that is different is that other nodes perform the work.

Changing this isn’t very glorious or filled with a lot of eureka! moments. It’s mostly just eating your vegetables and rearranging logic. The benefits, however, are substantial since, as it turned out, the majority of the calls that a client makes in a given period can be serviced by any node and therefore should not go to the location node.

The results
Okay, all the wall-of-text above was just to explain how the system works. Now that you’ve eaten your vegetables it’s time for the sugar-laden dessert. Let’s see the effect this had on Jita last week, comparing the performance before and after the phase #2 of these changes were deployed to Tranquility on 12 August.

Figure 2: Network traffic on the Jita node

In Figure 2 you can see the number of calls made onto the Jita node in four consecutive runs. After moving some services over to the character nodes, up to 80% of the calls were routed elsewhere freeing Jita up for important things like inventory operations and scams in local.

Figure 3: CPU utilization on the Jita node

As expected, moving services away from the Jita node had a good effect on CPU and has allowed us to scale Jita beyond the 1400 pilot limit that has capped its population for a while. Hopefully you should not be towed to another star system when you log in on Sunday evenings for a while.

Even though the metrics that we have gathered are for the Jita node, this change will have a positive effect on all loaded nodes in the cluster–with Jita and Fleet Fight nodes benefiting the most. The reason I use Jita here is that it has a very predictable load pattern whereas fleet fights are anything but. However, the same principles apply. Before this change you would be making something like 5-10 server calls to your location node to finish jumping, each one of these calls could take a long time to complete. Now you’ll be making something like 4, with the rest returning very quickly.

We’re hoping you’ll be able to tell the difference the next time you decide to invade your nearest friendly neighbor.

Also keep in mind that the other benefit of this type of offloading is that you don’t “feel” the lag as much. Take Jita for example. If you’re just browsing the market you don’t notice that it’s laggy since you’re not talking to that node, you’re talking to the market node. With the character node changes that we’re doing much of the user experience will be improved greatly since the buttons that you’re clicking end up on a lightly loaded node, even if clicking around in space is laggy. This can make a big difference in the overall playing experience.

We have been deploying the character node changes piecemeal to Tranquility throughout August and have a few more going out over the next few weeks. These should give you better performance in fleet fights as well as allow us to push Jita’s concurrent player boundaries further.

This is all well and good but keep in mind that ‘lag’ hasn’t been ‘fixed’ once and for all and it might not be in the foreseeable future, but we are pushing the boundaries of the cluster more aggressively these days than we have been for a while. Like I mentioned before, this is just one of the many things that we are working on right now. More services are being moved to character nodes this autumn and we will have plenty of more awesome fixes aimed at fleet fights coming in over the next weeks and months.

Fleet fight tests on Singularity

As stated before, it is very important for us to get as many people as possible involved when we are testing on the Singularity test server (SiSi). Please join in when the tests are advertised and help us test the performance improvements so that they can be deployed onto Tranquility as quickly as possible.

A word about fleet fight notifications

A fleet fight which happens on a node with a hundred other systems is not a good experience for anyone involved since often times the node only has 30% left of its CPU capacity when the fight starts. You can help make sure that the solar system you will be fighting in is on a dedicated node.

Use the Fleet Fight Notification form to let us know about a pending fleet fight. Sending in a petition is not the right way to report this, you must use the form. If you report the fight well in advance we can make sure that the fight is as smooth as we can make it.

Our next mass test on the Singularity test server will be held on Thursday, August 26, 2010 at 20:00 UTC and is expected to last 1.5 hours. We would like at least 300 pilots (500 is ideal). More information about the test can be found in this thread. Feedback about the test can be shared in this thread.

For those of you interested in learning more about mass tests and our most recent results, be sure to check out CCP Tanis’s newest dev blog.

As we’re all too painfully aware, when a few hundred pod pilots get together and exchange ammunition, the dreaded “lag monster” often comes around. There are a number of distinct phenomena that are referred to as lag. It is important to distinguish between them, as they typically have different causes/solutions (even if the symptoms seem the same, like “my guns didn’t work”). This blog is about the type called “module lag,” in which modules start to respond very poorly – sometimes for minutes late. Specifically, this blog is about the bug that causes this for repeats/deactivations, and why it’s not fixed on Tranquility (TQ) yet.

At the Council of Stellar Management (CSM) summit in June, the CSM made an excellent presentation of the issues around fleet-fight lag, specifically about the issues of modules becoming ‘stuck’ and the workarounds used by players. This provided useful insight into a problem which, until recently, we’d never been able to reproduce in a development environment. This made the problem very tangible, and gave us a great symptom to begin digging into.

Like all the best bugs, this one has layers. We’ll start at the first thing we noticed when digging at it: the system responsible for telling the server when modules should be turned off or repeated would get minutes behind in processing when fleet fights happen while other systems remain reasonably responsive. This system, named Dogma, handles module activation/repeat/deactivation, as well as the actual effects of those modules. It also handles the various skill/ship modifiers you see in the game.

This sort of lopsided performance should be easily tweaked. Tasks on the EVE servers use a time-sharing technique called cooperative multitasking which, in short, means that a task has to willingly yield execution to other tasks, otherwise it will run forever. In this case, it would seem the part of Dogma handling module repeat and deactivation was being too nice – yielding execution too much.

Looking at the code some, a theory emerged as to why. There was an error case that stuck out as odd – if an effect was supposed to be stopped or repeated, but the effect system itself didn’t agree that it was time yet, the code would throw up its hands and give up. If that error case gets hit, the processing loop would yield to other systems early. A code comment was very reassuring though – this error was supposedly “rare.”

The “rare” error happened 1.5 million times in the month of June, 2010 on TQ.

More information can be found here.

I like planning. I really do. I think calendars are pretty sexy and my OCD acts up when things aren’t color coded, have point values or do barrel rolls. This summer, I planned to sneak a few smaller items onto my teams backlog and positively surprise the Council of Stellar Management (CSM) when they got here. I ended up going to all these CSM summits, talking about features new and old, but never actually got to tell the CSM about some of the small things we had lined up. My plan didn’t work out, but the job got done, which is what matters.

Usually, smaller changes get bunched up and released with the coming expansion. This particular change is a little different though, as part of the code is needed for a bugfix, meaning we will be deploying it in the near future instead of this winter. The change we are introducing is going to end ghost datacore accumulation.

Currently, any character collecting research points from an agent will continue to do so, even after your account lapses due to inactivity. This is a pretty massive loophole to making substantial amounts of money and it is now being closed. When this change is deployed, characters will stop collecting research points when your account lapses into inactivity. You won’t lose the points you’ve accumulated up to that point, but simply won’t gain anymore. When you activate your account, your character will automatically begin earning RPs again. This change is long overdue and will hopefully benefit our active players.

On a sidenote, this was an issue our team picked off the CSMs list of priorities. It’s a great list that outlines both big and small concerns in the community and hopefully we can continue to address it. Summers are pretty quiet here, due to holidays and it’s a great time to go over the list and pick a few items for your team. Hopefully, we can bring you more items off the CSMs list, but till then, here’s one of them.

If anyone is interested, the CSM item is here.

The complete list is here.

Anyway, I should have more blogs coming up as we speed up for the winter release.

Till then, toodles.