Following the release of the Dominion expansion we started to get reports from players about an increase in “fleet lag.” Before we go any further, I would like to assure everyone that a taskforce of developers has been working on this continuously, but tracking down the source has proven a very difficult task.

So, what is the problem?
“Lag” is an ambiguous term and can refer to many things. The graphics in the client might be slowing it down, the network might be congested or a host of other issues may be coming into play. We have spent a long time trying to reproduce the issue on the test servers as well as monitoring fights as they have been unfolding on Tranquility, where we have received help from the players in the form of testing and bug reports. You might see names like “CCP Atlas”, “CCP GingerDude” and “CCP Warlock” poking about in Local. Don’t be alarmed, we are simply monitoring the node, looking for glitches. There are senior software engineers spending their evenings and weekends on Tranquility monitoring and measuring since we have not been able to see issues when testing because the sheer number of people required to trigger the problems and the specific ‘usage pattern’ of the game systems in question are outside what can currently be tested in the lab (although this is something that is being worked on).

What we are looking at now specifically is the fact that the performance of the sol nodes (the computers in the clusters handling the solar systems) appears to be worse than before. Even though CPU across the cluster has not increased with the Dominion expansion, each reinforced node is no longer able to handle the same number of people fighting as gracefully as it did before.

Additionally, we have the issue of the grid not loading when jumping into a system. Being stuck on the source side and not getting through the gate is one thing since you’re not in the battle yet, but if you actually get through the gate and arrive on the other side without managing to load the grid you could be at a disadvantage and possibly vulnerable because of your client’s inability to act. This can be detrimental to the fleet jumping into the system.

What is happening is that the client is requesting information about the simulation from the server but it’s not receiving it in a timely manner. There may be many minutes before the information is finally received as it is getting stuck in the lower levels of the sol node.

Anatomy of “grid not loading”
When the specific scenario of “grid not loading” occurs, its fingerprint on your client is quite specific (even though sadly we cannot see the effects in the server logs).

Refer to the following handy diagram to determine whether you have encountered the problem:

CLICK FOR LARGER VIEW

This is what it would typically look like if you’re jumping into a system which exhibits the issue. However, if you are logging into such a system and encounter this issue you will only have a black screen and a “loading” bar.

If we look at the image a bit you will see that the local channel, the right click menu and the system information are all updated to the new system while the nebula is that of the old system. Your overview will also be empty and no brackets in space. If you open up the monitor (control-shift-alt-m) it will have (at least) one outstanding call. This is the request for the simulation instance.

When you see this, the worst thing you can do is to click any buttons or type in local. It will only exacerbate the problem. Relogging will also not help and might even make the situation worse. The best you can do is wait it out (see the “Fleet fight survival guide” below).

Light at the end of the tunnel
We were finally able to reproduce this with a special fleet test on our Singularity test server on Jan 27th where over 400 people participated in helping us out. That was really fantastic to see and I would like to thank everyone who showed up.

On Jan 28th, we had a 1600 person fleet fight on Tranquility which our team monitored closely, keeping the node alive using methods that make our system admins faint. This was one of the biggest, if not the biggest, fleet fight ever in EVE (at least where the node survived the ordeal). This event allowed us to identify what was causing some of these glitches and deploy fixes live.

We have added a stop-gap measure in place which keeps nodes from dying during these high stress scenarios and we have some more permanent solutions in the pipes that will be deployed over the next few weeks.

How can I help?
There is a mass test scheduled for the Singularity test server on Thursday Feb 4th at 16:00 Eve time. Those of you who wish to help out, please come and provide us with client logs when you encounter the “grid not loading” problem or other issues. We will specifically be testing jumping in several solar systems trying out a few of the software solutions that have been implemented.

This requires running Logserver.exe from the client installation folder before you start the client. Your logs are extremely important, and each bug report you submit with this information will go a long way toward bringing us closer to a resolution or fix for this issue.

There is more information on this fleet test in this thread.

Additionally, if you are planning to attack a system we also highly recommend that you notify us at least 24 hours beforehand through the Fleet Fight Notification Form so that the system can be placed on a reinforced node. This will also give us the chance to mobilize people to be in the solar system with measuring devices firmly attached to our Polaris frigates.

Fleet Fight Lag Survival Guide
I’d like to talk a little bit about the best tactics we’ve found to use during periods of extreme server-side lag and methods to make the experience a little more bearable. Note that I am talking about server-side lag here, not low FPS or bad network conditions (for example, if you find yourself in a fleet fight on your 9600bps cell phone modem up on a glacier somewhere). The lag that this refers to is when there are hundreds of players fighting in a solar system and everyone is experiencing delays, where guns could take minutes to fire and there’s some gnarly rubberbanding. If you’ve encountered this type of lag you typically know the symptoms. It’s under these circumstances that the “grid not loading” issue manifests itself for example.

Do not hammer that button!
One of the most important things to keep in mind when you are in a system with heavy lag is that you should not hammer any buttons and you should try to execute as few commands as possible. If you are waiting for the grid to load the worst thing you can do is click buttons on the interface or chat in local. These actions will have a negative impact on your client’s ability to recover (more to the point, on your session on the server). If you have an outstanding call you should wait for it to complete, and it will complete eventually. It might take minutes but your gun will fire and clicking the button again will not help. Do not browse the market or look up contracts while waiting for the system to load. I know it’s hard but you have to leave the client alone while it waits for the calls to complete or you will add to the problem and can even cause the session to get stuck.

Use the Monitor
The network tab in the ingame monitor debugging tool (control-shift-alt-m) can help you understand whether your calls are completing in a timely manner. If there is a number in the “Outstanding” row that means that the server is busy executing your command. Wait for it to complete before executing another one!

Calls do time out
What I said about calls completing eventually is true… from a certain point of view. The call will be handled eventually by the server but it’s possible that the client has given up by then. Calls typically time out in 8 minutes and this is true of the request for the simulation instance (grid). If you have waited for 8 minutes for the grid to load you will not recover. If you have the monitor open and see the Outstanding number drop to 0 and your grid doesn”t load then you have no choice but to relog, and even that might not fix things if the server is still overloaded.

Note that if you get podded while in this state then the client should recover when your session is moved into your clone station.

Final words
We have come a long way in understanding the issues that have plagued large fights after Dominion and have gotten important insight into fleet fight lag as a whole. The problems we are experiencing post-Dominion are being worked on and they will be fixed soon, adding more space carnage to your battles.

I would like to ask that we keep the discussion thread attached to this blog productive and try to keep flaming to a minimum.

A Word from Customer Support
Support’s intervention in large fleet engagements in even a remotely fair manner is very problematic due to the complexity of the situation.

Under circumstances where lag is involved or elusive bugs that are difficult to establish as the deciding factor, CCP cannot justify passing judgment on who should have won or who should have lost. While a problem such as the one in focus here is known to affect some players in certain situations, the evidence available in server-side logs, when available at all, can be haphazard and arbitrary. As such, intervention on CCP’s part would risk being arbitrary as well.

Please understand that fleet fight reimbursement has always been very controversial and few issues have been discussed and argued in more detail within CCP. The conclusion has, however, always been to leave fleet battles alone rather than reimbursing them as a whole. We apologize for any inconvenience this may cause and we appreciate your patience as we work diligently towards resolving this issue.

The most mysterious and elusive of all the peoples of EVE, the Jovians number only a fraction of any of their neighbors, but their technological superiority makes them powerful beyond all proportion.

Although definitely human, the Jovians often seem to the other races as though they are not, the reason being that they embraced genetic engineering as the way to solve any and all the problems which plague the human race. Over the thousands of years since, the Jovians have experimented with every kind of genetic modification their technology allowed. As their powers grew, they began to believe they were capable of anything, and this led them into increasingly more bizarre mutations of their bodies and minds, a policy rigorously backed up by strict governmental control.

But one fateful moment in their history made them lose this control for a few generations, and the results were catastrophic. By this time the Jovians had begun interfering with their basic instincts, curbing their aggression and sexual instincts and cultivating strange new ones instead. Since the Shrouded Days, as the Jovians call their momentary social eclipse, they have been trying to put the pieces together again, but their DNA-structure has in many ways been damaged beyond repair. The consequence is the dreaded Jovian Disease. Genetic in nature, it is not infectious to other races, but among Jovians it causes a depression so deep and serious that the victim loses the will to live, and death results within a few days or weeks.

Despite this, the Jovians escaped the chaos that followed the closure of EVE remarkably well. Within the space of only a few centuries they had recovered, and were once again running a hi-tech society. They settled in a number of systems and founded an empire lasting for nine millennia, but even if the Jovians are by far the most technologically advanced of the races of EVE, they have still not recovered the splendour of their first empire. The disease within them keeps them in a reproductive straightjacket, preventing them from increasing their numbers sufficiently for their current empire to flourish.

The Jovians crave knowledge, any knowledge at all. Their superior technology has enabled them to infiltrate the other races with bugging devices and sensors, giving them unrivalled access to information, which they use to maintain their strong position among the races. The Jovians sell a lot of their advanced technology equipment to the other races and it is this, more than anything else, which keeps the others at bay.

Jovian society is mysterious and difficult to comprehend. For this and other reasons it remains very much closed to the other races, and few foreigners reside within the Jovian Empire.

The Gallenteans. Self-righteous, meddling, pompous and tiresome, or virile liberalists and defenders of the free world. Love them or hate them, you simply can’t ignore them. Everybody has an opinion on the Gallente Federation, it all depends from which side of the table you view them. For many, it is the Promised Land, where any dream can become a reality. Descendants of Tau Ceti Frenchmen, the Gallenteans remain strong believers in free will and human rights, despite numerous setbacks in their long history.

It has been said that, once you have seen the Crystal Boulevard in Caille you’ve seen it all. True, the view is spectacular, but if there’s one thing you can never see in its entirety, that is the Gallente Federation. You may travel its length and breadth, marvel at the Sunspiral on Troux, climb the Akat Mountains on tropical Intaki or thrill to the Mendre dancers on Sovicou. Wherever you go, you will always see something new and exciting, even when you visit the same place again. Gallente society is in a constant state of flux, vigorous, vibrant and progressive.

Few societies display such stark contrasts. Many of the wealthiest people in the world are Gallenteans, creating a constant demand for luxury goods. At the same time, the ranks of the poor number millions, because while the liberal market-driven economy and individual freedom may allow everybody the chance to advance to the top, they make it just as easy to plummet to the very bottom of the social ladder.

In the world of EVE, the Gallentean are the kings of entertainment, mass-producing everything from cheap porn-flicks to elaborate stage-shows for an ever-hungry public. They boast the most elaborate luxury space yachts, and the most glittering hotel reservoirs. Anything your mind or body could ever crave, the Gallenteans have plenty of it.

The Gallenteans are not alone in their Federation, whose boundaries are home to pockets of residents, varying in size and representing all the other races of EVE, most of whom left their own empires due to political or ideological differences, or simply in search of peace and prosperity. In addition to these there are two human races, the Intakis and the Mannars, both of whom the Gallenteans found while exploring and expanding their empire. Both were at a very primitive level when the Gallenteans found them, but since coming under the protection and guidance of the Gallenteans, both races have flourished and are today a full-fledged members of the Federation.

The Caldari were initially part of the Federation but deep-seated differences and mutual animosity between them and the Gallenteans drove them out to found their own empire. For a time, the two empires warred against each other, but as neither could gain sufficient advantage to claim victory, peace was settled in the end.

A tough, no-nonsense race, the Minmatars are a determined and independent people. Their home planet of Matar is a natural paradise, although centuries of abuse have taken much from its beauty.

For the Minmatars, the most important thing in life is to be able to take care of yourself on your own, and although kin and family play an important role in their society, they prefer identifying themselves by the clan or tribe to which they belong. A clan can have any number of people in it, and its size is largely dictated by the main activity of its members. Most specialize in one area of activity. While those who live on a planet can focus on agricultural or industrial activity, others who travel around the world of EVE concentrate on trading, pirating, and suchlike.

In the distant past the clans constantly warred against each other. Since then, however, Minmatars have learned that cooperation is more important, and although the clans still try to maintain their regional and ideological identities, they act as a single unit towards other races.

The fortunes of the Minmatars have ebbed and flowed continuously. At one time they had a flourishing empire with a level of mechanical excellence never before or since seen anywhere. Later, however, they had to endure centuries of enslavement, toiling and dying for the benefit of foreign masters. Today most of them have regained their freedom, but the legacy of their enslavement has been the diaspora of the race.

The Minmatars are the most numerous of all the races in the world of EVE, but their vast numbers are divided into many factions. While the Minmatar Republic is the official state, only a quarter of all Minmatars are part of it. The largest proportion, almost a third, are enslaved within the huge Amarr Empire, while a fifth resides within the Gallente Federation, creating a powerful political bloc which keeps relations between the Gallenteans and the Amarrians in a constant state of tension. The remainder, who are not part of any formal organization, live as freemen throughout the world. Many are itinerant labourers, roaming from one system to another in search for work. A fair number make their living on the darker side of the law, acting as pirates, smugglers and peddlers in all kinds of illegal goods, and many of the larger criminal groups in the world of EVE are run by Minmatars.

The largest of the empires in the world of EVE, Amarr spans 40% of the inhabited solar systems. The Amarr Emperor is the head of a ritualistic, authoritarian imperial state, and below him are the Five Heirs, the heads of the five royal families from which a new Emperor is chosen. The Emperor’s authority is unquestioned and absolute, but the archaic and bureaucratic system of government makes it difficult for him to exert his rule unless directly in person. Otherwise, the Five Heirs rule in his name, dividing the huge empire between them.

The Emperor and the Five Heirs can expect to live for at least 500 years. Extensive cyber-implants keep their frail bodies alive, even when their organs begin to fail. These cyber-enhancements date back many millennia, and have become a symbol of royal divinity in the eyes of the Amarrians.

Always a deeply religious people, religion remains of great importance to every Amarrian, a fervour which at various times has been responsible both for great good and great evil. Shortly after recovering from the closure of EVE, they began to expand their realm at the expense of neighbouring states. The nations they conquered were enslaved, a practice justified by their religion. Ever since, the Amarrians have enslaved every nation and race they have encountered, and today slavery is an essential part of Amarr society. This has, of course, tainted their relations with other races, especially the individualistic Gallenteans.

The Amarrians were the first of the races in EVE to re-discover Warp technology, notably Jump gate technology. After accomplishing this more than 2,000 years ago, they immediately began expanding to nearby solar systems, slowly building up their empire in the process. On the way, they encountered two human races, both of whom suffered the fate of being enslaved by the far more powerful Amarrians.

In recent years, however, the Amarrians have begun to run into serious opposition. First, they met the Gallente Federation. Although much smaller than the Amarr Empire, the Amarrians soon found the economic and military might of the Gallenteans a match for their own.

Soon after, the Jovians arrived on the scene and the Amarrians made a futile attempt to subjugate them, resulting in a humiliating defeat. To make matters worse, the Minmatars, enslaved for centuries by the Amarrians, used the opportunity to rebel against their slave-masters.

Since these fateful events almost two centuries ago, the Amarrians have learned restraint. They have slowed down their expansion and are less forceful in their dealings with other races, but still view themselves as the most powerful race in EVE, if only because of their sheer numbers.