Annaro – Confirmation that the Ducia Foundry team deployed to Annaro system included slave workers emerged last night, sparking controversy and overshadowing the successful rescue of 176 miners from the collapsed CBD Corporation asteroid mine. The Caldari Business Tribunal is reported to have been alerted by individuals within the Annaro police; they are investigating allegations that corporate and civil codes were broken by the local police corporation.

According to reports, the rescue team included 16 slaves, who carried out dangerous work on the surface of the asteroid. Those slaves have since been purchased and released by Amarr Holder Aszhar Freimala, in accordance with an agreement between Freimala, Ducia Foundry, and Annaro police commander Garr Diymir. Despite this agreement, the venture was still illegal under Caldari law, as the individuals in question were slaves during the rescue operation.

Sources within the Annaro police force have alleged that Commander Diymir gave his authority for the illegal venture to proceed and ordered several police patrols to change course away from the Amarr team’s ships and the asteroid. Tribunal investigators are interviewing senior officers in the Annaro police corporation. Commander Diymir is reported to have been suspended from duty, though he is cooperating with investigators. With the investigation appearing to be aimed at prosecuting Garr Diymir, local miners have begun a fund to help pay for his legal costs.

ZYL-FT, Geminate – The WI. deployment away from Geminate has left the region up for grabs by several new entrants, among which are Red Alliance [RED], SOLAR FLEET [SOLAR] and Ad Astra. Some of the allies of Atlas Alliance [ATLAS] in the war effort over the past few months are now settling into their newly claimed territories and using its resources to strengthen their positions.

With a long history of involvements in this area, Ad Astra. collaborated with ATLAS during the war and was in charge of opening up a second front against Wildly Inappropriate. [WI.] in “West” Geminate. They have now settled in the ZYL-FT Constellation. Paulo fr of Ad Astra. reported “We didn’t fleet with ATLAS/Cult of War/Gentlemen’s Club. The goal was to put pressure on WI. in another [part] of Geminate.” In that way, paulo explained, Ad Astra. could use the 250-ship fleets resulting from Atlas “Calls to Arms” to inflict as much damage as possible on Wildly Inappropriate’s infrastructure, destroying or reinforcing their towers. “We… kept pressure on them, R.A.G.E and Majesta Empire by roaming… preventing their pilots from making money.”

On the 26.01.112, the last three former WI. outposts in Geminate were taken over by Red Alliance and SOLAR FLEET, while Atlas has turned to the systems which had been occupied by Get Off My Lawn after WI.’s departure, particularly O-VWPB, which houses a valuable Technetium-yielding moon.

Red Alliance member Lux Nifertari explained that his alliance was blue to Atlas and SOLAR during the conflict. “ATLAS wanted fun [while] we and SOLAR wanted space.” Lux added that with their main goal achieved, RED now plans on using its part of Geminate to “power up” the Alliance. “People may ask – why Geminate? The answer is simple: we were allied with the Northern Coalition and tried to help all we could, but they left us without any help versus ATLAS in Insmother.”

In Insmother, Red Alliance undertook an aggressive campaign against some of the alliances affiliated with Atlas, until they were offered help by their former enemies. “They made the offer to help us handle some space. We connected with SOLAR and decided to attack Geminate. [People thought that] RED was dead [so we] attacked Geminate [to show that] RED is alive.”

According to Lux, RED plans on staying in Geminate for as long as possible and are currently focused on keeping blue standings in the region to allow the farming of resources.

Paulo fr of Ad Astra. confirmed that their main goal had been the removal of Wildly Inappropriate. from Geminate, but that his alliance would maintain blue standings with RED and SOLAR while they “make Geminate safer for all of us.” He further stated “[We plan] to reset standings soon and become nice neighbours.”

The future doesn’t look entirely rosy to paulo fr, however. “R.A.G.E and Majesta Empire should be back soon… the NC blob will be back [once] they are finished in Pure Blind. Pandemic Legion has also moved in to the V-MZW0 station. We prefer to save money for further campaigns and want to be there for the final fight for Geminate.”

GalNet References
Wildly Inappropriate Alliance Redeploys to Pure Blind

Following the release of the Dominion expansion we started to get reports from players about an increase in “fleet lag.” Before we go any further, I would like to assure everyone that a taskforce of developers has been working on this continuously, but tracking down the source has proven a very difficult task.

So, what is the problem?
“Lag” is an ambiguous term and can refer to many things. The graphics in the client might be slowing it down, the network might be congested or a host of other issues may be coming into play. We have spent a long time trying to reproduce the issue on the test servers as well as monitoring fights as they have been unfolding on Tranquility, where we have received help from the players in the form of testing and bug reports. You might see names like “CCP Atlas”, “CCP GingerDude” and “CCP Warlock” poking about in Local. Don’t be alarmed, we are simply monitoring the node, looking for glitches. There are senior software engineers spending their evenings and weekends on Tranquility monitoring and measuring since we have not been able to see issues when testing because the sheer number of people required to trigger the problems and the specific ‘usage pattern’ of the game systems in question are outside what can currently be tested in the lab (although this is something that is being worked on).

What we are looking at now specifically is the fact that the performance of the sol nodes (the computers in the clusters handling the solar systems) appears to be worse than before. Even though CPU across the cluster has not increased with the Dominion expansion, each reinforced node is no longer able to handle the same number of people fighting as gracefully as it did before.

Additionally, we have the issue of the grid not loading when jumping into a system. Being stuck on the source side and not getting through the gate is one thing since you’re not in the battle yet, but if you actually get through the gate and arrive on the other side without managing to load the grid you could be at a disadvantage and possibly vulnerable because of your client’s inability to act. This can be detrimental to the fleet jumping into the system.

What is happening is that the client is requesting information about the simulation from the server but it’s not receiving it in a timely manner. There may be many minutes before the information is finally received as it is getting stuck in the lower levels of the sol node.

Anatomy of “grid not loading”
When the specific scenario of “grid not loading” occurs, its fingerprint on your client is quite specific (even though sadly we cannot see the effects in the server logs).

Refer to the following handy diagram to determine whether you have encountered the problem:

CLICK FOR LARGER VIEW

This is what it would typically look like if you’re jumping into a system which exhibits the issue. However, if you are logging into such a system and encounter this issue you will only have a black screen and a “loading” bar.

If we look at the image a bit you will see that the local channel, the right click menu and the system information are all updated to the new system while the nebula is that of the old system. Your overview will also be empty and no brackets in space. If you open up the monitor (control-shift-alt-m) it will have (at least) one outstanding call. This is the request for the simulation instance.

When you see this, the worst thing you can do is to click any buttons or type in local. It will only exacerbate the problem. Relogging will also not help and might even make the situation worse. The best you can do is wait it out (see the “Fleet fight survival guide” below).

Light at the end of the tunnel
We were finally able to reproduce this with a special fleet test on our Singularity test server on Jan 27th where over 400 people participated in helping us out. That was really fantastic to see and I would like to thank everyone who showed up.

On Jan 28th, we had a 1600 person fleet fight on Tranquility which our team monitored closely, keeping the node alive using methods that make our system admins faint. This was one of the biggest, if not the biggest, fleet fight ever in EVE (at least where the node survived the ordeal). This event allowed us to identify what was causing some of these glitches and deploy fixes live.

We have added a stop-gap measure in place which keeps nodes from dying during these high stress scenarios and we have some more permanent solutions in the pipes that will be deployed over the next few weeks.

How can I help?
There is a mass test scheduled for the Singularity test server on Thursday Feb 4th at 16:00 Eve time. Those of you who wish to help out, please come and provide us with client logs when you encounter the “grid not loading” problem or other issues. We will specifically be testing jumping in several solar systems trying out a few of the software solutions that have been implemented.

This requires running Logserver.exe from the client installation folder before you start the client. Your logs are extremely important, and each bug report you submit with this information will go a long way toward bringing us closer to a resolution or fix for this issue.

There is more information on this fleet test in this thread.

Additionally, if you are planning to attack a system we also highly recommend that you notify us at least 24 hours beforehand through the Fleet Fight Notification Form so that the system can be placed on a reinforced node. This will also give us the chance to mobilize people to be in the solar system with measuring devices firmly attached to our Polaris frigates.

Fleet Fight Lag Survival Guide
I’d like to talk a little bit about the best tactics we’ve found to use during periods of extreme server-side lag and methods to make the experience a little more bearable. Note that I am talking about server-side lag here, not low FPS or bad network conditions (for example, if you find yourself in a fleet fight on your 9600bps cell phone modem up on a glacier somewhere). The lag that this refers to is when there are hundreds of players fighting in a solar system and everyone is experiencing delays, where guns could take minutes to fire and there’s some gnarly rubberbanding. If you’ve encountered this type of lag you typically know the symptoms. It’s under these circumstances that the “grid not loading” issue manifests itself for example.

Do not hammer that button!
One of the most important things to keep in mind when you are in a system with heavy lag is that you should not hammer any buttons and you should try to execute as few commands as possible. If you are waiting for the grid to load the worst thing you can do is click buttons on the interface or chat in local. These actions will have a negative impact on your client’s ability to recover (more to the point, on your session on the server). If you have an outstanding call you should wait for it to complete, and it will complete eventually. It might take minutes but your gun will fire and clicking the button again will not help. Do not browse the market or look up contracts while waiting for the system to load. I know it’s hard but you have to leave the client alone while it waits for the calls to complete or you will add to the problem and can even cause the session to get stuck.

Use the Monitor
The network tab in the ingame monitor debugging tool (control-shift-alt-m) can help you understand whether your calls are completing in a timely manner. If there is a number in the “Outstanding” row that means that the server is busy executing your command. Wait for it to complete before executing another one!

Calls do time out
What I said about calls completing eventually is true… from a certain point of view. The call will be handled eventually by the server but it’s possible that the client has given up by then. Calls typically time out in 8 minutes and this is true of the request for the simulation instance (grid). If you have waited for 8 minutes for the grid to load you will not recover. If you have the monitor open and see the Outstanding number drop to 0 and your grid doesn”t load then you have no choice but to relog, and even that might not fix things if the server is still overloaded.

Note that if you get podded while in this state then the client should recover when your session is moved into your clone station.

Final words
We have come a long way in understanding the issues that have plagued large fights after Dominion and have gotten important insight into fleet fight lag as a whole. The problems we are experiencing post-Dominion are being worked on and they will be fixed soon, adding more space carnage to your battles.

I would like to ask that we keep the discussion thread attached to this blog productive and try to keep flaming to a minimum.

A Word from Customer Support
Support’s intervention in large fleet engagements in even a remotely fair manner is very problematic due to the complexity of the situation.

Under circumstances where lag is involved or elusive bugs that are difficult to establish as the deciding factor, CCP cannot justify passing judgment on who should have won or who should have lost. While a problem such as the one in focus here is known to affect some players in certain situations, the evidence available in server-side logs, when available at all, can be haphazard and arbitrary. As such, intervention on CCP’s part would risk being arbitrary as well.

Please understand that fleet fight reimbursement has always been very controversial and few issues have been discussed and argued in more detail within CCP. The conclusion has, however, always been to leave fleet battles alone rather than reimbursing them as a whole. We apologize for any inconvenience this may cause and we appreciate your patience as we work diligently towards resolving this issue.

Sahtogas, The Bleak Lands – The Amarr system of Sahtogas fell to Minmatar occupancy yesterday at 14:55.

Sahtogas controls the only access point to the Amarr system of Haras, occupancy of which is currently fiercely contested between the Minmatar Tribal Liberation Force and the Amarr 24th Imperial Crusade.

The solar system is reported to house considerable mineral wealth, with surveys suggesting the presence of Promethium and several other valuable metals in the local moons.

While 24th Imperial Crusade forces had resisted Minmatar invasion of the system by protecting defensive installations throughout Tandoiras constellation, the final siege of the 24th Imperial Crusade bunker in the system slipped by with no active resistance and its fall passed unannounced on capsuleer GalNet channels.

31-MLU, Syndicate – Near 06:00 this morning, over 100 corporations were expelled from the GoonSwarm [OHGOD] alliance. Issachiar, pilot of a Hel-class supercarrier for GoonFleet, was caught by the dissolution of his alliance while within the shields of a starbase belonging to Battlestars corporation. The Hel was expelled from the shields where waiting forces from Cry Havoc [C H] fell upon the vessel.

When corporations began dropping out of GoonSwarm, Cry Havoc began receiving automated surrender messages. Mduncan0341 of C H described how those messages became a supercapital kill for him and his wingmates.

“We were on a roam, when we noticed Goonswarm alliance disbanding – our mailboxes were being flooded with surrender notices, since we have an active wardec against them. We ran back to Orvolle to ship into [remote repair battleships] since we knew they had a supercap staging [tower] in 31-MLU and we knew caps would bounce as they [arrived].”

While GoonFleet members VCtab and Apples Appleby claimed the Hel’s death was a desperate, deliberate and suicidal act from a pilot devasted by the loss of his alliance, eyewitnesses in Cry Havoc stated that the supercarrier was repelled by the Battlestars’ tower’s shield and recoiled a substantial distance, at least in terms of supercapital speeds – an accidental casualty of GoonSwarm’s shattering.

While the Hel at first managed to evade the C H fleet as they jumpbridged in, the ship was uncloaked at a nearby moon by a scouting Dramiel. It was tackled and the Cry Havoc raiding fleet quickly called in their standby capitals, including a pair of titans and a supercarrier of their own.

The former GoonSwarm corporations sent multiple waves of ships in an attempt to break the Hel out from the interdiction the C H fleet had put upon it, using Interdictor warp disruption “bubbles” and the infinite strength warp scrambling of Heavy Interdictors. The now-combined fleet from Cry Havoc and allies Ev0ke, however, was able to destroy the Hel in approximately 6 minutes and turn their attention to evacuating their own supercapitals before a heavy fleet from GoonSwarm’s so-called “CapSwarm” could respond. In the end, C H were able to depart the field with very light losses.

The expulsion of GoonSwarm’s corporations has caused the collapse of the alliance’s remaining sovereignty in Delve and Querious. (Other systems were lost on 26.01.112 when GoonFleet corporation was unable to make necessary upkeep payments.)

Multiple sources within the former GoonSwarm corporations have claimed that the corporations were expelled by karttoon, the former alliance leader and CEO of GoonFleet. Karttoon was allegedly one of three people who could have transferred funds to the correct division of the right corporate wallet to make the sovereignty payments, but he was absent at the time, as were the two other officers with wallet access.

Another source in GoonFleet speculated that mass demands for karttoon’s resignation in the wake of the lapsed upkeep payments in Delve may have motivated his alleged actions.

In addition to expelling the GoonSwarm corporations from the alliance shell, karttoon is alleged to have emptied the corporate and alliance wallet divisions that he could access, taking sums reported varyingly as 300 – 500 billion ISK in liquid funds and a possible trillion ISK in assets.

Apples Appleton claimed karttoon then went on to detonate several ships from the alliance dreadnought cache in order to claim the insurance, but other sources have queried that descriptions of the destroyed ships showed that their names spelled out an insulting message to karttoon.

In the wake of GoonSwarm’s dissolution into its constituent corporations, there is much speculation on what they will do next. A reported statement by Darius JOHNSON, former GoonSwarm leader and likely leadership candidate for any replacement, says:

“As many of you may have noticed we no longer have an alliance. The very gentleman who is CEO of fleet is the likely culprit so we’ll be dropping Goonfleet as well. Goonfleet is dead. Goonswarm is dead. We will now bask in the glory of Goonwaffe.”

GalNet References
Cry Havoc Battle Record
Lapsed Upkeep Payments Remap Null-Sec Sovereignty