Dedicated server performance

[Sibolo]

Hi all, I'm Sibolo from the BDR and we are having lots of trouble running our DS for our ATC Campaign.
ATC is a multiclan PvP campaign with 80 players played once a week.

During tests we have done, before the start of the campaign, our old server was showing great lag problems with very low server FPS and all strange of behaviours in game (scripts failing and so on) and it was really unplayable.
We tried different missions (both light non scripted missions and the well known Valhalla) just to understand if it had to do with the mission we were using and the server kept this lagging behaviour, so our attention was focused on the server performance. We discovered to have a lot of lost packets on our server connection, and the hosting company told us that they had recently setup a filter on UDP traffic to avoid hacker attacks.
We tried playing on another clan's server and found out that although the FPS were quite low, nobody had lags or any other problem.

So we decided to get a new server on this clan's hosting company and we bought a more costly and powerful option to try and be sure we would get good performance.
This is the new server specs:

Spoiler:

show spoiler

Intel® Xeon® E3-1245 Quadcore
incl. Hyper-Threading Technology
RAM16 GB DDR3 RAM ECCHard disks2 x 3 TB SATA 6 Gb/s HDD
7200 rpm (Software-RAID 1)
Enterprise classNIC1 GBit OnBoard
connected at 100 MBitBackup Space100 GBTrafficUnlimited*

Once our server was up we started testing again to find out we had the same lag problems. I investigated on the server and found out that it was only using one of the 8 cores (I mean really using, some threads were on the other cores but they were not working intensely) and infact, although we had very low server FPS around 1-3, the maximum CPU usage was only 15%.
So reading on some very interesting threads on this forum I found out that the server should have autodetected the number of cores, but that sometimes this fails so I put manually the Exthreads=7 and cpucount=4.
This showed a very different behaviour, now the server was actually using up to three cores and the overall cpu usage went up to around 30%.
So reading further I understood that since Hyperthraeding splits cpu's in two, in this case it would have been better to switch it off so that those 3 cores could have more power to them.
So now HT is off and the server can use during games up to 75% of the cpu power.
On our game this means that the FPS has greatly improved, infact during the same mission with almost the same number of players we have higher FPS which actually stays in the 30-40 zone....but suddenly it goes down to around zero, than after a second it comes back up to high values. This gives smooth game for about 30 secs and than a big lag to everybody: unplayable.

Further investigating showed that maybe those lags can be due to sudden big I/O operations so I thought of a disk bottleneck and started looking at the possibility of using a ramdisk and tried the Fancycache program, but we are still having these problems.

During the tests we have tried fiddling around with the server performance settings and we used the default values, the same values as our friend clan (which runs smooth but with low FPS) and also values from the Kelly's Heroes site who have a nice guide on these settings.

So now we are really lost, we are still paying both servers and any of your help would be greatly appreciated.

[.kju [PvPscene]]

a) post your server.cfg and basic.cfg please
b) your high player count tests have been with 1.59 or also with 1.60
c) did you try to optimize your mission?

[Sibolo]

Thank you for your reply:

a)
Server.cfg
Spoiler:

show spoiler

HostName="ATC SERVER - {BDR} ITA Clan";
Password="xxx";
PasswordAdmin="xxxx";
reportingIP="arma2pc.master.gamespy.com"; // This is the default setting. If you change this, your server might not turn up in the public list. Leave empty for private servers
logFile="serveratc_console.log"; // Tells arma-server where the logfile should go and what it should be called
timeStampFormat=full;

motd[] = {
"", "", "",
"Benvenuto nel server BDR",
"Welcome to BDR server, welcome to Arma Tactical Combat",
" ",
"Regole/Rules:",
"Linguaggio offensivo non tollerato /Offensive language is not tolerated",
"Teamkilling non tollerato/ Teamkilling is not tolerated",
" ",
"Buon Divertimento!! / Have fun!!",
"Raggiungici sul sito : www.bdrclan.com",
"Join us on :www.bdrclan.com, www.arma-tactical-combat.com",

"",
};

motdInterval = 5;





// JOINING RULES
checkfiles[]={

};

maxPlayers=100; // Maximum amount of players. Civilians and watchers, beholder, bystanders and so on also count as player.
Kickduplicate=1; // Each ArmA version has its onw ID. If kickduplicate is set to 1, a player will be kicked when he joins a server where another player with the same ID is playing.
verifySignatures=1; // Verifies the players files by checking them with the .bisign signatures. Works properly from 1.08 on
equalModRequired=0; // If set to 1, player has to use exactly the same -mod= startup parameter as the server.



// VOTING
voteMissionPlayers=1; // Tells the server how many people must connect so that it displays the mission selection screen.
voteThreshold=0.33; // 33% or more players need to vote for something, for example an admin or a new map, to become effective



// INGAME SETTINGS
disableVoN=1; // If set to 1, Voice over Net will not be available
vonCodecQuality=0; // Quality from 1 to 10
persistent=1; // If 1, missions still run on even after the last player disconnected.



// SCRIPTING ISSUES
onUserConnected=""; // self-explaining
onUserDisconnected="";
doubleIdDetected="";
regularCheck="";
//onDifferentData="";

// some ArmA specific stuff - signature verification

onUnsignedData = "kick (_this select 0)"; // unsigned data detected
onHackedData = "kick (_this select 0)"; // tampering of the signature detected
onDifferentData=""; // data with a valid signature, but different version than the one present on server detected
BattlEye = 0; //Server to use BattlEye system

Basic.cfg:
We have tried different configs including default values without solving our problems.
The config that I retain the best choice is the one I copied off Kelly's Heroes site but I am not sure it is:

Spoiler:

show spoiler

/*
Example ArmA2 configuration file
by [KH]Jman, 8th September 2011. http://www.kellys-heroes.eu
These example numbers are for a 2.5 - 3Ghz Quad Core Xeon on a 100mBit connection.
*/


/*
Bandwidth the server is guaranteed to have (in bps).
This value helps server to estimate bandwidth available.
Increasing it to too optimistic values can increase lag and CPU load
as too many messages will be sent but discarded. Default: 131072
*/
MinBandwidth=10000000;

/*
Bandwidth the server is guaranteed to never have.
This value helps the server to estimate bandwidth available.
*/
MaxBandwidth=2147483647;

/*
Maximum number of messages that can be sent in one simulation cycle.
Increasing this value can decrease lag on high upload bandwidth servers. Default: 128
*/
MaxMsgSend = 1024;

/*
Maximum size of guaranteed packet in bytes (without headers).
Small messages are packed to larger frames.
Guaranteed messages are used for non-repetitive events like shooting. Default: 512
*/
MaxSizeGuaranteed = 1024;

/*
Maximum size of non-guaranteed packet in bytes (without headers).
Non-guaranteed messages are used for repetitive updates like soldier or vehicle position.
Increasing this value may improve bandwidth requirement, but it may increase lag. Default: 256
*/
MaxSizeNonguaranteed = 64;


/*
Minimal error to send updates across network.
Using a smaller value can make units observed by binoculars or sniper rifle to move smoother.
Default: 0.01
*/
MinErrorToSend = 0.0040000002;

/*
Users with custom faces or custom sounds larger than this size are kicked when trying to connect.
Use this wisely as it can be the cause of alot of Join in Progress lag.
1600000 = 160k
*/
MaxCustomFileSize=0;

// EOF

b)
All our tests have been done on the 1.59 stable release. We now have moved to 1.60 stable only a few tests were conducted without much improvement but further testing is needed (not easy to keep 80 people up for testing...).

c)
Surely I've done what I could to optimize the mission scripts.
These were the first beleived cause of the problem, but then we tried different missions which had already been played in the past and had the same behaviour. Also tried some light missions, and the well known DAO Valhalla missions, still having problems. So this made us think it's mainly (but not only) a server issue.

[NoBrainer]

This is just to "clean up"

Server.cfg
motd[] = {
"", "", "",
"Benvenuto nel server BDR",
..........
"", <- not the last ","
};


It's in bits pr second. An of course it's a guesstimate

Basic.cfg
MinBandwidth=41943040; (40 mbps)
MaxBandwidth=104857600; (100 mbps)

and the ones to play with is these three I think
MaxMsgSend = 1024;
MaxSizeGuaranteed = 1024;
MaxSizeNonguaranteed = 64;

And why do you don't use default in v1.60 instead of:MinErrorToSend = 0.0040000002;

But the server probably won't use more than 4MB RAM.
You could set up the server with RAMDISK and put the most used addons in that RAMDISK.
Also you could use "mklink" to make links to the addons in RAMDISK.

After that you probably have to consider the mission.

[VisceralSyn]

Basic.cfg
MinBandwidth=41943040; (40 mbps)
MaxBandwidth=104857600; (100 mbps)

40Mbps is 40000000, 100Mbps is 100000000, (b) bits.
Since he states server is connected at 100Mbps, no reason not to use it all. There still is overhead, for 100mbps, should be around 3~8Mbps
MinBandwidth=92000000;
MaxBandwidth=100000000;

and the ones to play with is these three I think
MaxMsgSend = 1024;
MaxSizeGuaranteed = 1024;
[I]MaxSizeNonguaranteed = 64;

MaxMsgSend is how many messages, can be sent in a simulation cycle. this should allow you tweak more of the bandwidth. A message cannot be larger than 1,500Bytes (B), the only time you will see a full 1,500Byte message is in LAN. So after some conservative calculations, assume message will not be larger than 1,450~1,472; we will go with 1,450Bytes. 8,316 1,450Byte messages can be sent over 92Mbps.
MaxMsgSend=8192;

Theres no real reason to not use the default MaxSizeGuaranteed, or MaxSizeNonguaranteed. Since they are less than a full message size, the message is sent once its full at 1,450Bytes.

But the server probably won't use more than 4MB RAM.

Server will never use a mere 4MB of ram, it idles using around 100MB, and gets as large as 1,200MB

...Syn...

[.kju [PvPscene]]

The server.cfg and basic.cfg look OK.

As ...Syn... points out you could tweak the MaxXXX values. The MinErrorToSend is pretty important too.
Or use the defaults by not setting the values.

However the problem is to test with 80 impatient people.

My advice would be to try to get 2-3 servers ready to test for one night.
This way you may have a higher chance for success and you can see if one server is the problem.

Also prepare a second mission you know that works as backup.
To let people play this one instead if your tests fail - to keep them happy playing something at least.


One more idea: Run the mission on your server alone for a few minutes and then use #missions.
After that upload the mpStatistics.log here.

You said the server had low fps with the 80 player, didn't you? Anyway use #monitor 1 to get the info while testing.

[Das Attorney]

You should use:

verifySignatures = 2; 

And also, this value:

regularCheck=""; 

should be:

regularCheck="{}"; 

From the Wiki:

regularCheck is also known to cause sporadic (10 mins - 2 hrs) disconnects, terminating the client with "You were kicked off the game." on the client side and "Player Test disconnected." in the console log. To turn this function off, write regularCheck="{}";. But beware, this will also make the server more prone to cheating (even though most cheats are averted when connecting)

Probably doesn't help with the slowdowns, but just stuff I noticed.

[NoBrainer]

@...Syn...

So he will guarantied get 100mbit network?
I have never seen a 100mbit get 100mbit all the time.
I know my guesstimate for 40mbit is low, but then again, is just for A2 to calculate whatever it calculates.

I think your values are far off, but you may be more correct than me. Since I only have 12-18 people and only 26mbit network.

And of course 100mbit ain't 100000000 bits, since its 1k ain't 1000, but 1024.

I see I have written 4MB, but I mean 4GB of memory. But you guys got that one.
Then he has 12 GB RAM left to play with.

And the explanation for MaxMsgSend was good. But I doubt you're numbers will hold on "da intanet". But if he get's those values you say, it would be awesome.

[Sickboy]

Wasn't the regularCheck issues supposed to be resolved/improved with v1.60?
Is this is not the case?

[Alderman]

Wasn't the regularCheck issues supposed to be resolved/improved with v1.60?
Is this is not the case?

I am wondering this myself.