Jump to content

Photo
- - - - -

Dedicated server performance


  • Please log in to reply
99 replies to this topic
Sibolo
Sibolo

    Private First Class

  • Members
  • 22 posts

#1

Posted 07 January 2012 - 11:48

Hi all, I'm Sibolo from the BDR and we are having lots of trouble running our DS for our ATC Campaign.
ATC is a multiclan PvP campaign with 80 players played once a week.

During tests we have done, before the start of the campaign, our old server was showing great lag problems with very low server FPS and all strange of behaviours in game (scripts failing and so on) and it was really unplayable.
We tried different missions (both light non scripted missions and the well known Valhalla) just to understand if it had to do with the mission we were using and the server kept this lagging behaviour, so our attention was focused on the server performance. We discovered to have a lot of lost packets on our server connection, and the hosting company told us that they had recently setup a filter on UDP traffic to avoid hacker attacks.
We tried playing on another clan's server and found out that although the FPS were quite low, nobody had lags or any other problem.

So we decided to get a new server on this clan's hosting company and we bought a more costly and powerful option to try and be sure we would get good performance.
This is the new server specs:

Spoiler


Once our server was up we started testing again to find out we had the same lag problems. I investigated on the server and found out that it was only using one of the 8 cores (I mean really using, some threads were on the other cores but they were not working intensely) and infact, although we had very low server FPS around 1-3, the maximum CPU usage was only 15%.
So reading on some very interesting threads on this forum I found out that the server should have autodetected the number of cores, but that sometimes this fails so I put manually the Exthreads=7 and cpucount=4.
This showed a very different behaviour, now the server was actually using up to three cores and the overall cpu usage went up to around 30%.
So reading further I understood that since Hyperthraeding splits cpu's in two, in this case it would have been better to switch it off so that those 3 cores could have more power to them.
So now HT is off and the server can use during games up to 75% of the cpu power.
On our game this means that the FPS has greatly improved, infact during the same mission with almost the same number of players we have higher FPS which actually stays in the 30-40 zone....but suddenly it goes down to around zero, than after a second it comes back up to high values. This gives smooth game for about 30 secs and than a big lag to everybody: unplayable.

Further investigating showed that maybe those lags can be due to sudden big I/O operations so I thought of a disk bottleneck and started looking at the possibility of using a ramdisk and tried the Fancycache program, but we are still having these problems.

During the tests we have tried fiddling around with the server performance settings and we used the default values, the same values as our friend clan (which runs smooth but with low FPS) and also values from the Kelly's Heroes site who have a nice guide on these settings.

So now we are really lost, we are still paying both servers and any of your help would be greatly appreciated.

.kju -PvPscene-
.kju -PvPscene-

    Brigadier General

  • Members
  • 12275 posts

#2

Posted 14 January 2012 - 09:31

a) post your server.cfg and basic.cfg please
B) your high player count tests have been with 1.59 or also with 1.60
c) did you try to optimize your mission?



Current active projects: None :(

Maintained/assisted projects: IFA3, Blitzkrieg


Help: Got a crash? Report it! What is the RPT log file?


Sibolo
Sibolo

    Private First Class

  • Members
  • 22 posts

#3

Posted 14 January 2012 - 11:30

Thank you for your reply:

a)
Server.cfg
Spoiler


Basic.cfg:
We have tried different configs including default values without solving our problems.
The config that I retain the best choice is the one I copied off Kelly's Heroes site but I am not sure it is:

Spoiler


B)
All our tests have been done on the 1.59 stable release. We now have moved to 1.60 stable only a few tests were conducted without much improvement but further testing is needed (not easy to keep 80 people up for testing...).

c)
Surely I've done what I could to optimize the mission scripts.
These were the first beleived cause of the problem, but then we tried different missions which had already been played in the past and had the same behaviour. Also tried some light missions, and the well known DAO Valhalla missions, still having problems. So this made us think it's mainly (but not only) a server issue.

NoBrainer
NoBrainer

    Gunnery Sergeant

  • Members
  • 523 posts

#4

Posted 15 January 2012 - 20:19

This is just to "clean up"

Server.cfg

motd[] = {
"", "", "",
"Benvenuto nel server BDR",
..........
"", <- not the last ","
};


It's in bits pr second. An of course it's a guesstimate

Basic.cfg
MinBandwidth=41943040; (40 mbps)
MaxBandwidth=104857600; (100 mbps)

and the ones to play with is these three I think
MaxMsgSend = 1024;
MaxSizeGuaranteed = 1024;
MaxSizeNonguaranteed = 64;

And why do you don't use default in v1.60 instead of:MinErrorToSend = 0.0040000002;

But the server probably won't use more than 4MB RAM.
You could set up the server with RAMDISK and put the most used addons in that RAMDISK.
Also you could use "mklink" to make links to the addons in RAMDISK.

After that you probably have to consider the mission.
Artificial intelligence is no match for natural stupidity

NoPryl of Norway

VisceralSyn
VisceralSyn

    Sergeant

  • Members
  • 190 posts

#5

Posted 15 January 2012 - 22:23

Basic.cfg
MinBandwidth=41943040; (40 mbps)
MaxBandwidth=104857600; (100 mbps)


40Mbps is 40000000, 100Mbps is 100000000, (B) bits.
Since he states server is connected at 100Mbps, no reason not to use it all. There still is overhead, for 100mbps, should be around 3~8Mbps
MinBandwidth=92000000;
MaxBandwidth=100000000;

and the ones to play with is these three I think
MaxMsgSend = 1024;
MaxSizeGuaranteed = 1024;
[I]MaxSizeNonguaranteed = 64;


MaxMsgSend is how many messages, can be sent in a simulation cycle. this should allow you tweak more of the bandwidth. A message cannot be larger than 1,500Bytes (B), the only time you will see a full 1,500Byte message is in LAN. So after some conservative calculations, assume message will not be larger than 1,450~1,472; we will go with 1,450Bytes. 8,316 1,450Byte messages can be sent over 92Mbps.
MaxMsgSend=8192;

Theres no real reason to not use the default MaxSizeGuaranteed, or MaxSizeNonguaranteed. Since they are less than a full message size, the message is sent once its full at 1,450Bytes.

But the server probably won't use more than 4MB RAM.

Server will never use a mere 4MB of ram, it idles using around 100MB, and gets as large as 1,200MB

...Syn...

.kju -PvPscene-
.kju -PvPscene-

    Brigadier General

  • Members
  • 12275 posts

#6

Posted 16 January 2012 - 07:08

The server.cfg and basic.cfg look OK.

As ...Syn... points out you could tweak the MaxXXX values. The MinErrorToSend is pretty important too.
Or use the defaults by not setting the values.

However the problem is to test with 80 impatient people. :)

My advice would be to try to get 2-3 servers ready to test for one night.
This way you may have a higher chance for success and you can see if one server is the problem.

Also prepare a second mission you know that works as backup.
To let people play this one instead if your tests fail - to keep them happy playing something at least.


One more idea: Run the mission on your server alone for a few minutes and then use #missions.
After that upload the mpStatistics.log here.

You said the server had low fps with the 80 player, didn't you? Anyway use #monitor 1 to get the info while testing.



Current active projects: None :(

Maintained/assisted projects: IFA3, Blitzkrieg


Help: Got a crash? Report it! What is the RPT log file?


das attorney
das attorney

    Warrant Officer

  • Members
  • 2955 posts

#7

Posted 16 January 2012 - 07:41

You should use:

verifySignatures = 2;


And also, this value:

regularCheck="";


should be:

regularCheck="{}";


From the Wiki:

regularCheck is also known to cause sporadic (10 mins - 2 hrs) disconnects, terminating the client with "You were kicked off the game." on the client side and "Player Test disconnected." in the console log. To turn this function off, write regularCheck="{}";. But beware, this will also make the server more prone to cheating (even though most cheats are averted when connecting)


Probably doesn't help with the slowdowns, but just stuff I noticed. :)
Oblig sig

NoBrainer
NoBrainer

    Gunnery Sergeant

  • Members
  • 523 posts

#8

Posted 16 January 2012 - 13:05

@...Syn...

So he will guarantied get 100mbit network?
I have never seen a 100mbit get 100mbit all the time.
I know my guesstimate for 40mbit is low, but then again, is just for A2 to calculate whatever it calculates.

I think your values are far off, but you may be more correct than me. Since I only have 12-18 people and only 26mbit network.

And of course 100mbit ain't 100000000 bits, since its 1k ain't 1000, but 1024.

I see I have written 4MB, but I mean 4GB of memory. But you guys got that one.
Then he has 12 GB RAM left to play with.

And the explanation for MaxMsgSend was good. But I doubt you're numbers will hold on "da intanet". But if he get's those values you say, it would be awesome.
Artificial intelligence is no match for natural stupidity

NoPryl of Norway

sickboy
sickboy

    Colonel

  • Members
  • 9947 posts

#9

Posted 16 January 2012 - 13:16

Wasn't the regularCheck issues supposed to be resolved/improved with v1.60?
Is this is not the case?

Alderman
Alderman

    Staff Sergeant

  • Members
  • 276 posts

#10

Posted 17 January 2012 - 06:31

Wasn't the regularCheck issues supposed to be resolved/improved with v1.60?
Is this is not the case?


I am wondering this myself.
Executive Director / Chairman
Military Gaming Organization
http://militarygaming.org
Ph: (800) 601-2798 - Fax: (800) 601-2948

eddieck
eddieck

    Staff Sergeant

  • Members
  • 373 posts

#11

Posted 17 January 2012 - 07:25

Wasn't the regularCheck issues supposed to be resolved/improved with v1.60?
Is this is not the case?


It was supposed to be, but in my testing with the current 1.60 Linux DS, things have only improved slightly. It still randomly kicks legitimate players and causes a significant amount of desync.

I haven't created a CIT ticket because I don't have any actual information on it - no logs or anything that could indicate the source of the problem.

.kju -PvPscene-
.kju -PvPscene-

    Brigadier General

  • Members
  • 12275 posts

#12

Posted 17 January 2012 - 07:55

Do you have -netlog running (does that work in linux machines even)?



Current active projects: None :(

Maintained/assisted projects: IFA3, Blitzkrieg


Help: Got a crash? Report it! What is the RPT log file?


ArmaholicBR
ArmaholicBR

    Corporal

  • Members
  • 61 posts

#13

Posted 17 January 2012 - 08:39

I'm also having a performance problem with my server running RP missions..

I start a mission and everything seems fine...checking with monitor and server fps stays at 48 steady all the time, the it suddenly goes to 0-5fps like "freezing" and dsyncing players, and it comes back, sometimes really quick, sometimes not (2-15sec).

Cpu load stays under 50% all the time, then it drops to zero during that "freeze" period, and comes back again..I've checked rpt files and it all seems fine.

Has anyone seen this?

Spoiler


It's annoying and happens every couple minutes..

Edited by ArmaholicBR, 17 January 2012 - 08:45.


eddieck
eddieck

    Staff Sergeant

  • Members
  • 373 posts

#14

Posted 17 January 2012 - 09:28

Do you have -netlog running (does that work in linux machines even)?


It works on the Linux DS.

I'm also having a performance problem with my server running RP missions..


Use the Linux DS if you're running ZL.

Sibolo
Sibolo

    Private First Class

  • Members
  • 22 posts

#15

Posted 17 January 2012 - 11:04

I've read all your posts paying a lot of attention to all the matters.

One thing that really interested me, following your suggestion about the regularCheck command is this:

From the BIS WIKI:

If you do not include the regularCheck option or set regularCheck=""; it will be activated. This means the server checks files from time to time by hashing them and comparing the hash to the hash values of the clients. Since newer server versions this has lead to some lag spikes on certain systems, because the whole file is hashed in one burst. (The heavy I/O operation essentially blocking the whole server application for 1-5 secs, depending on the file size)
regularCheck is also known to cause sporadic (10 mins - 2 hrs) disconnects, terminating the client with "You were kicked off the game." on the client side and "Player Test disconnected." in the console log. To turn this function off, write regularCheck="{}";. But beware, this will also make the server more prone to cheating (even though most cheats are averted when connecting)


The description of the effects of the regularCheck on "some" systems is amazingly similar to what I've come up with in fact my own description of the problem in the first post was:
"we have higher FPS which actually stays in the 30-40 zone....but suddenly it goes down to around zero, than after a second it comes back up to high values. This gives smooth game for about 30 secs and than a big lag to everybody: unplayable.
Further investigating showed that maybe those lags can be due to sudden big I/O operations so I thought of a disk bottleneck and started looking at the possibility of using a ramdisk and tried the Fancycache program, but we are still having these problems."


How can I disable regularCheck?
Is it the regularCheck="{}"; that Das Attorney suggested?



Thank you.

ArmaholicBR
ArmaholicBR

    Corporal

  • Members
  • 61 posts

#16

Posted 17 January 2012 - 14:44

Use the Linux DS if you're running ZL.


No way to fix the "freezing" under Win2003? I can't format the server :(

I have an aplicaion running there that needs windows..

FSF_TargetZero
FSF_TargetZero

    Private First Class

  • Members
  • 32 posts

#17

Posted 17 January 2012 - 18:11

Server.cfg

persistent=1; // If 1, missions still run on even after the last player disconnected.

This option impacts high performances.
Clan FSF : www.clan-fsf.fr

Posted Image

eddieck
eddieck

    Staff Sergeant

  • Members
  • 373 posts

#18

Posted 17 January 2012 - 21:12

No way to fix the "freezing" under Win2003? I can't format the server :(


I wasn't able to fix it. The only thing I can suggest is to try changing -cpucount and -exthreads, but that may not work.

ArmaholicBR
ArmaholicBR

    Corporal

  • Members
  • 61 posts

#19

Posted 17 January 2012 - 21:22

Server.cfg

persistent=1; // If 1, missions still run on even after the last player disconnected.

This option impacts high performances.


Yes, persistent is turned on. The problem with performance could happen, but i think it isnt whats causing it.. As you can see, cpu doesnt even reach 50% load. Even with noone playing, i can see that the freeze still occurs, coming back to Life in a few seconds..

gonk
gonk

    First Sergeant

  • Members
  • 968 posts

#20

Posted 19 January 2012 - 12:33

I get this problem also (server)... after awhile it looks like the game migrates to one cpu... I have had cpu 1 go to max after a while and then re-run the same mission I have had cpu 4 go to max... very strange and I am interested in a tweak to get around this...
Intel Core i7 980X EXTREME, 24 Gig RAM, 240 Gig Revo x 2 SSD, 2 X EVGA (SC) 680GTX's, Win 7 64 Pro,
Single 27 inch Monitor, Oculus Rift V 1.0, Track IR 5, HOTAS Warthog, SIMPEDS Pedals..Arma3 Ready... not....:(