Jump to content
Sign in to follow this  
.kju

PvPscene Benchmark Suite

Recommended Posts

= PvPscene Benchmark Suite

The goal is to create a better system to benchmark the ArmA engine to

understand the effects of different game video settings, different hardware

components and system configuration.

The approach is to improve the suite step by step by adding

new benchmark scenarios, improving the benchmark system

and automating the process more and more.

The system is largely based on the work and insights from Mac

and his ArmA2 FPS analyzer - benchmarking tool.

Please show him your gratitude and motivate him to update the tool

to OA and extend the functionality. :)

The suite can be run fully automated and logs the result into the rpt file.

Sample output:

"_Preloading - PvPscene_BM_City (Chernarus)"
"avg fps	avg min	highest	lowest"
"20	14	26	5"

The suite contains 12 missions so far and takes 25 minutes to process.

You can also run a subset with configuration or copy the benchmark missions

to arma2\missions and run them one by one as singleplayer mission.

== Current scenarios

The current scenarios focus on the benchmarking the rendering only.

In other words no AI and no interaction - a simple camera scene moving along

some waypoints twice:

1) First a preloading run with faster speed.

2) Followed by the same run in normal speed.

All given scenarios enforce very high terrain detail (most dense grass/clutter

with highest radius) and 3000 viewdistance (10k for LongVD) to put enough

pressure on GPU and CPU.

The types so far are:

  • City
  • Coastline
  • OpenFields
  • Woods
  • LongViewDistance

on Chernarus, Takistan, Zargabad and ProvingGrounds.

== Possible new scenarios

  • Camera to zoom in and out to simulate RMB zoom.
  • Camera that simulates the optic view.
  • Non combat AI scenarios with different numbers of AI (10, 100, 250, 500, 1000).

== Install guide

1) Download the suite.

(file section at: http://dev-heaven.net/projects/clanbase/files)

2) Extract to the arma2 folder, like

c:\arma2\autotest\PvPscene_BM

3) Configure your arma2 path in _configureSettings.bat, like

(right click the _configuresettings file and choose edit)

set PATHTOARMA=c:\arma2

You can also select the set of missions to be run in there.

As a start it is best to run the complete suite.

You can also set custom startup parameters, if you want to:

set CUSTOMPARAMS="-profiles=c:\arma2" -window "-name=test"

4) Start the suite with the run.bat or runBeta.bat in .\arma2\autotest\PvPscene_BM.

5) Wait for the benchmarks to finish and check the rpt for the result.

6) You can run the suite again with different settings and compare the results yourself

or submit your results to the online database (no registration required).

== Notes

If you run arma in window mode, you have a look at the GPU use

while the benchmarks are run with a tool like MSI Afterburner.

You can follow the CPU use, if you have the task manager running with the

performance tab open.

=== Online Database

Please provide your values the following graphic settings along with your test results

to make the comparison between different systems and graphic settings useful.

For some graphic settings, you need to look in the arma2.cfg.

To find out your hardware, you can use the tool PC-Wizard for example.

To make the copy-paste action less effort, remove the duplicate lines in the rpt first.

You can do this with EditPadPro (trial/lite) for example nicely.

Just open the rpt / drag it into the text editor window, select in the menu 'extra',

and in the middle 'remove duplicate lines'.

The abbreviations for the graphic settings values are:

  • D - Disabled/Default
  • VL - Very Low
  • L - Low
  • N - Normal
  • H - High
  • VH - Very High

Edited by .kju [PvPscene]

Share this post


Link to post
Share on other sites

Lots of time? :) Download, unpack, configure - once. Can be done in 2 minutes.

The benchmarks are fully automated - you can go watch TV, eat or whatever meanwhile.

If you want a meaningful benchmark, you found one here.

Edited by .kju [PvPscene]

Share this post


Link to post
Share on other sites

:j:

there are 420 fields to fill out? is that a random number or by design? ;)

my benchmark results reflect gfx setting i use in normal gameplay.

Version 1.57.76815

Item str_disp_server_control listed twice

Warning: looped for animation: ca\wheeled\data\anim\uaz_cargo01_v0.rtm differs (looped now 0)! MoveName: kia_uaz_cargo02

Warning: looped for animation: ca\wheeled\data\anim\uaz_cargo01_v0.rtm differs (looped now 1)! MoveName: uaz_cargo02

"Type avg fps avg min highest lowest"

"Preloading 33 25 45 13 PvPscene_BM_City (Chernarus)"

"Type avg fps avg min highest lowest"

"Benchmark 37 30 52 12 PvPscene_BM_City (Chernarus)"

"Type avg fps avg min highest lowest"

"Preloading 42 32 62 10 PvPscene_BM_Coastline (Chernarus)"

"Type avg fps avg min highest lowest"

"Benchmark 45 36 68 14 PvPscene_BM_Coastline (Chernarus)"

"Type avg fps avg min highest lowest"

"Preloading 44 31 59 6 PvPscene_BM_OpenFields (Chernarus)"

"Type avg fps avg min highest lowest"

"Benchmark 47 37 72 8 PvPscene_BM_OpenFields (Chernarus)"

"Type avg fps avg min highest lowest"

"Preloading 46 33 59 11 PvPscene_BM_Woods (Chernarus)"

"Type avg fps avg min highest lowest"

"Benchmark 46 34 59 13 PvPscene_BM_Woods (Chernarus)"

"Type avg fps avg min highest lowest"

"Preloading 47 34 65 9 PvPscene_BM_City (takistan)"

"Type avg fps avg min highest lowest"

"Benchmark 51 43 65 11 PvPscene_BM_City (takistan)"

"Type avg fps avg min highest lowest"

"Preloading 45 28 66 12 PvPscene_BM_OpenFields (takistan)"

"Type avg fps avg min highest lowest"

"Benchmark 46 30 58 17 PvPscene_BM_OpenFields (takistan)"

"Type avg fps avg min highest lowest"

"Preloading 50 33 65 9 PvPscene_BM_Woods (takistan)"

"Type avg fps avg min highest lowest"

"Benchmark 52 36 68 18 PvPscene_BM_Woods (takistan)"

"Type avg fps avg min highest lowest"

"Preloading 41 32 45 11 PvPscene_BM_LongViewDistance (takistan)"

"Type avg fps avg min highest lowest"

"Benchmark 41 34 45 26 PvPscene_BM_LongViewDistance (takistan)"

"Type avg fps avg min highest lowest"

"Preloading 39 28 48 12 PvPscene_BM_City (Zargabad)"

"Type avg fps avg min highest lowest"

"Benchmark 42 34 70 14 PvPscene_BM_City (Zargabad)"

"Type avg fps avg min highest lowest"

"Preloading 36 23 49 13 PvPscene_BM_OpenFields (Zargabad)"

"Type avg fps avg min highest lowest"

"Benchmark 38 27 47 15 PvPscene_BM_OpenFields (Zargabad)"

"Type avg fps avg min highest lowest"

"Preloading 25 22 27 13 PvPscene_BM_LongViewDistance (Zargabad)"

"Type avg fps avg min highest lowest"

"Benchmark 25 23 28 19 PvPscene_BM_LongViewDistance (Zargabad)"

"Type avg fps avg min highest lowest"

"Preloading 33 21 38 13 PvPscene_BM_OpenFields (ProvingGrounds_PMC lite)"

"Type avg fps avg min highest lowest"

"Benchmark 33 22 44 8 PvPscene_BM_OpenFields (ProvingGrounds_PMC lite)"

Edited by [DirTyDeeDs]-Ziggy-

Share this post


Link to post
Share on other sites

Hey guys might be a stupid question but i can't set my shortcut up, can anyone give me a step by step on how to?

Edited by AC337

Share this post


Link to post
Share on other sites
Start the suite with the run.bat or runBeta.bat.
from within the ArmA 2/autotest/PvPscene_BM folder

also

right click the _configuresettings file and choose edit

looks like this

help-1.jpg

Edited by [DirTyDeeDs]-Ziggy-

Share this post


Link to post
Share on other sites
-Ziggy-;1858307']from within the ArmA 2/autotest/PvPscene_BM folder

also

right click the _configuresettings file and choose edit

looks like this

Thanks alot man!

Edited by Dwarden

Share this post


Link to post
Share on other sites

Nice work on this benchmark suite!

Since it forces high terrein details, and also VD, maybe make it enforce all

in game video options, and create .bats for a few different resolutions?

=====================================================================

== c:\arma2\arma2oa.exe

== "c:\arma2\arma2oa.exe" -nosplash -skipintro -world=empty -noFilePatching "-name=test" "-autotest=c:\arma2\autotest\PvPscene_BM\MissionSets\autotestAll.cfg"

=====================================================================

Exe timestamp: 2011/01/23 14:36:06

Current time: 2011/02/18 09:56:37

Version 1.57.76815

Item str_disp_server_control listed twice

Warning: looped for animation: ca\wheeled\data\anim\uaz_cargo01_v0.rtm differs (looped now 0)! MoveName: kia_uaz_cargo02

Warning: looped for animation: ca\wheeled\data\anim\uaz_cargo01_v0.rtm differs (looped now 1)! MoveName: uaz_cargo02

"Type avg fps avg min highest lowest"

"Preloading 42 32 52 13 PvPscene_BM_City (Chernarus)"

"Type avg fps avg min highest lowest"

"Benchmark 50 41 72 15 PvPscene_BM_City (Chernarus)"

"Type avg fps avg min highest lowest"

"Preloading 70 60 95 18 PvPscene_BM_Coastline (Chernarus)"

"Type avg fps avg min highest lowest"

"Benchmark 74 66 99 26 PvPscene_BM_Coastline (Chernarus)"

"Type avg fps avg min highest lowest"

"Preloading 72 60 86 23 PvPscene_BM_OpenFields (Chernarus)"

"Type avg fps avg min highest lowest"

"Benchmark 77 72 105 40 PvPscene_BM_OpenFields (Chernarus)"

"Type avg fps avg min highest lowest"

"Preloading 84 73 102 37 PvPscene_BM_Woods (Chernarus)"

"Type avg fps avg min highest lowest"

"Benchmark 86 80 103 31 PvPscene_BM_Woods (Chernarus)"

"Type avg fps avg min highest lowest"

"Preloading 66 53 87 11 PvPscene_BM_City (takistan)"

"Type avg fps avg min highest lowest"

"Benchmark 72 61 89 19 PvPscene_BM_City (takistan)"

"Type avg fps avg min highest lowest"

"Preloading 98 83 139 23 PvPscene_BM_OpenFields (takistan)"

"Type avg fps avg min highest lowest"

"Benchmark 99 89 129 38 PvPscene_BM_OpenFields (takistan)"

"Type avg fps avg min highest lowest"

"Preloading 104 90 139 20 PvPscene_BM_Woods (takistan)"

"Type avg fps avg min highest lowest"

"Benchmark 110 101 145 56 PvPscene_BM_Woods (takistan)"

"Type avg fps avg min highest lowest"

"Preloading 59 50 67 22 PvPscene_BM_LongViewDistance (takistan)"

"Type avg fps avg min highest lowest"

"Benchmark 62 52 70 42 PvPscene_BM_LongViewDistance (takistan)"

"Type avg fps avg min highest lowest"

"Preloading 55 44 73 13 PvPscene_BM_City (Zargabad)"

"Type avg fps avg min highest lowest"

"Benchmark 58 47 73 14 PvPscene_BM_City (Zargabad)"

"Type avg fps avg min highest lowest"

"Preloading 70 59 83 26 PvPscene_BM_OpenFields (Zargabad)"

"Type avg fps avg min highest lowest"

"Benchmark 74 66 85 26 PvPscene_BM_OpenFields (Zargabad)"

"Type avg fps avg min highest lowest"

"Preloading 31 28 35 2 PvPscene_BM_LongViewDistance (Zargabad)"

"Type avg fps avg min highest lowest"

"Benchmark 32 29 35 24 PvPscene_BM_LongViewDistance (Zargabad)"

"Type avg fps avg min highest lowest"

"Preloading 72 58 84 24 PvPscene_BM_OpenFields (ProvingGrounds_PMC lite)"

"Type avg fps avg min highest lowest"

"Benchmark 75 66 85 38 PvPscene_BM_OpenFields (ProvingGrounds_PMC lite)"

Edited by Birdseye

Share this post


Link to post
Share on other sites
= PvPscene Benchmark Suite

The goal is to create a better system to benchmark the ArmA engine to

understand the effects of different game video settings, different hardware

components and system configuration.

in line with the goal of this 'addon', please submit your results to the online database so they can be compared with other results.

Most of the fields can be copy pasted. ;)

Adding your information to the database will also express your appreciation to the author for the hard work that has gone into this effort. :clap:

Share this post


Link to post
Share on other sites

Is there an explanation of each field for the Database not sure what order my settings coniced with on I-L, O-AB. But I got my Bench's up there from Takistan run.

Also sorry Helo thought someone was typing hello!

LOL I found it, but I have no idea what settings I ran under I think just a stock Arma 2 run up used test as the profile and seeing as I dont have a test profile I would assume A2OA just setup what it felt was best for the game.

Think I will redo them and under a created Benchmark Profile.

And thanks for the sweet tool PvPScene!! :ok:

Edited by Rabble-Rouser

Share this post


Link to post
Share on other sites

Alrighty got a set of Bench's on there as Normal settings across the board next batch should be of my standard gameplay settings.

Lovin this tool set man very nice job!

Share this post


Link to post
Share on other sites

Thanks [DirTyDeeDs]-Ziggy-!

Updated the first post with several clarifications.

Birdseye this is possible - however to test different settings in one go is one major

feature of Mac's FPS analyzer. Lets motivate him to update it to make it a lot easier. :)

Rabble-Rouser you can find the explanations in the first post now.

Share this post


Link to post
Share on other sites

=== Online Database

The following graphic settings you are asked to submit in addition to your test results

to make the comparison between different systems and graphic settings useful.

i understand having at least one group of settings the same across the different benches to make comparisons meaningful, but why Windowed mode and 3D resolution 1280x960?

dont you think full screen and 3d resolution matching native resolution more appropriate?

I never play in windowed mode, and that 3d res is way off matching anything in my drop down.

I'd be happy to fulfill your request, but I hesitate to run a bench when I feel little can be gained from the test.

Please explain to this ignorant soul what the importance of those two particular settings is.

another question. online database. gfx settings. terrain detail field.

Since you are forcing Very High Terrain Detail in the benchmark, should all benchmarks report VH for that setting?

thanks

Edited by [DirTyDeeDs]-Ziggy-

Share this post


Link to post
Share on other sites

Well the wording is indeed confusing.

The screenshot was merely meant to explain the abbreviations like TD = texture detail.

It is NOT about recommended settings. Rewording it now.

The goal is to let people submit their current settings or those they prefer.

A direct comparison is something mid term and will be easier to achieve with Mac's tool.

PS: TD = texture detail. :) Terrain detail is not part of the screenshot.

PSS: Tomorrow a new layout with all result in one sheet will be available.

It will contain all results already submitted.

Edited by .kju [PvPscene]

Share this post


Link to post
Share on other sites

I dig the fact that you can compare what others run there settings at to help give a comparison on settings I have used or may want to use and see if it offers benefit or grief. Having one test to run everones settings the same would be handy just to set a standard for machine comparison. But every machine runs different wether you built it yourself and know it in and out or wether you bought it off a shelf and wonder why, with all the bloat ware, it is so slow.

I've been doing some scripting yesterday so I dint get around to doing a second test at what I actually run the game at for gaming. But adding it as a second comparison may shed light to others with a similar system as to what may be a better setting for them.

Share this post


Link to post
Share on other sites

The new version is up. Hopefully it is somewhat easier to use and provides a better overview.

Edited by .kju [PvPscene]

Share this post


Link to post
Share on other sites

Where do I find my results?

---------- Post added at 11:05 AM ---------- Previous post was at 10:30 AM ----------

NVM. I figured it out. But still dont know how to post them.

Share this post


Link to post
Share on other sites

Nice update in the database. It's easier now to post results, so I addded a few benchies.

Share this post


Link to post
Share on other sites

I've added three more runs of mine with different MFtRA (Max Frames to Render Ahead).

It basically says that for my system in terms of FPS there is no difference.

Right now I am using 1 as it makes mouse lag a non issue.

http://www.editgrid.com/user/kju/PvPscene_Benchmark_Suite

Share this post


Link to post
Share on other sites

Added my results with the latest version (1.59.79122).

It is actually considerably less FPS (10%) - even though I had PP disabled.

PvPscene_Benchmark_Suite

Edit: Yet it might be due to higher LODs shown overall and less LOD switching. FPS alone does not tell the whole picture.

Edited by .kju [PvPscene]

Share this post


Link to post
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
Sign in to follow this  

×