Server ability : URL forwarding of MISSION content

[Dwarden]

I'm creating this thread with poll to obtain some feedback on one 'possible' feature for ArmA 2 (with very rare chance this to appear also for ArmA 1) server / client.

Original thread with POLL was here : http://www.flashpoint1985.com/cgi-bin/ikonboard311/ikonboard.cgi?act=ST;f=57;t=77288

The feature is option for server to define file source forward for MP missions content (HTTP) and client's ability to download that content then from defined source.

Description:
The Arma 1 (and most likely 2 too) engine could easily support this feature ,
as MISSION related content is possible to load after engine start (contrary to map or other type of game changeable content like models).
So technically this feature would require that instead of downloading MISSION file from ArmA *1/2 game server directly
there will be inserted of small&simple portion of code telling game to download file from server defined URL location (if enabled server side).
The routine about file placement and check if present/loadable etc. is then same/similar to already existing part of code.

i split the poll into these sections tho i would like users to post with comments why they for or against this useful feature:

1. server admin/owner/donator + support this feature
2. just user + supporting this feature
3. anyone against this feature

thanks everyone for taking part

p.s. once again just in other words for these who not understand above :smile_o:

actual situation :

1. client w/o mission establish connection with server
2. mission file is downloaded from server using server-client connection
3. when mission file is downloaded and not corrupt then client loads it into memory and enters missions

positivies:
+ already done

negatives:
- will use bandwidth block of game server for mission distribution instead of game traffic
- try imagine popular public server with 1k Unique Users / week download several 5MB missions ...ouch

proposed situation:

1. client w/o mission establish connection with server
2. server tells client there is possible download location at http://www.somewebsite.net/folder/arma2mission.pbo (this part must be coded in)
3. clients starts wget on background and waits for error code or timeout on error code (this part must be coded in)
4. if error code is success then file is placed into mission directory like usual and loaded as usual (code already present ingame)
5. if file not match (already code present ingame) or if error code is fail / timeout then client may fall-back to try download from server using already existing method (this part must be coded in with reusing already present code)
6. when mission file is downloaded and not corrupt then client loads it into memory and enters mission


*not need to use external opensource wget and use own portion of code, it was just example

negatives:
- some code must be added but not much (as most of code needed is already in)

positives:
+ offloading the mission download to different source than gameserver

...

[.kju [PvPscene]]

What is/are the benefit(s)?

I do imagine at least one, yet please go on.

[Dwarden]

erm i though the benefits are clear ...

actual situation w/o this feature mean that any player w/o the mission in question is downloading it from the server:

which means :
- higher bw demand on server resulting to e.g. higher cost (bw block limits)
- less bw for players already on server (possible issues with quality of gameplay)

today most of modern MP oriented games and engines already offer this (e.g. engines from trio Epic, Valve, ID) for map/mission content

the problem turns worse in moment You use 'big/huge' mission and run popular server (100s to 1000s unique players)

with the feature you would then:

+ define mission source URL independently on the server location
+ save your server bw block
+ increase the quality of gameplay on server

in short instead of 'wasting' precious bandwidth on your  server (not everyone have dedicated server line beyond T2) You can utilize some ultra fast public mirror hosting mission file(s)

[.kju [PvPscene]]

while i agree that BI is engine is lacking so many aspects most
other mp engines have, i doubt whether the effort for this would
be well spent.

the engine needs to handle several things for his or you make it
very crude (unique identification, error handling, fallback, etc).

instead the should focus on proper configuration, addon download
(game restart is ok), more stable / resistant to errors, optimization,
linux server etc etc

[Dwarden]

there is no need to complicate things and that's so neat on this feature

game server:

add query string with Mission_downloadable == "true"/"false"

add query string with Mission_download_URL == "*"

game client:

if client sees true for Mission_downloadable
then client uses Mission_download_URL and auto adds "http://" to URL defined
then client executes e.g. wget to download URL+mission filename (+ needed switches)
then client waits till error returned or file downloaded or other error returns or else timeout
then client tries load mission like now (same part of code used for actual download from server)

yet if any of above fails client can easily fallback to already existing direct download from game server code
also client can use file operations code already present

that's all ... simpler it can't be

p.s. don't mismatch this with any addon content ...
that's completely different and unrelevant unrelated to this feature

[walker]

Hi all

I think Q has completely misunderstood and indeed obfuscating the simplicity and the importance of this proposal for Join In Progress (JIP) And how many ways this will improve ArmA.

JIP Lag and Mission load speeds
Basically this addresses Mission load speeds and JIP Lag.

It does so in three aspects.

1) When you first play a mission or when a mission is updated, you all have to download the mission. If it is one of the big servers say 60 plus players it causes loads of problems: It can take up to 10 minutes, people get bored, there are dropped connections adding to the load if they half way through a download.

Downloading the mission in advance removes this problem.
Torrents might be a useful additional location for missions then just put up a server index page for the torrents. That way a clan can trickle their servers mission and their addons, no need for emails to tell people to update addons either, it all becomes automated.

2) JIP lag for existing clients/players on a server when some one joins a server. The server has to slow to send the whole mission file which with sound and even video files to every new client. We have all experienced the dreaded Lag in a helo coming into the Hot LZ where we all crash and do not even know why, as a new player from a different continent joins with a mission they have never downloaded before (it is always polite ask on Team Speak if you can JIP in) JIP lag for a pilot is a killer.

Downloading the mission in advance removes this problem because all downloads are done separate from the server, mission load JIP Lag is removed, synchro Lag still exists but more on that later.


3) JIP lag for a new client/player is removed, meaning you join the game faster. Synchro Lag still exists but more on that later.

Synchro Lag
This is outside what Dwarden is proposing. This mostly affects big missions and Persistent Server missions like Warfare and Evolution and only when Joining In Progress

When you JIP into a mission that has been going for some time the mission environment has changed. Units have moved/changed position from the mission download positions, bushes trees lampposts have been knocked down by bad drivers (mostly me that, sorry) buildings blown up ammo load outs are all different people shoot bullets, lots of dead bodies where the bullets hit people, smoke, explosions etc.

What is needed for Synchro Lag is a server farm architecture essentially the ability to use client as a server. Nothing special just an additional server that acts as a dumb client to the main server and can be used by JIP players to synchronize to.

Another solution may be to use peer-to-peer updating from players in game, but there are a whole bunch of technical issues as pert of that route.

Anyway I think we should give Dwarden our full support on this.

Kind Regards walker

[.kju [PvPscene]]

1.15 fixed the mission download problem.
IC with 100 players had no longer any problem during JIP, mission
start, long mission download.

at least to our experience this is no longer true.
bandwith is the least issue of the A1 DS imo. the other stated
ones are way more important imo.
so i see this as a very minor issue in these days.

again you need to have the mission still on your server, make
a compare if the URL is offering the same file.

[Dwarden]

@<hidden> the client already compares the file (if you have the mission already) and if not it redownloads it ...
so again that part of code is already included in client ...

and please don&#39;t think in scope of &#39;rare server&#39; with &#39;uber fast&#39; line running &#39;small&#39; well spread mission for &#39;limited&#39; number of clients ...

this is aimed at:

- public server
- 64-128 or more player server
- mission size is 1 MB or bigger

for starter let say You have 2000 unique players / month and you rotate 10 missions with total size 25MB (avg 2.5MB/mission) and let say each mission gets 2 builds per month

if 25% of players already got these missions other way

then bw block needed is &#39;only&#39; 0.75*2000*25*2 = 75000 MB
(just for mission itself downloads)

now let&#39;s not even try imagine very popular server with way more and lot bigger missions and even more often updated ones ...

in short something what ArmA 2 try aims to be http://forums.bistudio.com/oldsmileys/smile_o.gif right ?

@<hidden> walker You not lightened my argument either http://forums.bistudio.com/oldsmileys/smile_o.gif
as some of the issues (or most) were resolved in public 1.15 beta already

anyway You still right that &#39;cluttering&#39; the server bandwidth on server where is some &#39;limit&#39; (be it max bw block or actual throughput) still exist and that&#39;s exactly what above suggested feature remove

[Lt.Kettch]

I am with Dwarden on this one

[SWAT_BigBear]

Dwarden,
You know I am for this, since I use this form of "redirect" for RvS and CoD servers. http://forums.bistudio.com/oldsmileys/wink_o.gif

[ombas]

i&#39;m all for an automatic http redirect for the user.  whatever makes it easier to get on and improves the play experience is a good thing for everybody.  i emphasize AUTO http redirect.

keep up the good work&#33;

[[HUD]Dorph]

I was a counterstrike-source server operater when this was introduced to css, and it took a massive bandwidth-load off the servers, this sounds like a good idear.

[Maddmatt]

Agreed. Downloading the mission straight from the server is:

1. Slow, can be a problem with large missions. Having to sit and wait for the mission to download at speeds much slower than your connection could handle is frustrating.
2. Uses bandwidth that should be used to run the game. Might cause desync in some cases.

Maybe it could be done for addons, although I don&#39;t know if I would be happy with loads of low-quality addons being forced on me. And that would be more complicated to implement.

[Max Power]

Agreed. Downloading the mission straight from the server is:

1. Slow, can be a problem with large missions. Having to sit and wait for the mission to download at speeds much slower than your connection could handle is frustrating.
2. Uses bandwidth that should be used to run the game. Might cause desync in some cases.

Maybe it could be done for addons, although I don&#39;t know if I would be happy with loads of low-quality addons being forced on me. And that would be more complicated to implement.
It would be too hard to unify everyone&#39;s addon structure and it would hog up your hard drive. you think your arma dir is big now? Imagine if you had every popular addon multiple times, depending on the tastes of the server admins&#33;

[Dwarden]

i repeat it again please don&#39;t mismatch this feature with addons / maps etc. downloading which may seems similar but is absolutely different ...

that would mean radical changes into engine which is exactly opposite where feature above aims ...

so keep it simple as it is ...

[Brendon]

Thank you walker for explaining it. I don&#39;t think English is the Topic creators main language, so didn&#39;t understand him too much. http://forums.bistudio.com/oldsmileys/confused_o.gif

Anyways, I support the idea as a user, anything to stop JIP lag as it can really kill a fire-fight and it also makes it quicker to join since it won&#39;t use the servers resources but its own.

[Spooner]

Although I&#39;m in favour of this, have you considered using ArmAlib to do this as a community side (tiny missions that are downloaded normally from the server, but that downloads the mission-specific files from an external source, be that HTTP or database)? Certainly not happening for ArmA before A2 at the very least, so this might be a better option, though getting the average user to use something like Armalib is harder than getting them to predownload the mission, in all likelyhood.

[bluevein]

This Idea never even crossed my mind but what a good one.

I&#39;m with you 100%, server owner/admin & user +

I hope BI concider this. http://forums.bistudio.com/oldsmileys/notworthy.gif

[Dwarden]

Although I&#39;m in favour of this, have you considered using ArmAlib to do this as a community side (tiny missions that are downloaded normally from the server, but that downloads the mission-specific files from an external source, be that HTTP or database)? Certainly not happening for ArmA before A2 at the very least, so this might be a better option, though getting the average user to use something like Armalib is harder than getting them to predownload the mission, in all likelyhood.
the point is simple feature out of box as part of the game client/server operations ...

anything what needs user interaction with 3rd party compoenents just for mission download forwarding
downplays such feature into oblivion ...

[bravo 6]

Dwarden, to be honest i also had some difficulties to understand the main idea of the pool, i feel its not quite clear yet.

I believe there are also more people that would like to participate on this but they just don&#39;t understand these kinda techniques.


So from what i understand to have better performance it would need 2 servers, one for the game it self and another server(s) for hosting missions (optional/resource). These way people would download missions from different sources and prevent lag.

Is this the idea?

If so, i still don&#39;t know what option to choose in the pool http://forums.bistudio.com/oldsmileys/confused_o.gif

[Dwarden]

what you not understand ?

ok i updated the first post and it&#39;s under the P.S. so read and tell me if You now able understand what i got in mind http://forums.bistudio.com/oldsmileys/smile_o.gif

and about the 2 servers need ...

no need for two servers ...
if there is UBLIC and FAST http/ftp mirror for missions then You &#39;lighten&#39; the load off your server to that download mirror ...

yet the option to download mission from Your server still exist in case the mirror goes down or client fails to connect to it or correctly download mission file

just note nearly every MP game now supports this ...

[Alex72]

Wasn&#39;t too clear for me either to understand fully lol. But if its about taking stress away from the server running the missions and improving JIP im all for it.

Sounded scary with one of the negatives though. If server rotating heeps of maps me as a user will have to sit back and just take all of them before joining. Dont know if i got that right?

But if this was a feature out of the box server hosts would know not to use too big rotations. So that wouldn&#39;t be a problem. Small note in the manual (or extra sheet if manual is done) could explain this easy. The more solid the core game is - the better. Good job Dwarden coming up with the idea.

PS. Hope i got your point somewhat correct.

Regards
Alex

[Dwarden]

http://forums.bistudio.com/oldsmileys/sad_o.gif fail again http://forums.bistudio.com/oldsmileys/sad_o.gif

you don&#39;t need download all the heap of maps ... just the one you joining in ...

now in ArmA if you download modified Warfare with 25MB size it&#39;s downloaded from the server it&#39;s hosted on ...

with this server admin may decide that you may download it from somewhere else (usually faster) which helps game server too ...

http://forums.bistudio.com/oldsmileys/crazy_o.gif http://forums.bistudio.com/oldsmileys/banghead.gif ... where is that smilie with smiley running around crazy screaming http://forums.bistudio.com/oldsmileys/pistols.gif

[TeRp]

Well, I understood you perfectly and I&#39;m supporting your idea. http://forums.bistudio.com/oldsmileys/smile_o.gif

//type fixed

[Alex72]

@<hidden>

LOL&#33; Sorry mate for making you nuts. http://forums.bistudio.com/oldsmileys/biggrin_o.gif

Misunderstood the mission rotation bit. Thaught you wrote it was one of the downsides to have to download it all. No harm done. Still supporting. Even more now.

Alex

[Dwarden]

72 @<hidden> Mar. 21 2009,22:35)]@<hidden>

LOL&#33; Sorry mate for making you nuts. http://forums.bistudio.com/oldsmileys/biggrin_o.gif

Misunderstood the mission rotation bit. Thaught you wrote it was one of the downsides to have to download it all. No harm done. Still supporting. Even more now.

Alex
no problemo http://forums.bistudio.com/oldsmileys/smile_o.gif that&#39;s why i rewrote example once more just for sure http://forums.bistudio.com/oldsmileys/wink_o.gif

[tomcat_]

that&#39;s a very good idea and i support it 100%.

However, i guess there will be an option to redirect to a website or just download straight from the server?

The reason i&#39;m mentioning that...is that by adding, changing, updating missions...someone has to be doing the same on the website..as well as the server itself...which might lead to having one version for the server another for a website...

that&#39;s the only negative i can see....

i know i&#39;m going a bit off topic here...but i really loved the BHD, Joint operations (novalogic) way of managing maps...where you had to upload them on the server...and clients didn&#39;t have to download the new missions....certainly requires different netcode and lots of coding...but i thought i should slip my suggestion in there...:)

[Johnnie_Walker]

Totally support this.

Downloading addons from within the game would also be very handy, just need to promt the user.
Adding a new shortcut would be just perfect.

[Dwarden]

Totally support this.

Downloading addons from within the game would also be very handy, just need to promt the user.
Adding a new shortcut would be just perfect.
downloading addons is different and too complex change
(and as long you think in way how ArmA 1 operate)

that&#39;s why this is is (repeating myself again) only and


ONLY for mission downloading

NOT for maps
NOT for addons


when i think about some (part of) code is in theory already inside ArmA 1 - used for squad xml and logo

also same code could be usable as optional way for downloading custom files (face/sound)

[Johnnie_Walker]

Totally support this.

Downloading addons from within the game would also be very handy, just need to promt the user.
Adding a new shortcut would be just perfect.
downloading addons is different and too complex change
(and as long you think in way how ArmA 1 operate)

that&#39;s why this is is (repeating myself again) only and


ONLY for mission downloading

NOT for maps
NOT for addons


when i think about some (part of) code is in theory already inside ArmA 1 - used for squad xml and logo

also same code could be usable as optional way for downloading custom files (face/sound)
I didn&#39;t mean to load addons "on the fly", just being able to provide a download link for the addons. So the user just have to click yes and the needed addons are downloaded to a default addon folder, and game is restarted with the brand new addons loaded.

Not too complex if you ask me.

[rundll.exe]

Totally support this.

Downloading addons from within the game would also be very handy, just need to promt the user.
Adding a new shortcut would be just perfect.
downloading addons is different and too complex change
(and as long you think in way how ArmA 1 operate)

that&#39;s why this is is (repeating myself again) only and


ONLY for mission downloading

NOT for maps
NOT for addons


when i think about some (part of) code is in theory already inside ArmA 1 - used for squad xml and logo

also same code could be usable as optional way for downloading custom files (face/sound)
I didn&#39;t mean to load addons "on the fly", just being able to provide a download link for the addons. So the user just have to click yes and the needed addons are downloaded to a default addon folder, and game is restarted with the brand new addons loaded.

Not too complex if you ask me.
And if that doesnt make it: Just a simple link that pops up to a website that has the addons. Thats better for users that like control over their addons


A problem would be the updating process. If the missions and/or addons change, they need to be updated on the http server aswell. If this doesnt happen automaticly, it would be waaaay to much work for server admins to make it happen. So there should be an additional tool that synchronises the missions between the two servers (by ftp or smth)

Resulting in quite a complicated feature for server admins... But I guess its worth it for the big servers.

[Dwarden]

please do not &#39;hijack&#39; the thread purpose related to mission download URL forwarding with addons / maps download solutions ...

purpose of this thread is to get &#39;extremely simple&#39; and &#39;powerful&#39; solution with &#39;minimum&#39; coding into engine ...

while what you want is maybe powerful and quite simple but needs &#39;lot of coding and changes&#39; ...

[Yoma]

I&#39;m all for this mission download url redirect.
Whatever happens, it will lead to less bandwidth usage for mission downloads on game server and less opportunity for lagspikes due to players connecting.

Including something that would be able to download addons would also be awesome and avoid people needing 3rd party client applications like the one i&#39;m making, but i understand it&#39;s outside the scope of what can be done in reasonable time.

What would be really really nice if you could include a means for players to be redirected to an url using default browser upon failing to load a mission due to missing addons.

So a server can define mission urls but also one "missing mods" url so people can get linked to the stuff they need.

They could then download the stuff they need manually or via a 3rd party app like mine.

Maybe you could pop up a message saying "BIS is in no way responsible for the content of the site we are trying to open. Do you want to open the "missing addons" page?

This approach would also be quite minimalistic in code changes yet be very very powerfull.

[Yoma]

Another potential future benefit of this feature is that someone could make a tool that checks addons a server runs combined with an online repository of mission files.

That could make "installing new missions" on a server a totally different experience.
Imagine a client tool that can
-get new missions from a public repository that use addons you have installed on your server
-link that info in your server config file
-reload your server config/restart server.

This way the server admins job of getting new nice missions "on the server" could radically change and the players game experience would be something evolving on a day to day basis.

[Dwarden]

impressive after nearly 2 months

someone voted against this simple&useful feature and yet failed to post why ...

p.s. yomma i suggested several times that ArmA/ArmA 2 could benefit from &#39;out of box&#39; lobby running prior the game

ideally same opensource lobby which is used by opensource RTS Spring
(simple, easy, working, contats server list browser, chat, IM,
custom content (maps,addons etc.) downloads via bittorrent, http or ftp etc.)

let me quote my old post (it may be wrong on some stuff but in general sense right)


due to fact that engines needs to compile addons (if i remember right) e.g. cpp while starting it&#39;s NOT possible to load addons with missions later

would simply mean total change in way how engine is coded (major rewrite) ...


i remember the compression of PBO was removed (it was there in OFP as form of LZH) due to way how streamed data are used and as it caused load on engine ...


BUT ... now please let the .PBO idea fade away and concetrate on same aspect but from different angle :
distribution

mission PBO files could be compressed (LZMA from 7-zip, http://www.7-zip.org/sdk.html ) on server for delivery before send to client and decompressed on client into MPmissioncache after download done

now this could be even more improved with URL forwarding (so let say arma2.info / ofpec.com holds major huge ultra fast FTP/HTTP server cache with multiple worldwide mirrors full of missions which clients will use to download from (told by server, used e.g. MD5/SHA1/SIGNature hash)

and now about mods / models / maps etc PBOs ...
to avoid rewrite of engine all what game needs is LOBBY running prior game itself and server reporting correct hashes of PBOs needed

the lobby could be used for usual stuff like server listing,chatting and &#33;&#33;&#33; stuff downloading prior joining server + correct mods usage (thanks to file hashes it prevents duplicates in cache or using same name and different content etc) ...

if You want to understood what i mean i suggest You check RTS open source Spring
http://spring.clan-sy.com/
they use own lobby system
http://trac.springlobby.info/
http://spring.clan-sy.com/websvn....ient%2F (http://spring.clan-sy.com/websvn/log.php?repname=spring&path=%2Ftrunk%2FLobby%2FTASClient%2F)
they even have &#39;module&#39; plugin which uses allows use of Bittorrent (and optional HTTP) to download/upload content
http://trac.caspring.org/wiki/sd

ofcourse this lobby must be integral part of game since day zero

what i&#39;m trying to say there are very effective routes w/o need to overhaul way how engine works and rewrite whole engine

[Yoma]

p.s. yomma i suggested several times that ArmA/ArmA 2 could benefit from &#39;out of box&#39; lobby running prior the game
Well i totally agree. But i do understand the developers wanting to focus on the game first and for all.

Actually BIS should make something like the last version of my tool and deploy it out of the box, but better ;-), with full clan support functionality etc.

Anyway i&#39;ll do my best to provide *something* if that isn&#39;t the case. I know it won&#39;t be the best, but there&#39;s no harm in trying...

[Alex72]

Good idea Yoma. Would love to see a tool like yours out of the box. Would make life so easy for everyone.

Alex.

[SWAT_BigBear]

Content deleted, due to wrong thread discussion.

[Dwarden]

please don't bring MOD / MAP install / downloads discussion into this thread ...
make new one and don't clutter it with too complex demands ...

this thread is for mission downloads URL forward (http/ftp ...) instead of direct download from game server ...

i mean it's so hard to understood (after i explained it like 4 times ?) ...

mission URI-forwarding&download is simple easy task to add into engine while mods/maps mean either change the engine too much or add new lobby sw ...

[Crowe]

Hi Dwarden,

you mentioned a good idea.

I see 2 little Problems with this system:

When the webserver is down, or delivers a corrupted mission file
When the mission on arma server is different from the webserver file

[Dwarden]

Hi Dwarden,

you mentioned a good idea.



I see 2 little Problems with this system:

When the webserver is down, or delivers a corrupted mission file
When the mission on arma server is different from the webserver file


these are no problem at all and i already covered and explained this :

1. when webserver is down the URI download code return 'error' and fallback to download from server
already existing code

2. when mission differs then the code again fallback to re-download from server (it's already now present if you have mission file name abc.pbo and join server with abc.pbo yet different content it overwrites the first one inside the MPmissioncache)
already existing code

problem solved ... we keep rolling over

[Crowe]

So bi devs are already implementing it?

btw. What will happen when you forward a URL like http://127.0.0.1 ??
The clients will fall back and download the mission from the armaserver i guess?

[Dwarden]

So bi devs are already implementing it?

Great :)

no, this suggestion / feature request covers the missing part of URL forward download but the code to 'resolve' your described issues is already present (just for other purpose)

[S!fkaIaC]

these are no problem at all and i already covered and explained this :

1. when webserver is down the URI download code return 'error' and fallback to download from server
already existing code

2. when mission differs then the code again fallback to re-download from server (it's already now present if you have mission file name abc.pbo and join server with abc.pbo yet different content it overwrites the first one inside the MPmissioncache)
already existing code

problem solved ... we keep rolling over

Does the existing functionality supporting also load sharing mechanisms?

e.g.:

We have Gameserver GS + FTP1 + FTP2 + FTP3

Request # 1 should be answered with URLs of

FTP1-FTP2-FTP3-GS

#2 with

FTP2-FTP3-FTP1-GS

#3 with

FTP3-FTP1-FTP2-GS


...and so on and the client should be able to go in consecutive order through the list and making a defined amount of attempt(s) to grab the requested content there.

Another requirement is a constant sync of the hashs of all FTPs and the GS.

If on files differs from the GS, the releted IP/URL should be taken from the round-robin list.

[Dwarden]

it would be enough to offer one URI to forward than none if the URL forward fail the game can easily fallback to obtain mission files directly from game server ...

ofcourse
it would be nice to have option to
e.g. provide primaryURLforward and secondaryURLforward locations ...

yet stay with the KISS (keep it stupid simple) principle :)

[S!fkaIaC]

It is KISS, what I described is nothing else then one possible load sharing method at DNS.

[Cross]

Sorry for OT but quickly going thru the whole thread, I lol'ed @<hidden> Dwardens efforts to make himself clear and to differentiate it from addon issue...and going nuts in the end :icon_lol:
It's a sad lol though....

Anyway...here are my points..
1- This may be a good solution for rental servers where they rent 1 CPU and 1 RAM block and 25Mbps connection from a rental co. Basically 1/4 of a server. Or for non-dedicated servers. Or dedicated servers operated from home with limited connectivity. This again a small % of MP community, I believe.

2- I dont think it is a problem for most DediServers. Nowadays most of the proper dedi servers are at least with Quad-core and 100mbps conn with 1-3TB monthly BW limit.

3-Arma1 does not fully use the available bandwidth of a 100mbps line. In a 30 person coop or domination session I've seen figures around 8-12Mbps outbound.

4-I support the idea of "mission-downloading should be separated".

Therefore my input or counter-suggestion would be
In JIP instances, where the mission has already started, ArmA client should start a background ftp/http transfer task/service upon/before connection to download the mission file from the same server's, missions folder. This should be done before harassing the Arma Server and should be thru a separate service launched for this purpose only.
This does not require any URL or anything and the process will not bring any extra-load to the Arma server/client itself.

Yet I think the biggest cause of lag due to JIP comes from the synchronization of client with the server on battlefield & soldier status

[Victor]

This would be a really good system.

[Dwarden]

Sorry for OT but quickly going thru the whole thread, I lol'ed @<hidden> Dwardens efforts to make himself clear and to differentiate it from addon issue...and going nuts in the end :icon_lol:
It's a sad lol though....

Anyway...here are my points..
1- This may be a good solution for rental servers where they rent 1 CPU and 1 RAM block and 25Mbps connection from a rental co. Basically 1/4 of a server. Or for non-dedicated servers. Or dedicated servers operated from home with limited connectivity. This again a small % of MP community, I believe.

2- I dont think it is a problem for most DediServers. Nowadays most of the proper dedi servers are at least with Quad-core and 100mbps conn with 1-3TB monthly BW limit.

3-Arma1 does not fully use the available bandwidth of a 100mbps line. In a 30 person coop or domination session I've seen figures around 8-12Mbps outbound.

4-I support the idea of "mission-downloading should be separated".

Therefore my input or counter-suggestion would be
In JIP instances, where the mission has already started, ArmA client should start a background ftp/http transfer task/service upon/before connection to download the mission file from the same server's, missions folder. This should be done before harassing the Arma Server and should be thru a separate service launched for this purpose only.
This does not require any URL or anything and the process will not bring any extra-load to the Arma server/client itself.

Yet I think the biggest cause of lag due to JIP comes from the synchronization of client with the server on battlefield & soldier status

remember my aim is to help even these who don't have huge bw blocks, fast lines (e.g. home server owners, these who got some usable broadband and wanna play over weekend) etc...
offloading mission download by forward is simple and effective optimization ...

going nuts ? ... yeah after trying explaining it for 20th time i want see you to not reach 'nuts' state :D

[S!fkaIaC]

going nuts ? ... yeah after trying explaining it for 20th time i want see you to not reach 'nuts' state

well, now I understand how it works in BIS and why simple changes never become available. One skilled dev makes a good and easy to implement suggestion, the other devs commenting his approach and making other suggestions...in the end the initial dev is in the madhouse and we got a sheep addon instead of a T80 :smiley-punched:

[Yoma]

and we got a sheep addon instead of a T80 :smiley-punched:

But... i love sheep:whistle:

[walker]

Hi all

WHOOPS SORRY S!fkaIaC
This is what happens when you read the quoted post rather than the original

I will leave my post here as quoted italized example of my own stupidity


For the record I would prefer BIS only made one generic tank then allowed the community to MOD in from there. I see little point in wasting BIS's programmers on making another T80.

I much prefer BIS to concentrate on adding capability such as new Animal Forms or add in FLIR, and Dialogues to vehicles so that people can mod in the functionality.

As I have said quite often I would prefer BIS work on making money out of creating the simulation and tools to allow the community to create content.

The Real Virtuality Engine is BIS's core product. In a Gold Rush the people who make the real money are those who make and sell the shovels.

BUT I recognise the need for BIS to feed their souls and make their Military simulator.

Adding in additional server functions and maybe setting up ArmA to work in a server farm architecture is more important that a particular tank such as a T80 that any half decent modder could make in a weekend or two in their bedroom.

That said I think the use of multiple cores is a step in that direction but Server Farms to allow ArmA to be a 24/7 MMOG is the way to go.

Go back a few pages to read the context ;) :o :o


Kind Regards walker

[S!fkaIaC]

Yome: But... i love sheep You really looooove them ? :D Are you from Ireland or New Zealand? lol



Walker: WHOOPS SORRY S!fkaIaC
This is what happens when you read the quoted post rather than the original

Walker, I was not directly refering to your post, I was laugthing about poor Dwarden and his attempts to keep his original aim.

I was kidding about the way we turn Dwardens initial idea into something different and he is going nuts over this and I suggested it is similar in BIS that to many devs involved making a sheep out of a T80 if everyone can comment the design :eek: and some of the devs end up in the madhouse.

[Dwarden]

yeah some people are turning lil easy code tweak into major rewrite :) ...
and what about stop driving this thread offtopic and stay on topic of simple server improvement ? :)

[S!fkaIaC]

Maybe I am wrong, but most replies supporting your suggestion, me included.
So what you gonna do with this?

The poll in the old forum is still valid?

So if majority is supporting you, whats next? Request to Suma/Marek/BIS?

[=SF=Hyrax]

a fast redirect would be a life saver

[Defunkt]

There are good reasons (as detailed above) why most games support this. It should be a trivial change, for the server nothing more than delivering a URL and a checksum, that's it. A little more to do for the client (support for http:// downloads) but still not much. If the resource is missing or the checksum is wrong both fallback to the current behaviour.

Of course this should be done.

[GossamerSolid]

It'd be great if server admins would be able to define a web server address so new joiners would download the mission from the web server instead.

Source engine and UE2 + UE3 support this (probably more engines).

I just find that the server will choke if too many people are downloading the mission.

[GossamerSolid]

I see this post now, I for one really support this.

I recently included mando missile mod into my mission and the mission is now 9mb large. When new players come and download it from us, it lags the crap out of everybody playing (which gets really annoying). Then if it was offloaded to an HTTP server, then we could keep playing uninterrupted and the download times would be much faster.

Another thing for the future could be addon downloading, but I guess Yoma already does that pretty good.

[Dwarden]

re-added poll ... feel free to take part !

[GossamerSolid]

re-added poll ... feel free to take part !

Oh, I have

I really hope this gets implemented. Will make everything 100% bandwidth wise dealing with missions.

[PiratRu]

I have made how here is written! Has registered two lines in a file server.cfg.

Mission_downloadable == "true"
Mission_download_URL == "link"

But on mine it hasn't given any result. There can be I have registered not there? Prompt please.

[CarlGustaffa]

Wow, must have completely missed this thread. Is this one in the game yet?

[PiratRu]

I apologize in advance, if such theme already is! But it is heavy to me to see so much a lot of information. Therefore I have made a separate theme where I can discuss the problem - can to somebody too will help.

All problem that the server often falls! I have noticed that falls while the player is connected to the server! I where here already met that about switching of the player for закачки on a separate host or a site. Here such two lines I have found and have inserted into a file server.cfg.

Mission_downloadable == "true";
Mission_download_URL == "http://link/mpmissions";

But on mine hasn't given any result. These lines need to be registered in other file?
Can be registers absolutely on another?
If who can prompt - please prompt!

If certainly it is possible, at the same time a restriction line on ping.

The server costs on Linux

Thankful in advance to all!

[Dwarden]

Wow, must have completely missed this thread. Is this one in the game yet?

not yet

[GossamerSolid]

There isn't a redirect download feature yet (To my knowledge), though it was talked about and seemed to be a good feature.

As for high ping kick, it's configurable in your server's BEServer.cfg



RConPassword *****
MaxPing 300

[PiratRu]

Tnx GossamerS!

BEServer.cfg is placed in profiles and battleye folder in server?

[CarlGustaffa]

Domination example, where mission tends to be all kinds of variations.
1. Check mission cache. If same as server, this is used (no download time i.e. on reconnects).
2. Check cache based on server name. If matching, is copied from server cache to mission cache (no download time, i.e. on servers you play a lot).
3. Check all other server caches. If matching, is copied from found server cache to current server cache and mission cache (no download time, i.e. on an unmodified version).
4. If server has redirect, download the mission to server cache and when complete copy to mission cache.
5. If it doesn't download mission from server at slower speed, to server cache, and when complete copy to mission cache.

[PiratRu]

Very much pieces of good advice CarlGustaffa!
But how to me to learn, where the information about кеше is stored?
If I can limit loading of cards from the server what variable needs to be registered?

[alef]

About code, do you use already libcurl? "You may also freely use curl and libcurl in your commercial projects." (http://curl.haxx.se/docs/copyright.html)
About the algorithm and implementation, I would prefer an array of download URL strings for the mission,
... to be put in description.ext,
... and select randomly each of those.
For each, the user have to be asked if he wants to download from that server.
Cache the answer and put it into his ArmA2OAProfile

[bhaz]

I can't see any valid argument as to why this shouldn't be implemented. The majority of games that require custom content upon joining a server already have features similar to this. The benefits completely and thoroughly destroy the detriments, and at the time of this posting, 97.89% of those that voted agree. Though I'd imagine an engine change as big as this would break beta compatibility with the current patch?

I completely agree with alef's idea - an array of URL strings. Though I don't think it should be stored in the mission's description file. Even then, how would you read a mission's description.ext file if you don't have the mission? This also puts unnecessary effort onto the mission developer, having to find and maintain their own hosting.

This should be chosen by the admin of the server in question (server.cfg). It would make sense for example that a server provide download links suitable to its location, with fallback links further in the array in case a server goes down.

[nuxil]

Wow.
Dwarden if you implement this it will be like a gift sendt from heaven.

even if you have like 100 in ping to a server but fail to downloade the mission with in a sertan timeframe battleye will kick you off with a timeout.

this would overcome the problem with big mission and battleye.

:yay:

[Terox]

I have voted for this option but am unlikely to use it.
The reason for this is.
Implementing an option is always a good thing, if you dont want to use it, then it doesnt affect you, if you do have reason to use it, then it's there for you.

With the arguments on server load and bandwidth useage being a major part of this discussion then i would like to point out the following, which is defintely the case scenario for us and I would suspect many other servers out there.

Game servers tend to have more powerful processors than web servers
Game servers either rental or colocations tend to have more available monthly bandwidth allowances than some web providers.
Game servers tend to have faster connection speeds than web servers.
ArmA Bandwidth useage per client is relatively small and nowhere near utilises the bandwidth available to a dedicated game server hosted at a data center or a fast line

So I feel downloading from a web host may increase the timeframe from engine startup to actually playing in game.

I find it hard to believe that JIP lag is caused by downloading mission files, it's probably to do (And i am guessing but based on limited research) with transferring data about the locations and states of objects in the realtime environment to the JIP client and if this is the case, then hosting the mission files on a web server is not going to have any positive effect on this as some believe.

I also think time could be better spent on improvements elsewhere as Kju originally stated.
Using this methodology for Addons and mods however would be extremely useful and could be pursued at some point in the future when this has been proven to work for ArmA.
It's alrightb saying that every other MP game out there uses this methodology, however they all have different engines and work in different ways, so a comparison here really isnt relevant.

If this were to be implemented then for ease of use, automatic synchronisation between the web server host folder and the game server mpmissions folder should occur otherwise either the Webhost will have to be updated manually which may often be overlooked or scripts written to do this, which not everyone is capable of doing.

On a more positive note, if it helps some server admins with low bandwidth restrictions or poor connections out then, thats all the better. So go for it.

[Pulverizer]

Why not off-load the JIP data to a webserver as well, so only the last minute or two's worth of game changes need to be sent using the game server's bw, instead of potentially many hours'. It would be sent incrementally as changes occur, not instantaneously, thus costing almost no bw at all.

[Terox]

Why not off-load the JIP data to a webserver as well, so only the last minute or two's worth of game changes need to be sent using the game server's bw, instead of potentially many hours'. It would be sent incrementally as changes occur, not instantaneously, thus costing almost no bw at all.

The data the JIP client is given, is the data at that moment in time of the client actually joining the game, eg current status and locations of objects and values for certain variables that are different from Game start There is no need to store this data for any other time other than game start and at JIP point as it is constantly changing and would simply have to be redownloaded again at JIP point. In your method, I presume the system would suffer as if a JIP client was connecting every 2 minutes etc.

For example
@<hidden> Mission start: Variable = 1
@<hidden> WebHost store point Variable - 2,3,4,5,6,7 etc etc
@<hidden> JIP Point Variable = 23

Why would the JIP client ever need to redefine the variable value more than once when he connects

[Dwarden]

Terox as many games already have this ability, web hosters already offer for free to mirror the mission content

in fact one of reasons many hosters increase the prices of A2/OA hosting is because of the way higher bw block used by server itself

anyway this one of many improvements considered not the only one

[Pulverizer]

Why would the JIP client ever need to redefine the variable value more than once when he connects

The system would have to overwrite data where needed, not store a million changes to the same variable of course :P

It is possible to write to a certain offset with FTP, right?

Obviously this would not save any bw on constantly moving units etc. I was thinking about changes to much more static things such as fallen trees, damaged buildings, dead bodies, ammo taken from cargo and so on.

Getting the sync correct would be the difficult part. Ie, knowing exactly what data the connecting client needs updated. I guess the webserver JIP file would have to be locked every other minute or something. The client would report which "version" of JIP data he has upon connecting and the game server would know what needs to be refreshed from there.

[PiratRu]

Awful this English!
Not absolutely I have understood all, but I will re-read once again. Somebody can show on an example of one mission. What, how, where and where to register?

[larsiano]

If i understand correctly this option will give the players a popup where to get the mission? In that case i hope you can auto- redirect them to a side URL for the download. This could be a great feature for private servers like the one i run that struggle with bandwidth. Will it include an addon download option that would be lovely :)

[=WFL= Sgt Bilko]

I'm all for flexibility. It would probably be a game server variable specifying redirected or local server download (and a list of valid URLs if it is redirected). "Redirected" in this context, would mean that mission will be downloaded automatically BUT from a specified URL rather than from local game server.

Actual URLs should be set by server admins in game server config not by the mission makers.

This is how it's been in many other games for a longtime and works quite well.
It used to be that the local server download rate in other games were capped to same as player data rate to avoid bandwidth hogging. With the introduction of redirected download you could (depending on web hosting)theoretically reach your ISP download rate without any impact of game server traffic for other players.

JIP (in the sense of local game world initialization) is another beast not related to missions/mods download.

And it would be nice to see, not only missions, but also mods downloaded this way (though I don't know if A2 game engine would allow for that).

Sorry if this has all been covered already, I confess I didn't read through the complete thread :)

[Dwarden]

If i understand correctly this option will give the players a popup where to get the mission? In that case i hope you can auto- redirect them to a side URL for the download. This could be a great feature for private servers like the one i run that struggle with bandwidth. Will it include an addon download option that would be lovely :)

no popup, the aim is to be completely seamless download on background

if it fails it will utilize the same method as now

anyway when and if ... no ETA

[Yoma]

It's a good idea. Just do it :)

[PiratRu]

Guys, please... :confused:
Show please on an example!
As well as that where to write. To be honest, you at all don't represent, as these falling of the server out of the blue have already bothered.
I have noticed that more often the server falls at connection of the new player.

There is a site where I and can lay out cards! But in any way I will not understand where and what to write, what the new connected were swung not by servers. Was already tired to experiment.

[=WFL= Sgt Bilko]

Guys, please... :confused:
Show please on an example!
As well as that where to write. To be honest, you at all don't represent, as these falling of the server out of the blue have already bothered.
I have noticed that more often the server falls at connection of the new player.

There is a site where I and can lay out cards! But in any way I will not understand where and what to write, what the new connected were swung not by servers. Was already tired to experiment.The redirected download feature is not yet implemented. I'm sure there will be a "How-To" when it is in place. This thread is only to get community ideas and opinions.

[GossamerSolid]

This would be great if it would be included in the upcoming 1.55 patch.

[BearBison]

this would be a great idea for us server owners and also just the players.

[red29]

I'm going to be brutally honest:

Why wouldn't BI offer it? What could possibly be the downside of offering a simple redirect to download mission files, especially when it can be enabled or disabled at will? Why wouldn't BI just take the initiative to do something so easy that would do so much good for the community, instead of creating a poll that probably wasted 10x the effort of just coding it?

[Hatsune Miku]

I'm going to be brutally honest:

Why wouldn't BI offer it? What could possibly be the downside of offering a simple redirect to download mission files, especially when it can be enabled or disabled at will? Why wouldn't BI just take the initiative to do something so easy that would do so much good for the community, instead of creating a poll that probably wasted 10x the effort of just coding it?

Design procedures, probably - although trully it is quite a basic and essential feature that should have been implemented a long time ago, which you can clearly see the positive effects when observing the source engine handling this very same issue.

[Dwarden]

it's still on TODO list definitely not forgotten as some might fear

[Banshee]

We've used this for Enemy-Territory servers in the past. It seems a good solution for public servers with frequently changing big missions and limited bandwith.

With that feature you could place the mission file on a CDN or filehoster. For example Dropbox, Github, Rapidshare or simply your own webserver. This allready works with the Six-Updater by the way trough YML-files and a custom repository.


This will stress the network connection of the gameserver less than now.

I think resources should go into other stuff, but if you want, go ahead. :)

[wohali]

In support - very helpful as you mention in other engines (including Unreal). It'd help significantly if related assets to a mission could be downloaded without the use of another tool, such as Six.

[Dirty Haz]

Maybe make it optional by server admin/owner.

[Alderman]

/This is needed, for many of the reasons already stated and understood by BI. This is one thing I definitely will not have a dissenting opinion on.

[Double Doppler]

Would this solve the problem of mission files over 4 MB not being downloaded properly? If so I am supporting it, but only if it can be used by the server admin's discretion.

[GossamerSolid]

It's probably not going to happen at this stage, let's hope for A3 implementation.

[Dwarden]

things may happen, even as test-bed for the A3 before getting it there ... so never say never until A3 is out :)

[GossamerSolid]

things may happen, even as test-bed for the A3 before getting it there ... so never say never until A3 is out :)

"Never", there I said it :p

Well if this does make it to A2, I would be pleased.

[[HUD]Dorph]

This would be awsome - Customizing mission contents in mp an still having players that dont get autobooted.
This can only be a big win for whole comunity.

[SWAT_BigBear]

things may happen, even as test-bed for the A3 before getting it there ... so never say never until A3 is out :)
-ZxHAZChcYU


Edit..lol, go start reading the 1st couple of pages in this thread, this new forums make us look uber geeky.

[nomad_man]

Man, some people are driving me nuts in this thread. They didn't read the original message or explanations.

If you are indifferent to the answer, than don't post a reply.
If you are ok with the way things are setup, all you will have to do is change a config option.

Everyone else will benefit from having missions hosted at a different location. This will save you bandwidth and potentially speed up your downloads, provided you don't pick a slow mirror.

If there is a mission that you play out of the box, someone can create a mirror everyone else can link to it.

I say it's a definite yes.

[WoODoO]

BIS can take place to provide, mirror for this files also :)

[inque]

Maybe BIS can also consider Steam Workshop (http://steamcommunity.com/workshop/) as a means for delivering maps and mods in ArmA III? It would definitely make it easier for people to manage mods and maps

[Sickboy]

As Banshee mentioned, having the feature available out of the box could be welcome,
yet you can have this already TODAY, incl Mod distribution, TeamSpeak server joining, and many many more things, by deploying Six Updater (http://www.six-updater.net/p/about.html).

Many communities are already enjoying pain-free Many-Mod, Many-Mission, Many-Player games on their servers by deploying SU.
Give it a try yourself, while waiting for out-of-the-box functionality.

[Dwarden]

this feature is completely unrelated to both last posts .. why derail the thread?

it's about simple feature, simple way to make engine use instant miror (http/ftp) for just mission

IF mirror==true THEN downloadFROMmirror ELSE downloadASnow
IF mirrorERRORLEVEL THEN downloadASnow

simple, clean and dandy ... so please don't derail KISS concept thread with overcomplex solution (no matter how 'great')

[nomad_man]

So, this thread has been open for a little while. While six-updated has provided a decent way of delivering addons it has been shown that it is complex and people who are less technically savvy have a lot of problems using it. I would still prefer the automated delivery of addons and maps to the clients via other mechanisms.

Speaking of which, this thread has been open for a long time now, and by the vote results it seems like people would like that to be implemented. Any idea if anyone is working on that?

[Byrgesen]

Awsome idea, i really hope it becomes reality.

Question:
Would ArmaHolic be one of the webhosts for missions? It seem like it would be an ideal place to have the FTP server, as its already in place.
But i dunno if it can handle this amount of traffic tbh.
It would just make sense to upload a mission to a centralized FTP, like ArmaHolic, insted of having 1000's of webhosts with different missions and stuff.

Im just thinking about the first guy who stuffs a keylogger/malware into a mission and uses a dodgy webhost to distribute it. Would that be to easy if its not centralized?