PDA

View Full Version : Tool for class discovery?

TeTeT
Dec 15 2011, 08:22
Hello,

is there a tool to discover all the classes defined in a mod? Lately I've been looking at the unsung mod and haven't discovered any class documentation on their site and wonder if there's a generic way to export the class structure of a mod to some sort of text file, probably using any markup language.

I guess one can unpbo and unrap the config.bin's and then parse the config.cpp for the needed information. Or use a class viewer from arma itself and use the rpt file for copy'n'paste. Does a tool that automates this already exist?

Sickboy
Dec 15 2011, 08:30
I'm doing some for the official game versions and for ACE mod over at http://browser.dev-heaven.net

The project can also run locally, and then you can import any AllInOne to get the same results, but it's not really setup for usage by 3rd parties at this time.

You can generate an AllInOne config when the mod is loaded, and extract your info easily from that: http://community.bistudio.com/wiki/AllInOne_Config

TeTeT
Dec 15 2011, 09:34
Thanks for the pointer! I used the dumpConfig script from http://www.ofpec.com/forum/index.php?topic=35236.0 and now got a all in one config file.

Cloned your browser via git clone git://git.dev-heaven.net/six-configparser-web. Now on to setting it up locally, no idea if ruby on rails works fine on ubuntu 10.04 lts...

Sickboy
Dec 15 2011, 12:58
Should work fine on Ubuntu, however some utilities are only available for windows, like BI tools.

For config import you'll need at least 1 bi tool, but you can do do this part on Windows.

But as said before, the project is not setup to be used by anyone but me, and I do not have the time to support you currently.

At least a short version of what needs to be done:
- cfgConvert.exe from BI-Tools binpbo needs to be available. Currently expected to be in PATH, or in the folder where you run ConfigParser.rb from
- AllInOneConfig generated
- Generated file should be cleaned up with trimconfig.rb
- Cleaned file should be placed in a folder as config.hpp together with these templates: config_template.hpp, CfgDefaultWeapons.hpp, CfgDefaultVehicles.hpp
from within that folder, run ruby path\to\six-configparser-web\lib\configparser.rb

The above has to be done on a windows machine. Below can be on linux:

- six-configparser-web Database created and schema loaded
- Outputted config.yml should be placed in the root of six-configparser-web project and from the root should be ran: rake sync

I have updated the repository, added trimconfig.rb, and my frankenstein script autodump.rb that runs the game, exports config from game to rpt, import from rpt to xml, and from xml to yaml :D

Windows Ruby @<hidden> http://www.rubyinstaller.org/

BTW, I am working on a much improved version of the parser using sbsmac's ARMA assembly for .NET, but it's a nice to have - rainy sunday..

TeTeT
Dec 15 2011, 13:23
Cheers, thanks for providing the additional scripts! So far I had good success with using all sorts of Windows arma tools with wine, for example cpbo, extractpbo, texview, unRap are all working.

But I cannot even get the rails server up, as it's missing some goldberg table. I guess goldberg is responsible for the layout? I followed the info in vendor/plugins/goldberg/README.rdoc and run 'rake goldberg:install' and 'rake goldberg:theme[goldberg]', but still get this error when connecting to localhost:3000:

spindler@<hidden>:~/Work/Arma2/tools/six-configparser-web$ruby script/server => Booting WEBrick => Rails 2.3.12 application starting on http://0.0.0.0:3000 NOTE: SourceIndex.new(hash) is deprecated; From /var/lib/gems/1.8/gems/rails-2.3.12/lib/rails/vendor_gem_source_index.rb:100:in new'. Cannot find or read /home/spindler/Work/Arma2/tools/six-configparser-web/config/newrelic.yml => Call with -d to detach => Ctrl-C to shutdown server [2011-12-15 14:16:16] INFO WEBrick 1.3.1 [2011-12-15 14:16:16] INFO ruby 1.8.7 (2011-06-30) [x86_64-linux] [2011-12-15 14:16:22] INFO WEBrick::HTTPServer#start: pid=25126 port=3000 Processing Goldberg::ContentPagesController#view_default (for 127.0.0.1 at 2011-12-15 14:16:30) [GET] Goldberg::SystemSettings Load (0.7ms) SELECT * FROM "goldberg_system_settings" LIMIT 1 ActiveRecord::StatementInvalid (Could not find table 'sessions'): /usr/lib/ruby/1.8/webrick/httpserver.rb:104:in service' /usr/lib/ruby/1.8/webrick/httpserver.rb:65:in run' /usr/lib/ruby/1.8/webrick/server.rb:173:in start_thread' /usr/lib/ruby/1.8/webrick/server.rb:162:in start' /usr/lib/ruby/1.8/webrick/server.rb:162:in start_thread' /usr/lib/ruby/1.8/webrick/server.rb:95:in start' /usr/lib/ruby/1.8/webrick/server.rb:92:in each' /usr/lib/ruby/1.8/webrick/server.rb:92:in start' /usr/lib/ruby/1.8/webrick/server.rb:23:in start' /usr/lib/ruby/1.8/webrick/server.rb:82:in start' Rendered rescues/_trace (26.8ms) Rendered rescues/_request_and_response (0.9ms) Rendering rescues/layout (internal_server_error) Sickboy Dec 15 2011, 13:29 You probably have migrated the goldberg structure but not the actual structure for this rails project. Probably best to remove the DB, create again, and run rake db:schema:load this should migrate both. Otherwise first rake db:migrate and then rake goldberg:migrate That reminds me, when you are succesfully logged in, you will have to edit the goldberg permissions in the CMS, and give yourself access to all the controllers.. TeTeT Dec 15 2011, 13:52 Thanks, one step further. Now it fails with 'undefined method site_name' for nil:NilClass'. I guess I missed another setup step. Given that I've actually never touched rails and ruby only on the surface, no surprise it's a bumby road. BTW if you're low on time, this is certainly not a pressing topic at all. Sickboy Dec 15 2011, 14:10 Missing default goldberg system data, try: rake goldberg:load_bootstrap TeTeT Dec 15 2011, 17:23 Tried load_bootstrap, even after deleting the db again, but had no success. I haven't really found any good README on how to get this goldberg stuff running ...  I just did a plain goldberg / ruby install and it worked: 3716 rails toast 3717 ls 3718 cd vendor/plugins/ 3719 ls 3720 cd goldberg/ 3721 ls 3722 cd rai 3723 ls 3724 cd toast/ 3725 git init 3726 git submodule add git://github.com/lardawge/goldberg.git vendor/plugins/goldberg 3727 ls 3728 find . -iname goldberg 3729 vi config/database.yml 3730 rake goldberg:install 3731 rake goldberg:theme[goldberg] 3732 ls 3733 ruby script/server ---------- Post added at 06:23 PM ---------- Previous post was at 05:40 PM ---------- I skipped the server setup for now and tried to convert the hpp to yaml. But this failed also:$ ruby ConfigParser.rb config
Parsing
ConfigParser.rb:359:in initialize': No such file or directory - config.xml (Errno::ENOENT)
from ConfigParser.rb:359:in open'
from ConfigParser.rb:359:in read_xml'
from ConfigParser.rb:254:in parse!'
from ConfigParser.rb:580
from ConfigParser.rb:578:in each'
from ConfigParser.rb:578

I then created an empty config.xml file (touch config.xml) and the next error popped up:

\$ ruby ConfigParser.rb config
Parsing
Processing mainhash
ConfigParser.rb:523:in process_element': undefined method attributes' for nil:NilClass (NoMethodError)
from ConfigParser.rb:266:in parse!'
from ConfigParser.rb:580
from ConfigParser.rb:578:in each'
from ConfigParser.rb:578

I have those files in the working directory, plus dozens of empty artefacts from the failed parse:

-rw-r--r-- 1 spindler spindler 521 2011-12-15 10:55 mysql_utf8.rb
-rw-r--r-- 1 spindler spindler 14650 2011-12-15 10:55 ConfigParser.rb
-rw-r--r-- 1 spindler spindler 4815439 2011-12-15 18:12 config.hpp
-rw-r--r-- 1 spindler spindler 2304 2011-12-15 18:13 config_template.hpp
-rw-r--r-- 1 spindler spindler 1494 2011-12-15 18:13 CfgDefaultWeapons.hpp
-rw-r--r-- 1 spindler spindler 38713 2011-12-15 18:13 CfgDefaultVehicles.hpp

Sickboy
Dec 15 2011, 17:24
You're missing CfgConvert.exe from the BI Tools
Location for me is: C:\Program Files (x86)\Bohemia Interactive\Tools\BinMake\cfgconvert

BI Tools @<hidden> http://community.bistudio.com/wiki/BI_Tools_2.5

Will get back to you on the Goldberg part ;)

TeTeT
Dec 15 2011, 17:40
Thanks, I added a script named ConfigConvert.exe to my path, but the parser calls cfgConvert. After renaming the script, I got a config.yml. But they only contain a few classes it seems. Maybe because almost all entries are saved to 'other.hpp'?

Sickboy
Dec 15 2011, 17:44
Please copy the output to www.pastie.org or similar.

Not all root classes are imported currently, these are; http://www.pastie.org/3022124
You did Dump, then Trim, and the output of that put as config.hpp?
Otherwise perhaps put up the whole folder on some filespace.

Re Goldberg, this works:
rake db:create
rake db:migrate
rake goldberg:migrate
rake goldberg:install

If you get any errors please paste em as well.

TeTeT
Dec 15 2011, 18:54
trimconfig didn't work for me, so I skipped it. Probably the dump output I copied isn't of a format that the parser understands? Here is the dump I try to process:
http://tspindler.de/arma/UNS-classes.zip
I called trimconfig like this:

ruby trimconfig.rb config.hpp tmp

I'll try the recommended setup for the server next, thanks.

Sickboy
Dec 15 2011, 20:02
To get the proper dump:
- Delete arma2oa.rpt
- Start game, with dumpconfig mission, and initiate the dump
- When dump complete, take arma2oa.rpt, feed it to trimconfig.rb, rename the output file to config.hpp
- Use this config.hpp in the folder with the templates and CfgConvert.exe
- run ConfigParser.rb from the folder
- config.yml should be full of the classes i've listed earlier

TeTeT
Dec 16 2011, 11:49
I've now used a different dumpconfig script, found on armaholic (dumpConfig.7z) that dumps the config into the rpt and can be parsed. I had to change the newline chars with "tr -d '\15\32'", so the rpt could be used with trimconfig.rb. When converting the config.hpp I discovered that I need a 2 GB virtual machine, as the system was running out of memory with 1GB. The subsequent 'rake sync' took quite a while, I'd say around one hour or more to complete. Probably because a sqlite db on a vm is not particularly fast.

Now the data seems to be imported, but I cannot access it. In the setup of goldberg I found some cfg* pages and made them publicly available. However, when I access cfg_weapons I get this error: http://pastebin.ubuntu.com/772078/. As it mentions ACE and ACEX I wonder if I need to disable that somehow, as I hadn't had it loaded when doing the dump.

Is there a way to setup the browser to use a different port? A grep revealed that there's a setting in vendor/plugins/goldberg/db/SystemSettings.yml. I would love to have two servers running, one for Vietnam: the experience, one for the Unsung mod.

Sickboy
Dec 16 2011, 12:58
Nice! I have fixed the error, please pull and restart the app.

Performance is indeed limited by SQlite and VM, but also on MySQL it is somewhat slow.
I'll try to profile it soon and see what can be done about the performance.

You can specify the port to use when starting the webserver: ruby script/server --port=21212

You can add as many Mods/Configs as you want though, no need to run on multiple ports.
Except that the webrick webserver that is used in this configuration is designed for single-user (development/test).
For production you're recommended to run Passenger with Nginx, or Apache2: http://www.modrails.com/

BTW there's several rake tasks included that generate lists or diffs, with specific purposes. You can leverage the powerful back-end to do the same for your needs.

If you need any functionality or want to report bugs / feedback, please find the issue tracker @<hidden> http://dev-heaven.net/projects/six-configparser-web/issues

TeTeT
Dec 16 2011, 13:10
Thanks for your help again! Unfortunately even with the new trunk pulled, it still fails with RuntimeError (Called id for nil, which would mistakenly be 4 -- if you really wanted the id of nil, use object_id):. Maybe something is still not configured correctly? I've changed the properties of cfg_vehicles and cfg_weapons in administration > setup > controllers / actions. In that menu I also see some missing classes: main, modpermissions, repositories, serverpermissions, servers, sixgroups, sixusers.

Sickboy
Dec 16 2011, 14:08
Those models have no controllers, that's by design.

You are correct re the id error - my mistake, please pull again.

TeTeT
Dec 16 2011, 14:28
Still no joy, wants to access a non existing method, I'd say: http://pastebin.ubuntu.com/772232/

Sickboy
Dec 16 2011, 14:30
Sorry, not my day, evidently... Please pull again :)

TeTeT
Dec 16 2011, 14:42
Works! Thanks a lot for all of your help, wouldn't have gone anywhere without it.

How would I now use one server instance for multiple configs? E.g. one for VTE, one for Unsung?

On the 'not my day', trust me I know exactly how that feels and I had plenty of them!

Sickboy
Dec 16 2011, 14:51
NP :)

You can manage versions and mods at /versions and /mods
Relationship: Mods has_many Versions

So you can make a VTE mod, and an Unsung mod, and attach versions to these mods, depending where the version belongs :)

You can simply prepare a new config.hpp of another mod, and run the rake sync task again.
This will create a new Version entry, the version id can be used at each page, e.g /cfg_vehicles/classlist/?version=1 etc.

On the classlist pages you find the ModVersion selector up top, e.g see: http://browser.dev-heaven.net/cfg_weapons/classlist
There (on the official Config Browser site) should be a lot of ACE versions in there and several game versions ;)

TeTeT
Dec 16 2011, 15:36
Seems the parsing of the unsung config fails:

File CfgVehicles.hpp, line 24602: /Root/CfgVehicles.LAND_uns_shopOld_01: Undefined base class 'LAND_uns_bar'
Config : some input after EndOfFile.
W:\C\El\ParamFile\paramFile.cpp(753) : Class destroyed, but still locked
ConfigParser.rb:359:in initialize': No such file or directory - config.xml (Errno::ENOENT)
from ConfigParser.rb:359:in open'
from ConfigParser.rb:254:in parse!'
from ConfigParser.rb:580
from ConfigParser.rb:578:in each'
from ConfigParser.rb:578

The config.hpp is available at http://tspindler.de/arma/config.zip

Seems the ordering of classes is important, e.g. if a class inherits from one that is defined later, the parse fails. Do you parse in one pass? I guess to support the config I got for unsung one needs to use two-pass parsing, first getting the names, then constructing the object tree and in the second pass putting the data in.

Sickboy
Dec 17 2011, 09:54
That's rather odd, never really have had an AllInOne config export where the classes inside rootclasses would be mixed up,
seems strange - the config should be dumped in the order it is in memory, and you'd expect the order to be correct for the inheritance etc to work.
However perhaps their inheritance tree and requiredAddons tree is not 100% valid, while the game can handle this, the other utilities cannot.

The issue is not with my parser but with the CfgConvert.exe utility who converts the config into an XML, while my parser reads the XML. Currently the conversion to XML fails.

The config.hpp is simply not a valid config. Perhaps you can fix it up manually if there are not that many classes.
Otherwise the dumpconfig script should be reviewed to see if it can be improved, and write classes in the correct order.

Of course if we go more ninja, we could write our own .hpp parser, or a pre-parser that would re-order the classes or so, but not really have time for this currently.

TeTeT
Dec 17 2011, 13:15
I guess the dump file is just too broken. I re-ordered a dozen classes so the inheritance order is respected, but then I bump into this problem:

Processing mainhash
Proc
Proc2
Processing WeaponAim: Pos10Vector
Processing Level0: Pos10Vector
Processing Target: Pos10Vector
Processing Velocity: Pos10Vector
Processing EventHandlers: EventHandlers
Processing AnimationSources: AnimationSources
Processing ViewPilot: ViewPilot
Processing Turrets: Turrets
Processing LEFT_TURRET: MainTurret
Processing AnimationSources: AnimationSources
Processing Turrets: Turrets
Processing MainTurret: MainTurret
Processing ViewOptics: ViewOptics
ConfigParser.rb:164:in find_class': stack level too deep (SystemStackError)
from ConfigParser.rb:165:in find_class'
from ConfigParser.rb:161:in find_class'
from ConfigParser.rb:307:in proc2'
from ConfigParser.rb:91:in each_pair'
from ConfigParser.rb:91:in each'
from ConfigParser.rb:91:in each_pair'
from ConfigParser.rb:300:in proc2'
from ConfigParser.rb:310:in proc2'
... 19 levels...
from ConfigParser.rb:273:in parse!'
from ConfigParser.rb:580
from ConfigParser.rb:578:in `each'
from ConfigParser.rb:578

There are a dozen or more definitions like 'ViewOptics : ViewOptics' - I suspect this means inherit from the upper class? I checked the other dump I have and there are the same definitions, but processed without a problem. Probably there's some sort of circular inheritance setup with the unsung mod, class a : b, class b : c, class c : a, or something alike.

I removed the 'ViewOptics : ViewOptics' declarations for a test and then I ran out of stack space upon 'NewTurret'. Even after setting stacksize in ulimit to 100k, it run out of stacksize. Guess the config is just too broken :(

Sickboy
Dec 20 2011, 07:14
Please create a ticket on the issue @<hidden> http://dev-heaven.net/projects/six-configparser-web/issues
When I have some time coming days I will try to look into it.

TeTeT
Dec 20 2011, 07:39
Cheers, here's the bug report:
http://dev-heaven.net/issues/27207

Thanks a lot, but be aware that this is a low priority issue, if any priority at all.