Tool for class discovery?

[TeTeT]

Hello,

is there a tool to discover all the classes defined in a mod? Lately I've been looking at the unsung mod and haven't discovered any class documentation on their site and wonder if there's a generic way to export the class structure of a mod to some sort of text file, probably using any markup language.

I guess one can unpbo and unrap the config.bin's and then parse the config.cpp for the needed information. Or use a class viewer from arma itself and use the rpt file for copy'n'paste. Does a tool that automates this already exist?

[Sickboy]

I'm doing some for the official game versions and for ACE mod over at http://browser.dev-heaven.net

The project can also run locally, and then you can import any AllInOne to get the same results, but it's not really setup for usage by 3rd parties at this time.

You can generate an AllInOne config when the mod is loaded, and extract your info easily from that: http://community.bistudio.com/wiki/AllInOne_Config

[TeTeT]

Thanks for the pointer! I used the dumpConfig script from http://www.ofpec.com/forum/index.php?topic=35236.0 and now got a all in one config file.
[edit]
Cloned your browser via git clone git://git.dev-heaven.net/six-configparser-web. Now on to setting it up locally, no idea if ruby on rails works fine on ubuntu 10.04 lts...

[Sickboy]

Should work fine on Ubuntu, however some utilities are only available for windows, like BI tools.

For config import you'll need at least 1 bi tool, but you can do do this part on Windows.

But as said before, the project is not setup to be used by anyone but me, and I do not have the time to support you currently.

At least a short version of what needs to be done:
- cfgConvert.exe from BI-Tools binpbo needs to be available. Currently expected to be in PATH, or in the folder where you run ConfigParser.rb from
- AllInOneConfig generated
- Generated file should be cleaned up with trimconfig.rb
- Cleaned file should be placed in a folder as config.hpp together with these templates: config_template.hpp, CfgDefaultWeapons.hpp, CfgDefaultVehicles.hpp
from within that folder, run ruby path\to\six-configparser-web\lib\configparser.rb

The above has to be done on a windows machine. Below can be on linux:

- six-configparser-web Database created and schema loaded
- Outputted config.yml should be placed in the root of six-configparser-web project and from the root should be ran: rake sync

I have updated the repository, added trimconfig.rb, and my frankenstein script autodump.rb that runs the game, exports config from game to rpt, import from rpt to xml, and from xml to yaml :D

Windows Ruby @<hidden> http://www.rubyinstaller.org/


BTW, I am working on a much improved version of the parser using sbsmac's ARMA assembly for .NET, but it's a nice to have - rainy sunday..

[TeTeT]

Cheers, thanks for providing the additional scripts! So far I had good success with using all sorts of Windows arma tools with wine, for example cpbo, extractpbo, texview, unRap are all working.

But I cannot even get the rails server up, as it's missing some goldberg table. I guess goldberg is responsible for the layout? I followed the info in vendor/plugins/goldberg/README.rdoc and run 'rake goldberg:install' and 'rake goldberg:theme[goldberg]', but still get this error when connecting to localhost:3000:



spindler@<hidden>:~/Work/Arma2/tools/six-configparser-web$ ruby script/server
=> Booting WEBrick
=> Rails 2.3.12 application starting on http://0.0.0.0:3000
NOTE: SourceIndex.new(hash) is deprecated; From /var/lib/gems/1.8/gems/rails-2.3.12/lib/rails/vendor_gem_source_index.rb:100:in `new'.
Cannot find or read /home/spindler/Work/Arma2/tools/six-configparser-web/config/newrelic.yml
=> Call with -d to detach
=> Ctrl-C to shutdown server
[2011-12-15 14:16:16] INFO WEBrick 1.3.1
[2011-12-15 14:16:16] INFO ruby 1.8.7 (2011-06-30) [x86_64-linux]
[2011-12-15 14:16:22] INFO WEBrick::HTTPServer#start: pid=25126 port=3000


Processing Goldberg::ContentPagesController#view_default (for 127.0.0.1 at 2011-12-15 14:16:30) [GET]
Goldberg::SystemSettings Load (0.7ms) SELECT * FROM "goldberg_system_settings" LIMIT 1

ActiveRecord::StatementInvalid (Could not find table 'sessions'):
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'

Rendered rescues/_trace (26.8ms)
Rendered rescues/_request_and_response (0.9ms)
Rendering rescues/layout (internal_server_error)

[Sickboy]

You probably have migrated the goldberg structure but not the actual structure for this rails project.

Probably best to remove the DB, create again, and run rake db:schema:load
this should migrate both. Otherwise first rake db:migrate and then rake goldberg:migrate

That reminds me, when you are succesfully logged in, you will have to edit the goldberg permissions in the CMS, and give yourself access to all the controllers..

[TeTeT]

Thanks, one step further. Now it fails with 'undefined method `site_name' for nil:NilClass'. I guess I missed another setup step. Given that I've actually never touched rails and ruby only on the surface, no surprise it's a bumby road.

BTW if you're low on time, this is certainly not a pressing topic at all.

[Sickboy]

Missing default goldberg system data, try: rake goldberg:load_bootstrap

[TeTeT]

Tried load_bootstrap, even after deleting the db again, but had no success. I haven't really found any good README on how to get this goldberg stuff running ...
[edit]
I just did a plain goldberg / ruby install and it worked:



3716 rails toast
3717 ls
3718 cd vendor/plugins/
3719 ls
3720 cd goldberg/
3721 ls
3722 cd rai
3723 ls
3724 cd toast/
3725 git init
3726 git submodule add git://github.com/lardawge/goldberg.git vendor/plugins/goldberg
3727 ls
3728 find . -iname goldberg
3729 vi config/database.yml
3730 rake goldberg:install
3731 rake goldberg:theme[goldberg]
3732 ls
3733 ruby script/server


---------- Post added at 06:23 PM ---------- Previous post was at 05:40 PM ----------

I skipped the server setup for now and tried to convert the hpp to yaml. But this failed also:


$ ruby ConfigParser.rb config
Parsing
ConfigParser.rb:359:in `initialize': No such file or directory - config.xml (Errno::ENOENT)
from ConfigParser.rb:359:in `open'
from ConfigParser.rb:359:in `read_xml'
from ConfigParser.rb:254:in `parse!'
from ConfigParser.rb:580
from ConfigParser.rb:578:in `each'
from ConfigParser.rb:578


I then created an empty config.xml file (touch config.xml) and the next error popped up:


$ ruby ConfigParser.rb config
Parsing
Processing mainhash
ConfigParser.rb:523:in `process_element': undefined method `attributes' for nil:NilClass (NoMethodError)
from ConfigParser.rb:266:in `parse!'
from ConfigParser.rb:580
from ConfigParser.rb:578:in `each'
from ConfigParser.rb:578


I have those files in the working directory, plus dozens of empty artefacts from the failed parse:



-rw-r--r-- 1 spindler spindler 521 2011-12-15 10:55 mysql_utf8.rb
-rw-r--r-- 1 spindler spindler 14650 2011-12-15 10:55 ConfigParser.rb
-rw-r--r-- 1 spindler spindler 4815439 2011-12-15 18:12 config.hpp
-rw-r--r-- 1 spindler spindler 2304 2011-12-15 18:13 config_template.hpp
-rw-r--r-- 1 spindler spindler 1494 2011-12-15 18:13 CfgDefaultWeapons.hpp
-rw-r--r-- 1 spindler spindler 38713 2011-12-15 18:13 CfgDefaultVehicles.hpp

[Sickboy]

You're missing CfgConvert.exe from the BI Tools
Location for me is: C:\Program Files (x86)\Bohemia Interactive\Tools\BinMake\cfgconvert

BI Tools @<hidden> http://community.bistudio.com/wiki/BI_Tools_2.5

Will get back to you on the Goldberg part ;)

[TeTeT]

Thanks, I added a script named ConfigConvert.exe to my path, but the parser calls cfgConvert. After renaming the script, I got a config.yml. But they only contain a few classes it seems. Maybe because almost all entries are saved to 'other.hpp'?

[Sickboy]

Please copy the output to www.pastie.org or similar.
What about the XML?

Not all root classes are imported currently, these are; http://www.pastie.org/3022124
You did Dump, then Trim, and the output of that put as config.hpp?
Otherwise perhaps put up the whole folder on some filespace.


Re Goldberg, this works:
rake db:create
rake db:migrate
rake goldberg:migrate
rake goldberg:install

If you get any errors please paste em as well.

[TeTeT]

trimconfig didn't work for me, so I skipped it. Probably the dump output I copied isn't of a format that the parser understands? Here is the dump I try to process:
http://tspindler.de/arma/UNS-classes.zip
I called trimconfig like this:

ruby trimconfig.rb config.hpp tmp

I'll try the recommended setup for the server next, thanks.

[Sickboy]

To get the proper dump:
- Delete arma2oa.rpt
- Start game, with dumpconfig mission, and initiate the dump
- When dump complete, take arma2oa.rpt, feed it to trimconfig.rb, rename the output file to config.hpp
- Use this config.hpp in the folder with the templates and CfgConvert.exe
- run ConfigParser.rb from the folder
- config.yml should be full of the classes i've listed earlier

[TeTeT]

I've now used a different dumpconfig script, found on armaholic (dumpConfig.7z) that dumps the config into the rpt and can be parsed. I had to change the newline chars with "tr -d '\15\32'", so the rpt could be used with trimconfig.rb. When converting the config.hpp I discovered that I need a 2 GB virtual machine, as the system was running out of memory with 1GB. The subsequent 'rake sync' took quite a while, I'd say around one hour or more to complete. Probably because a sqlite db on a vm is not particularly fast.

Now the data seems to be imported, but I cannot access it. In the setup of goldberg I found some cfg* pages and made them publicly available. However, when I access cfg_weapons I get this error: http://pastebin.ubuntu.com/772078/. As it mentions ACE and ACEX I wonder if I need to disable that somehow, as I hadn't had it loaded when doing the dump.

Is there a way to setup the browser to use a different port? A grep revealed that there's a setting in vendor/plugins/goldberg/db/SystemSettings.yml. I would love to have two servers running, one for Vietnam: the experience, one for the Unsung mod.

[Sickboy]

Nice! I have fixed the error, please pull and restart the app.

Performance is indeed limited by SQlite and VM, but also on MySQL it is somewhat slow.
I'll try to profile it soon and see what can be done about the performance.

You can specify the port to use when starting the webserver: ruby script/server --port=21212

You can add as many Mods/Configs as you want though, no need to run on multiple ports.
Except that the webrick webserver that is used in this configuration is designed for single-user (development/test).
For production you're recommended to run Passenger with Nginx, or Apache2: http://www.modrails.com/

BTW there's several rake tasks included that generate lists or diffs, with specific purposes. You can leverage the powerful back-end to do the same for your needs.
Check lib\tasks.

If you need any functionality or want to report bugs / feedback, please find the issue tracker @<hidden> http://dev-heaven.net/projects/six-configparser-web/issues

[TeTeT]

Thanks for your help again! Unfortunately even with the new trunk pulled, it still fails with RuntimeError (Called id for nil, which would mistakenly be 4 -- if you really wanted the id of nil, use object_id):. Maybe something is still not configured correctly? I've changed the properties of cfg_vehicles and cfg_weapons in administration > setup > controllers / actions. In that menu I also see some missing classes: main, modpermissions, repositories, serverpermissions, servers, sixgroups, sixusers.

[Sickboy]

Those models have no controllers, that's by design.

You are correct re the id error - my mistake, please pull again.

[TeTeT]

Still no joy, wants to access a non existing method, I'd say: http://pastebin.ubuntu.com/772232/

[Sickboy]

Sorry, not my day, evidently... Please pull again :)

[TeTeT]

Works! Thanks a lot for all of your help, wouldn't have gone anywhere without it.

How would I now use one server instance for multiple configs? E.g. one for VTE, one for Unsung?

On the 'not my day', trust me I know exactly how that feels and I had plenty of them!

[Sickboy]

NP :)

You can manage versions and mods at /versions and /mods
Relationship: Mods has_many Versions

So you can make a VTE mod, and an Unsung mod, and attach versions to these mods, depending where the version belongs :)

You can simply prepare a new config.hpp of another mod, and run the rake sync task again.
This will create a new Version entry, the version id can be used at each page, e.g /cfg_vehicles/classlist/?version=1 etc.

On the classlist pages you find the ModVersion selector up top, e.g see: http://browser.dev-heaven.net/cfg_weapons/classlist
There (on the official Config Browser site) should be a lot of ACE versions in there and several game versions ;)

[TeTeT]

Seems the parsing of the unsung config fails:


File CfgVehicles.hpp, line 24602: /Root/CfgVehicles.LAND_uns_shopOld_01: Undefined base class 'LAND_uns_bar'
Config : some input after EndOfFile.
Error reading binary file 'config_tmp.hpp'
W:\C\El\ParamFile\paramFile.cpp(753) : Class destroyed, but still locked
ConfigParser.rb:359:in `initialize': No such file or directory - config.xml (Errno::ENOENT)
from ConfigParser.rb:359:in `open'
from ConfigParser.rb:359:in `read_xml'
from ConfigParser.rb:254:in `parse!'
from ConfigParser.rb:580
from ConfigParser.rb:578:in `each'
from ConfigParser.rb:578


The config.hpp is available at http://tspindler.de/arma/config.zip

[edit]
Seems the ordering of classes is important, e.g. if a class inherits from one that is defined later, the parse fails. Do you parse in one pass? I guess to support the config I got for unsung one needs to use two-pass parsing, first getting the names, then constructing the object tree and in the second pass putting the data in.

[Sickboy]

That's rather odd, never really have had an AllInOne config export where the classes inside rootclasses would be mixed up,
seems strange - the config should be dumped in the order it is in memory, and you'd expect the order to be correct for the inheritance etc to work.
However perhaps their inheritance tree and requiredAddons tree is not 100% valid, while the game can handle this, the other utilities cannot.

The issue is not with my parser but with the CfgConvert.exe utility who converts the config into an XML, while my parser reads the XML. Currently the conversion to XML fails.

The config.hpp is simply not a valid config. Perhaps you can fix it up manually if there are not that many classes.
Otherwise the dumpconfig script should be reviewed to see if it can be improved, and write classes in the correct order.

Of course if we go more ninja, we could write our own .hpp parser, or a pre-parser that would re-order the classes or so, but not really have time for this currently.

[TeTeT]

I guess the dump file is just too broken. I re-ordered a dozen classes so the inheritance order is respected, but then I bump into this problem:



Processing mainhash
Proc
Proc2
Processing WeaponAim: Pos10Vector
Processing Level0: Pos10Vector
Processing Target: Pos10Vector
Processing Velocity: Pos10Vector
Processing EventHandlers: EventHandlers
Processing AnimationSources: AnimationSources
Processing ViewPilot: ViewPilot
Processing Turrets: Turrets
Processing LEFT_TURRET: MainTurret
Processing AnimationSources: AnimationSources
Processing Turrets: Turrets
Processing MainTurret: MainTurret
Processing ViewOptics: ViewOptics
ConfigParser.rb:164:in `find_class': stack level too deep (SystemStackError)
from ConfigParser.rb:165:in `find_class'
from ConfigParser.rb:161:in `find_class'
from ConfigParser.rb:307:in `proc2'
from ConfigParser.rb:91:in `each_pair'
from ConfigParser.rb:91:in `each'
from ConfigParser.rb:91:in `each_pair'
from ConfigParser.rb:300:in `proc2'
from ConfigParser.rb:310:in `proc2'
... 19 levels...
from ConfigParser.rb:273:in `parse!'
from ConfigParser.rb:580
from ConfigParser.rb:578:in `each'
from ConfigParser.rb:578


There are a dozen or more definitions like 'ViewOptics : ViewOptics' - I suspect this means inherit from the upper class? I checked the other dump I have and there are the same definitions, but processed without a problem. Probably there's some sort of circular inheritance setup with the unsung mod, class a : b, class b : c, class c : a, or something alike.

[edit]
I removed the 'ViewOptics : ViewOptics' declarations for a test and then I ran out of stack space upon 'NewTurret'. Even after setting stacksize in ulimit to 100k, it run out of stacksize. Guess the config is just too broken :(

[Sickboy]

Please create a ticket on the issue @<hidden> http://dev-heaven.net/projects/six-configparser-web/issues
When I have some time coming days I will try to look into it.

[TeTeT]

Cheers, here's the bug report:
http://dev-heaven.net/issues/27207

Thanks a lot, but be aware that this is a low priority issue, if any priority at all.