Yesterday i've had a trivial problem that created me a lot of headache.
My layers.cfg was perfect, my textures was good and all was fine, but in my map i have seen only one texture on mask, the texture applied to 0,0,0 (black) mask colour. Changing the texture on this layer worked fine (no rvmat or texture issue), but all other textures was not shown at all.
This happened in Bulldozer and in the compiled Arma2 world, with and without clutter.
The layer pictures in data\layers was good, but i cannot view any other texture than the sand one (my coastal/sea texture applied to black mask colour).
My map size is 10240 meters, my mask and satpic is 10240 but i've used a reduced one (1024x1024) for debugging and map creation purposes (calculator has been used properly and problem happened with 5x5, 10x10 and 40x40 texture size).
Well, the problem was in my case the mask size was TOO SMALL.
All was perfect but nothing worked. I've finally tried to increase mask to 2048x2048and all worked fine!
Maybe the problem is trivial, but it took 2 days for me to get out. Hope this helps