I think torrents are ideal solution. BIS and other reliable volunteers would just host trackers which doesn't require much bandwidth and game servers would act as seeds (with custom upload limits of course). Lobbies for Spring RTS (real-time strategy engine) already does it (SpringLobby and Zero-K). How should clients participate is a question. I suggest that clients should only seed mods that they're just using (playing) with configurable limits and possibility to opt-out from seeding.
Another thing are versions and updates. Downloading 3GB mod again for one-line fix isn't good. This could be solved by seeding also deltas. Yum package manager does it too (I've no idea how it deals with compression).