I would go with Collada - maybe it is slower to load, but it can contain a lot of informations, can be easily adapted to any needs and interpreter is easy to write.
Almost all games go with their own format. Then you just need to write a converter from Collada (for example) to your own binary format. The advantage is that loading them is lighting fast since you can format the data in any way you want.
After reading this I think I'll follow your advice theagentd. It'd suit my needs and beyond if I wrote a converter back to Collada.
Thanks for the quick replies!