You're doing it wrong. Focus on high-level optimizations first (like culling), only if you're out of any useable high level optimizations and the design is 'finished' go crazy with low-level optimizations. There are of course exceptions to this advice, but this is not the case.
After doing proper optimizations you may also realize it doesn't require high end GPU you're developing for and could be useable also for mere mortals too.
This isn't focused on FPS performance but on VRAM usage. Your point still stands though. I'll focus on other more important stuff. Thanks to everyone.