1. Here we recommend OpenGL which handles most of this anyway on hardware.
2. The amount of performance increase would be pathetic. Your time would be better spent optimising something else.
3. It makes code messy.
4. If it were a good idea, it would already be done.
Simply, if you are at the stage where you need to optimise colour operations, you need to be using OpenGL.
I totally agree with what you have said, except I believe the performance gain could be huge in some scenarios. In my swimming pool project, there is a place where the camera is able to look at the bottom of the pool through a foggy window. To render each pixel on the bottom of the pool, I have to literally unpack and pack the color data 4 times. That definitely a lot of overhead there