If you're using vertex arrays, you're sending the data to the card every time you call glColorPointer/glVertexPointer or with every glDrawArrays/glDrawElements call (I think this depends on how smart the graphics card is, and when it does the transfer). If you are using VBOs, there will probably be less of a performance hit, since all the buffers are on the graphics card anyway.

If you consider OpenGL as a client-server model, enabling the color buffer doesn't change the amount of work being done server side when determining a vertex's color, since it would just take the color from the last call to glColor3f.  There would be more work on the client side (or in the bridge between client and server), to change this color with each vertex.

For many purposes, I don't think you'll see much of a performance hit compared to the draw command itself, or using more detailed geometry.  However, I think it's usually recommended to take the practice of not enabling things that you don't need to.  So if you don't use the colors or texture coordinates for a draw command, then don't have them enabled that time around.  It would be fastest of all to sort by what needs to be enabled so that these changes are across multiple calls to the draw methods.