video memory is normally accessed through AGP. This might slow down graphics in some circumstances. I have a cheap motherboard that uses system RAM for video and, although sharing RAM between CPU and video is generally not optimal, there is no transfer of data through the AGP. This makes my cheap motherboard very fast when writing/reading on 'video' RAM. The same might apply to your linux box.
Actually, I might be partially wroung here. My motherboard might still use AGP circuitry to negotiate the reading/writing on the video part of RAM, which might slow things down a tad. It might even send the data through the bus at 1x, 2x or4x speed.
In any case, this could still apply to your linux box, as it could have a completely different and straight forward architecture. I miss the old Atari ST days, that machine had all of system RAM available to video, working at twice the speed (16MHz!!!) of the CPU, the memory was accesed by both on alternate cycles. So the CPU never had to wait to use RAM. Anybody have any 4GHz DIMMs?
PD: Me, I didn't mean to piss me off by correcting myself, so don't flame me.