For faster access, you have a couple options. One is to grab the raw int array backed by the BufferedImage, which is the fastest:
int pixels = ((DataBufferInt)img.getRaster().getDataBuffer()).getData();
The problem is, sometimes ImageIO will load an image using a type other than TYPE_INT_ARGB. So it might be backed by a byte array, or an int array of TYPE_INT_RGB. A simple solution is to create a new image of the necessary type. The benefit of this method is that you can modify the pixel array to change how the image is rendered.
In your case, since you don't need to render the buffered image, you might rather use this (which will do the type conversion for you):
int pixels = img.getRGB(0, 0, imgWidth, imgHeight, null, 0, imgWidth);
As for how to implement something like this.. there are many ways. I like using enums for small projects. You could also use EnumMap or HashMap to query the colors. Here is one possible implementation:http://www.java-gaming.org/?action=pastebin&id=432