Well only the diagonal values should be 0.1 so I expected it to scale the vertecies down but then the 4,4 value in the matrix should be 1.

I got it working now but it is funny to see that if a point is (5, 4, 0, 0.1) then gl draws it like (50, 40, 0, 1)

gl doesn't draw based on pixel values. It draws based on window size. If you try drawing something with no projection matrix setup, the origin will be the center of the window. Positive x-axis right, positive y-axis up. Point with of coordinates (0, 0) would be right at the center of the window. Point (1,1) would be the most upper right pixel of the screen. Point (-1,-1) would be most bottom left pixel. Basically, gl understands everything with float values 1f being equal half of the window size.

This does mean that if your window size is say 500,100 that if you tell opengl to render a square at 0,0 of coordinates 1,1 it would render a square which would take up 250 pixels horizontally and only 50 pixels vertically.

You need to include window width/height ratio in your projection matrix to make opengl work properly.