No, you can't just bind it to 127.0.0.1 and expect it to work for everyone in the world. You'll have to bind to the machine's IP(ipconfig or ifconfig depending on windows / linux) or you use 0.0.0.0 which binds to all addresses available. Then for the client, you connect to the external network IP.
Lolz Agro, they're all the same I thought.

1 2 3
| clientSocket = new Socket("127.0.0.1", clientPort); clientSocket = new Socket("0.0.0.0", clientPort); clientSocket = new Socket("localhost", clientPort); |
Maybe you misread

.
I asked, if I were to program a basic client/server pair and do all the testing for it only on '127.0.0.1', would it work the same if I were to host the server on a 'VPS'?
The VPS's IP would be the 'external network IP'.
(Which the client connects to)