ServerSocket.bind() hanging?

[elias]

Hi,

Implementing loopback networking for TT I ran into the problem that the ServerSocket.bind call can hang under some circumstances:

1   2   3   4

ServerSocketChannel server_channel = ServerSocketChannel.open();         server_channel.configureBlocking(false);         SocketAddress address = new InetSocketAddress(port);         server_channel.socket().bind(address);

It turns out that the last line hangs for several minutes in SUSE 9 when the firewall is activated. After that, the socket works normally and can be connected to. It seems like some request is timing out, but which one exactly? The firewall is accepting all outgoing connections.

- elias

[blahblahblahh]

Timeouts that long are usually due to DNS timeouts (or automatic authentication - e.g. "incorrect password" retries in broken Windows Networking implementations like NT4, 95 and 98 used to have timeouts measured in minutes ... I take it you aren't using any remote authentication?)

AFAIAA all the "features" in java where it uses DNS (e.g. automatic unwanted reverse lookups) have by now been classed as bugs and fixed , so I would suggest it's not a problem within java.

Are you using IPChains (EDIT: or IPtables, same difference )? If so, what is your ruleset, and have you narrowed down which rule(s) is causing the problem?

Stab in the dark: If you're using the default "security features" there are a large number of things that are being done automatically, some of which are really stupid, and often the distro won't actually tell you what it's done, let alone allow you to pick-n-mix (the linux distros often don't seem to be able to appreciate there's a middle ground between kernel-programmers and 5-year-olds in their user base!). If so, you'll need to run through the list of "helpful" things that are being done. There are all sorts of advanced options by now to do things like artificially delay the initial response to certain ports / packet sigs in order to weaken DoS attacks whilst still allowing legitimate traffic through.

[elias]

More info: I'm using the default firewall in SUSE 9, which means iptables with god knows how many rules. I've tried to track the offending rule by making the firewall log every dropped _and_ accepted packet to syslog. However, no log entry is ever generated.

The real problem is that I can't see how the firewall should ever be involved when a port is bound to? I have no problem connecting to the port after it is bound (a few minutes later).

Note that all the normal linux services restart almost instantly, hinting that native applications can bind ports without problems. Of course, most of them probably run on priviledged ports and I'm not sure if that gives extra permissions.

- elias

Games published by our own members! Check 'em out!

[elias]

I did a minimal test program showing the problem:

1   2   3   4   5   6   7   8   9   10   11   12

import java.net.*;  import java.io.*;  public class TestServer {          public static void main(String[] args) {                  try {                          new ServerSocket(21000);                  } catch (IOException e) {                          System.exit(10);                  }                  System.out.println("Er der flere chips?");          } }

And it hangs a few minutes as expected.

- elias

[blahblahblahh]

This probably won't help much, but have you tried "stop"ping one of your native services (e.g. FTP), then (as root) running your simple java test on the port that service was on? I'm sure it won't have any positive effect, but if it did...

It's been a while since I used SuseConfig but it was pretty comprehensive - does it have options to "downgrade" the secureness of your system?

Also, how are you turning the firewall off? (I'm just suspicious that perhaps when you turn it off, more is happening than just stopping iptables).

Finally, can you dump your iptables and post, or is it REALLY huge (more than about 30 rules)? IIRC command is something like

iptables -vL

Oh, and one more thing; are there any delays when you run that last command? If so, do they vanish when you do "iptables -nvL" (or NvL? - arrgh; hate case-sensitivity!)

[elias]

I did a corresponding test in native C with the same port:

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32   33   34   35   36   37

#include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <stdio.h> int main() {       struct sockaddr_in address;       int desc = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);       int retval;       if (desc == -1) {             perror("socket() failed");             return 10;       }       address.sin_family = AF_INET;       address.sin_port = htons(21000);       address.sin_addr.s_addr = INADDR_ANY;       retval = bind(desc, (struct sockaddr*) &address, sizeof(address));       if (retval != 0) {             perror("bind() failed");             return 20;       }       retval = listen(desc, 5);       if (retval == -1) {             perror("listen() failed");             return 35;       }       printf("Socket is listening\n");       struct sockaddr_in accept_addr;       int size = sizeof(accept_addr);       int accept_desc = accept(desc, (struct sockaddr*) &accept_addr, &size);       if (accept_desc == -1) {             perror("accept failed");             return 30;       }       close(desc);       return 0; }

Need I say that it works without delay? After the listen() call, the port is open in a "netstat -l -n --inet".

BTW, blahblahblah you've probably debugged a lot of networking applications, and I'm curious as to which tools you're using to simulate different network environments? Specifically, I'd like to specify the mean lag, bandwidth and packet loss on my ethernet devices - 100 Mbit LAN testing is rather far from the rogue internet environment.

- elias

[elias]

If I shut down the ftp service and run the minimal test on port 21, the program still hangs.

There's no security granularity in the firewall config in Yast2. I stop the firewall with "rcSuSEfirewall2 stop", and I'm not sure what else that script does.

I tried flushing the rules with iptables --flush and checked that there were no rules left with iptables --list. And the testprogram still hangs, so the problem must be somethinge else, right? Now, the core firewall script is unfortunately miles long and I have little chance of figuring it out from that.

But what could the difference between the C and java code sample be?

- elias

[Golthar]

How many IP's does your system have?
Perhaps the C variant binds to just one IP and the Java variant tries to bind to all?

I had a problem with my Linux server like this once after I specified the right interface/IP adress to bind on, all was fine

[elias]

No, specifying a particular interface doesn't seem to work :/

- elias

[Golthar]

Sorry it was not helpful.

By the way it sounds, one of the interfaces is having an hard time binding.
Did you try doing: demsg after finaly binding?

It opens the last kernel logs, I usualy find things like dropped packets in there, perhaps it shows you more networking data