c - Avoiding TIME_WAIT

Question

I'm trying to avoid TIME_WAIT in a client. I connect and then set O_NONBLOCK and SO_REUSEADDR. I call read until it returns 0. When read returns 0, the errno is also 0. I interpreted this as a sign that the server closed the connection. However, if I call close, the socket is set to TIME_WAIT, as confirmed by netstat.

Since I make a number of connections to the same host / port, I eventually start seeing "Address in use" errors (see http://hea-www.harvard.edu/~fine/Tech/addrinuse.html).

Should I be calling close after read returns 0? If I don't will the file descriptor be released?

Answer 1

The side that is that one that initiated the closing of the connection is the one that ends up in the TIME_WAIT state. read() returning 0 is supposed to indicate that the server closed the socket first, so yes - this should mean that the TIME_WAIT ends up on the server side, and the client goes through LAST_ACK.

At the end of the day, you can't avoid a TIME_WAIT state. Even if you succeed in moving it from the client to the server side, you still can't re-use that (server host, server port, client host, client port) tuple until the TIME_WAIT is over (regardless of which side it's on).

Since three parts of that tuple are fixed in your scenario (server host, server port, client host), you really only have these options:

• Try to make more client ports available. Some operating systems only use a small range of the available ports for "ephemeral ports" by default (I'm not sure about OSX in this regard). If that's the case, see if you can change the range with a configuration tweak in the OS, or alternatively have the application hunt for a working port with bind()/connect() in a loop until the connection works.

• Expand the number of client host values available, by using multiple IP addresses on your client. You'll have to have the application bind() to one of these IP addresses specifically though.

• Expand the number of server host/server port values available, by using multiple ports and/or IP addresses on the server. The client will need to pick one to connect to (round robin, random, etc).

• Probably the best option, if it's doable: refactor your protocol so that connections that are finished aren't closed, but go into an "idle" state so they can be re-used later, instead of opening up a new connection (like HTTP keep-alive).