Intermittent bug with asyncio and MS Edge
On Mon, Mar 23, 2020 at 8:03 PM Frank Millman <frank at chagford.com> wrote:
> On 2020-03-22 12:11 PM, Chris Angelico wrote:
> > On Sun, Mar 22, 2020 at 8:30 PM Frank Millman <frank at chagford.com> wrote:
> >> On 2020-03-22 10:45 AM, Chris Angelico wrote:
> > If you can recreate the problem with a single socket and multiple
> > requests, that would be extremely helpful. I also think it's highly
> > likely that this is the case.
> I am working on a stripped-down version, but I realise there are a few
> things I have not grasped.
> Hope you don't mind, but can you scan through what follows and tell me
> if I am on the right lines?
> Both the client and the server can send a header with 'Keep-alive', but
> what does it actually mean?
> If the client sends it, does that mean that it wants the server to keep
> the connection open, and only close it when the client closes it from
> the other end?
> Conversely, if the server sends it, does it mean that it wants the
> client to keep the connection open? If so, under what condition would
> the server close the connection (other than a timeout). Should the
> server only send 'Keep-alive' if it receives 'Keep-alive'?
I'm pretty sure the server should never say "keep-alive" if the client
said "close". (There's a bit of an oddity in that the default changed
between, I think, HTTP 1.0 and 1.1. But that's a minor discrepancy.)
Generally, you'll see one of these options:
1) Client says "Connection: close". This means "I'm only gonna be
doing this one request, then we're done". Good for non-browser usage,
automated scripts etc.
2) Client says "Keep-Alive", server says "Close". This means the
client would be okay with connection reuse, but the server's saying no
(maybe it doesn't support it, or maybe it's in a soft shutdown state
where it wants to end connections, or whatever). After the current
request, the connection will be closed.
3) Both ends say "Connection: Keep-Alive". After the request, the
socket isn't closed, and can be reused for some other request.
I'm not sure how a client would (or should) handle it if the server
says "Connection: Keep-Alive" after the client said "Connection:
Close". Probably the client would just close its end of the socket and
that'd be that.
> In my program, when I send a file in response to a GET, I send a header
> with 'Keep-alive', then I send the file, then I close the connection.
> This now seems wrong. It could even be the cause of my bug, but then why
> has it only appeared now? Both Edge and Chrome send 'Keep-alive' headers.
It mightn't be fundamentally WRONG (there needs to be *some* way to
signal that you're now done), but it would probably be suboptimal. If
you're always going to close the connection after one request, just
send Connection: Close.
> If I am thinking along the right lines, then the exchange should go like
> this -
> Client sends request, with 'Keep-alive' header.
> Server sends response, but does not close the connection. The server
> should reply with a 'Keep-alive' header. If it does not, the client will
> close the connection, in which case the server will also close.
> Assuming that they both send 'Keep-alive', the onus is on the client to
> close the connection when it has no more requests. The server should
> have a timeout in case the client goes away.
I think either end is allowed to close the connection.
> Assuming that the above is correct, the client will rely on
> 'Content-length' to determine when it has received the entire request.
> If the client has more than one request, it will send the first, wait
> for the response, when fully received as per the 'Content-length' it
> will send the next one, until all requests have been sent and all
> responses received, at which point it will close the connection.
Yep. Which makes things very interesting (for serene values of
"interesting", such as "we're all gonna die") when you want to use
connection pooling with something that fundamentally doesn't have a
length, such as a chunked transfer. Which you are doing. That may or
may not actually be significant.
> All this assumes only one connection. Alternatively the client could
> open multiple connections for the requests. In that case it would make
> sense to use 'Connection: Close', so that the server can close the
> connection straight away, making it available for reuse.
What you'll often see is that the client has some sort of connection
limit, and reuses those same connections. For instance, it might have
a maximum of four connections per server (where "server" has to
include the name as well as its IP and port, due to TLS SNI), so it
requests an HTML page on one connection (with keep-alive), parses
that, and then fires off one request on the reused connection and
three others on new connections. As those requests complete, it sends
new requests on the same sockets.
> If this all makes sense, I should write two versions of the client
> program, one using a single connection, and one using a pool of connections.
Possibly! I think you'll most likely see that one of those behaves
perfectly normally, and you only trigger the issue in the other. So
you could move forward with just one test program.