We recently switched from ELB to ELB2/ALBs and occasionally our go http/2 clients are seeing GOAWAY messages from our Application Load Balancers which I'm unable to explain. The target group servers only support http/1.1 and our load balancers should always have at least one healthy server in rotation.
I can reliably reproduce the GOAWAY when registering a new instance in the ALB. The ALB returns GOAWAY when the target is in the "Initial" state. Further, even though the ALB responds with GOAWAY the request successfully makes it to the other instance registered in the target group. So, given instances web0 and web1, if I deregister web0 and re-register that target I can reliably reproduce the GOAWAY if I make a request while web0 is in "Initial". However our logs show that web1 successfully handled the request.
Our client is a Go program using http.DefaultClient. I can reproduce this behavior using both Go 1.7 and 1.8beta2.
When this occurs our client logs more details about the HTTP/2 response:
http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=NO_ERROR, debug=""
I'd like to better understand what's going on here. Should either the go http2 package or our code automatically handle the GOAWAY by retrying the request? I'm not familiar enough with http2 to know if GOAWAY is expected, which implies that our Go client shouldn't handle it as an error condition, or if this indicates that something is going wrong at the ALB.
The GOAWAY frame carries three pieces of information that could help you troubleshooting the issue:
+-+-------------------------------------------------------------+
|R| Last-Stream-ID (31) |
+-+-------------------------------------------------------------+
| Error Code (32) |
+---------------------------------------------------------------+
| Additional Debug Data (*) |
+---------------------------------------------------------------+
GOAWAY frame with Last-Stream-ID with NO_ERROR, to let the client know that a shutdown is about to come, then after some time, send another GOAWAY frame with Last-Stream-ID set to the actually last processed ID. So that the client knows what's been passed on. Here's the relevant extract, from RFC7540, 6.8 GOAWAYA server that is attempting to gracefully shut down a connection SHOULD send an initial GOAWAY frame with the last stream identifier set to 2^31-1 and a NO_ERROR code. This signals to the client that a shutdown is imminent and that initiating further requests is prohibited. After allowing time for any in-flight stream creation (at least one round-trip time), the server can send another GOAWAY frame with an updated last stream identifier. This ensures that a connection can be cleanly shut down without losing requests.
found an upper-case letter in header name when an upper case letter was found in a header name.http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=NO_ERROR, debug=""
Since the server is sending NO_ERROR, your client should simply try to reconnect, and not treat the message as an error.
As to why the ALB is sending GOAWAYs... I'm not sure, can you give us more details about that?
@frederik-deweerdt answer should be accepted as the answer, regarding Application Load Balancers specifically, here is the answer from an AWS Forums post for similar question, https://forums.aws.amazon.com/thread.jspa?messageID=771883򼜫
The HTTP/2 GOAWAY response your clients are receiving are connections that are being closed gracefully by the Application Load Balancer. Application Load Balancers will generally allow idle connections to last until the configured idle timeout, which has a default of 60 seconds. However, there are some conditions which can trigger closing of idle connections. On an HTTP/1.1 connection, an outstanding request is allowed to complete, and then the TCP connection is torn down normally. On an HTTP/2 connection, the load balancer initiates a close of these connections gracefully by sending an HTTP/2 GOAWAY. Per RFC 7540 "GOAWAY allows an endpoint to gracefully stop accepting new streams while still finishing processing of previously established streams". The client should respond by completing in progress requests, closing the connection and reconnecting if needed. Application Load Balancer will log the HTTP status for each request in the access log, and not the connection state closing signal.
Clients experiencing errors due to having received an HTTP/2 GOAWAY should be checked to ensure they are fully compliant with HTTP/2 RFCs.
You can read more about HTTP/2 GOAWAY method in section 6.8 of RFC 7540. https://tools.ietf.org/html/rfc7540#section-6.8
Please let us know if you have any further questions about this or other behavior of Elastic Load Balancers.