Client connection leak with chunked and compressed responses

Version: 3.0.7

TL/DR: dropwizard-default HTTP client leaks connections if the response is both chunked and compressed

Reproduction steps: run this app: https://github.com/eddsteel/zipped-chunked



Hi folks,



I'm seeing a connection leak in the HTTP client of a dropwizard app when calling another dropwizard app if the response is both chunked and compressed. We noticed it in the kotlin-based dropwizard 3.0 client app when upgrading the kotlin-based dropwizard server from DW 2.1 to 3.0 but I think the issue could be older (I'll explain). It seems to be 100% of the request connections are held open (in our dev environment metrics we see a 1:1 correlation of requests to a specific endpoint and leased connections, until the client app falls over).



I have reproduced the issue in a [Java application]( https://github.com/eddsteel/zipped-chunked) with DW 3.0.7, using a single application as both client and server. I've tried to simplify/remove as much of our code as possible leaving only dropwizard core.



More background: the connection leak was discovered in the client after upgrading the server. When I investigated locally I discovered that the "working" combination of 3.0 client and 2.1 server was not actually using `gzip` encoding, and the problem only appeared with the 3.0 server when the client was using compression. Configuring it not to use gzip also fixed the problem with connections leaking. Since the 2.1/3.0 combo never used gzip it's possible the problem would exist with 2.1 too.



## Reproduction



Anyway, to focus on the reproducible case with the latest v3 DW as both client and server. It should behave the same way as our systems as far as client creation and chunked data handling go, but with a lot of layers removed. The app has two endpoints to simulate server and client. `POST /` produces a ChunkedOutput response (just one chunk). `GET /check` will call `POST /` 1025 times and read/parse the full body. If compression is enabled (via application config) the 1025th attempt will fail due to an exhausted connection pool. None of the connections have been released.



```sh

[nix-shell:~/src/zipped-chunked]$ java -Ddw.client.compress=false -jar target/webapp-1.0-SNAPSHOT.jar server config.yml >/dev/null &

[1] 28790



[nix-shell:~/src/zipped-chunked]$ curl http://localhost:8080/check

done

[nix-shell:~/src/zipped-chunked]$ kill 28790



[nix-shell:~/src/zipped-chunked]$ java -Ddw.client.compress=true -jar target/webapp-1.0-SNAPSHOT.jar server config.yml >/dev/null &



[nix-shell:~/src/zipped-chunked]$ curl http://localhost:8080/check

{"code":500,"message":"Connections exhausted"}

[nix-shell:~/src/zipped-chunked]$ curl -v http://localhost:8080 -d {}

*   Trying [::1]:8080...

* Connected to localhost (::1) port 8080

> POST / HTTP/1.1

> Host: localhost:8080

> User-Agent: curl/8.4.0

> Accept: */*

> Content-Length: 2

> Content-Type: application/x-www-form-urlencoded

> 

< HTTP/1.1 200 OK

< Date: Thu, 06 Jun 2024 16:44:07 GMT

< Content-Type: application/json

< Vary: Accept-Encoding

< Transfer-Encoding: chunked

< 

{"label":"test","integers":[1,2,3],"integerCount":3}

* Connection #0 to host localhost left intact



[nix-shell:~/src/zipped-chunked]$ curl --compressed -v http://localhost:8080 -d {}

*   Trying [::1]:8080...

* Connected to localhost (::1) port 8080

> POST / HTTP/1.1

> Host: localhost:8080

> User-Agent: curl/8.4.0

> Accept: */*

> Accept-Encoding: deflate, gzip

> Content-Length: 2

> Content-Type: application/x-www-form-urlencoded

>

< HTTP/1.1 200 OK

< Date: Thu, 06 Jun 2024 16:44:52 GMT

< Content-Type: application/json

< Vary: Accept-Encoding

< Content-Encoding: gzip

< Transfer-Encoding: chunked

<

{"label":"test","integers":[1,2,3],"integerCount":3}

* Connection #0 to host localhost left intact



```



## Investigation



As far as I can tell both the ChunkedOutput and ChunkedInput objects are correctly closed in the application code. The ChunkedInput with [try with resources](https://github.com/eddsteel/zipped-chunked/blob/628b7d7c065a519bba31ebf60ce76320a0086414/src/main/java/com/eddsteel/resources/ClientResource.java#L27), and the ChunkedOutput [directly](https://github.com/eddsteel/zipped-chunked/blob/main/src/main/java/com/eddsteel/resources/ServerResource.java#L31). In the `compressed=false` case, the [EOFSensorInputStream](https://github.com/apache/httpcomponents-core/blob/rel/v5.2.4/httpcore5/src/main/java/org/apache/hc/core5/http/io/EofSensorInputStream.java)'s EOF watcher's `eofDetected` method is called and the connection is released ([here](https://github.com/apache/httpcomponents-client/blob/rel/v5.3.1/httpclient5/src/main/java/org/apache/hc/client5/http/impl/classic/ResponseEntityProxy.java#L98)). It seems that when a GZIPInputStream is involved, `-1` is never received and the connection is never released.



I took a trace dump from `checkEOF` for both versions and also logged each byte received from the breakpoint.



### compressed



```

"pool-3-thread-7 - GET /check@4758" prio=5 tid=0x1b nid=NA runnable

  java.lang.Thread.State: RUNNABLE

	  at org.apache.hc.core5.http.io.EofSensorInputStream.checkEOF(EofSensorInputStream.java:195)

	  at org.apache.hc.core5.http.io.EofSensorInputStream.read(EofSensorInputStream.java:119)

	  at java.util.zip.CheckedInputStream.read(CheckedInputStream.java:59)

	  at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:266)

	  at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258)

	  at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164)

	  at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)

	  at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91)

	  at org.apache.hc.client5.http.entity.GZIPInputStreamFactory.create(GZIPInputStreamFactory.java:61)

	  at org.apache.hc.client5.http.entity.LazyDecompressingInputStream.initWrapper(LazyDecompressingInputStream.java:51)

	  at org.apache.hc.client5.http.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:57)

	  at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:69)

	  at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream.read(ReaderInterceptorExecutor.java:263)

	  at org.glassfish.jersey.client.ChunkedInput$AbstractBoundaryParser.readChunk(ChunkedInput.java:112)

	  at org.glassfish.jersey.client.ChunkedInput.read(ChunkedInput.java:471)

	  at com.eddsteel.resources.ClientResource.callServer(ClientResource.java:30)

```



This gets the following bytes: `31, 16, 139, 8, 0, 6, 57`, which seems to only be the GZIP header from `GZIPInputStream.readHeader` in the constructor. However the application code _is_ eventually getting the full body and parsing the JSON.



### not compressed



This code gets the full body, ending with a -1.



```

"pool-3-thread-7 - GET /check@4741" prio=5 tid=0x1b nid=NA runnable

  java.lang.Thread.State: RUNNABLE

	  at org.apache.hc.core5.http.io.EofSensorInputStream.checkEOF(EofSensorInputStream.java:197)

	  at org.apache.hc.core5.http.io.EofSensorInputStream.read(EofSensorInputStream.java:119)

	  at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:69)

	  at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream.read(ReaderInterceptorExecutor.java:263)

	  at org.glassfish.jersey.client.ChunkedInput$AbstractBoundaryParser.readChunk(ChunkedInput.java:112)

	  at org.glassfish.jersey.client.ChunkedInput.read(ChunkedInput.java:471)

	  at com.eddsteel.resources.ClientResource.callServer(ClientResource.java:30)

```



Bytes received: (the full expected JSON + -1): `123, 34, 108, 97, 98, 101, 108, 34, 58, 34, 116, 101,

115, 116, 34, 44, 34, 105, 110, 116, 101, 103, 101, 114, 115, 34, 58, 91, 49, 44, 50, 44, 51, 93, 44, 

34, 105, 110, 116, 101, 103, 101, 114, 67, 111, 117, 110, 116, 34, 58, 51, 125, 13, 10, -1`





I think either the way the streams are layered, or the behavior of `LazyDecompressingInputStream`, is causing the `EofSensorInputStream` to be bypassed.



## Next steps



The code at fault seems to be within Apache HTTPComponents or possibly how Jersey configures it, but I wanted to ask here first since you're the experts in combining these libraries:



1. Is this familiar to anyone?

2. Is there anything wrong in the example code wrt ChunkedInput/ChunkedOutput or compression configuration?

3. Should I be directly closing/ releasing connections somehow, and not relying on the EOF detection?

4. Does it seem like a bug?

5. Anything more I should do to isolate it?

6. Should I report to Apache folks directly?



I've not had any luck with any of the other jersey connector providers so far (even making a request to this endpoint successfully), but it might be an option.

[]