gRPC Communication Issues with Network Intermediaries (Proxies, Load Balancers, Antivirus)

[Devil2-O]

Hi All,

We recently migrated from CORBA to gRPC and while the transition was generally smooth, we're experiencing significant communication issues when network intermediaries are involved, such as proxies, reverse proxies, load balancers, and antivirus software.

Background:
Our application is a standalone system that handles large message payloads. When our server sends data to clients, these network intermediaries are causing several problems:

Issues Encountered:

1. Stream termination: Intermediaries are closing streams prematurely
2. TCP connection drops: Connections are being killed unexpectedly
3. Timeout-based resets: Streams are reset after certain time thresholds, even when data is actively being transmitted
4. Incomplete data delivery: Clients receive "end-of-midstream" or "stream closed" errors before receiving complete payloads

Root Cause Analysis:
Previously with CORBA, we never encountered these issues since CORBA uses the IIOP protocol. However, since gRPC uses HTTP/2, our communication is now subject to HTTP-based network policies and restrictions that these intermediaries enforce.

Current Situation:

• Suggesting network configuration changes isn't a permanent solution for us
• Our server needs to send large payloads and we cannot change our entire architecture immediately
• We considered streaming the data, but this still uses a single HTTP/2 stream, which encounters the same intermediary restrictions

Question:
Since these HTTP port restrictions are implemented for security reasons, we cannot continuously ask network administrators to modify their configurations.

Are there any gRPC-specific solutions or best practices to mitigate these issues when dealing with network intermediaries that have strict HTTP policies?

Any suggestions or guidance would be greatly appreciated.

Thanks!

[shivaspeaks]

RCA is not helping. Help us with some more details such as logs, client and server side configuration, do the failures happen after a consistent amount of time or after a certain amount of data is transferred, maybe some logs will be good for us to understand.

Are there any gRPC-specific solutions or best practices to mitigate these issues when dealing with network intermediaries that have strict HTTP policies?

Check this out for best practices https://grpc.io/docs/guides/performance/

gRPC allows users to fine-tune a lot of pieces in transport settings such as keepalives, flow control window, message limits. Perhaps you need to fine-tune according to your project.

[Devil2-O]

We couldn't conduct a thorough RCA as we're unable to reproduce this issue internally, and we don't have access to the intermediary's name or configuration details. The only information we have is the error that occurs when the server is sending data and the stream gets terminated. From gRPC logging, we can see that packets sent from the server are not reaching the client.

Exception stack:
io.grpc.StatusRuntimeException: INTERNAL: Encountered end-of-stream mid-frame at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268) at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249) at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167) at chs.grpc.GrpcDataServiceGrpc$GrpcDataServiceBlockingStub.refresh(GrpcDataServiceGrpc.java:14822) at chs.utility.dataservices.grpc.GrpcDataServiceDelegator.refresh(GrpcDataServiceDelegator.java:3547) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

From gRPC fine logging:

There is an antivirus software on the client machine that is acting as an intermediary and appears to be stopping the packets, causing the stream to end. We are unable to determine the exact cause. We initially thought there might be some timeout on the RPC, so we configured our application to send smaller data chunks from the server side instead of large chunks, but this didn't help either. I'm not sure how much this information helps.

Regarding the client and server configuration - we're using a general setup where the inbound message size is set to maximum, and we use blocking stubs for most RPCs. Nothing special in the configuration.

Questions:
Have you seen any cases like this where intermediaries impose significant restrictions that disrupt communication? If so, how should we handle it? If there are any resources or documentation available, please point us in the right direction.

Thanks!

[shivaspeaks]

Which version of grpc-java are you using?

[Devil2-O]

We are using 1.77.0, and recently upgraded to 1.79.0.