Preface

The cause is a recent project left over from history, and some new requirements need to be added. During the stress test on this machine, it was found that after running for a period of time in the concurrent 600 state, the 500 error would start to appear occasionally. It may be that the old project uses fewer people (B2B projects), and no one has reported this problem before after the actual deployment. I roughly tracked the log and found that the system is calling a third-party service abnormally. There are many reasons for this situation. Carefully look at the Exception message from the exception stack, narrow the scope of the problem and verify it. This time it throws java.net.SocketException: Too many open files. It indicates that too many socket handles have been opened on the server and the upper limit is exceeded (usually 1024). In this case, a new network connection cannot be established.

Troubleshoot

Experienced programmers will call the netstat command at this time (pressure measurement cannot be interrupted),It is found that there are a large number of TCP links in the ESTABLISHED state, and a small number of TCP links in the CLOSE-WAIT state. Going to the source code again, the remote call part is because the code is too old, and org.apache.commons.httpclient.HttpClient is used, and a new instance will be linked for each call.

 

try {  
  client.executeMethod(method);  
  byte[] responseBody = null;  
    
  responseBody = method.getResponseBody();  
    
} catch (HttpException e) {  
  // TODO Auto-generated catch block  
  e.printStackTrace();  
} catch (IOException e) {  
  // TODO Auto-generated catch block  
  e.printStackTrace();  
}finally{  
  method.releaseConnection();  
    
}  

At first glance, there seems to be no problem. Although this method has overhead for each remote call, it is reasonable to release the resources every time they are used up. The current concurrency is not enough to cause the socket handle to be insufficient. But in fact, the socket is not really closed.Keep-alive of HTTP and TCPAccording to the document, if HttpClient does not initiate a close actively, the link will be maintained for a period of time, and the link is not reused. During the maintenance period, as soon as other concurrency comes in, it may throw an exception that the handle is not enough.
There is even more serious, the TCP link has entered the CLOSE_WAIT state, refer to the figure below

TCP-CLOSE four-way handshake

, Because some abnormal server initiates FIN, the requester passively closes and enters CLOSE-WAIT, but did not receive the last handshake message, causing SOCKET to remain in this state (generally passive close will last for 2 hours)

 

Approach：

 

HttpClient client = new HttpClient(new HttpClientParams(),new SimpleHttpConnectionManager(true));

Further exploration (RestTemplate and ClosableHttpClient)

The above approach is equivalent to closing HttpClient every time it runs out, which avoids this exception to a certain extent, but each new\close process consumes a lot of JVM memory, which greatly affects performance to a certain extent. At this time, a connection needs to be introduced. Pool, we can look at ClosableHttpClient, one of the simplest creation methods:

 

HttpClients.custom()
                .evictExpiredConnections()
                .evictIdleConnections(30, TimeUnit.SECONDS)
                .build()

ClosableHttpClient will create a connection pool with a size of 5 by default (for infrequent RPC calls). End-to-end links can be reused. Two methods related to evict are configured. On the one hand, it is used to handle abnormal links in the CLOSE_WAIT state. The aspect is used to process the link of IDLE state, and its internal source code will start a timing task to detect.

 

image.png

 

Spring WebClient encapsulates the RestTempate dedicated to restful requests, and actually uses ClosableHttpClient internally. For Clients with connection pools, it is best to use the singleton mode, and configure the appropriate connection pool size and configuration according to the amount of calls The timeout period, etc., don't make more complaints, here is an example:

 

@Configuration
public class RestClientConfiguration {

    /**
     * create ClosablehttpClient
     *
     * @return httpClient
     */
    @Bean
    public RestTemplate restTemplate() throws KeyStoreException, NoSuchAlgorithmException, KeyManagementException {

        //https config
        TrustStrategy acceptingTrustStrategy = (X509Certificate[] chain, String authType) -> true;
        SSLContext sslContext = org.apache.http.ssl.SSLContexts.custom()
                .loadTrustMaterial(null, acceptingTrustStrategy)
                .build();

        SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(sslContext
                ,null, null, NoopHostnameVerifier.INSTANCE);

        Registry<ConnectionSocketFactory> registry = RegistryBuilder.<ConnectionSocketFactory>create()
                .register("http", PlainConnectionSocketFactory.getSocketFactory())
                .register("https", csf)
                .build();

        PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager(registry);
        //Maximum number of connections 3000
        connectionManager.setMaxTotal(3000);
                 //Number of routing links 400
        connectionManager.setDefaultMaxPerRoute(400);
        RequestConfig requestConfig = RequestConfig.custom()
                .setSocketTimeout(60000)
                .setConnectTimeout(60000)
                .setConnectionRequestTimeout(10000)
                .build();

        HttpComponentsClientHttpRequestFactory requestFactory =
                new HttpComponentsClientHttpRequestFactory();

        CloseableHttpClient httpClient = HttpClients.custom().setDefaultRequestConfig(requestConfig)
                .setConnectionManager(connectionManager)
                .evictExpiredConnections()
                .evictIdleConnections(30, TimeUnit.SECONDS)
                .build();
        requestFactory.setHttpClient(httpClient);
        return new RestTemplate(requestFactory);
    }
}

OF: Cheng _listen
Link: https://www.jianshu.com/p/82f30208ae56
Source: Jianshu
The copyright belongs to the author. For commercial reprints, please contact the author for authorization. For non-commercial reprints, please indicate the source.