Getting this error out of a sudden java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
07:45 26 May 2019

I'm using httpcore-nio-4.4.5.jar. I'm using the elasticsearch RestHighLevelClient to interact with our elasticsearch servers. This all works fine except for some time we get I/O reactor stopped error out of a sudden.

Everything seems fine on ES side. No strange behaviour.

That's how i'm initializing my ES client.

public synchronized RestHighLevelClient getHighLevelClient() throws ManagerException {
        if (highLevelClient != null) {
            return highLevelClient;
        }

        Map map = getEsServers(esAddresses);

        HttpHost[] hosts = new HttpHost[map.size()];

        int i = 0;

        for (Map.Entry entry : map.entrySet()) {
            hosts[i++] = new HttpHost(entry.getKey(), entry.getValue(), "http");
            LOGGER.info(entry.getKey() + " " + entry.getValue());
        }

        RestClientBuilder restClientBuilder = RestClient.builder(hosts);
        highLevelClient = customizeHttpClient(restClientBuilder);
        return highLevelClient;
    }
public RestHighLevelClient customizeHttpClient(RestClientBuilder restClientBuilder) {
        Header[] defaultHeaders = new Header[2];
        defaultHeaders[0] = new BasicHeader("Authorization", "Basic YTph");
        defaultHeaders[1] = new BasicHeader("Accept", "application/json");

        restClientBuilder.setDefaultHeaders(defaultHeaders);

        restClientBuilder.setMaxRetryTimeoutMillis(MAX_RETRY_TIMEOUT_MILLIS);

        restClientBuilder.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder
                .setConnectTimeout(CONNECT_TIMEOUT_MILLIS)
                .setSocketTimeout(SOCKET_TIMEOUT_MILLIS)
                .setConnectionRequestTimeout(CONNECTION_REQUEST_TIMEOUT_MILLIS));

        restClientBuilder.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder
                .setMaxConnPerRoute(MAX_CONN_PER_ROUTE)
                .setMaxConnTotal(MAX_CONN_TOTAL));

        return new RestHighLevelClient(restClientBuilder);
    }

So basically first I get the following stacktrace

java.lang.IllegalStateException: I/O reactor has been shut down
        at org.apache.http.util.Asserts.check(Asserts.java:34) 
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:224) 
        at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:434) 
        at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:276) 
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:266) 
        at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:363) 
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:125) 
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:346) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:328) 
        at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:271) 
        at org.elasticsearch.client.RestHighLevelClient.performRequestAsync(RestHighLevelClient.java:537) 
        at org.elasticsearch.client.RestHighLevelClient.performRequestAsyncAndParseEntity(RestHighLevelClient.java:515) 
        at org.elasticsearch.client.RestHighLevelClient.searchAsync(RestHighLevelClient.java:400) 

and after that no Timeout just the following exception continuously until i restart my servers.

java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
    at org.apache.http.util.Asserts.check(Asserts.java:46)  
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)  
    at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:346)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:328)  
    at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:271)  
    at org.elasticsearch.client.RestHighLevelClient.performRequestAsync(RestHighLevelClient.java:537)  
    at org.elasticsearch.client.RestHighLevelClient.performRequestAsyncAndParseEntity(RestHighLevelClient.java:515)  
    at org.elasticsearch.client.RestHighLevelClient.searchAsync(RestHighLevelClient.java:400)  ```
java elasticsearch nio apache-httpasyncclient