Flakey tests in gitlab pipeline with selenium webdriver v4.34.0

10:58 15 Jul 2025

We’ve got issues using selenium-webdriver when running tests in a gitlab pipeline. Our component tests have become more flakey recently. As more tests have been added, the flakiness has become more apparent.

The tests run OK locally.

What we have is a NodeJs, express, nunjucks UI, running mocha tests using a selenium webdriver. While the UI does have links to other services there's a 'fake' service set up to provide responses to any external calls. So for the tests we're running docker containers for the UI, the fake service, localstack and selenium-chrome.

The selenium-web driver version is "4.34.0"

The docker image we’re using is 'selenium/standalone-chromium:137.0@sha256:f3bde4afe83cb361025dfab27ad6ce9aa12eacfa5204ec17b06ff5d1940777a3'

And I have also tried 'selenium/standalone-chromium:138.0@sha256:7e431c5aeed78f06f146076f023e02f543f0c936fb619b64617915a5c0f19f34'

The tests were stable at an earlier docker image version.

I'm now getting I'm getting failures like

1) /referrals/assign/bulk - When no row is selected
       "before each" hook for "Should show the error message when no rows selected":
     WebDriverError: unknown error: net::ERR_CONNECTION_REFUSED
  (Session info: chrome=137.0.7151.68)
      at Object.throwDecodedError (node_modules/selenium-webdriver/lib/error.js:523:15)
      at parseHttpResponse (node_modules/selenium-webdriver/lib/http.js:524:13)
      at Executor.execute (node_modules/selenium-webdriver/lib/http.js:456:28)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async thenableWebDriverProxy.execute (node_modules/selenium-webdriver/lib/webdriver.js:745:17)
      at async selectLot (test/helper/index.js:217:3)
      at async Context. (test/component/assign-referrals/assign-referrals-error.spec.js:18:5)

There's been a round of refactoring - so the tests are better separated into discrete files and smaller pipeline jobs etc. They do seem better than originally where it seemed that one failure caused multiple others.

However, while the tests can now run in the pipeline successfully, many times an odd test (or two) fails. It's not the same test that fails each time. Re-running the job can ensure the test then passes.

Say the tests have this beforeEach hook

beforeEach(async () => {
  driver = getDriver('open-tasks-csrf');
  await selectLot(driver);
});

Logging has determined that the driver is set up, but the ‘selectLot’ processing doesn’t complete and doesn’t get beyond driver.get(…)

async function selectLot(driver, lot = 'lot_1') {
  const uiServer = config.get('test.ui_server');
  const contextPath = config.get('app.path');
  await driver.get(`${uiServer}${contextPath}`);
…

In the logs, we see the processing up to that point, the failure and then the afterEach hook is called.

For completeness, our getDriver() is

function getDriver(identifier = 'default', timeouts = { implicit: 8000, pageLoad: 8000, script: 8000 }) {
  nextPort++;
  const chromeOptions = new chrome.Options();
  chromeOptions.addArguments([
    '--no-sandbox',
    '--disable-gpu',
    '--headless=new',
    `--remote-debugging-port=${nextPort}`,
    '--verbose',
  ]);

  const capabilities = Capabilities.chrome();

  capabilities.setProxy(null);

  const driver = new Builder()
    .forBrowser('chrome')
    .usingServer(config.get('test.selenium_server'))
    .withCapabilities(capabilities)
    .setChromeOptions(chromeOptions)
    .build();

  driver.manage().setTimeouts(timeouts);
  driver.manage().deleteAllCookies();
  return driver;
}

The issues started around the time we introduced csrf verification alongside a lot of other updates. However, skipping the csrf verification for the majority of tests hasn't cured the problem.

We have increased the timeouts and logging the time taken for the set up doesn't indicate that the processing is usually taking times approaching the timeout limits.

When we were using the selenium/standalone-chromium:137.0 docker image, we did try setting the browser version to 137 in the chrome options.

I'm running out of ideas of where to look / what to try next so - please - wondering if anyone has experienced something similar or can suggest a way forward.

Edited following Corey's suggestion about getting more logging. The port 9223 matches a failing test. At the end I can see LocalNode.stopTimedOutSession. Is it as simple as that - some process occasionally takes longer than one of these timeouts - timeouts = { implicit: 8000, pageLoad: 8000, script: 8000 }. Or is there some setting for the docker container I need to add?

09:55:07.968 INFO [LocalNode.newSession] - Session created by the Node. Id: 15fa91435a6875ce8d8616ebdbec310f, Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 138.0.7204.92, chrome: {chromedriverVersion: 138.0.7204.92 (f079b9bc781e..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:9223}, goog:loggingPrefs: {BROWSER: ALL, DRIVER: ALL}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://localhost:4444/session..., se:cdpVersion: 138.0.7204.92, se:containerName: f3cd6f111622, se:deleteSessionOnUi: true, se:downloadsEnabled: true, se:noVncPort: 7900, se:vnc: ws://localhost:4444/session..., se:vncEnabled: true, se:vncLocalAddress: ws://172.18.0.3:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
09:55:07.975 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: 15fa91435a6875ce8d8616ebdbec310f 
 Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 138.0.7204.92, chrome: {chromedriverVersion: 138.0.7204.92 (f079b9bc781e..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:9223}, goog:loggingPrefs: {BROWSER: ALL, DRIVER: ALL}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://localhost:4444/session..., se:cdpVersion: 138.0.7204.92, se:containerName: f3cd6f111622, se:deleteSessionOnUi: true, se:downloadsEnabled: true, se:noVncPort: 7900, se:vnc: ws://localhost:4444/session..., se:vncEnabled: true, se:vncLocalAddress: ws://172.18.0.3:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
09:55:07.982 DEBUG [RequestConverter.channelInactive] - Channel became inactive.
09:55:07.991 DEBUG [LoggingHandler.channelRead] - [id: 0xdd76b823, L:/[0:0:0:0:0:0:0:0]:4444] READ: [id: 0x7de8b6bc, L:/172.18.0.3:4444 - R:/172.18.0.1:57244]
09:55:07.991 DEBUG [LoggingHandler.channelRead] - [id: 0xdd76b823, L:/[0:0:0:0:0:0:0:0]:4444] READ: [id: 0x35f2f92f, L:/172.18.0.3:4444 - R:/172.18.0.1:57258]
09:55:07.992 DEBUG [LoggingHandler.channelRead] - [id: 0xdd76b823, L:/[0:0:0:0:0:0:0:0]:4444] READ: [id: 0x8a534c23, L:/172.18.0.3:4444 - R:/172.18.0.1:57262]
09:55:07.992 DEBUG [LoggingHandler.channelReadComplete] - [id: 0xdd76b823, L:/[0:0:0:0:0:0:0:0]:4444] READ COMPLETE
09:55:07.996 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
DELETE /session/15fa91435a6875ce8d8616ebdbec310f/cookie HTTP/1.1
Accept: application/json; charset=utf-8
User-Agent: selenium/4.34.0 (js linux)
Content-Length: 0
Host: docker:4444
Connection: close
09:55:07.996 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
DELETE /session/15fa91435a6875ce8d8616ebdbec310f/cookie HTTP/1.1
Accept: application/json; charset=utf-8
User-Agent: selenium/4.34.0 (js linux)
Content-Length: 0
Host: docker:4444
Connection: close
09:55:07.996 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /session/15fa91435a6875ce8d8616ebdbec310f/url HTTP/1.1
Accept: application/json; charset=utf-8
User-Agent: selenium/4.34.0 (js linux)
Content-Length: 38
Content-Type: application/json;charset=UTF-8
Host: docker:4444
Connection: close
09:55:07.996 DEBUG [RequestConverter.channelRead0] - Incoming message: EmptyLastHttpContent
09:55:07.996 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /session/15fa91435a6875ce8d8616ebdbec310f/timeouts HTTP/1.1
Accept: application/json; charset=utf-8
User-Agent: selenium/4.34.0 (js linux)
Content-Length: 47
Content-Type: application/json;charset=UTF-8
Host: docker:4444
Connection: close
09:55:07.996 DEBUG [RequestConverter.channelRead0] - End of http request: EmptyLastHttpContent
09:55:07.996 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /session/15fa91435a6875ce8d8616ebdbec310f/url HTTP/1.1
Accept: application/json; charset=utf-8
User-Agent: selenium/4.34.0 (js linux)
Content-Length: 38
Content-Type: application/json;charset=UTF-8
Host: docker:4444
Connection: close
09:55:07.997 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /session/15fa91435a6875ce8d8616ebdbec310f/timeouts HTTP/1.1
Accept: application/json; charset=utf-8
User-Agent: selenium/4.34.0 (js linux)
Content-Length: 47
Content-Type: application/json;charset=UTF-8
Host: docker:4444
Connection: close
09:55:07.997 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultLastHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 38, cap: 38/38, unwrapped: PooledUnsafeDirectByteBuf(ridx: 286, widx: 286, cap: 2048)), decoderResult: success)
09:55:07.997 DEBUG [RequestConverter.channelRead0] - End of http request: DefaultLastHttpContent(data: PooledSlicedByteBuf(ridx: 38, widx: 38, cap: 38/38, unwrapped: PooledUnsafeDirectByteBuf(ridx: 286, widx: 286, cap: 2048)), decoderResult: success)
09:55:07.997 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultLastHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 47, cap: 47/47, unwrapped: PooledUnsafeDirectByteBuf(ridx: 300, widx: 300, cap: 2048)), decoderResult: success)
09:55:07.998 DEBUG [RequestConverter.channelRead0] - End of http request: DefaultLastHttpContent(data: PooledSlicedByteBuf(ridx: 47, widx: 47, cap: 47/47, unwrapped: PooledUnsafeDirectByteBuf(ridx: 300, widx: 300, cap: 2048)), decoderResult: success)
09:55:07.998 DEBUG [HttpTracing.inject] - Injecting (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f/cookie into org.openqa.selenium.remote.tracing.empty.NullSpan@7757194a at org.openqa.selenium.grid.router.HandleSession:180
09:55:07.998 DEBUG [HttpTracing.inject] - Injecting (POST) /session/15fa91435a6875ce8d8616ebdbec310f/url into org.openqa.selenium.remote.tracing.empty.NullSpan@383aaf62 at org.openqa.selenium.grid.router.HandleSession:180
09:55:07.998 DEBUG [HttpTracing.inject] - Injecting (POST) /session/15fa91435a6875ce8d8616ebdbec310f/timeouts into org.openqa.selenium.remote.tracing.empty.NullSpan@2b3c9954 at org.openqa.selenium.grid.router.HandleSession:180
09:55:08.002 DEBUG [HttpTracing.inject] - Injecting (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f/cookie into org.openqa.selenium.remote.tracing.empty.NullSpan@7db70240 at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler:85
09:55:08.002 DEBUG [HttpTracing.inject] - Injecting (POST) /session/15fa91435a6875ce8d8616ebdbec310f/timeouts into org.openqa.selenium.remote.tracing.empty.NullSpan@1b7bf85a at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler:85
09:55:08.002 DEBUG [HttpTracing.inject] - Injecting (POST) /session/15fa91435a6875ce8d8616ebdbec310f/url into org.openqa.selenium.remote.tracing.empty.NullSpan@f6430be at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler:85
09:55:08.003 DEBUG [JdkHttpClient.execute0] - Executing request: (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f/cookie
09:55:08.003 DEBUG [JdkHttpClient.execute0] - Executing request: (POST) /session/15fa91435a6875ce8d8616ebdbec310f/timeouts
09:55:08.003 DEBUG [JdkHttpClient.execute0] - Executing request: (POST) /session/15fa91435a6875ce8d8616ebdbec310f/url
09:55:08.006 DEBUG [JdkHttpClient.execute0] - Ending request (POST) /session/15fa91435a6875ce8d8616ebdbec310f/timeouts in 3ms
09:55:08.008 DEBUG [RequestConverter.channelInactive] - Channel became inactive.
09:55:08.112 DEBUG [JdkHttpClient.execute0] - Ending request (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f/cookie in 109ms
09:55:08.114 DEBUG [RequestConverter.channelInactive] - Channel became inactive.
09:55:08.256 DEBUG [JdkHttpClient.execute0] - Ending request (POST) /session/15fa91435a6875ce8d8616ebdbec310f/url in 253ms
09:55:08.259 DEBUG [RequestConverter.channelInactive] - Channel became inactive.
09:55:08.263 DEBUG [LoggingHandler.channelRead] - [id: 0xdd76b823, L:/[0:0:0:0:0:0:0:0]:4444] READ: [id: 0xc2271bc0, L:/172.18.0.3:4444 - R:/172.18.0.1:57264]
09:55:08.263 DEBUG [LoggingHandler.channelReadComplete] - [id: 0xdd76b823, L:/[0:0:0:0:0:0:0:0]:4444] READ COMPLETE
09:55:08.267 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
DELETE /session/15fa91435a6875ce8d8616ebdbec310f/window HTTP/1.1
Accept: application/json; charset=utf-8
User-Agent: selenium/4.34.0 (js linux)
Content-Length: 0
Host: docker:4444
Connection: close
09:55:08.267 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
DELETE /session/15fa91435a6875ce8d8616ebdbec310f/window HTTP/1.1
Accept: application/json; charset=utf-8
User-Agent: selenium/4.34.0 (js linux)
Content-Length: 0
Host: docker:4444
Connection: close
09:55:08.267 DEBUG [RequestConverter.channelRead0] - Incoming message: EmptyLastHttpContent
09:55:08.267 DEBUG [RequestConverter.channelRead0] - End of http request: EmptyLastHttpContent
09:55:08.268 DEBUG [HttpTracing.inject] - Injecting (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f/window into org.openqa.selenium.remote.tracing.empty.NullSpan@7e2f4044 at org.openqa.selenium.grid.router.HandleSession:180
09:55:08.268 DEBUG [HttpTracing.inject] - Injecting (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f/window into org.openqa.selenium.remote.tracing.empty.NullSpan@5dd29542 at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler:85
09:55:08.269 DEBUG [JdkHttpClient.execute0] - Executing request: (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f/window
09:55:08.378 DEBUG [JdkHttpClient.execute0] - Ending request (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f/window in 109ms
09:55:08.381 DEBUG [RequestConverter.channelInactive] - Channel became inactive.
09:55:08.383 DEBUG [LoggingHandler.channelRead] - [id: 0xdd76b823, L:/[0:0:0:0:0:0:0:0]:4444] READ: [id: 0xed494aa4, L:/172.18.0.3:4444 - R:/172.18.0.1:57272]
09:55:08.384 DEBUG [LoggingHandler.channelReadComplete] - [id: 0xdd76b823, L:/[0:0:0:0:0:0:0:0]:4444] READ COMPLETE
09:55:08.388 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
DELETE /session/15fa91435a6875ce8d8616ebdbec310f HTTP/1.1
Accept: application/json; charset=utf-8
User-Agent: selenium/4.34.0 (js linux)
Content-Length: 0
Host: docker:4444
Connection: close
09:55:08.388 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
DELETE /session/15fa91435a6875ce8d8616ebdbec310f HTTP/1.1
Accept: application/json; charset=utf-8
User-Agent: selenium/4.34.0 (js linux)
Content-Length: 0
Host: docker:4444
Connection: close
09:55:08.389 DEBUG [RequestConverter.channelRead0] - Incoming message: EmptyLastHttpContent
09:55:08.389 DEBUG [RequestConverter.channelRead0] - End of http request: EmptyLastHttpContent
09:55:08.389 DEBUG [HttpTracing.inject] - Injecting (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f into org.openqa.selenium.remote.tracing.empty.NullSpan@379d1d8f at org.openqa.selenium.grid.router.HandleSession:180
09:55:08.390 DEBUG [HttpTracing.inject] - Injecting (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f into org.openqa.selenium.remote.tracing.empty.NullSpan@2bd17697 at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler:85
09:55:08.391 DEBUG [JdkHttpClient.execute0] - Executing request: (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f
09:55:08.393 DEBUG [JdkHttpClient.execute0] - Ending request (DELETE) /session/15fa91435a6875ce8d8616ebdbec310f in 2ms
09:55:08.393 DEBUG [UrlChecker.waitUntilUnavailable] - Waiting for http://localhost:32679/shutdown
09:55:08.394 DEBUG [UrlChecker.lambda$waitUntilUnavailable$2] - Polling http://localhost:32679/shutdown
09:55:08.394 FINEST [HttpURLConnection.plainConnect0] - ProxySelector Request for http://localhost:32679/shutdown
09:55:08.395 FINEST [KeepAliveCache$ClientVector.get] - cached HttpClient was idle for 995
09:55:08.395 FINEST [HttpClient.logFinest] - KeepAlive stream retrieved from the cache, sun.net.www.http.HttpClient(http://localhost:32679/status)
09:55:08.395 FINEST [HttpURLConnection.plainConnect0] - Proxy used: DIRECT
09:55:08.395 DEBUG [HttpURLConnection.writeRequests] - sun.net.www.MessageHeader@5d5a7cc25 pairs: {GET /shutdown HTTP/1.1: null}{User-Agent: Java/21.0.7}{Host: localhost:32679}{Accept: */*}{Connection: keep-alive}
09:55:08.396 FINEST [HttpClient.logFinest] - KeepAlive stream used: http://localhost:32679/shutdown
09:55:08.396 DEBUG [HttpURLConnection.getInputStream0] - sun.net.www.MessageHeader@309175113 pairs: {null: HTTP/1.1 200 OK}{Content-Length: 40}{Content-Type: application/json; charset=utf-8}
09:55:08.398 DEBUG [ExternalProcess$Builder.lambda$start$0] - completed to copy the output of process 168
09:55:08.406 DEBUG [UrlChecker.lambda$waitUntilUnavailable$2] - Polling http://localhost:32679/shutdown
09:55:08.406 FINEST [HttpURLConnection.plainConnect0] - ProxySelector Request for http://localhost:32679/shutdown
09:55:08.406 FINEST [KeepAliveCache$ClientVector.get] - cached HttpClient was idle for 10
09:55:08.406 FINEST [HttpClient.logFinest] - KeepAlive stream retrieved from the cache, sun.net.www.http.HttpClient(http://localhost:32679/shutdown)
09:55:08.407 FINEST [HttpURLConnection.plainConnect0] - Proxy used: DIRECT
09:55:08.407 DEBUG [HttpURLConnection.writeRequests] - sun.net.www.MessageHeader@7bc7f7f35 pairs: {GET /shutdown HTTP/1.1: null}{User-Agent: Java/21.0.7}{Host: localhost:32679}{Accept: */*}{Connection: keep-alive}
09:55:08.411 INFO [LocalNode.stopTimedOutSession] - Session id 15fa91435a6875ce8d8616ebdbec310f is stopping on demand...
09:55:08.413 INFO [LocalSessionMap.remove] - Deleted session from local Session Map, Id: 15fa91435a6875ce8d8616ebdbec310f

javascript node.js selenium-webdriver mocha.js

Your Answer

Privacy & Cookie Consent