Lost connections and connect retry

Lost connections

If the connection of a session ends without the client closing it, for example because the server crashed, was stopped or its container restarted, the client detects it. Every following keyword call in that session fails with:

ConnectionError: Connection to 'tcp://127.0.0.1:8271' was lost: the connection was closed by the server

The cause is “the connection was closed by the server” when the server closed the connection cleanly. Otherwise it is the type and message of the error that ended the connection, for example ConnectionResetError: [Errno 104] Connection reset by peer. For a stdio: URI, the exit code of the server process is added, for example (server process exited with code 3).

A keyword that is running when the connection is lost fails with:

ConnectionError: Connection to 'tcp://127.0.0.1:8271' was lost while running keyword 'Greet': the connection was closed by the server

The client does not send that keyword again, since the server may already have run it.

Without connect retry, the client does not reconnect on its own. The lost session is kept until it ends according to rpc_scope, and the first keyword call after that starts a new session, which connects again and imports the remote library anew. With rpc_scope=TEST, only the rest of the current test fails; with GLOBAL, every later call of the run fails. Ending a session on the client side, at the end of its rpc_scope, is never reported as a lost connection. See Sessions and rpc_scope.

Connect retry

Connect retry lets the client wait for a server that is not reachable yet, for example while its container starts or restarts. It is off by default; enable it with rpc_connect_retry_timeout:

*** Settings ***
Library    JsonRpcRemote    tcp://127.0.0.1:8271    MyLibrary
...        rpc_connect_retry_timeout=1 minute    rpc_connect_retry_interval=2s

With connect retry, a failed connection attempt is repeated every rpc_connect_retry_interval until it succeeds or rpc_connect_retry_timeout has passed. This applies to every connection: at the library import, when a new session starts, and when the client replaces a lost session. Each attempt is still limited by rpc_timeout.

If the time passes without a connection, the call fails with an error such as:

ConnectionError: Could not connect to 'tcp://127.0.0.1:8271' within 1 minute (31 attempts): ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 8271)

The client waits like that only once per outage: until a connection succeeds again, every further call makes a single attempt and fails right away if the server is still unreachable. After the next successful connection, a new outage gets the full time again. A connection that needed more than one attempt is logged at INFO level.

Example: a server that is still starting

In this example, the suite starts before the server, as with a container that is still starting. The library is MyLibrary from Getting started, and the files are in docs/examples/connect-retry in the repository. The library import tries to connect every half second, and succeeds once the server is up:

greet.robot

*** Settings ***
Library    JsonRpcRemote    uri=tcp://127.0.0.1:8271    library_name=MyLibrary
...        rpc_connect_retry_timeout=1 minute    rpc_connect_retry_interval=0.5s


*** Test Cases ***
Greet Waits For The Server
    ${greeting}=    Greet    World
    Should Be Equal    ${greeting}    Hello, World!

Without rpc_connect_retry_timeout, the import would fail at once with ConnectionRefusedError.

Reconnecting after a lost connection

With connect retry, the next keyword call after a lost connection replaces the session: it connects again and imports the remote library anew. The client then logs a warning such as:

Connection to 'tcp://127.0.0.1:8271' was lost: the connection was closed by the server. Reconnected after 12.4s (7 attempts); the remote library was imported again, so its server-side state was reset.

If the server restarted, nothing of the earlier library instance is left: whatever it kept in memory, such as an open application or a login, is gone. If the server kept running, a library with the global scope keeps its instance, as Sessions and rpc_scope describes. A keyword that was running when the connection was lost still fails and is not sent again.

Robot Framework registers the remote library’s keywords once, at the library import. After a reconnect, the client therefore compares the library definition, its keywords and their arguments, documentation and sources, with the one from the import. If the server came back with a different definition, for example after an update of the library, the call fails with:

ConnectionError: Reconnected to 'tcp://127.0.0.1:8271', but the library definition changed since the library was imported (keywords added: New Keyword); the library cannot be used until its session ends according to rpc_scope.

Every further call fails the same way, without connecting, until the session ends according to rpc_scope. A new session after that is not compared.