NO-JIRA: add Loki readiness check and restart logic to observability tests

Loki's ingester can enter a shutdown state over time, causing it to
reject all writes with HTTP 503 while still responding to queries.
This adds a readiness check before running observability tests and
automatically restarts the Loki container if it's unhealthy.

Also removes unnecessary retry wrappers around Loki queries since
the suite setup now ensures Loki is ready before tests begin.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED

Author: agullon

test/bin/manage_loki.sh

@@ -14,7 +14,7 @@ DEFAULT_HOST_PORT="3100"
 
 usage() {
     cat - <<EOF
-${BASH_SOURCE[0]} (start|stop) [port]
+${BASH_SOURCE[0]} (start|stop|restart) [port]
 
   -h           Show this help.
 
@@ -25,6 +25,8 @@ start [port]: Start Loki.
 stop: Stop Loki.
             The container name is assumed to be loki.
 
+restart: Restart the Loki container.
+
 EOF
 }
 
@@ -52,6 +54,12 @@ action_start() {
         "${LOKI_IMAGE}" > /dev/null
 }
 
+action_restart() {
+    local container_name="loki"
+    echo "Restarting Loki container ${container_name}"
+    podman restart "${container_name}"
+}
+
 if [ $# -eq 0 ]; then
     usage
     exit 1
@@ -60,7 +68,7 @@ action="${1}"
 shift
 
 case "${action}" in
-    start|stop)
+    start|stop|restart)
         "action_${action}" "$@"
         ;;
     -h)

test/resources/loki.py

@@ -68,6 +68,15 @@ def _print_results(results: dict) -> None:
                 _log(f"{log_line}")
 
 
+def check_loki_ready(host: str, port: int) -> None:
+    url = f"http://{host}:{port}/ready"
+    _log(f"Checking Loki readiness at {url}")
+    response = requests.get(url, timeout=5)
+    _log(f"Loki readiness response: {response.status_code} {response.text.strip()}")
+    if response.status_code != 200:
+        raise Exception(f"Loki is not ready: {response.status_code} {response.text.strip()}")
+
+
 def check_loki_query(host: str, port: int, query: str, limit: int = 10) -> None:
     try:
         from robot.libraries.BuiltIn import BuiltIn

test/suites/optional/observability.robot

@@ -72,6 +72,7 @@ Setup Suite And Prepare Test Host
     Command Should Work    sudo firewall-cmd --reload
     # Configure observability settings
     Check Required Observability Variables
+    Ensure Loki Is Ready
     Set Test OTEL Configuration
     # We need to do something to the cluster to generate new kube events
     Create Hello MicroShift Pod
@@ -88,6 +89,20 @@ Check Required Observability Variables
     ${string_value}    Convert To String    ${LOKI_HOST}
     Should Not Be Empty    ${string_value}    LOKI_HOST variable is required
 
+Ensure Loki Is Ready
+    [Documentation]    Check if Loki's ingester is healthy, restart the container if not.
+    ...    Loki's ingester can enter a shutdown state over time, causing it to
+    ...    reject all writes with HTTP 503 while still responding to queries.
+    FOR    ${attempt}    IN RANGE    5
+        ${status}=    Run Keyword And Return Status
+        ...    Check Loki Ready    ${LOKI_HOST}    ${LOKI_PORT}
+        IF    ${status}    RETURN
+        Log    Loki is not ready (attempt ${attempt}), restarting container    console=True
+        Local Command Should Work    ./bin/manage_loki.sh restart
+        Sleep    2s
+    END
+    Fail    Loki did not become ready after 5 restart attempts
+
 Set Test OTEL Configuration
     [Documentation]    Set Test OTEL Configuration