Implement Lambda runtime init error reporting

[joe4dev]

Summary

When a Lambda runtime exits unexpectedly or throws an error during initialization, LocalStack previously received no callback and would wait until the environment timeout. This PR adds two complementary error-reporting paths so LocalStack immediately receives a structured ErrorResponse instead of timing out, and also adds RIE-side support for the init-phase timeout retry protocol.

Changes

cmd/localstack/events.go — new

LocalStackEventsAPI wraps telemetry.StandaloneEventsAPI and overrides SendInvokeStart to emit the synthetic START RequestId: ... Version: ... log line at the AWS-faithful point: after any inline suppressed-init (if init timed out or failed on-demand) and before the runtime handles the invocation. rapidcore calls SendInvokeStart at the right moment in doInvoke (internal/lambda/rapid/handlers.go), so this produces the correct AWS log ordering without any custom sequencing.

internal/lambda/rapidcore/server_localstack.go — new

Contains LocalStack-specific additions to the rapidcore Server that must live in package rapidcore (to access unexported channels and state helpers) but are kept out of server.go (which is vendored from upstream and must stay byte-identical to avoid rebase conflicts):

• InitCompletionResponse — carries the structured init failure cause (error type and message) so callers can report it to LocalStack
• interpretInitFailure — maps an InitFailure to the sentinel error and structured cause (mirrors the upstream awaitInitialized body without modifying it)
• AwaitInitializedWithTimeout — behaves like AwaitInitialized but returns early if init does not complete within the given timeout; on timeout returns timedOut=true without consuming the init-failures channel, so a subsequent invoke's Reserve()/awaitInitialized() can still observe the init outcome and trigger the suppressed init

cmd/localstack/custom_interop.go — modified

New fields on CustomInteropServer:

• initStart time.Time — set in Init(), used to compute Init Duration
• warmStart bool — flipped on the first invoke; Init Duration is only emitted on the first invocation
• initTimedOut atomic.Bool — set by ReportInitTimeout; suppresses Init Duration on the folded-invoke REPORT (the timed-out init was already reported separately)
• initErrorForwarded atomic.Bool — set in SendInitErrorResponse before forwarding; prevents SendInitError from sending a duplicate error status for the same failed init
• initErrorType atomic.Value — stores rapidcore's scrubbed fatal error type (e.g. Runtime.Unknown) when init failed; used to render INIT_REPORT(phase=invoke) and REPORT Status/Error Type for the on-demand folded-into-invoke path
• onDemand bool — true for on-demand functions; controls the init-failure reporting model

New methods:

• ReportInitTimeout() — emits INIT_REPORT Init Duration: ... Phase: init Status: timeout and sets initTimedOut
• SendInitError(errType, errMsg) — crash-path fallback: reports a structured init failure to LocalStack when the runtime died without calling /init/error (e.g. crash, sys.exit, invalid entrypoint); no-op if SendInitErrorResponse already forwarded the error

On-demand vs PC/SnapStart/Managed Instances split in SendInitErrorResponse:

AWS handles init failures differently depending on the function type:

• On-demand: a failed cold-start init is folded into the first invocation. SendInitErrorResponse records the error type in initErrorType but does not call /status/error; the function signals ready so LocalStack dispatches the first invoke, which surfaces the error with the full INIT_REPORT(phase=invoke)/START/END/REPORT log envelope.
• PC/SnapStart/MI: init failures are reported at provisioning time via /status/error. SendInitErrorResponse decodes the runtime's payload into a map[string]any (not a typed struct) to preserve all fields verbatim — in particular an empty-but-present "stackTrace": [] (e.g. Runtime.HandlerNotFound) that omitempty would drop on re-marshal — then injects requestId and POSTs to LocalStack.

Other custom_interop.go changes:

• NewCustomInteropServer accepts a pre-created *LocalStackAdapter (shared with the events API in main.go)
• Timeout error message format: "RequestId: <id> Error: Task timed out after X seconds" with ErrorType: "Sandbox.Timedout" and Status: timeout in the REPORT line
• All HTTP response bodies are now closed (fixes connection leaks on SendStatus, SendLogs, SendResult)

cmd/localstack/main.go — modified

• Creates LocalStackAdapter upfront and passes it to both NewLocalStackEventsAPI and NewCustomInteropServer
• Wires lsEventsAPI into the sandbox via SetEventsAPI
• Replaces AwaitInitialized() with AwaitInitializedWithTimeout() using LOCALSTACK_INIT_PHASE_TIMEOUT (default 10 s, overridden by LocalStack for PC/SnapStart/MI via execution_environment.py in localstack-pro)
• On timeout: calls ReportInitTimeout() and launches an async Reset("initTimeout") so rapidcore re-runs the init as a suppressed init on the first invocation
• On on-demand init failure (ErrInitDoneFailed): signals ready and lets the first invoke surface the cached error
• On PC/SnapStart/MI failure or unexpected error: calls SendInitError (no-op if SendInitErrorResponse already ran) and exits

cmd/localstack/awsutil.go — modified

• PrintEndReports gains a status string parameter
• Init Duration is placed after Max Memory Used in the REPORT line, matching the AWS field order
• status (e.g. "Status: timeout" or "Status: error\tError Type: Runtime.Unknown") is appended after Init Duration

cmd/localstack/logs.go — modified

Documents why bare CR characters in captured runtime output are intentionally not rewritten to LF: AWS splits CloudWatch log events on LF only, so print("a\rb") must stay as one event "a\rb". Converting CR to LF here would wrongly split such records (see test_multi_line_prints).

internal/lsapi/types.go — modified

ErrorResponse.RequestId changed from string to *string with omitempty: a pointer-to-empty-string serializes as "" (required for init errors, which always include a requestId field), while nil is omitted (used when no request ID is known).

Tests

Covered by the integration tests in localstack/localstack-pro#7293:

Scenario	Test
Exception raised during module import	test_lambda_runtime_error
sys.exit() called during init	test_lambda_runtime_exit
Missing AWS_LAMBDA_EXEC_WRAPPER script	test_lambda_runtime_wrapper_not_found
Handler function does not exist in module	test_lambda_handler_not_found
Init phase exceeds 10 s → transparent retry with function timeout	test_lambda_timeout_init_phase

Related

Depends on #101
Closes DEVX-1