feat(admin): keep server errors where a restart cannot take them
Release / Build and publish image (push) Failing after 12m56s

The panel's watcher reported four 5xx today — twice from one address at
11:49, twice from another at 10:45 — and there is no way to find out what
any of them were. verdande logs the real cause of every 500, but on a Rune
the log belongs to the container and the container is replaced on every
restart. So an operator is told that something broke and given nothing to
act on, which is the worst of both: alarming and useless.

Every 500 now writes a row: what the handler was attempting in its own
words, the error it hit, the method and path, who was asking, and the
request id that ties it back to the log line for whoever still has it. The
"what" is the part a status code cannot give — "list projects" and "move
task" are the difference between a fault you can find and one you can only
count.

Written best-effort and never checked. This is the path that is already
reporting a failure, and a diagnostic that can itself fail the request
would turn one broken screen into two.

Threading the request through `internal()` and `storeError()` touched 108
call sites across 19 files, which is the price of not having had it there
from the start. It is mechanical and the compiler checked all of it.

Kept for thirty days by the nightly sweep: long enough to answer "what
happened last week", short enough that a fault loop cannot fill the volume
the tasks live on.

Read at Indstillinger → Fejl, administrators only — it carries paths,
account names and the server's own error strings. The empty state says
"Ingen fejl registreret. Det er den rigtige tilstand", because a blank
panel should not read as a broken one.

The test drops a table and asks for the endpoint that reads it: the same
code path a disk error takes, and the row that comes back has to name the
operation, the account and the request.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kristian Wind
2026-08-17 22:58:02 +02:00
co-authored by Claude Opus 5
parent 4be934ccc1
commit c2e2d66e76
31 changed files with 634 additions and 177 deletions
+77
View File
@@ -0,0 +1,77 @@
package store
import (
"context"
"time"
)
// A ServerError is one 500 the API answered, kept so it can be read after the
// container that produced it has gone.
type ServerError struct {
ID string
At time.Time
Method string
Path string
Status int
What string
Message string
UserID string
UserName string
RequestID string
}
// RecordError stores one failure.
//
// Errors here are dropped rather than returned: this is called from the path that
// is already reporting a failure to somebody, and a diagnostic that can itself
// fail the request would turn one broken screen into two.
func (db *DB) RecordError(ctx context.Context, e ServerError) {
var user any
if e.UserID != "" {
user = e.UserID
}
_, _ = db.ExecContext(ctx,
`INSERT INTO error_log (id, at, method, path, status, what, message, user_id, request_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
NewID(), time.Now().Unix(), e.Method, truncate(e.Path, 500), e.Status,
truncate(e.What, 200), truncate(e.Message, 2000), user, truncate(e.RequestID, 100))
}
// ListErrors returns the most recent failures, newest first.
func (db *DB) ListErrors(ctx context.Context, limit int) ([]ServerError, error) {
rows, err := db.QueryContext(ctx, `
SELECT e.id, e.at, e.method, e.path, e.status, e.what, e.message,
COALESCE(e.user_id, ''), COALESCE(u.name, ''), e.request_id
FROM error_log e
LEFT JOIN users u ON u.id = e.user_id
ORDER BY e.at DESC
LIMIT ?`, limit)
if err != nil {
return nil, err
}
defer rows.Close()
out := []ServerError{}
for rows.Next() {
var e ServerError
var at int64
if err := rows.Scan(&e.ID, &at, &e.Method, &e.Path, &e.Status, &e.What,
&e.Message, &e.UserID, &e.UserName, &e.RequestID); err != nil {
return nil, err
}
e.At = time.Unix(at, 0).UTC()
out = append(out, e)
}
return out, rows.Err()
}
// PurgeOldErrors is run by the nightly sweep. A diagnostic that grows without
// limit is a disk that fills for a reason nobody expected.
func (db *DB) PurgeOldErrors(ctx context.Context, keep time.Duration) (int64, error) {
res, err := db.ExecContext(ctx,
`DELETE FROM error_log WHERE at < ?`, time.Now().Add(-keep).Unix())
if err != nil {
return 0, err
}
return res.RowsAffected()
}
@@ -0,0 +1,27 @@
-- Server errors, kept where they can outlive the container that produced them.
--
-- verdande already logs the real cause of every 500 to stdout. On a Rune that is
-- the container's log, and every restart starts a new one — so the panel can
-- report "HTTP 5xx, twice, at 11:49" long after the line explaining it has gone.
-- An operator is then told that something broke and given no way to find out what.
--
-- Bounded by the sweeper rather than by size: this is a diagnostic, and one that
-- grows without limit is a disk that fills for a reason nobody expected.
CREATE TABLE error_log (
id TEXT PRIMARY KEY,
at INTEGER NOT NULL,
method TEXT NOT NULL,
path TEXT NOT NULL,
status INTEGER NOT NULL,
-- what: the operation the handler was attempting, in the words it uses in the
-- log — "list projects", "move task". The detail a stack trace would give and
-- a status code cannot.
what TEXT NOT NULL,
message TEXT NOT NULL,
-- Whose request it was, so a fault that only one account hits is visible as
-- one. NULL for anything that failed before authentication.
user_id TEXT REFERENCES users (id) ON DELETE SET NULL,
request_id TEXT NOT NULL DEFAULT ''
);
CREATE INDEX idx_error_log_at ON error_log (at DESC);