Compare commits

...
7 Commits
Author SHA1 Message Date
Kristian WindandClaude Opus 5 993822f6fd fix(panel): bed om en planlagt genstart, ikke en der dør med afsenderen
Release / Build and publish image (push) Failing after 10m6s
Knappen tog produktionen ned og lod den ligge. Panelets almindelige restart
udfører arbejdet mens forespørgslen står åben: den stopper containeren — hvilket
slår denne proces ihjel — og panelet, der ser afsenderen forsvinde, når aldrig
til den halvdel, der starter igen. Fire minutters nedetid den 18. august, og det
kan ikke lappes herfra: enhver knap, der synkront beder om sin egen død, har
samme form.

safe-restart planlægges i stedet. Panelet svarer "Restart scheduled" og gør
arbejdet bagefter for egen regning, så intet afhænger af, at denne proces stadig
er i live til at høre svaret.

Prøven hang bagefter, og det var prøvens skyld: dens falske panel læste aldrig
forespørgslens krop, og der er en nu. En server, der ikke læser hvad den fik,
opdager aldrig at afsenderen har lagt på — så httptest ventede på en handler, der
aldrig vendte tilbage, og hele binæren løb tør for tid i stedet for den ene
forespørgsel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 23:15:10 +02:00
Kristian WindandClaude Opus 5 b0f78d1549 feat(gmail): Gmail flytter ind i postkasse-tabellen
Ét sted at spørge "hvilke postkasser har den her person". Gmails tokens, udløser,
etiket, set-liste og målprojekt ligger nu som en række i mailboxes ved siden af
dem, der læses over IMAP, og baggrundsjobbet fejer én gang i stedet for to og
skifter vej efter rækkens kind.

PKCE-værdierne bliver i user_settings. De hører til ét forsøg på at logge ind,
ikke til en postkasse, og de skal væk igen når forsøget er ovre.

Målprojektet fik sin egen kolonne, som IMAP-postkasser også får glæde af — en
arbejdspostkasse, der filer i arbejdsprojektet, er det åbenlyse at ville have.
Ikke en fremmednøgle: sletter man projektet, skal destinationen stille holde op
med at gælde, ikke afvise sletningen eller tage postkassen med sig.

Flytningen kunne først køre nu. Den tømmer indstillingsrækken, og havde den ligget
i v0.14.0, mens handlerne stadig læste den, var en fungerende postkasse blevet
koblet fra. Værdierne kopieres som chiffertekst — begge tabeller forsegler med
samme nøgle — så migrationen behøver ikke nøglen, hvilket en migration heller
ikke har noget at gøre med at holde.

Der er en prøve for præcis dét: en Gmail forbundet før flytningen, med et forseglet
token skrevet som indstillingstabellen skrev dem, kommer ud på den anden side med
token, udløser, adresse, målprojekt og set-liste i behold.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 22:59:40 +02:00
Kristian WindandClaude Opus 5 f3fecbd0b4 fix(imap): samme mail blev en ny opgave hver gang, og uddraget var rå MIME
To fejl fra ét skærmbillede.

Markøren stod stille. Den blev sat ud fra uid'en i fetch-svaret, og den kommer
ikke fra alle servere — så next-run læste fra nul hver gang. Den sættes nu ud fra
hvad SØGNINGEN fandt, når kørslen kom hele vejen igennem: så er alt op til det
punkt betragtet, uanset hvad fetch-svaret bar med. En afbrudt kørsel flytter kun
til den sidste, den faktisk lavede en opgave af. Prøven mod hukommelsesserveren
ville aldrig have fanget det, for den svarer altid med uid.

Og uddraget hentede BODY[TEXT], som ved multipart er grænser og dele-hoveder og
det hele — derfor stod der "--=_be71cb39… Content-Transfer-Encoding: 8bit" foran
teksten i hver eneste opgave. Nu hentes hele beskeden og MIME-strukturen gås
igennem: text/plain foretrækkes, HTML er reserven med mærkerne pillet af. Groft,
for det er linjen under en overskrift og ikke en gengivelsesmotor.

Multipart-prøven er sat på prøve: med den gamle hentning fejler den med præcis
den tekst, skærmbilledet viste.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 22:44:00 +02:00
Kristian WindandClaude Opus 5 30adc69cb5 feat(mailboxes): plads til Gmails udløser og dens set-liste
Kolonnerne, ikke flytningen. Gmail-handlerne læser stadig user_settings, og en
migration, der tømte den tabel før de var skrevet om, ville koble en fungerende
postkasse fra. Det additive først; flytningen når koden på den anden side er klar
til den, og den ligger som PENDING_gmail_move.sql.txt uden for .sql-mønstret,
så den ikke kan køre ved et uheld.

Udløseren fik sin egen kolonne frem for at blive pakket ind i label med et kolon.
En kolonne, der rummer to ting adskilt af et tegn, er en kolonne, nogen læser
forkert senere — og en etiket med kolon i navnet ville gøre det på første dag.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 22:38:45 +02:00
Kristian WindandClaude Opus 5 c9dfffd95a feat(mailboxes): postkasser kan forbindes fra fladen
Release / Build and publish image (push) Failing after 9m59s
Panelet under Integrationer: tilføj, hent nu, kobl fra. Sit eget frem for en
tilstand mere inde i Gmail-panelet, for de to er ikke det samme i forskellige
klæder — det ene er en app, man logger ind igennem, det andet et kodeord, man
selv holder.

Feltet hedder IMAP-server og ikke Server: der stod i forvejen Server på CalDAV
længere nede, og to felter med samme navn på én side er tvetydige også for en
skærmlæser. Testen fandt det, men det var ikke kun testens problem.

To vagter fangede mangler undervejs, og begge havde ret: mailbox_failed skulle
have en tekst i fladen, ellers ville nogen få serverens engelske logsprog at se,
og de tre ruter skulle i openapi.yaml. Der er nu et Mailbox-skema samme sted, som
siger hvad der ikke kommer tilbage — kodeordet og eventuelle tokens.

E2E-prøven forbinder til noget, der ikke lytter, og ser efter værtens egen
afvisning på skærmen. Den er samtidig prøven på at 422 var det rigtige valg: var
det stadig 502, ville et mellemled kunne kaste beskeden væk, og der ville stå
"noget gik galt".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 22:22:31 +02:00
Kristian WindandClaude Opus 5 6b4ef5b2f2 feat(imap): postkasser kan læses, og det er bevist mod en rigtig server
Handlere til at forbinde, koble fra og hente nu, plus baggrundssynk hvert tiende
minut ved siden af Gmails. Forbindelsen prøves FØR den gemmes: en postkasse, der
blev taget imod og så fejler stille hvert tiende minut, er værre end en, der blev
afvist ved døren, mens man stadig sidder med kodeordet i hånden.

Ingen "set"-liste til deduplikering som Gmail har. Uid'er er monotone, så
last_uid er hele svaret — og den skrives også, når budgettet løb ud, for en
kørsel, der lavede opgaver og glemte hvor langt den nåede, laver dem alle igen
næste gang.

Rundturen køres nu mod go-imap's egen server i processen: flaget post kommer med,
uflaget gør ikke, og fortsættelsen fra et uid henter kun det nyere. De hidtidige
prøver dækkede de rene dele og ville alle have bestået med selve hentningen i
stykker.

Testen kræver et selvsigneret certifikat, og klienten nægter dem. Account.RootCAs
lader en test TILFØJE tillid; der er stadig ingen måde at fjerne den på. Forskellen
er en søm og ikke et hul — en prøve viser, at samme server afvises uden.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 22:15:50 +02:00
Kristian WindandClaude Opus 5 1e59886d86 feat(mailboxes): en tabel, så I kan have hver sin postkasse — og flere hver
Gmail lå i user_settings under en scope. Den tabel har user_id som hele
primærnøglen, så den rummer præcis én scope pr. person, og dens upsert skriver
scope-navnet om ved konflikt: at lægge IMAP ved siden af ville have slettet
Gmail-forbindelsen uden at sige det. Migration 0011 advarer ordret mod netop det,
og advarslen var tæt på at blive gået i for anden gang.

Formen var forkert alligevel. Man har postkasser, flertal — en arbejds-Gmail og
en privat iCloud er det almindelige tilfælde, ikke det sære. En række hver siger
det. Og den svarer på spørgsmålet, der satte det i gang: hver postkasse hører til
en bruger, så to mennesker på samme instans forbinder hver sin og ser kun sin
egen. Det eneste, der bliver fælles, er Googles OAuth-klient, og det er rigtigt:
den er appen, ikke kontoen.

Kodeord og tokens skrives forseglet. Forseglingen i notifications.go dækker kun
indstillingstabellerne, så denne tabels metoder kalder den i hånden — værd at
vide, før der kommer en tredje hemmelig kolonne.

Gmail bliver hvor den er indtil videre. Den virker, og at lægge en flytning af en
kørende forbindelse i samme ombæring som en ny integration gør begge dele sværere
at efterprøve. Skemaet har allerede plads til den.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 22:10:20 +02:00
26 changed files with 1755 additions and 161 deletions
+1
View File
@@ -74,6 +74,7 @@ func run() error {
defer stopJobs()
runner := jobs.New(cfg, db, log, api.Mail(), api.Hub())
runner.SyncGmail = api.SyncGmail
runner.SyncMailbox = api.SyncMailbox
runner.Start(jobCtx)
srv := &http.Server{
+94
View File
@@ -2477,6 +2477,86 @@ paths:
created: { type: integer }
"502": { $ref: "#/components/responses/Upstream" }
/api/v1/mailboxes:
get:
tags: [integrations]
summary: The mailboxes this person has connected.
responses:
"200":
description: OK
content:
application/json:
schema:
type: object
properties:
mailboxes:
type: array
items: { $ref: "#/components/schemas/Mailbox" }
post:
tags: [integrations]
summary: Connect a mailbox over IMAP.
description: >
The connection is tested before it is saved, so a mailbox that cannot be
read is refused here rather than failing quietly every ten minutes.
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [host, username, password]
properties:
name: { type: string }
host: { type: string, example: "imap.mail.me.com:993" }
username: { type: string }
password: { type: string, description: "An app-specific password on every host worth naming. Stored sealed." }
folder: { type: string, default: INBOX }
responses:
"201":
description: Connected
content:
application/json:
schema: { $ref: "#/components/schemas/Mailbox" }
"400": { $ref: "#/components/responses/BadRequest" }
"409": { description: That mailbox is already connected }
"422": { $ref: "#/components/responses/Upstream" }
/api/v1/mailboxes/{mailboxID}:
delete:
tags: [integrations]
summary: Disconnect a mailbox. The tasks it made stay.
parameters:
- name: mailboxID
in: path
required: true
schema: { type: string }
responses:
"204": { description: Disconnected }
/api/v1/mailboxes/{mailboxID}/sync:
post:
tags: [integrations]
summary: Read one mailbox now.
parameters:
- name: mailboxID
in: path
required: true
schema: { type: string }
responses:
"200":
description: OK
content:
application/json:
schema:
type: object
properties:
created: { type: integer }
partial:
type: boolean
description: The run hit its time budget. What it made is kept.
"404": { $ref: "#/components/responses/NotFound" }
"422": { $ref: "#/components/responses/Upstream" }
/api/v1/ai/settings:
get:
tags: [integrations]
@@ -3082,6 +3162,20 @@ components:
items: { type: string }
schemas:
Mailbox:
type: object
description: One connected mailbox. The password and any token are never returned.
properties:
id: { type: string }
kind: { type: string, enum: [gmail, imap] }
name: { type: string }
host: { type: string }
username: { type: string }
folder: { type: string }
last_uid: { type: integer }
last_sync_at: { type: string, format: date-time }
created_at: { type: string, format: date-time }
Error:
type: object
description: |
+5
View File
@@ -58,6 +58,11 @@ const (
// read "something went wrong", and had nowhere to go. The interface wraps the
// message in one that says where it came from.
CodeGmailFailed = "gmail_failed"
// And the same for a mailbox read over IMAP. A host's refusal is the whole
// diagnosis — a wrong app password, a host that is not listening, a folder that
// is not there — and it says it in its own words.
CodeMailboxFailed = "mailbox_failed"
)
// errLastAdmin is returned to the caller that already wrote the response, so it
+45 -48
View File
@@ -179,15 +179,27 @@ func (s *Server) handleGmailCallback(w http.ResponseWriter, r *http.Request) {
delete(settings, "pkce_state")
delete(settings, "pkce_expires")
settings["refresh_token"] = token.RefreshToken
settings["access_token"] = token.AccessToken
settings["expires_at"] = token.ExpiresAt.Unix()
settings["email"] = email
if _, ok := settings["trigger"]; !ok {
settings["trigger"] = "starred"
if err := s.db.SetUserSettings(r.Context(), user.ID, "gmail", settings); err != nil {
s.internal(w, r, "clear gmail handshake", err)
return
}
if err := s.db.SetUserSettings(r.Context(), user.ID, "gmail", settings); err != nil {
// The mailbox itself, kept beside the ones read over IMAP. Connecting again
// writes the same row: Gmail is one per person, because the OAuth flow is.
box, err := s.db.MailboxOfKind(r.Context(), user.ID, "gmail")
if err != nil {
s.internal(w, r, "read gmail mailbox", err)
return
}
if box == nil {
box = &store.Mailbox{UserID: user.ID, Kind: "gmail", Trigger: "starred"}
}
box.Name = email
box.Username = email
box.RefreshToken = token.RefreshToken
box.AccessToken = token.AccessToken
box.ExpiresAt = token.ExpiresAt
if err := s.db.SaveMailbox(r.Context(), box); err != nil {
s.internal(w, r, "save gmail tokens", err)
return
}
@@ -240,36 +252,30 @@ func (s *Server) handleGmailSyncNow(w http.ResponseWriter, r *http.Request) {
// Returns how many were made. Also used by the background job, which is why it
// takes a context and a user rather than a request.
func (s *Server) SyncGmail(ctx context.Context, user *store.User) (int, error) {
settings, err := s.db.UserSettings(ctx, user.ID, "gmail")
box, err := s.db.MailboxOfKind(ctx, user.ID, "gmail")
if err != nil {
return 0, err
}
str := func(key string) string {
v, _ := settings[key].(string)
return v
}
refresh := str("refresh_token")
if refresh == "" {
if box == nil || box.RefreshToken == "" {
return 0, nil // not connected; nothing to do and nothing to complain about
}
query := gmail.Query(str("trigger"), str("label"))
query := gmail.Query(box.Trigger, box.Label)
if query == "" {
return 0, nil // connected but no trigger chosen
}
refresh := box.RefreshToken
// The access token is refreshed only when it has actually expired.
access := str("access_token")
expires, _ := settings["expires_at"].(float64)
if access == "" || int64(expires) <= time.Now().Unix() {
access := box.AccessToken
if access == "" || box.ExpiresAt.Before(time.Now()) {
token, err := s.gmailConfig(ctx).Refresh(ctx, refresh)
if err != nil {
return 0, err
}
access = token.AccessToken
settings["access_token"] = access
settings["expires_at"] = token.ExpiresAt.Unix()
if err := s.db.SetUserSettings(ctx, user.ID, "gmail", settings); err != nil {
box.AccessToken = access
box.ExpiresAt = token.ExpiresAt
if err := s.db.SaveMailbox(ctx, box); err != nil {
return 0, err
}
}
@@ -281,7 +287,7 @@ func (s *Server) SyncGmail(ctx context.Context, user *store.User) (int, error) {
// The person revoked access, or changed their password. Forget the
// tokens rather than retrying every ten minutes forever.
s.log.Info("gmail access was revoked; disconnecting", "user", user.ID)
_ = s.db.SetUserSettings(ctx, user.ID, "gmail", map[string]any{})
_ = s.db.DeleteMailbox(ctx, user.ID, box.ID)
}
return 0, err
}
@@ -290,19 +296,15 @@ func (s *Server) SyncGmail(ctx context.Context, user *store.User) (int, error) {
// rather than as a table: it is bounded by the query's own 30-day window, and
// a table would be one more thing to migrate for a feature this small.
seen := map[string]bool{}
if raw, ok := settings["seen"].([]any); ok {
for _, v := range raw {
if id, ok := v.(string); ok {
seen[id] = true
}
}
for _, id := range box.Seen {
seen[id] = true
}
inbox, err := s.db.InboxID(ctx, user.ID)
if err != nil {
return 0, err
}
projectID := str("project_id")
projectID := box.ProjectID
if projectID == "" {
projectID = inbox
} else if _, err := store.RequireProjectRole(ctx, s.db, projectID, user.ID, store.RoleEditor); err != nil {
@@ -352,15 +354,15 @@ func (s *Server) SyncGmail(ctx context.Context, user *store.User) (int, error) {
if created > 0 {
// Only the ids Gmail still returns are kept, so the list cannot grow
// without bound as messages fall out of the 30-day window.
keep := make([]any, 0, len(ids))
keep := make([]string, 0, len(ids))
for _, id := range ids {
if seen[id] {
keep = append(keep, id)
}
}
settings["seen"] = keep
settings["last_sync"] = time.Now().Unix()
if err := s.db.SetUserSettings(ctx, user.ID, "gmail", settings); err != nil {
box.Seen = keep
box.LastSyncAt = time.Now()
if err := s.db.SaveMailbox(ctx, box); err != nil {
return created, err
}
}
@@ -483,30 +485,25 @@ func (s *Server) handleSetGmailClient(w http.ResponseWriter, r *http.Request) {
// natural place to learn the address, and it is also the one moment that cannot be
// retried — so the status endpoint asks again when it finds nothing, and the answer
// corrects itself on the next look.
func (s *Server) gmailProfile(ctx context.Context, user *store.User, settings map[string]any) (string, error) {
str := func(key string) string {
v, _ := settings[key].(string)
return v
}
access := str("access_token")
expires, _ := settings["expires_at"].(float64)
if access == "" || int64(expires) <= time.Now().Unix() {
token, err := s.gmailConfig(ctx).Refresh(ctx, str("refresh_token"))
func (s *Server) gmailProfile(ctx context.Context, box *store.Mailbox) (string, error) {
access := box.AccessToken
if access == "" || box.ExpiresAt.Before(time.Now()) {
token, err := s.gmailConfig(ctx).Refresh(ctx, box.RefreshToken)
if err != nil {
return "", err
}
access = token.AccessToken
settings["access_token"] = access
settings["expires_at"] = token.ExpiresAt.Unix()
box.AccessToken = access
box.ExpiresAt = token.ExpiresAt
}
email, err := s.gmailClient(access).Profile(ctx)
if err != nil {
return "", err
}
settings["email"] = email
if err := s.db.SetUserSettings(ctx, user.ID, "gmail", settings); err != nil {
box.Name = email
box.Username = email
if err := s.db.SaveMailbox(ctx, box); err != nil {
return "", err
}
return email, nil
+6 -5
View File
@@ -1,6 +1,7 @@
package httpapi
import (
"github.com/kristianwind/verdande/internal/store"
"net/http"
"net/http/httptest"
"net/url"
@@ -38,8 +39,8 @@ func TestAGmailFailureSaysWhatGmailSaidAndIsRecorded(t *testing.T) {
t.Fatal(err)
}
// Connected, with a refresh token that will be refused.
if err := ts.db.SetUserSettings(t.Context(), user.ID, "gmail", map[string]any{
"refresh_token": "spent", "trigger": "starred",
if err := ts.db.SaveMailbox(t.Context(), &store.Mailbox{
UserID: user.ID, Kind: "gmail", RefreshToken: "spent", Trigger: "starred",
}); err != nil {
t.Fatal(err)
}
@@ -111,9 +112,9 @@ func TestASlowMailboxDoesNotHangTheRequest(t *testing.T) {
}
// A token that has not expired, so the run goes straight to the message list
// rather than stopping at a refresh.
if err := ts.db.SetUserSettings(t.Context(), user.ID, "gmail", map[string]any{
"refresh_token": "r", "access_token": "a", "trigger": "starred",
"expires_at": time.Now().Add(time.Hour).Unix(),
if err := ts.db.SaveMailbox(t.Context(), &store.Mailbox{
UserID: user.ID, Kind: "gmail", RefreshToken: "r", AccessToken: "a",
Trigger: "starred", ExpiresAt: time.Now().Add(time.Hour),
}); err != nil {
t.Fatal(err)
}
+42 -26
View File
@@ -370,27 +370,28 @@ type gmailSettings struct {
}
func (s *Server) handleGetGmail(w http.ResponseWriter, r *http.Request) {
values, err := s.db.UserSettings(r.Context(), userFrom(r.Context()).ID, "gmail")
user := userFrom(r.Context())
box, err := s.db.MailboxOfKind(r.Context(), user.ID, "gmail")
if err != nil {
s.internal(w, r, "gmail settings", err)
s.internal(w, r, "gmail mailbox", err)
return
}
str := func(key string) string {
v, _ := values[key].(string)
return v
if box == nil {
writeJSON(w, http.StatusOK, gmailSettings{})
return
}
// The mailbox address, fetched again if it is missing.
//
// It is stored at connect time, and the call that fetches it can fail — a
// slow token, a scope Google had not applied yet. That failure used to be a
// log line and an empty string, so the page said "an unknown account" and went
// on saying it forever: the only way back was to disconnect and reconnect,
// which nobody would guess. Asking again here costs one request on a page
// nobody opens often, and it means the wrong answer lasts until the next look
// rather than until somebody reconnects.
email := str("email")
if email == "" && str("refresh_token") != "" {
if fetched, err := s.gmailProfile(r.Context(), userFrom(r.Context()), values); err == nil {
// It is stored at connect time, and the call that fetches it can fail — a slow
// token, a scope Google had not applied yet. That failure used to be a log line
// and an empty string, so the page said "an unknown account" and went on saying
// it forever: the only way back was to disconnect and reconnect, which nobody
// would guess. Asking again here costs one request on a page nobody opens
// often, and it means the wrong answer lasts until the next look.
email := box.Username
if email == "" && box.RefreshToken != "" {
if fetched, err := s.gmailProfile(r.Context(), box); err == nil {
email = fetched
} else {
s.log.Warn("gmail profile", "err", err)
@@ -398,10 +399,10 @@ func (s *Server) handleGetGmail(w http.ResponseWriter, r *http.Request) {
}
writeJSON(w, http.StatusOK, gmailSettings{
Connected: str("refresh_token") != "",
Connected: box.RefreshToken != "",
Email: email,
Trigger: str("trigger"),
Label: str("label"),
Trigger: box.Trigger,
Label: box.Label,
})
}
@@ -412,27 +413,42 @@ func (s *Server) handleSetGmail(w http.ResponseWriter, r *http.Request) {
}
user := userFrom(r.Context())
values, err := s.db.UserSettings(r.Context(), user.ID, "gmail")
box, err := s.db.MailboxOfKind(r.Context(), user.ID, "gmail")
if err != nil {
s.internal(w, r, "gmail settings", err)
s.internal(w, r, "gmail mailbox", err)
return
}
values["trigger"] = req.Trigger
values["label"] = req.Label
if box == nil {
// Choosing what makes a task before connecting is allowed, and the choice is
// kept: the row is the mailbox with no credential in it yet, which reads as
// not connected everywhere that asks and is skipped by the sweep.
box = &store.Mailbox{UserID: user.ID, Kind: "gmail"}
}
box.Trigger = req.Trigger
box.Label = req.Label
if err := s.db.SetUserSettings(r.Context(), user.ID, "gmail", values); err != nil {
if err := s.db.SaveMailbox(r.Context(), box); err != nil {
s.internal(w, r, "save gmail settings", err)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleDisconnectGmail forgets the connection, tokens and all.
// handleDisconnectGmail forgets the connection, tokens and all. The tasks it made
// stay: they are the person's work now, not the mailbox's.
func (s *Server) handleDisconnectGmail(w http.ResponseWriter, r *http.Request) {
if err := s.db.SetUserSettings(r.Context(), userFrom(r.Context()).ID, "gmail", map[string]any{}); err != nil {
s.internal(w, r, "disconnect gmail", err)
user := userFrom(r.Context())
box, err := s.db.MailboxOfKind(r.Context(), user.ID, "gmail")
if err != nil {
s.internal(w, r, "gmail mailbox", err)
return
}
if box != nil {
if err := s.db.DeleteMailbox(r.Context(), user.ID, box.ID); err != nil {
s.internal(w, r, "disconnect gmail", err)
return
}
}
w.WriteHeader(http.StatusNoContent)
}
+219
View File
@@ -0,0 +1,219 @@
package httpapi
import (
"context"
"errors"
"fmt"
"net/http"
"strings"
"time"
"github.com/go-chi/chi/v5"
"github.com/kristianwind/verdande/internal/imap"
"github.com/kristianwind/verdande/internal/store"
)
// defaultMailboxSyncBudget bounds one read of one mailbox when somebody is waiting.
// The same reasoning as Gmail's: a mailbox that has stopped answering must not
// hold a request open until whatever sits in front of this server answers for it.
const defaultMailboxSyncBudget = 25 * time.Second
func (s *Server) handleListMailboxes(w http.ResponseWriter, r *http.Request) {
user := userFrom(r.Context())
list, err := s.db.Mailboxes(r.Context(), user.ID)
if err != nil {
s.internal(w, r, "list mailboxes", err)
return
}
if list == nil {
list = []store.Mailbox{}
}
writeJSON(w, http.StatusOK, map[string]any{"mailboxes": list})
}
// handleAddMailbox connects one, after proving it can actually be read.
//
// Tested before it is saved, not after: a mailbox that was accepted and then
// silently fails every ten minutes is worse than one that refused at the door,
// where the person is still holding the password they just typed.
func (s *Server) handleAddMailbox(w http.ResponseWriter, r *http.Request) {
var body struct {
Name string `json:"name"`
Host string `json:"host"`
Username string `json:"username"`
Password string `json:"password"`
Folder string `json:"folder"`
}
if err := decodeJSON(w, r, &body); err != nil {
return
}
body.Host = strings.TrimSpace(body.Host)
body.Username = strings.TrimSpace(body.Username)
if body.Host == "" || body.Username == "" || body.Password == "" {
writeError(w, http.StatusBadRequest, CodeValidation,
"a mailbox needs a host, a username and a password")
return
}
// A host with no port is the mistake everyone makes once, and 993 is the answer
// for every host worth naming.
if !strings.Contains(body.Host, ":") {
body.Host += ":993"
}
account := imap.Account{
Host: body.Host, Username: body.Username,
Password: body.Password, Folder: body.Folder,
}
client, err := imap.Dial(account)
if err != nil {
s.upstream(w, r, CodeMailboxFailed, "connect mailbox", err)
return
}
client.Close()
user := userFrom(r.Context())
m := &store.Mailbox{
UserID: user.ID, Kind: "imap", Name: strings.TrimSpace(body.Name),
Host: body.Host, Username: body.Username,
Password: body.Password, Folder: body.Folder,
}
if m.Name == "" {
m.Name = body.Username
}
if err := s.db.SaveMailbox(r.Context(), m); err != nil {
// The unique index. Saying which mailbox is already here beats "constraint
// failed", which is true and useless.
if strings.Contains(err.Error(), "UNIQUE") {
writeError(w, http.StatusConflict, CodeValidation,
fmt.Sprintf("%s on %s is already connected", body.Username, body.Host))
return
}
s.internal(w, r, "save mailbox", err)
return
}
writeJSON(w, http.StatusCreated, m)
}
func (s *Server) handleDeleteMailbox(w http.ResponseWriter, r *http.Request) {
user := userFrom(r.Context())
if err := s.db.DeleteMailbox(r.Context(), user.ID, chi.URLParam(r, "mailboxID")); err != nil {
s.internal(w, r, "delete mailbox", err)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleSyncMailbox reads one mailbox now, within a budget of its own.
func (s *Server) handleSyncMailbox(w http.ResponseWriter, r *http.Request) {
user := userFrom(r.Context())
m, err := s.db.Mailbox(r.Context(), user.ID, chi.URLParam(r, "mailboxID"))
if err != nil {
s.internal(w, r, "read mailbox", err)
return
}
if m == nil {
writeError(w, http.StatusNotFound, CodeNotFound, "no such mailbox")
return
}
ctx, cancel := context.WithTimeout(r.Context(), defaultMailboxSyncBudget)
defer cancel()
created, err := s.SyncMailbox(ctx, user, m)
if err != nil {
// Whatever was made by the deadline is kept: each task is committed as it
// is made, so a short budget is a shorter run rather than a lost one.
if errors.Is(err, context.DeadlineExceeded) {
writeJSON(w, http.StatusOK, map[string]any{"created": created, "partial": true})
return
}
s.upstream(w, r, CodeMailboxFailed, "sync mailbox", err)
return
}
writeJSON(w, http.StatusOK, map[string]any{"created": created})
}
// SyncMailbox turns the flagged mail newer than the last run into tasks.
//
// Exported because the background job calls it too, on the same terms and without
// a budget: nobody is waiting there.
func (s *Server) SyncMailbox(ctx context.Context, user *store.User, m *store.Mailbox) (int, error) {
client, err := imap.Dial(imap.Account{
Host: m.Host, Username: m.Username, Password: m.Password, Folder: m.Folder,
})
if err != nil {
return 0, err
}
defer client.Close()
messages, searched, err := client.Since(m.LastUID, 25)
if err != nil {
return 0, err
}
if len(messages) == 0 {
return 0, nil
}
projectID, err := s.db.InboxID(ctx, user.ID)
if err != nil {
return 0, err
}
created := 0
highest := m.LastUID
for _, msg := range messages {
if ctx.Err() != nil {
// What was made so far is already committed and counted, and the marker
// below has to be written for it: a run that created tasks and forgot how
// far it got would make them all again on the next pass.
break
}
subject := msg.Subject
if subject == "" {
subject = "(uden emne)"
}
content := subject
if msg.From != "" {
content = msg.From + ": " + subject
}
task := &store.Task{
ProjectID: projectID,
Content: content,
Description: msg.Snippet,
Priority: 4,
CreatedBy: user.ID,
}
if err := s.db.CreateTask(ctx, task, nil); err != nil {
s.log.Warn("mailbox create task", "err", err, "mailbox", m.ID)
continue
}
created++
if msg.UID > highest {
highest = msg.UID
}
s.hub.Publish(projectID, "task.created", toTaskJSON(*task))
}
// A run that got all the way through moves the marker to the whole of what the
// search considered, not merely to the last message it happened to read. That
// is what stops the same mail becoming a task again on the next run when the
// server did not put a uid in the fetch reply.
if ctx.Err() == nil && searched > highest {
highest = searched
}
// Written even when the loop was cut short, and outside the request's context
// so a cancelled request still records what it did.
if highest > m.LastUID {
if err := s.db.MarkMailboxRead(context.WithoutCancel(ctx), m.ID, highest, time.Now()); err != nil {
s.log.Warn("mark mailbox read", "err", err, "mailbox", m.ID)
}
}
if ctx.Err() != nil {
return created, ctx.Err()
}
return created, nil
}
+16 -2
View File
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"net/http"
"strings"
"time"
)
@@ -72,13 +73,26 @@ func (s *Server) handleRestartFromPanel(w http.ResponseWriter, r *http.Request)
ctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), 10*time.Second)
defer cancel()
url := fmt.Sprintf("%s/api/servers/%s/restart", s.cfg.PanelURL, s.cfg.PanelServerID)
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, nil)
// safe-restart rather than restart, and the difference is not politeness.
//
// The plain endpoint does its work while the request is open: it stops the
// container — which kills this process — and the panel, seeing the caller
// vanish, never gets to the starting half. That took production down and left
// it down for four minutes on 18 August, and it cannot be fixed from this side:
// any button that synchronously asks for its own death has the same shape.
//
// safe-restart is scheduled. The panel answers "Restart scheduled" and does the
// work on its own afterwards, so nothing depends on this process still being
// alive to hear the reply.
url := fmt.Sprintf("%s/api/servers/%s/safe-restart", s.cfg.PanelURL, s.cfg.PanelServerID)
body := strings.NewReader(`{"backup_first":false,"target_id":""}`)
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, body)
if err != nil {
s.internal(w, r, "build panel request", err)
return
}
req.Header.Set("Authorization", "Bearer "+s.cfg.PanelToken)
req.Header.Set("Content-Type", "application/json")
resp, err := (&http.Client{Timeout: 10 * time.Second}).Do(req)
if err != nil {
+7
View File
@@ -1,6 +1,7 @@
package httpapi
import (
"io"
"net/http"
"net/http/httptest"
"strings"
@@ -15,6 +16,11 @@ import (
// request has to end either way, or whatever sits in front answers for it.
func TestAPanelThatNeverAnswersStillEndsTheRequest(t *testing.T) {
panel := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Drained first. A real server reads what it was sent, and one that does not
// never learns the caller has hung up — httptest.Server.Close then waits for
// a handler that will never return, and the whole binary times out instead
// of the one request.
_, _ = io.Copy(io.Discard, r.Body)
<-r.Context().Done()
}))
defer panel.Close()
@@ -56,6 +62,7 @@ func TestAPanelThatNeverAnswersStillEndsTheRequest(t *testing.T) {
// And a panel that says no must say so in this server's own words, not in a proxy's.
func TestAPanelThatRefusesSaysSoAsJSON(t *testing.T) {
panel := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = io.Copy(io.Discard, r.Body)
w.WriteHeader(http.StatusForbidden)
}))
defer panel.Close()
+10
View File
@@ -260,6 +260,16 @@ func New(cfg *config.Config, db *store.DB, log *slog.Logger, web fs.FS) *Server
r.Get("/version", s.handleVersion)
// Mailboxes read over IMAP — iCloud, Fastmail, any ordinary host. Each
// belongs to the person who connected it, so there is no admin gate
// here and no shared registration: the credential is theirs.
r.Route("/mailboxes", func(r chi.Router) {
r.Get("/", s.handleListMailboxes)
r.Post("/", s.handleAddMailbox)
r.Delete("/{mailboxID}", s.handleDeleteMailbox)
r.Post("/{mailboxID}/sync", s.handleSyncMailbox)
})
r.Route("/gmail", func(r chi.Router) {
r.Get("/", s.handleGetGmail)
r.Put("/", s.handleSetGmail)
+133 -23
View File
@@ -16,14 +16,18 @@
package imap
import (
"bytes"
"crypto/tls"
"crypto/x509"
"fmt"
"io"
"net"
"strings"
"time"
"github.com/emersion/go-imap/v2"
"github.com/emersion/go-imap/v2/imapclient"
"github.com/emersion/go-message"
)
// Message is one mail, reduced to what a task needs.
@@ -41,6 +45,13 @@ type Account struct {
Username string
Password string // an app-specific password on every host worth naming
Folder string // "INBOX" when empty
// RootCAs adds a certificate authority to trust, on top of the system's. It
// exists so a test can put a real IMAP server behind a self-signed certificate
// and still be refused by everything else — the field can only widen trust,
// never switch verification off, which is the difference between a seam and a
// hole. Nil in production, where the system's roots are the answer.
RootCAs *x509.CertPool
}
// Dial opens a TLS connection and logs in. The caller closes it.
@@ -55,7 +66,11 @@ func Dial(a Account) (*Client, error) {
}
c, err := imapclient.DialTLS(a.Host, &imapclient.Options{
TLSConfig: &tls.Config{ServerName: host, MinVersion: tls.VersionTLS12},
TLSConfig: &tls.Config{
ServerName: host,
MinVersion: tls.VersionTLS12,
RootCAs: a.RootCAs,
},
})
if err != nil {
return nil, fmt.Errorf("connect to %s: %w", a.Host, err)
@@ -96,9 +111,13 @@ func (c *Client) Close() error {
// that consumed it would be taking something away. Flagging a mail is a deliberate
// act with no other meaning, which is exactly what is wanted — the same reason the
// Gmail side reads stars.
func (c *Client) Since(uid uint32, limit int) ([]Message, error) {
// The second return is the highest uid the search considered. A caller that got
// through every message can move its marker to that, and then it does not matter
// whether the server put a uid in each fetch reply — which some do not, and which
// is how the same mail became a task again on every run.
func (c *Client) Since(uid uint32, limit int) ([]Message, uint32, error) {
if _, err := c.c.Select(c.folder, &imap.SelectOptions{ReadOnly: true}).Wait(); err != nil {
return nil, fmt.Errorf("open %s: %w", c.folder, err)
return nil, 0, fmt.Errorf("open %s: %w", c.folder, err)
}
// From uid+1 to the end. A mailbox with nothing new answers with no uids
@@ -112,12 +131,12 @@ func (c *Client) Since(uid uint32, limit int) ([]Message, error) {
}
found, err := c.c.UIDSearch(criteria, nil).Wait()
if err != nil {
return nil, fmt.Errorf("search %s: %w", c.folder, err)
return nil, 0, fmt.Errorf("search %s: %w", c.folder, err)
}
uids := found.AllUIDs()
if len(uids) == 0 {
return nil, nil
return nil, uid, nil
}
if limit > 0 && len(uids) > limit {
// The newest ones. A mailbox flagged over years would otherwise make the
@@ -127,29 +146,51 @@ func (c *Client) Since(uid uint32, limit int) ([]Message, error) {
want := imap.UIDSetNum(uids...)
fetch := &imap.FetchOptions{
Envelope: true,
UID: true,
BodySection: []*imap.FetchItemBodySection{{Specifier: imap.PartSpecifierText, Peek: true}},
Envelope: true,
UID: true,
// The whole message, headers included, so the MIME structure can be walked.
// TEXT alone gives the body as bytes — boundaries and part headers and all —
// which is what put "--=_be71cb39… Content-Transfer-Encoding: 8bit" at the
// front of every task made from a multipart mail.
BodySection: []*imap.FetchItemBodySection{{Peek: true}},
}
var highest uint32
for _, u := range uids {
if uint32(u) > highest {
highest = uint32(u)
}
}
msgs, err := c.c.Fetch(want, fetch).Collect()
if err != nil {
return nil, fmt.Errorf("fetch from %s: %w", c.folder, err)
return nil, 0, fmt.Errorf("fetch from %s: %w", c.folder, err)
}
out := make([]Message, 0, len(msgs))
for _, m := range msgs {
for i, m := range msgs {
if m.Envelope == nil {
continue
}
// The uid comes from the search when the fetch reply does not carry one.
// A missing uid is not cosmetic: it is the marker the next run starts from,
// and a marker stuck at zero makes every message arrive again on every run.
// That is what happened against a real server while the tests stayed green,
// because the server they run against always answers with it.
msgUID := uint32(m.UID)
if msgUID == 0 && i < len(uids) {
msgUID = uint32(uids[i])
}
out = append(out, Message{
UID: uint32(m.UID),
UID: msgUID,
From: address(m.Envelope.From),
Subject: strings.TrimSpace(m.Envelope.Subject),
Snippet: snippet(m),
Date: m.Envelope.Date,
})
}
return out, nil
return out, highest, nil
}
// address renders the first sender as "Name <addr>", or just the address.
@@ -165,23 +206,92 @@ func address(list []imap.Address) string {
return a.Name + " <" + addr + ">"
}
// snippet is the first readable line or so of the body.
// snippet is the first readable text in the mail.
//
// No MIME walk and no HTML stripping yet: this is what shows under a task's title,
// and a wrong-but-short line is a smaller problem than a parser that has to be
// right about every mail ever sent. The full mail stays where it is.
// It walks the MIME structure rather than taking the body as bytes. text/plain is
// preferred and HTML is the fallback with its tags stripped — crudely, because
// this is the line under a task's title and not a rendering engine. The mail
// itself stays where it is.
const snippetLimit = 400
func snippet(m *imapclient.FetchMessageBuffer) string {
for _, body := range m.BodySection {
text := strings.TrimSpace(string(body.Bytes))
if text == "" {
continue
if text := readable(body.Bytes); text != "" {
return cut(text)
}
if len(text) > snippetLimit {
text = text[:snippetLimit] + "…"
}
return text
}
return ""
}
func readable(raw []byte) string {
entity, err := message.Read(bytes.NewReader(raw))
if err != nil {
// Not something this parser recognises. The raw bytes read badly but read,
// and a mail is not worth losing over its own headers.
return strings.TrimSpace(string(raw))
}
return strings.TrimSpace(fromEntity(entity, 0))
}
// fromEntity descends into a multipart mail. The depth guard is for the mail that
// nests parts inside parts inside parts — rare, and cheaper to bound than to trust.
func fromEntity(entity *message.Entity, depth int) string {
if depth > 4 {
return ""
}
if parts := entity.MultipartReader(); parts != nil {
var html string
for {
part, err := parts.NextPart()
if err != nil {
break
}
if text := fromEntity(part, depth+1); text != "" {
kind, _, _ := part.Header.ContentType()
if kind != "text/html" {
return text
}
if html == "" {
html = text
}
}
}
return html
}
body, _ := io.ReadAll(entity.Body)
kind, _, _ := entity.Header.ContentType()
if kind == "text/html" {
return stripTags(string(body))
}
return strings.TrimSpace(string(body))
}
// stripTags removes markup without pretending to understand it. Enough for one
// line of preview, and honest about being no more than that.
func stripTags(html string) string {
var b strings.Builder
depth := 0
for _, r := range html {
switch {
case r == '<':
depth++
case r == '>':
if depth > 0 {
depth--
}
case depth == 0:
b.WriteRune(r)
}
}
return strings.Join(strings.Fields(b.String()), " ")
}
func cut(text string) string {
runes := []rune(text)
if len(runes) > snippetLimit {
return string(runes[:snippetLimit]) + "…"
}
return text
}
+264
View File
@@ -0,0 +1,264 @@
package imap
import (
"crypto/ecdsa"
"crypto/elliptic"
"crypto/rand"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"math/big"
"net"
"strings"
"testing"
"time"
"github.com/emersion/go-imap/v2"
"github.com/emersion/go-imap/v2/imapserver"
"github.com/emersion/go-imap/v2/imapserver/imapmemserver"
)
// The round trip, against a real IMAP server rather than a stub of one. The
// earlier tests cover the pure parts — what a sender renders as, where a snippet
// is cut — and would all have passed with the fetch itself broken.
func TestFlaggedMailComesBackAsMessages(t *testing.T) {
addr, pool := serveMailbox(t, []mail{
{subject: "Faktura 4711", from: "Anders Bo <anders@example.dk>", body: "Vedhæftet.", flagged: true},
{subject: "Nyhedsbrev", from: "noreply@example.com", body: "Læs mere", flagged: false},
{subject: "Kontrakt", from: "jura@example.dk", body: "Til underskrift", flagged: true},
})
client, err := Dial(Account{
Host: addr, Username: "kw", Password: "hemmelig", RootCAs: pool,
})
if err != nil {
t.Fatal(err)
}
defer client.Close()
msgs, _, err := client.Since(0, 25)
if err != nil {
t.Fatal(err)
}
// Two of the three. The newsletter is unflagged, and flagging is the whole
// signal: unread is a state the owner is still using.
if len(msgs) != 2 {
t.Fatalf("got %d messages, want the 2 flagged ones", len(msgs))
}
if msgs[0].Subject != "Faktura 4711" {
t.Errorf("first subject is %q", msgs[0].Subject)
}
if !strings.Contains(msgs[0].From, "anders@example.dk") {
t.Errorf("first sender is %q", msgs[0].From)
}
if msgs[0].UID == 0 {
t.Error("no uid came back, so a second run would fetch everything again")
}
// And reading on from a marker returns only what is newer, which is the whole
// of the deduplication.
rest, _, err := client.Since(msgs[0].UID, 25)
if err != nil {
t.Fatal(err)
}
if len(rest) != 1 || rest[0].Subject != "Kontrakt" {
t.Fatalf("reading on from uid %d gave %d messages", msgs[0].UID, len(rest))
}
}
func TestAWrongPasswordIsRefusedClearly(t *testing.T) {
addr, pool := serveMailbox(t, nil)
_, err := Dial(Account{Host: addr, Username: "kw", Password: "forkert", RootCAs: pool})
if err == nil {
t.Fatal("a wrong password was accepted")
}
// Hosts say everything from AUTHENTICATIONFAILED to a sentence about the web,
// so the reply names the host and the user rather than repeating their words.
if !strings.Contains(err.Error(), "kw") {
t.Errorf("the error does not say whose login was refused: %v", err)
}
}
// A certificate this process signed itself must not be enough on its own.
func TestAnUntrustedCertificateIsRefused(t *testing.T) {
addr, _ := serveMailbox(t, nil)
if _, err := Dial(Account{Host: addr, Username: "kw", Password: "hemmelig"}); err == nil {
t.Fatal("a self-signed certificate was accepted without being trusted")
}
}
type mail struct {
subject, from, body string
flagged bool
// raw replaces the whole message when set, which is the only way to get a
// genuinely multipart one in.
raw string
}
// serveMailbox starts an in-process IMAP server over TLS and returns its address
// and the pool that trusts it.
func serveMailbox(t *testing.T, mails []mail) (string, *x509.CertPool) {
t.Helper()
cert, pool := selfSigned(t)
memServer := imapmemserver.New()
user := imapmemserver.NewUser("kw", "hemmelig")
if err := user.Create("INBOX", nil); err != nil {
t.Fatal(err)
}
memServer.AddUser(user)
for _, m := range mails {
flags := []imap.Flag{}
if m.flagged {
flags = append(flags, imap.FlagFlagged)
}
raw := m.raw
if raw == "" {
raw = "From: " + m.from + "\r\nSubject: " + m.subject +
"\r\nDate: Tue, 18 Aug 2026 10:00:00 +0200\r\n\r\n" + m.body + "\r\n"
}
if _, err := user.Append("INBOX", strings.NewReader(raw), &imap.AppendOptions{
Flags: flags,
}); err != nil {
t.Fatal(err)
}
}
srv := imapserver.New(&imapserver.Options{
NewSession: func(_ *imapserver.Conn) (imapserver.Session, *imapserver.GreetingData, error) {
return memServer.NewSession(), nil, nil
},
TLSConfig: &tls.Config{Certificates: []tls.Certificate{cert}},
InsecureAuth: false,
})
t.Cleanup(func() { srv.Close() })
ln, err := tls.Listen("tcp", "127.0.0.1:0", &tls.Config{Certificates: []tls.Certificate{cert}})
if err != nil {
t.Fatal(err)
}
go srv.Serve(ln)
// "localhost", not the loopback address: the certificate names a host, and the
// client checks that name — as it should.
_, port, _ := net.SplitHostPort(ln.Addr().String())
return net.JoinHostPort("localhost", port), pool
}
func selfSigned(t *testing.T) (tls.Certificate, *x509.CertPool) {
t.Helper()
key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
if err != nil {
t.Fatal(err)
}
template := x509.Certificate{
SerialNumber: big.NewInt(1),
Subject: pkix.Name{CommonName: "localhost"},
DNSNames: []string{"localhost"},
IPAddresses: []net.IP{net.ParseIP("127.0.0.1")},
NotBefore: time.Now().Add(-time.Hour),
NotAfter: time.Now().Add(time.Hour),
KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
IsCA: true,
BasicConstraintsValid: true,
}
der, err := x509.CreateCertificate(rand.Reader, &template, &template, &key.PublicKey, key)
if err != nil {
t.Fatal(err)
}
certPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
keyDER, err := x509.MarshalECPrivateKey(key)
if err != nil {
t.Fatal(err)
}
keyPEM := pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER})
cert, err := tls.X509KeyPair(certPEM, keyPEM)
if err != nil {
t.Fatal(err)
}
pool := x509.NewCertPool()
pool.AppendCertsFromPEM(certPEM)
return cert, pool
}
// The bug this was written for: every press of "Fetch now" made the same mail
// into a task again. The marker the next run starts from is the uid, and if it
// comes back as zero the marker never moves.
func TestEveryMessageCarriesAUID(t *testing.T) {
addr, pool := serveMailbox(t, []mail{
{subject: "En", from: "a@example.dk", body: "en", flagged: true},
{subject: "To", from: "b@example.dk", body: "to", flagged: true},
})
client, err := Dial(Account{Host: addr, Username: "kw", Password: "hemmelig", RootCAs: pool})
if err != nil {
t.Fatal(err)
}
defer client.Close()
msgs, _, err := client.Since(0, 25)
if err != nil {
t.Fatal(err)
}
for _, m := range msgs {
if m.UID == 0 {
t.Fatalf("%q came back without a uid; the marker would never move and "+
"every run would make it into a task again", m.Subject)
}
}
if msgs[0].UID == msgs[1].UID {
t.Error("two messages share a uid")
}
}
// The other half of the same report: the snippet under a task's title read
// "--=_be71cb39… Content-Transfer-Encoding: 8bit Content-Type: text/plain".
func TestAMultipartMailReadsAsItsText(t *testing.T) {
const boundary = "=_be71cb3975bdfbc60a16327cb2f49c23"
body := "This is a multi-part message.\r\n" +
"--" + boundary + "\r\n" +
"Content-Transfer-Encoding: 8bit\r\n" +
"Content-Type: text/plain; charset=UTF-8; format=flowed\r\n\r\n" +
"Hej igen,\r\n\r\nVi har normalt 3 slags. :)\r\n" +
"--" + boundary + "\r\n" +
"Content-Type: text/html; charset=UTF-8\r\n\r\n" +
"<html><body><p>Hej igen</p></body></html>\r\n" +
"--" + boundary + "--\r\n"
addr, pool := serveMailbox(t, []mail{{
flagged: true,
raw: "Content-Type: multipart/alternative; boundary=\"" + boundary + "\"\r\n" +
"From: kontakt@mikrobagt.dk\r\nSubject: Re: Kaffebestilling\r\n" +
"Date: Tue, 18 Aug 2026 10:00:00 +0200\r\n\r\n" + body,
}})
client, err := Dial(Account{Host: addr, Username: "kw", Password: "hemmelig", RootCAs: pool})
if err != nil {
t.Fatal(err)
}
defer client.Close()
msgs, _, err := client.Since(0, 25)
if err != nil {
t.Fatal(err)
}
if len(msgs) != 1 {
t.Fatalf("got %d messages", len(msgs))
}
got := msgs[0].Snippet
for _, junk := range []string{boundary, "Content-Transfer-Encoding", "Content-Type", "<html>", "<p>"} {
if strings.Contains(got, junk) {
t.Errorf("the snippet still carries %q: %q", junk, got)
}
}
if !strings.HasPrefix(got, "Hej igen") {
t.Errorf("the snippet does not start with the text of the mail: %q", got)
}
}
+44 -13
View File
@@ -37,6 +37,10 @@ type Runner struct {
// startable from main.
SyncGmail func(ctx context.Context, user *store.User) (int, error)
// SyncMailbox reads one IMAP mailbox. Supplied by the HTTP layer for the same
// reason as SyncGmail: it owns turning a message into a task.
SyncMailbox func(ctx context.Context, user *store.User, m *store.Mailbox) (int, error)
wg sync.WaitGroup
}
@@ -60,7 +64,7 @@ func (r *Runner) Start(ctx context.Context) {
// need a public webhook and a Cloud Pub/Sub topic, which is a lot of Google
// account for a to-do app. Ten minutes is fast enough for something somebody
// starred and slow enough not to look like abuse.
r.every(ctx, "gmail", 10*time.Minute, r.syncGmailAccounts)
r.every(ctx, "mailboxes", 10*time.Minute, r.syncMailboxes)
}
func (r *Runner) Wait() { r.wg.Wait() }
@@ -247,26 +251,53 @@ func rotateBackups(cfg *config.Config, log *slog.Logger) error {
// --- gmail ---------------------------------------------------------------------------
// syncGmailAccounts polls every connected mailbox.
// syncMailboxes polls every connected mailbox, Gmail and IMAP alike.
//
// One at a time, and a failure on one account does not stop the others: a revoked
// token or an expired refresh is a problem for that person alone.
func (r *Runner) syncGmailAccounts(ctx context.Context) error {
if r.SyncGmail == nil {
// One sweep rather than two, now that both live in the same table. Nobody is
// waiting, so no budget of its own — the request handler has one because a person
// is watching a spinner. A host that has gone quiet costs this run its own dial
// timeout and nothing else: the loop moves on, per mailbox and per person, so one
// bad password does not stop everybody else's mail.
func (r *Runner) syncMailboxes(ctx context.Context) error {
if r.SyncMailbox == nil {
return nil
}
users, err := r.db.UsersWithGmail(ctx)
userIDs, err := r.db.UsersWithMailboxes(ctx)
if err != nil {
return err
}
for _, user := range users {
created, err := r.SyncGmail(ctx, &user)
if err != nil {
r.log.Warn("gmail sync", "err", err, "user", user.ID)
for _, id := range userIDs {
user, err := r.db.UserByID(ctx, id)
if err != nil || user == nil {
continue
}
if created > 0 {
r.log.Info("tasks created from gmail", "user", user.ID, "count", created)
boxes, err := r.db.Mailboxes(ctx, id)
if err != nil {
r.log.Warn("list mailboxes", "err", err, "user", id)
continue
}
for i := range boxes {
var created int
var err error
// Gmail is fetched over HTTP with a token, IMAP over a connection with a
// password. Which one is a property of the row, not of the loop.
if boxes[i].Kind == "gmail" {
if r.SyncGmail == nil {
continue
}
created, err = r.SyncGmail(ctx, user)
} else {
created, err = r.SyncMailbox(ctx, user, &boxes[i])
}
if err != nil {
r.log.Warn("mailbox sync", "err", err, "user", id,
"kind", boxes[i].Kind, "mailbox", boxes[i].ID)
continue
}
if created > 0 {
r.log.Info("tasks created from a mailbox", "user", id,
"kind", boxes[i].Kind, "count", created)
}
}
}
return nil
+253
View File
@@ -0,0 +1,253 @@
package store
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"time"
)
// Mailbox is one connected mailbox belonging to one person.
type Mailbox struct {
ID string `json:"id"`
UserID string `json:"-"`
Kind string `json:"kind"` // "gmail" or "imap"
Name string `json:"name"`
Host string `json:"host,omitempty"`
Username string `json:"username,omitempty"`
Password string `json:"-"` // never leaves the server
Folder string `json:"folder,omitempty"`
RefreshToken string `json:"-"`
AccessToken string `json:"-"`
ExpiresAt time.Time `json:"-"`
Label string `json:"label,omitempty"`
// ProjectID is where this mailbox's tasks land. Empty means the inbox, and so
// does an id the person can no longer write to.
ProjectID string `json:"project_id,omitempty"`
Trigger string `json:"trigger,omitempty"` // "starred" or "label"
// Seen is the message ids Gmail has already turned into tasks. IMAP does not
// need it — uids are monotonic, so LastUID says the same in one number — but
// Gmail's query is a rolling window rather than a marker.
Seen []string `json:"-"`
LastUID uint32 `json:"last_uid,omitempty"`
LastSyncAt time.Time `json:"last_sync_at,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
const mailboxColumns = `id, user_id, kind, name, host, username, password, folder,
refresh_token, access_token, expires_at, label, trigger_kind, seen, project_id,
last_uid, last_sync_at, created_at`
// Mailboxes returns everything one person has connected, oldest first.
func (db *DB) Mailboxes(ctx context.Context, userID string) ([]Mailbox, error) {
rows, err := db.QueryContext(ctx,
`SELECT `+mailboxColumns+` FROM mailboxes WHERE user_id = ? ORDER BY created_at`, userID)
if err != nil {
return nil, err
}
defer rows.Close()
var out []Mailbox
for rows.Next() {
m, err := db.scanMailbox(rows)
if err != nil {
return nil, err
}
out = append(out, m)
}
return out, rows.Err()
}
// Mailbox returns one, and only to the person it belongs to: the id is enough to
// find a row, so the owner is part of the question rather than checked after.
func (db *DB) Mailbox(ctx context.Context, userID, id string) (*Mailbox, error) {
row := db.QueryRowContext(ctx,
`SELECT `+mailboxColumns+` FROM mailboxes WHERE id = ? AND user_id = ?`, id, userID)
m, err := db.scanMailbox(row)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
return &m, nil
}
// MailboxOfKind returns a person's mailbox of one kind, or nil. Gmail is one per
// person because the OAuth flow is: connecting again replaces what was there.
func (db *DB) MailboxOfKind(ctx context.Context, userID, kind string) (*Mailbox, error) {
row := db.QueryRowContext(ctx,
`SELECT `+mailboxColumns+` FROM mailboxes WHERE user_id = ? AND kind = ?`, userID, kind)
m, err := db.scanMailbox(row)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
return &m, nil
}
// SaveMailbox writes one, sealing the two columns that must not travel with a
// backup. Called by hand rather than by the settings tables' own seal, which does
// not reach this table.
func (db *DB) SaveMailbox(ctx context.Context, m *Mailbox) error {
if m.ID == "" {
m.ID = NewID()
}
if m.CreatedAt.IsZero() {
m.CreatedAt = time.Now()
}
if m.Folder == "" {
m.Folder = "INBOX"
}
password, err := db.sealValue(m.Password)
if err != nil {
return fmt.Errorf("seal password: %w", err)
}
refresh, err := db.sealValue(m.RefreshToken)
if err != nil {
return fmt.Errorf("seal refresh_token: %w", err)
}
access, err := db.sealValue(m.AccessToken)
if err != nil {
return fmt.Errorf("seal access_token: %w", err)
}
// Not sealed: which mails have been read is not a secret, and a column of
// ciphertext here would only make the table harder to look at.
seen, err := json.Marshal(m.Seen)
if err != nil {
return err
}
if m.Trigger == "" {
m.Trigger = "starred"
}
_, err = db.ExecContext(ctx, `
INSERT INTO mailboxes (`+mailboxColumns+`)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (id) DO UPDATE SET
name = excluded.name,
host = excluded.host,
username = excluded.username,
password = excluded.password,
folder = excluded.folder,
refresh_token = excluded.refresh_token,
access_token = excluded.access_token,
expires_at = excluded.expires_at,
label = excluded.label,
trigger_kind = excluded.trigger_kind,
seen = excluded.seen,
project_id = excluded.project_id,
last_uid = excluded.last_uid,
last_sync_at = excluded.last_sync_at`,
m.ID, m.UserID, m.Kind, m.Name, m.Host, m.Username, password, m.Folder,
refresh, access, unixOrZero(m.ExpiresAt), m.Label, m.Trigger, string(seen), m.ProjectID,
m.LastUID, unixOrZero(m.LastSyncAt), m.CreatedAt.Unix())
return err
}
// MarkMailboxRead records how far a run got. Written separately from the rest so
// a sync never rewrites the credentials it is holding in memory.
func (db *DB) MarkMailboxRead(ctx context.Context, id string, lastUID uint32, at time.Time) error {
_, err := db.ExecContext(ctx,
`UPDATE mailboxes SET last_uid = ?, last_sync_at = ? WHERE id = ?`,
lastUID, at.Unix(), id)
return err
}
// DeleteMailbox disconnects one. The tasks it made stay: they are the person's
// work now, not the mailbox's.
func (db *DB) DeleteMailbox(ctx context.Context, userID, id string) error {
_, err := db.ExecContext(ctx, `DELETE FROM mailboxes WHERE id = ? AND user_id = ?`, id, userID)
return err
}
// UsersWithMailboxes returns everybody the sweep has to visit.
func (db *DB) UsersWithMailboxes(ctx context.Context) ([]string, error) {
rows, err := db.QueryContext(ctx, `SELECT DISTINCT user_id FROM mailboxes`)
if err != nil {
return nil, err
}
defer rows.Close()
var ids []string
for rows.Next() {
var id string
if err := rows.Scan(&id); err != nil {
return nil, err
}
ids = append(ids, id)
}
return ids, rows.Err()
}
type scanner interface {
Scan(dest ...any) error
}
func (db *DB) scanMailbox(row scanner) (Mailbox, error) {
var m Mailbox
var expires, lastSync, created int64
var seen string
err := row.Scan(&m.ID, &m.UserID, &m.Kind, &m.Name, &m.Host, &m.Username,
&m.Password, &m.Folder, &m.RefreshToken, &m.AccessToken, &expires,
&m.Label, &m.Trigger, &seen, &m.ProjectID, &m.LastUID, &lastSync, &created)
if err != nil {
return m, err
}
// A row that somehow holds nonsense here is not worth failing a whole sync
// over: an empty list means everything looks new, which is recoverable.
_ = json.Unmarshal([]byte(seen), &m.Seen)
for _, field := range []*string{&m.Password, &m.RefreshToken, &m.AccessToken} {
plain, err := db.unsealValue(*field)
if err != nil {
return m, err
}
*field = plain
}
m.ExpiresAt = timeOrZero(expires)
m.LastSyncAt = timeOrZero(lastSync)
m.CreatedAt = time.Unix(created, 0)
return m, nil
}
// sealValue and unsealValue are the single-string versions of the map helpers in
// notifications.go. Without a key they pass the value through, which is what the
// tests that never set one expect.
func (db *DB) sealValue(v string) (string, error) {
if db.box == nil {
return v, nil
}
return db.box.Seal(v)
}
func (db *DB) unsealValue(v string) (string, error) {
if db.box == nil {
return v, nil
}
return db.box.Unseal(v)
}
func unixOrZero(t time.Time) int64 {
if t.IsZero() {
return 0
}
return t.Unix()
}
func timeOrZero(unix int64) time.Time {
if unix == 0 {
return time.Time{}
}
return time.Unix(unix, 0)
}
+264
View File
@@ -0,0 +1,264 @@
package store
import (
"context"
"path/filepath"
"strings"
"testing"
"time"
"github.com/kristianwind/verdande/internal/secret"
)
func TestAMailboxIsStoredSealedAndComesBackWhole(t *testing.T) {
db, userID := sealedStore(t)
ctx := context.Background()
const password = "abcd-efgh-ijkl-mnop"
m := &Mailbox{
UserID: userID,
Kind: "imap",
Name: "iCloud",
Host: "imap.mail.me.com:993",
Username: "kw@icloud.com",
Password: password,
}
if err := db.SaveMailbox(ctx, m); err != nil {
t.Fatal(err)
}
var stored string
if err := db.QueryRowContext(ctx,
`SELECT password FROM mailboxes WHERE id = ?`, m.ID).Scan(&stored); err != nil {
t.Fatal(err)
}
if strings.Contains(stored, password) {
t.Fatalf("the password is in the row as typed: %s", stored)
}
back, err := db.Mailbox(ctx, userID, m.ID)
if err != nil {
t.Fatal(err)
}
if back == nil {
t.Fatal("the mailbox was not found")
}
if back.Password != password {
t.Errorf("got password %q", back.Password)
}
if back.Folder != "INBOX" {
t.Errorf("the folder defaulted to %q, want INBOX", back.Folder)
}
}
// The question that prompted the table: two people on one instance, each with
// their own mailbox, neither seeing the other's.
func TestAMailboxBelongsToOnePerson(t *testing.T) {
db, kristian := sealedStore(t)
ctx := context.Background()
andreas := &User{Email: "andreas@example.dk", Name: "Andreas"}
if err := db.CreateUser(ctx, andreas, "Indbakke"); err != nil {
t.Fatal(err)
}
mine := &Mailbox{UserID: kristian, Kind: "imap", Host: "imap.mail.me.com:993", Username: "kw"}
theirs := &Mailbox{UserID: andreas.ID, Kind: "imap", Host: "imap.fastmail.com:993", Username: "andreas"}
for _, m := range []*Mailbox{mine, theirs} {
if err := db.SaveMailbox(ctx, m); err != nil {
t.Fatal(err)
}
}
list, err := db.Mailboxes(ctx, kristian)
if err != nil {
t.Fatal(err)
}
if len(list) != 1 || list[0].Username != "kw" {
t.Fatalf("got %d mailboxes for Kristian, want only his own", len(list))
}
// And the id alone is not enough to reach somebody else's.
other, err := db.Mailbox(ctx, kristian, theirs.ID)
if err != nil {
t.Fatal(err)
}
if other != nil {
t.Error("one person read another's mailbox by its id")
}
}
// Several mailboxes each is the ordinary case — a work Gmail and a private
// iCloud — and is the whole reason this is a table rather than a setting.
func TestOnePersonCanHaveSeveral(t *testing.T) {
db, userID := sealedStore(t)
ctx := context.Background()
for _, host := range []string{"imap.mail.me.com:993", "imap.fastmail.com:993"} {
if err := db.SaveMailbox(ctx, &Mailbox{
UserID: userID, Kind: "imap", Host: host, Username: "kw",
}); err != nil {
t.Fatal(err)
}
}
list, err := db.Mailboxes(ctx, userID)
if err != nil {
t.Fatal(err)
}
if len(list) != 2 {
t.Fatalf("got %d mailboxes, want 2", len(list))
}
}
// The same mailbox twice would make two tasks out of every mail.
func TestTheSameMailboxTwiceIsRefused(t *testing.T) {
db, userID := sealedStore(t)
ctx := context.Background()
first := &Mailbox{UserID: userID, Kind: "imap", Host: "imap.mail.me.com:993", Username: "kw"}
if err := db.SaveMailbox(ctx, first); err != nil {
t.Fatal(err)
}
again := &Mailbox{UserID: userID, Kind: "imap", Host: "imap.mail.me.com:993", Username: "kw"}
if err := db.SaveMailbox(ctx, again); err == nil {
t.Error("the same host and username was accepted a second time")
}
}
func TestReadingProgressIsRemembered(t *testing.T) {
db, userID := sealedStore(t)
ctx := context.Background()
m := &Mailbox{UserID: userID, Kind: "imap", Host: "imap.mail.me.com:993", Username: "kw"}
if err := db.SaveMailbox(ctx, m); err != nil {
t.Fatal(err)
}
at := time.Now().Truncate(time.Second)
if err := db.MarkMailboxRead(ctx, m.ID, 4711, at); err != nil {
t.Fatal(err)
}
back, err := db.Mailbox(ctx, userID, m.ID)
if err != nil {
t.Fatal(err)
}
if back.LastUID != 4711 {
t.Errorf("last uid is %d", back.LastUID)
}
if !back.LastSyncAt.Equal(at) {
t.Errorf("last sync is %v, want %v", back.LastSyncAt, at)
}
// The credentials it was holding must not have been rewritten as a side effect.
if back.Host != "imap.mail.me.com:993" {
t.Errorf("the host changed to %q", back.Host)
}
}
// Disconnecting takes the mailbox, not the work it produced.
func TestDisconnectingKeepsTheTasks(t *testing.T) {
db, userID := sealedStore(t)
ctx := context.Background()
m := &Mailbox{UserID: userID, Kind: "imap", Host: "imap.mail.me.com:993", Username: "kw"}
if err := db.SaveMailbox(ctx, m); err != nil {
t.Fatal(err)
}
if err := db.DeleteMailbox(ctx, userID, m.ID); err != nil {
t.Fatal(err)
}
list, err := db.Mailboxes(ctx, userID)
if err != nil {
t.Fatal(err)
}
if len(list) != 0 {
t.Errorf("the mailbox is still connected")
}
}
// The migration that this whole move stands or falls on: a Gmail connected before
// there was a mailboxes table must come out the other side still connected, with
// its token readable and its settings intact. Nobody should have to reconnect
// because the server tidied up.
func TestAGmailConnectedBeforeTheMoveSurvivesIt(t *testing.T) {
dir := t.TempDir()
box, err := secret.Open("", t.TempDir())
if err != nil {
t.Fatal(err)
}
// A database at the state before 0017, with a sealed token written the way the
// settings table wrote them.
db, userID := storeAt(t, dir, box)
sealed, err := box.Seal("1//0-the-original-refresh-token")
if err != nil {
t.Fatal(err)
}
values := `{"refresh_token":"` + sealed + `","trigger":"label","label":"Verdande",` +
`"email":"kw@nolimit.dk","seen":["abc","def"],"project_id":"p-1"}`
if _, err := db.ExecContext(context.Background(), `
INSERT INTO user_settings (user_id, scope, values_json, updated_at)
VALUES (?, 'gmail', ?, 0)
ON CONFLICT (user_id) DO UPDATE SET scope = 'gmail', values_json = excluded.values_json`,
userID, values); err != nil {
t.Fatal(err)
}
// 0017 has already run on this database, so the row is moved by hand here in
// the same statement the migration uses. What is being tested is the shape it
// lands in and that the ciphertext survived the copy.
if _, err := db.ExecContext(context.Background(), `
INSERT INTO mailboxes (id, user_id, kind, name, host, username, password, folder,
refresh_token, access_token, expires_at, label, trigger_kind, seen, project_id,
last_uid, last_sync_at, created_at)
SELECT lower(hex(randomblob(16))), s.user_id, 'gmail',
json_extract(s.values_json, '$.email'), '', json_extract(s.values_json, '$.email'),
'', 'INBOX', json_extract(s.values_json, '$.refresh_token'), '', 0,
json_extract(s.values_json, '$.label'), json_extract(s.values_json, '$.trigger'),
json_extract(s.values_json, '$.seen'), json_extract(s.values_json, '$.project_id'),
0, 0, unixepoch()
FROM user_settings s WHERE s.scope = 'gmail'`); err != nil {
t.Fatal(err)
}
moved, err := db.MailboxOfKind(context.Background(), userID, "gmail")
if err != nil {
t.Fatal(err)
}
if moved == nil {
t.Fatal("the Gmail connection did not come across")
}
// The whole point: the token was copied as ciphertext and still opens.
if moved.RefreshToken != "1//0-the-original-refresh-token" {
t.Errorf("the token came out as %q", moved.RefreshToken)
}
if moved.Trigger != "label" || moved.Label != "Verdande" {
t.Errorf("the trigger came out as %q/%q", moved.Trigger, moved.Label)
}
if moved.Username != "kw@nolimit.dk" {
t.Errorf("the address came out as %q", moved.Username)
}
if moved.ProjectID != "p-1" {
t.Errorf("the destination project came out as %q", moved.ProjectID)
}
// And the seen list, or every mail in the window becomes a task again.
if len(moved.Seen) != 2 || moved.Seen[0] != "abc" {
t.Errorf("the seen list came out as %v", moved.Seen)
}
}
// storeAt opens a database in a named directory with a given key, so a test can
// keep the key while the database is reopened.
func storeAt(t *testing.T, dir string, box *secret.Box) (*DB, string) {
t.Helper()
db, err := Open(filepath.Join(dir, "test.db"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { db.Close() })
db.UseSecrets(box)
user := &User{Email: "kristian@example.dk", Name: "Kristian"}
if err := db.CreateUser(context.Background(), user, "Indbakke"); err != nil {
t.Fatal(err)
}
return db, user.ID
}
@@ -0,0 +1,57 @@
-- Mailboxes: one row per connected mailbox, not one setting per person.
--
-- Gmail lived in `user_settings` under scope 'gmail'. That table has `user_id` as
-- its whole primary key, so it holds exactly one scope per person and its upsert
-- rewrites the scope name on conflict — adding IMAP beside Gmail there would have
-- deleted the Gmail connection without saying so. Migration 0011 warns about this
-- in as many words, and the warning was nearly walked into a second time.
--
-- The shape is wrong for the thing anyway. A person has mailboxes, plural: a work
-- Gmail and a private iCloud is the ordinary case, not the exotic one. A row each
-- says that. It also answers the question that prompted this — every mailbox
-- belongs to a user, so two people on the same instance connect their own and see
-- only their own. The one piece that stays instance-wide is the Google OAuth
-- client, which is a single registration everybody signs in through, and that is
-- correct: it is the app, not the account.
--
-- `kind` rather than two tables. What differs is how the credential is obtained —
-- OAuth for Gmail, a password for IMAP — and what differs is a nullable column,
-- not a schema. What is the same is everything the sync loop touches: which
-- mailbox, how far it has read, when it last ran.
--
-- `password` and `refresh_token` are written sealed; see internal/secret. The
-- store's own seal/unseal only covers the settings tables, so this table's methods
-- call them by hand — a fact worth knowing before adding a third secret column.
--
-- `last_uid` is IMAP's place-marker and means nothing for Gmail, which keeps its
-- own by query. Zero is "nothing read yet" for both.
CREATE TABLE mailboxes (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL REFERENCES users (id) ON DELETE CASCADE,
kind TEXT NOT NULL CHECK (kind IN ('gmail', 'imap')),
name TEXT NOT NULL DEFAULT '',
-- IMAP
host TEXT NOT NULL DEFAULT '',
username TEXT NOT NULL DEFAULT '',
password TEXT NOT NULL DEFAULT '',
folder TEXT NOT NULL DEFAULT 'INBOX',
-- Gmail
refresh_token TEXT NOT NULL DEFAULT '',
access_token TEXT NOT NULL DEFAULT '',
expires_at INTEGER NOT NULL DEFAULT 0,
label TEXT NOT NULL DEFAULT '',
last_uid INTEGER NOT NULL DEFAULT 0,
last_sync_at INTEGER NOT NULL DEFAULT 0,
created_at INTEGER NOT NULL
);
-- The sweep asks "whose mailboxes are due", so it reads by user.
CREATE INDEX mailboxes_by_user ON mailboxes (user_id);
-- The same mailbox twice is a mistake, not a choice: two rows would make two
-- tasks out of every mail. Gmail's uniqueness is the account, IMAP's is the host
-- and the username together, and `kind` keeps the two from colliding.
CREATE UNIQUE INDEX mailboxes_once ON mailboxes (user_id, kind, host, username);
@@ -0,0 +1,22 @@
-- Room for Gmail beside the others.
--
-- Only the two columns. Moving the data is a separate migration on purpose: the
-- Gmail handlers still read user_settings, and a migration that emptied that table
-- before they were rewritten would disconnect a working mailbox. Additive first,
-- and the move when the code on the other side is ready for it.
--
-- Gmail lived in `user_settings` under scope 'gmail' because it was the only
-- integration of its kind. Now there are two, and two places to look for the same
-- question — which mailboxes does this person have — is one place too many.
--
-- `seen` is Gmail's own deduplication: a list of message ids it has already turned
-- into tasks. IMAP does not need it, because uids are monotonic and `last_uid`
-- answers the same question in one number. Gmail's query is a 30-day window rather
-- than a marker, so the list is what it has.
ALTER TABLE mailboxes ADD COLUMN seen TEXT NOT NULL DEFAULT '[]';
-- What makes a mail into a task: 'starred', or 'label' with the name in `label`.
-- Its own column rather than packed into `label` with a separator — a column that
-- holds two things joined by a colon is a column somebody parses wrongly later,
-- and a label named with a colon in it would do it on the first day.
ALTER TABLE mailboxes ADD COLUMN trigger_kind TEXT NOT NULL DEFAULT 'starred';
@@ -0,0 +1,12 @@
-- Where a mailbox's tasks land.
--
-- Gmail carried this as `project_id` in its settings blob. It belongs to the
-- mailbox, and it is worth having for the ones read over IMAP too: a work mailbox
-- filing into the work project is the obvious thing to want, and the alternative
-- is everything landing in the inbox to be moved by hand.
--
-- Empty means the person's inbox, which is also what an id that no longer names a
-- project they can write to falls back to. Not a foreign key for that reason: a
-- deleted project should quietly stop being the destination, not refuse the delete
-- or take the mailbox down with it.
ALTER TABLE mailboxes ADD COLUMN project_id TEXT NOT NULL DEFAULT '';
@@ -0,0 +1,46 @@
-- Gmail's rows move in beside the ones read over IMAP.
--
-- Held back until now on purpose: this empties the settings row, and running it
-- while the handlers still read that row would have disconnected a working
-- mailbox. internal/httpapi now reads the mailboxes table, so the move is safe.
--
-- The values move across still sealed. Both tables seal with the same key and
-- store the same string, so copying the ciphertext verbatim is not a shortcut
-- around the encryption — it is the encryption, untouched. That also means this
-- migration does not need the key, which a migration has no business holding.
--
-- `username` is the address the account reports, and it is what makes the unique
-- index meaningful. It may be empty on rows that never got that far; two such rows
-- cannot exist for one person anyway, because the settings table could only hold
-- one.
INSERT INTO mailboxes (
id, user_id, kind, name, host, username, password, folder,
refresh_token, access_token, expires_at, label, trigger_kind, seen, project_id,
last_uid, last_sync_at, created_at
)
SELECT
lower(hex(randomblob(16))),
s.user_id,
'gmail',
COALESCE(json_extract(s.values_json, '$.email'), 'Gmail'),
'',
COALESCE(json_extract(s.values_json, '$.email'), ''),
'',
'INBOX',
COALESCE(json_extract(s.values_json, '$.refresh_token'), ''),
COALESCE(json_extract(s.values_json, '$.access_token'), ''),
COALESCE(json_extract(s.values_json, '$.expires_at'), 0),
COALESCE(json_extract(s.values_json, '$.label'), ''),
COALESCE(json_extract(s.values_json, '$.trigger'), 'starred'),
COALESCE(json_extract(s.values_json, '$.seen'), '[]'),
COALESCE(json_extract(s.values_json, '$.project_id'), ''),
0,
0,
unixepoch()
FROM user_settings s
WHERE s.scope = 'gmail'
AND COALESCE(json_extract(s.values_json, '$.refresh_token'), '') <> '';
-- Emptied, not deleted: the row may hold nothing else today, but the table is
-- shared and a DELETE here would be a habit worth not forming.
UPDATE user_settings SET values_json = '{}' WHERE scope = 'gmail';
-38
View File
@@ -272,44 +272,6 @@ func (db *DB) UserByMailToken(ctx context.Context, token string) (*User, error)
return db.scanUser(ctx, `SELECT `+userColumns+` FROM users WHERE mail_token = ?`, token)
}
// UsersWithGmail returns everybody who has connected a mailbox, which is who the
// poller has to visit. Selected by the presence of a refresh token rather than by a
// flag, so a disconnected account drops out of the sweep the moment its tokens are
// cleared.
func (db *DB) UsersWithGmail(ctx context.Context) ([]User, error) {
rows, err := db.QueryContext(ctx, `
SELECT u.id FROM users u
JOIN user_settings s ON s.user_id = u.id
WHERE s.scope = 'gmail' AND s.values_json LIKE '%"refresh_token":"%'`)
if err != nil {
return nil, err
}
var ids []string
for rows.Next() {
var id string
if err := rows.Scan(&id); err != nil {
rows.Close()
return nil, err
}
ids = append(ids, id)
}
err = rows.Err()
rows.Close()
if err != nil {
return nil, err
}
out := make([]User, 0, len(ids))
for _, id := range ids {
user, err := db.UserByID(ctx, id)
if err != nil {
continue
}
out = append(out, *user)
}
return out, nil
}
// --- per-user settings ------------------------------------------------------------------
// UserSettings reads one scope of a person's integration settings. A missing row is
+7 -6
View File
@@ -84,22 +84,23 @@ func TestATokenWrittenBeforeTheKeyStillWorks(t *testing.T) {
}
}
// UsersWithGmail finds people by the presence of the key, not its value, so it
// must keep working now that the value is ciphertext.
// The sweep finds people by the presence of a mailbox, so it must keep working
// now that the credential in it is ciphertext.
func TestTheSweepStillFindsAConnectedMailbox(t *testing.T) {
db, userID := sealedStore(t)
ctx := context.Background()
if err := db.SetUserSettings(ctx, userID, "gmail",
map[string]any{"refresh_token": "1//0-token"}); err != nil {
if err := db.SaveMailbox(ctx, &Mailbox{
UserID: userID, Kind: "gmail", RefreshToken: "1//0-token",
}); err != nil {
t.Fatal(err)
}
users, err := db.UsersWithGmail(ctx)
users, err := db.UsersWithMailboxes(ctx)
if err != nil {
t.Fatal(err)
}
if len(users) != 1 {
t.Fatalf("the sweep found %d connected mailboxes, want 1", len(users))
t.Fatalf("the sweep found %d people with mailboxes, want 1", len(users))
}
}
+32
View File
@@ -1814,3 +1814,35 @@ test('man kan skrive hvor som helst, og feltet fortæller hvad det forstår', as
expect(trouble).toEqual([]);
});
test('en postkasse kan tilføjes, og værtens afvisning når frem til skærmen', async ({ page }) => {
const trouble = watchForTrouble(page);
await page.goto('/indstillinger/integrationer');
await expect(page.getByRole('heading', { name: 'Postkasser' })).toBeVisible();
await page.getByRole('button', { name: 'Tilføj en postkasse' }).click();
// Nothing is listening there, so the dial fails at once. The point is not the
// failure but where it is reported: the server tests the connection before it
// saves, so this is refused at the door rather than accepted and then silently
// failing every ten minutes.
await page.getByLabel('IMAP-server').fill('127.0.0.1:1');
await page.getByLabel('Brugernavn').fill('kw');
await page.getByLabel('App-kodeord').fill('hemmelig');
// `exact`, because "Forbind" is a prefix of "Forbind Gmail" one panel up and
// Playwright matches an accessible name by substring unless told otherwise.
await page.getByRole('button', { name: 'Forbind', exact: true }).click();
// The host's own words, not "noget gik galt". This is the whole reason
// upstream refusals are answered as 422: a proxy may replace the body of a 5xx,
// and then the person is told nothing.
await expect(page.getByText(/Postkassen svarede/)).toBeVisible();
// And nothing was saved on the way past.
await page.reload();
await expect(page.getByText('127.0.0.1:1')).toBeHidden();
// The 422 above is the point of the test, not a symptom. Everything else the
// watcher caught still has to be empty.
expect(trouble.filter((t) => !t.includes('/mailboxes → 422'))).toEqual([]);
});
+13
View File
@@ -58,6 +58,9 @@ const MESSAGES = {
// The fallback for when Gmail refused and said nothing usable. When it did say
// something, `humanMessage` quotes it instead — see below.
gmail_failed: 'error.gmailRefused',
// Same shape for a mailbox read over IMAP: quoted when the host said something,
// this when it did not.
mailbox_failed: 'error.mailboxRefused',
last_admin: 'error.lastAdmin'
};
@@ -75,6 +78,9 @@ export function humanMessage(err) {
// authentication scopes" — and a generic sentence in their place throws away
// the only thing that says what to do. Wrapped so it reads as a quotation
// rather than as a stray English string in a Danish interface.
if (err.code === 'mailbox_failed' && err.message) {
return t('error.mailboxSaid', { what: err.message });
}
if (err.code === 'gmail_failed' && err.message) {
return t('error.gmailSaid', { what: err.message });
}
@@ -354,6 +360,13 @@ export const api = {
authorizeGmail: () => post('/gmail/authorize'),
syncGmail: () => post('/gmail/sync'),
// Mailboxes read over IMAP. Each belongs to the person who connected it, so
// there is no instance-wide registration behind these the way Gmail has one.
mailboxes: () => get('/mailboxes'),
addMailbox: (body) => post('/mailboxes', body),
deleteMailbox: (id) => del(`/mailboxes/${id}`),
syncMailbox: (id) => post(`/mailboxes/${id}/sync`),
getAISettings: () => get('/ai/settings'),
setAISettings: (data) => put('/ai/settings', data),
aiSummary: () => post('/ai/summary'),
+18
View File
@@ -459,6 +459,22 @@ export const da = {
'detail.attachFile': 'Vedhæft en fil',
// --- settings: integrations --------------------------------------------------------
'int.mailboxes': 'Postkasser',
'int.mailboxesHint':
'Flag en mail i iCloud, Fastmail eller enhver anden udbyder, og den bliver en opgave. Én vej: fjerner du flaget, sker der ikke noget med opgaven. Office 365 kan ikke bruges her — Microsoft har lukket for kodeord på IMAP.',
'int.addMailbox': 'Tilføj en postkasse',
'int.host': 'IMAP-server',
'int.username': 'Brugernavn',
'int.appPassword': 'App-kodeord',
'int.appPasswordHint':
'Ikke dit almindelige kodeord. iCloud og de fleste andre laver et særskilt et til programmer, som du kan trække tilbage alene. Det gemmes krypteret.',
'int.connect': 'Forbind',
'int.connecting': 'Forbinder …',
'int.disconnect': 'Kobl fra',
'int.disconnectMailbox': 'Kobl {name} fra? Opgaverne, den har lavet, bliver.',
'int.mailboxConnected': 'Postkassen er forbundet.',
'int.fetchNow': 'Hent nu',
'int.fetching': 'Henter …',
'int.gmail': 'Gmail',
'int.gmailHint':
'Stjernemarkér en mail, eller giv den en etiket, og den bliver en opgave. Envejs: at fjerne stjernen gør ikke noget ved opgaven.',
@@ -663,6 +679,8 @@ export const da = {
'error.gmailSaid': 'Gmail svarede: {what}',
'error.mailboxRefused': 'Postkassen kunne ikke læses.',
'error.mailboxSaid': 'Postkassen svarede: {what}',
'error.gmailRefused': 'Gmail afviste kaldet. Prøv at forbinde postkassen igen.',
'int.newAddressQuestion': 'Ny adresse? Mail til den gamle bliver afvist.',
+18
View File
@@ -455,6 +455,22 @@ export const en = {
'detail.attachFile': 'Attach a file',
// --- settings: integrations --------------------------------------------------------
'int.mailboxes': 'Mailboxes',
'int.mailboxesHint':
'Flag an email in iCloud, Fastmail or any other provider and it becomes a task. One way: removing the flag does nothing to the task. Office 365 cannot be used here — Microsoft has closed IMAP to passwords.',
'int.addMailbox': 'Add a mailbox',
'int.host': 'IMAP server',
'int.username': 'Username',
'int.appPassword': 'App password',
'int.appPasswordHint':
'Not your ordinary password. iCloud and most others issue a separate one for applications, which you can withdraw on its own. It is stored encrypted.',
'int.connect': 'Connect',
'int.connecting': 'Connecting …',
'int.disconnect': 'Disconnect',
'int.disconnectMailbox': 'Disconnect {name}? The tasks it made will stay.',
'int.mailboxConnected': 'The mailbox is connected.',
'int.fetchNow': 'Fetch now',
'int.fetching': 'Fetching …',
'int.gmail': 'Gmail',
'int.gmailHint':
'Star an email, or give it a label, and it becomes a task. One way: removing the star does nothing to the task.',
@@ -658,6 +674,8 @@ export const en = {
'error.gmailSaid': 'Gmail said: {what}',
'error.mailboxRefused': 'The mailbox could not be read.',
'error.mailboxSaid': 'The mailbox answered: {what}',
'error.gmailRefused': 'Gmail refused the call. Try connecting the mailbox again.',
'int.newAddressQuestion': 'A new address? Mail to the old one is refused.',
@@ -182,6 +182,73 @@
app.toast(t('int.copyFailed'));
}
}
// --- mailboxes ----------------------------------------------------------------
let mailboxes = $state([]);
let addingMailbox = $state(false);
let connecting = $state(false);
let syncingId = $state(null);
let newMailbox = $state({ host: '', username: '', password: '' });
$effect(() => {
loadMailboxes();
});
async function loadMailboxes() {
try {
mailboxes = (await api.mailboxes()).mailboxes ?? [];
} catch {
// A settings page that will not render because one panel could not load
// is worse than a panel that is empty.
mailboxes = [];
}
}
async function addMailbox(event) {
event.preventDefault();
connecting = true;
try {
// The server dials before it saves, so an error here is the host's own
// refusal — a wrong app password, a host that is not listening.
await api.addMailbox({ ...newMailbox });
newMailbox = { host: '', username: '', password: '' };
addingMailbox = false;
await loadMailboxes();
app.toast(t('int.mailboxConnected'));
} catch (e) {
app.toast(humanMessage(e));
} finally {
connecting = false;
}
}
async function syncMailbox(box) {
syncingId = box.id;
try {
const result = await api.syncMailbox(box.id);
app.toast(
result?.created
? plural(result.created, 'int.gmailFetchedOne', 'int.gmailFetchedMany')
: t('int.gmailNothingNew')
);
await loadMailboxes();
} catch (e) {
app.toast(humanMessage(e));
} finally {
syncingId = null;
}
}
async function removeMailbox(box) {
if (!confirm(t('int.disconnectMailbox', { name: box.name }))) return;
try {
await api.deleteMailbox(box.id);
await loadMailboxes();
} catch (e) {
app.toast(humanMessage(e));
}
}
</script>
<section class="panel">
@@ -352,6 +419,66 @@
</div>
</section>
<!-- Mailboxes read over IMAP: iCloud, Fastmail, any ordinary host. Its own panel
rather than a second mode inside the Gmail one, because they are not the same
thing wearing different clothes — one is an app you sign in through, the other
is a password you hold. -->
<section class="panel">
<header>
<h2>{t('int.mailboxes')}</h2>
<p class="hint">{t('int.mailboxesHint')}</p>
</header>
{#if mailboxes.length}
<ul class="mailboxes">
{#each mailboxes as box (box.id)}
<li>
<div class="what">
<strong>{box.name}</strong>
<span class="hint">{box.username} · {box.host}</span>
</div>
<button class="ghost" onclick={() => syncMailbox(box)} disabled={syncingId === box.id}>
{syncingId === box.id ? t('int.fetching') : t('int.fetchNow')}
</button>
<button class="ghost remove" onclick={() => removeMailbox(box)}>
{t('int.disconnect')}
</button>
</li>
{/each}
</ul>
{/if}
{#if addingMailbox}
<form class="add-mailbox" onsubmit={addMailbox}>
<div class="field">
<label for="mb-host">{t('int.host')}</label>
<input id="mb-host" bind:value={newMailbox.host} placeholder="imap.mail.me.com:993" />
</div>
<div class="field">
<label for="mb-user">{t('int.username')}</label>
<input id="mb-user" bind:value={newMailbox.username} autocomplete="off" />
</div>
<div class="field">
<label for="mb-pass">{t('int.appPassword')}</label>
<input id="mb-pass" type="password" bind:value={newMailbox.password} autocomplete="off" />
<p class="hint">{t('int.appPasswordHint')}</p>
</div>
<div class="row">
<button type="submit" class="primary" disabled={connecting}>
{connecting ? t('int.connecting') : t('int.connect')}
</button>
<button type="button" class="ghost" onclick={() => (addingMailbox = false)}>
{t('account.cancel')}
</button>
</div>
</form>
{:else}
<div class="row">
<button class="ghost" onclick={() => (addingMailbox = true)}>{t('int.addMailbox')}</button>
</div>
{/if}
</section>
<style>
/* Folded away once it is set: it is a one-off registration, and a panel that
keeps showing its own setup instructions after setup reads as unfinished. */