verdande/NOTES-SPEC.md

# Notes, as verdande builds them

A description of one working notes system: the data model, the syntax, the search,
the rendering contract, the import and export formats, and — more useful than any
of it — the decisions that look arbitrary until you know what they cost.

It is written to be lifted into another program. Nothing here depends on Go,
SQLite or Svelte except where it says so; where an answer was forced by the tool,
that is called out so you can pick a different one on purpose rather than by
accident.

---

## Read this part first: what to ask before you build

**This document is not a drop-in. It describes a notes system that lives inside a
task app with projects, roles, a trash and a search box, and about a third of its
design is inherited from that.** Lifting it into a program with different
surroundings without deciding which parts to keep produces something that works
for a week and then fights you.

**If you are an assistant working from this file: ask these questions and get
answers before writing code.** Do not guess a default. Each one changes what you
build, not just how you build it.

### About the surroundings

1. **Does the host app already have a place to file things** — projects, folders,
   notebooks, tags? verdande deliberately gave notes no hierarchy of their own and
   filed them under the projects that already existed. If your app has nothing,
   you need an answer for where a note lives, and "a folder tree" is the expensive
   one.
2. **Is there more than one person?** Sharing, roles and "who wrote this" run
   through the whole model below. Single-user changes the schema and removes about
   half the permission surface.
3. **Is there already a trash, and does it expire?** Notes lean on one. Without
   it, deleting is either permanent or you are building a trash.
4. **Is there an existing full-text index?** The search here is FTS5-shaped. On
   Postgres it is `tsvector`, on a client-side app it is probably a scan, and the
   ranking section below changes accordingly.

### About the notes themselves

5. **Markdown, rich text, or blocks?** verdande stores Markdown and renders it.
   That decision is defended below; it is also the one that most constrains
   everything else.
6. **Is the first line the title, or is there a title field?** Both work. Deriving
   it means one less thing to keep in step and one less thing a person can leave
   blank, and it forces the rule "the first line is a heading" on the editor.
7. **What links to what?** `#project`, `[[note]]` and task references exist here
   because the app has projects and tasks. Yours may have none of them, or others.
8. **Do notes carry files?** Attachments bring content-type handling, a store, an
   authorisation path and an XSS surface. Skipping them is a legitimate answer.

### About scale and language

9. **How many notes, realistically?** Several choices below flip at around ten
   thousand: the list payload, the search fallback, rendering the whole list.
10. **Which languages?** The folding described under *Search* is Danish. Every
    language has its own version of that problem and the generic answer (strip
    diacritics) is wrong for several of them.
11. **Where does the text end up besides the screen?** Search indexes, AI
    features, exports and backups each read it. Plaintext makes all of them cheap
    and makes a stolen backup expensive — see *What is still open*.

### About integration, specifically

12. **Who owns the note's identity — you or the other system?** If notes will sync
    with something else, ids and timestamps are the whole problem, and the
    `updated_at` rule below becomes the first thing you have to change.
13. **Is this a read of somebody else's notes, or the store of record?** A copy
    needs conflict rules; a store of record needs export.
14. **What happens to a note whose project, task or link target is deleted?**
    Answered here as "the link goes dead and the note is untouched". If your host
    app cascades instead, say so before the first delete.

---

## The shape of a note

```
id           an opaque string, unique
project_id   nullable — where it is filed, or loose
title        derived, never set by a caller
body         Markdown, the whole note
created_by   nullable — survives the account being deleted
pinned       favourite; the list puts these first
archived_at  nullable timestamp — put away, not thrown away
deleted_at   nullable timestamp — the trash
created_at
updated_at
```

### Markdown in a column

Not rich text, and not a format of our own.

Export was a requirement before a line was written, and **if the stored form is
the exported form there is no conversion left to lose anything in.** A note stays
readable by any editor on the day the program is gone. It is also what makes
search and anything an AI does over notes cheap: both read text.

The cost is real and you should know it before choosing: everything the editor can
do has to survive a round trip through Markdown, and anything Markdown cannot say,
the editor must not offer. There is no place to hide a feature.

### The title is the first line

Always, and re-derived on every save — not "when it is empty".

Derived once and then left alone, a title goes stale the moment somebody rewrites
the opening of a note, and the list ends up calling a note by a name that is
nowhere in it. **The column is a cache of the body, not a field beside it**, and no
caller can set it, so every caller gets the same answer.

This forces one editor rule: a new note opens on its title, and the first line is
styled as a heading. Otherwise a note whose first line is a sentence is a note
named by its first sentence.

### Archive and trash are two different bits

The trash says *this was a mistake, and in thirty days it is gone*. The archive
says *this is finished, and I do not want to read past it.*

A twelve-hundred-note import is exactly the case where most notes are worth
keeping and almost none are worth looking at. Without an archive, the list is the
whole archive forever, and finding this week's note means reading past a decade.

Both are nullable timestamps rather than booleans: *when was this put away* answers
questions a flag cannot, and it costs the same byte.

---

## Filing: no hierarchy of its own

A note belongs to a project, or to nobody.

The reasoning is worth repeating because it is easy to talk yourself out of:
projects already exist, are already shared, already have roles, already have a
trash. **A second hierarchy to file things in is one too many**, and a note filed
under a project inherits every one of those answers instead of needing its own.

- A note with a project: everybody who can read the project can read the note; an
  editor can change it.
- A note without one: the author's, and nobody else's.

Sharing a note *is* filing it in a project. There is no access list of its own —
a second way to give somebody access is a second place to get it wrong, and the
two would disagree the day one of them changed.

---

## Links: three kinds, parsed out of the text

The body is the truth. Links are an index over it, rebuilt on every save.

| Syntax | Means | Notes |
|---|---|---|
| `#project` | a project | Same characters the task quick-add parser uses, on purpose |
| `[[a note]]` | another note, by title | The spelling every note app has settled on |
| `/opgave/<uuid>` or `task:<uuid>` | a task | What a link pasted out of the app carries |

Patterns as used here:

```
project   (?:^|\s)#([\p{L}\p{N}_-]{1,64})
note      \[\[([^\]\n]{1,200})\]\]
task      (?:/opgave/|task:)([0-9a-fA-F-]{36})
```

Three decisions inside that table:

- **`#Firma` in a note means the same project as `#Firma` in the task box.** Not a
  second kind of tag that happens to look alike. That is the whole of "tag a note
  and see it in the project".
- **The link table is torn down and rebuilt on save, not diffed.** A body is small
  and the set is small; a diff is a second description of the same fact, and it
  can disagree with the text. That is the one thing this must never do.
- **The target is not a foreign key.** A link to something deleted is a fact about
  the note. It should read as a dead link rather than vanish, and it must not
  block the delete.

The index exists so the question can be asked **backwards**: *what has been written
about this task?* Answering that by searching every note's prose would be too slow
to leave a panel open on.

---

## Search

Three layers, in the order they answer.

### 1. Full-text, ranked

FTS5 over `title`, `body` and a folded copy, with `unicode61 remove_diacritics 2`.

Ranking is bm25 with **the title weighted ten times the body**. A word in the title
is what the note is *about*; the same word once in a long body is a mention.

This matters more than it sounds. Ordering by "last touched" is invisible with ten
notes and useless with twelve hundred imported the same evening: searching "Tesla"
answered with a note about nginx first and the note actually called "Tesla Model Y"
second — which reads as a search that does not work, because from where the person
is sitting it is one.

### 2. Folding, for the language you are in

`remove_diacritics` does not touch `ø`, `æ` or `å` — Unicode considers them
letters in their own right, not accented forms. So a Dane cannot find "grøn" by
typing "gron", which is how people type when the keyboard is not theirs.

The answer here is a generated column carrying a transliterated copy:

```
ø,Ø → o     æ,Æ → ae     å,Å → aa
```

indexed alongside the body and **weighted the same as the body** — otherwise a
folded match outranks an exact one.

> **Ask, do not copy.** This mapping is Danish. German wants `ü → ue`, Swedish
> disagrees with Danish about `ä`, and Turkish has a dotless `ı` that breaks
> naive lowercasing entirely. Get the mapping from somebody who types the
> language.

### 3. A word inside a word

An index is built of tokens, and a token is whatever the tokenizer decided was a
word. `gamerule keepInventory true` is three of them, and the middle one is
`keepinventory` — so a search for "keep inventory" asks for two words that are
nowhere and gets nothing, while the note sits right there.

A prefix query gets half of it: `keep*` matches, `inventory*` cannot, and the two
are ANDed, so the half that works is thrown away by the half that cannot.

**This is not exotic in a notes app.** A decade of notes is full of commands,
identifiers and file names, and every one of them is several words the index
thinks is one.

So: the index answers first; when it has nothing, the text is read, and every term
must appear somewhere in the note, in any word. A `LIKE '%term%'` over the folded
column covers title, body and language folding at once.

What it costs: a leading `%` cannot use an index, so it scans. At twelve hundred
notes that is a few megabytes and a millisecond or two, and it only runs when the
fast path already came back empty — which is exactly the search that was about to
disappoint somebody. **At a hundred thousand notes it is the wrong shape**, and the
answer then is a trigram index, not a bigger scan.

It does not rank. There is no match to rank — that is the whole situation.

---

## The list, and why the body is empty in it

A list showing a title and ninety characters of preview must not fetch whole
notes. Twelve hundred notes came to 3.9 MB, of which 98 % was body nobody could
see.

The important half is *how* it is trimmed: **the body is emptied, not shortened.**

A field called `body` holding a *piece* of the body is a trap. The editor opens
it, the autosave pause writes it back, and the note is now cut down to its own
excerpt — destroyed by nothing more than having been looked at. Send `preview` as
its own field and leave `body` empty; the client sees the empty body and fetches
the whole note.

That costs one more separation, and it is not optional:

> **Which row is selected and which note is loaded are two different things.**

The editor loads its text when the note's id changes. Hand it the excerpt first
and the whole note second and the id is the same both times — so the second one
never renders, and somebody sits looking at a note that had a title a moment ago.

Highlight the row immediately from the list; give the editor the note only when it
is whole.

Rendering cost, separately: with twelve hundred rows, first paint took 2.1 s. CSS
`content-visibility: auto` with an intrinsic-size guess took it to 0.95 s with no
virtual list, no scroll maths, and no list that can fall out of step with itself.
Try that before building windowing.

---

## The editor contract

Rich text on top, Markdown underneath, and a source view that shows the file.

The supported subset — deliberately small, because everything in it has to survive
the round trip:

| Block | Markdown |
|---|---|
| Title | `# ` |
| Heading | `## ` |
| Subheading | `### ` |
| Quote | `> ` |
| Code block | ```` ``` ```` with an optional language |
| Bullet and numbered lists | `- `, `1. `, nested |

Inline: `**bold**`, `*italic*`, `~~strikethrough~~`, `` `code` ``, `<u>underline</u>`,
and images as `![alt](url)`.

Five rules that were each learned the hard way:

1. **Every line becomes exactly one block.** Markdown normally joins consecutive
   lines into a paragraph. Here it must not: the person typed a line and expects
   to get it back, and an editor that silently reflows what somebody wrote is an
   editor they stop trusting.
2. **Load on id change, never while typing.** Writing to the editor's HTML puts
   the caret back at the start, which mid-sentence is unusable.
3. **One definition of what a code block contains.** A copy button inside the
   `<pre>` and a syntax highlighter reading `textContent` will disagree, and the
   difference ends up in the file — a click on "Copy" wrote the word *Copied* into
   the code and the next save put it on disk. Have one function that says what a
   block holds, used by the highlighter and the Markdown serialiser both.
4. **A note that cannot be rendered must not take the pane with it.** One note
   arriving in an unexpected shape threw inside the editor's reactive effect, and
   from then on *no* note would open — the previous one just stayed on screen. Put
   a guard around the render, fall back to showing the raw text, and let the rest
   of the pane keep working.
5. **Escaping between tags and escaping inside an attribute are different jobs.**
   An escaper written for text between two tags handles `&`, `<`, `>` — and not
   quotes, because between tags they are harmless. Inside an attribute they close
   it. `![" onerror="alert(1)](…)` is the whole exploit.

---

## Attachments

Images pasted or dragged into a note, stored content-addressed by hash, referenced
from the body as ordinary Markdown images.

The security shape, which is the part worth copying:

- **Served as `application/octet-stream` with `nosniff` by default.** Attachments
  are the one place a user supplies content another user opens. An uploaded SVG or
  HTML file rendered inline runs its own script on your origin with the session
  cookie attached.
- **One exception, as a named allowlist:** a short list of raster image types
  served inline as their exact type — `png`, `jpeg`, `gif`, `webp`, `avif`, `heic`.
  A note carries its pictures in the text, and served as octet-stream every one of
  them is a broken image.
- **The list is named one type at a time, never matched on an `image/` prefix.**
  The prefix lets `image/svg+xml` through, and an SVG is a document that can run
  script.
- **The stored path is checked against the files directory again at read time.** A
  traversal that somehow got into the database must not become one that reads
  `/etc/passwd`.
- **Reaching an attachment follows the note, which follows its project.** Same
  chain as everything else. Do not let a client hand you the parent id and take it
  on trust.

---

## Import and export

The same format both ways: **a zip of Markdown files**, one per note, with a YAML
front matter block.

```markdown
---
created: 2019-04-02T10:15:00Z
modified: 2024-11-30T21:04:00Z
---

# Møde med Anders

Han vil gerne have kaffe hver uge.
```

- `created` / `created_at` / `date` and `modified` / `updated` / `updated_at` are
  all accepted on the way in. Obsidian and Bear write this block; so should you.
- Images travel in the zip beside the notes, and the links in the text are
  relative — **a note and its pictures must end up in the same archive**, which
  matters the moment an export is split into parts.
- File names collide: two notes called "Møde" are two notes, and a zip with one
  entry twice is a zip that loses one.

Limits that a real import taught, all of them cheap to add and expensive to omit:

- A ceiling on the upload, on the number of entries, and on the **unpacked** size.
  64 MB of zip expanded to 66 GB is a normal shape for an attack.
- A per-file limit as well as a whole-archive budget: one enormous photograph and
  ten thousand small ones are different attacks with the same ending.
- The entry name is never used as a path. Store by hash; the name is only a key
  for rewriting links.
- **Count what you skip, and say so.** A note whose name contained an ellipsis was
  dropped silently by a filter looking for `..` — one out of 1141, found only by
  counting at both ends.

### Timestamps on the way in

An importer needs to write `created_at` and `updated_at` from the file, and the
normal save path must refuse to let it.

`updated_at` means *when did this program last write this note*. Let any caller set
it and the column means whatever the last writer felt like. Give the importer its
own narrow entry point instead, and say in the comment why it exists.

Without those dates, a move of twelve hundred notes is a pile that was all written
the same evening — and the order they were written in is half of what an archive
is.

---

## What is still open here

Written down because a specification that only lists what works is a specification
that lies by omission.

- **Note bodies are plaintext at rest.** They hold API keys and passwords, and
  they are in every backup anybody can download. Encrypting them is bigger than it
  sounds: the body is in the full-text index and the folded column is generated
  from it, so both are a second plaintext copy in the same database. Encrypt and
  both have to go, and search has to be rewritten to decrypt in application code —
  fine at a thousand notes, a full scan per keystroke at ten thousand, at which
  point the answer is a keyed index (an HMAC per word).
- **Import never merges.** Every run creates new notes. Say so out loud in the UI.
- **There is no sync.** Export and the API are the way out.

---

## A short list of the traps, for the person in a hurry

1. A `body` field holding an excerpt will be saved over the real one.
2. Selected row and loaded note are two different pieces of state.
3. The title must be re-derived on every save or it goes stale.
4. Diacritic stripping is not folding, and folding is language-specific.
5. A tokenizer cannot see a word inside a word; decide what you do about it.
6. An attachment served inline is a script on your origin unless you allowlist by
   exact type and send `nosniff`.
7. Escaping for text and escaping for attributes are different functions.
8. One render failure must not be allowed to freeze the whole pane.
9. An unpacked-size limit is not optional on an archive import.
10. Let the importer set timestamps; let nobody else.