diff --git a/NOTES-SPEC.md b/NOTES-SPEC.md new file mode 100644 index 0000000..db56136 --- /dev/null +++ b/NOTES-SPEC.md @@ -0,0 +1,437 @@ +# Notes, as verdande builds them + +A description of one working notes system: the data model, the syntax, the search, +the rendering contract, the import and export formats, and — more useful than any +of it — the decisions that look arbitrary until you know what they cost. + +It is written to be lifted into another program. Nothing here depends on Go, +SQLite or Svelte except where it says so; where an answer was forced by the tool, +that is called out so you can pick a different one on purpose rather than by +accident. + +--- + +## Read this part first: what to ask before you build + +**This document is not a drop-in. It describes a notes system that lives inside a +task app with projects, roles, a trash and a search box, and about a third of its +design is inherited from that.** Lifting it into a program with different +surroundings without deciding which parts to keep produces something that works +for a week and then fights you. + +**If you are an assistant working from this file: ask these questions and get +answers before writing code.** Do not guess a default. Each one changes what you +build, not just how you build it. + +### About the surroundings + +1. **Does the host app already have a place to file things** — projects, folders, + notebooks, tags? verdande deliberately gave notes no hierarchy of their own and + filed them under the projects that already existed. If your app has nothing, + you need an answer for where a note lives, and "a folder tree" is the expensive + one. +2. **Is there more than one person?** Sharing, roles and "who wrote this" run + through the whole model below. Single-user changes the schema and removes about + half the permission surface. +3. **Is there already a trash, and does it expire?** Notes lean on one. Without + it, deleting is either permanent or you are building a trash. +4. **Is there an existing full-text index?** The search here is FTS5-shaped. On + Postgres it is `tsvector`, on a client-side app it is probably a scan, and the + ranking section below changes accordingly. + +### About the notes themselves + +5. **Markdown, rich text, or blocks?** verdande stores Markdown and renders it. + That decision is defended below; it is also the one that most constrains + everything else. +6. **Is the first line the title, or is there a title field?** Both work. Deriving + it means one less thing to keep in step and one less thing a person can leave + blank, and it forces the rule "the first line is a heading" on the editor. +7. **What links to what?** `#project`, `[[note]]` and task references exist here + because the app has projects and tasks. Yours may have none of them, or others. +8. **Do notes carry files?** Attachments bring content-type handling, a store, an + authorisation path and an XSS surface. Skipping them is a legitimate answer. + +### About scale and language + +9. **How many notes, realistically?** Several choices below flip at around ten + thousand: the list payload, the search fallback, rendering the whole list. +10. **Which languages?** The folding described under *Search* is Danish. Every + language has its own version of that problem and the generic answer (strip + diacritics) is wrong for several of them. +11. **Where does the text end up besides the screen?** Search indexes, AI + features, exports and backups each read it. Plaintext makes all of them cheap + and makes a stolen backup expensive — see *What is still open*. + +### About integration, specifically + +12. **Who owns the note's identity — you or the other system?** If notes will sync + with something else, ids and timestamps are the whole problem, and the + `updated_at` rule below becomes the first thing you have to change. +13. **Is this a read of somebody else's notes, or the store of record?** A copy + needs conflict rules; a store of record needs export. +14. **What happens to a note whose project, task or link target is deleted?** + Answered here as "the link goes dead and the note is untouched". If your host + app cascades instead, say so before the first delete. + +--- + +## The shape of a note + +``` +id an opaque string, unique +project_id nullable — where it is filed, or loose +title derived, never set by a caller +body Markdown, the whole note +created_by nullable — survives the account being deleted +pinned favourite; the list puts these first +archived_at nullable timestamp — put away, not thrown away +deleted_at nullable timestamp — the trash +created_at +updated_at +``` + +### Markdown in a column + +Not rich text, and not a format of our own. + +Export was a requirement before a line was written, and **if the stored form is +the exported form there is no conversion left to lose anything in.** A note stays +readable by any editor on the day the program is gone. It is also what makes +search and anything an AI does over notes cheap: both read text. + +The cost is real and you should know it before choosing: everything the editor can +do has to survive a round trip through Markdown, and anything Markdown cannot say, +the editor must not offer. There is no place to hide a feature. + +### The title is the first line + +Always, and re-derived on every save — not "when it is empty". + +Derived once and then left alone, a title goes stale the moment somebody rewrites +the opening of a note, and the list ends up calling a note by a name that is +nowhere in it. **The column is a cache of the body, not a field beside it**, and no +caller can set it, so every caller gets the same answer. + +This forces one editor rule: a new note opens on its title, and the first line is +styled as a heading. Otherwise a note whose first line is a sentence is a note +named by its first sentence. + +### Archive and trash are two different bits + +The trash says *this was a mistake, and in thirty days it is gone*. The archive +says *this is finished, and I do not want to read past it.* + +A twelve-hundred-note import is exactly the case where most notes are worth +keeping and almost none are worth looking at. Without an archive, the list is the +whole archive forever, and finding this week's note means reading past a decade. + +Both are nullable timestamps rather than booleans: *when was this put away* answers +questions a flag cannot, and it costs the same byte. + +--- + +## Filing: no hierarchy of its own + +A note belongs to a project, or to nobody. + +The reasoning is worth repeating because it is easy to talk yourself out of: +projects already exist, are already shared, already have roles, already have a +trash. **A second hierarchy to file things in is one too many**, and a note filed +under a project inherits every one of those answers instead of needing its own. + +- A note with a project: everybody who can read the project can read the note; an + editor can change it. +- A note without one: the author's, and nobody else's. + +Sharing a note *is* filing it in a project. There is no access list of its own — +a second way to give somebody access is a second place to get it wrong, and the +two would disagree the day one of them changed. + +--- + +## Links: three kinds, parsed out of the text + +The body is the truth. Links are an index over it, rebuilt on every save. + +| Syntax | Means | Notes | +|---|---|---| +| `#project` | a project | Same characters the task quick-add parser uses, on purpose | +| `[[a note]]` | another note, by title | The spelling every note app has settled on | +| `/opgave/` or `task:` | a task | What a link pasted out of the app carries | + +Patterns as used here: + +``` +project (?:^|\s)#([\p{L}\p{N}_-]{1,64}) +note \[\[([^\]\n]{1,200})\]\] +task (?:/opgave/|task:)([0-9a-fA-F-]{36}) +``` + +Three decisions inside that table: + +- **`#Firma` in a note means the same project as `#Firma` in the task box.** Not a + second kind of tag that happens to look alike. That is the whole of "tag a note + and see it in the project". +- **The link table is torn down and rebuilt on save, not diffed.** A body is small + and the set is small; a diff is a second description of the same fact, and it + can disagree with the text. That is the one thing this must never do. +- **The target is not a foreign key.** A link to something deleted is a fact about + the note. It should read as a dead link rather than vanish, and it must not + block the delete. + +The index exists so the question can be asked **backwards**: *what has been written +about this task?* Answering that by searching every note's prose would be too slow +to leave a panel open on. + +--- + +## Search + +Three layers, in the order they answer. + +### 1. Full-text, ranked + +FTS5 over `title`, `body` and a folded copy, with `unicode61 remove_diacritics 2`. + +Ranking is bm25 with **the title weighted ten times the body**. A word in the title +is what the note is *about*; the same word once in a long body is a mention. + +This matters more than it sounds. Ordering by "last touched" is invisible with ten +notes and useless with twelve hundred imported the same evening: searching "Tesla" +answered with a note about nginx first and the note actually called "Tesla Model Y" +second — which reads as a search that does not work, because from where the person +is sitting it is one. + +### 2. Folding, for the language you are in + +`remove_diacritics` does not touch `ø`, `æ` or `å` — Unicode considers them +letters in their own right, not accented forms. So a Dane cannot find "grøn" by +typing "gron", which is how people type when the keyboard is not theirs. + +The answer here is a generated column carrying a transliterated copy: + +``` +ø,Ø → o æ,Æ → ae å,Å → aa +``` + +indexed alongside the body and **weighted the same as the body** — otherwise a +folded match outranks an exact one. + +> **Ask, do not copy.** This mapping is Danish. German wants `ü → ue`, Swedish +> disagrees with Danish about `ä`, and Turkish has a dotless `ı` that breaks +> naive lowercasing entirely. Get the mapping from somebody who types the +> language. + +### 3. A word inside a word + +An index is built of tokens, and a token is whatever the tokenizer decided was a +word. `gamerule keepInventory true` is three of them, and the middle one is +`keepinventory` — so a search for "keep inventory" asks for two words that are +nowhere and gets nothing, while the note sits right there. + +A prefix query gets half of it: `keep*` matches, `inventory*` cannot, and the two +are ANDed, so the half that works is thrown away by the half that cannot. + +**This is not exotic in a notes app.** A decade of notes is full of commands, +identifiers and file names, and every one of them is several words the index +thinks is one. + +So: the index answers first; when it has nothing, the text is read, and every term +must appear somewhere in the note, in any word. A `LIKE '%term%'` over the folded +column covers title, body and language folding at once. + +What it costs: a leading `%` cannot use an index, so it scans. At twelve hundred +notes that is a few megabytes and a millisecond or two, and it only runs when the +fast path already came back empty — which is exactly the search that was about to +disappoint somebody. **At a hundred thousand notes it is the wrong shape**, and the +answer then is a trigram index, not a bigger scan. + +It does not rank. There is no match to rank — that is the whole situation. + +--- + +## The list, and why the body is empty in it + +A list showing a title and ninety characters of preview must not fetch whole +notes. Twelve hundred notes came to 3.9 MB, of which 98 % was body nobody could +see. + +The important half is *how* it is trimmed: **the body is emptied, not shortened.** + +A field called `body` holding a *piece* of the body is a trap. The editor opens +it, the autosave pause writes it back, and the note is now cut down to its own +excerpt — destroyed by nothing more than having been looked at. Send `preview` as +its own field and leave `body` empty; the client sees the empty body and fetches +the whole note. + +That costs one more separation, and it is not optional: + +> **Which row is selected and which note is loaded are two different things.** + +The editor loads its text when the note's id changes. Hand it the excerpt first +and the whole note second and the id is the same both times — so the second one +never renders, and somebody sits looking at a note that had a title a moment ago. + +Highlight the row immediately from the list; give the editor the note only when it +is whole. + +Rendering cost, separately: with twelve hundred rows, first paint took 2.1 s. CSS +`content-visibility: auto` with an intrinsic-size guess took it to 0.95 s with no +virtual list, no scroll maths, and no list that can fall out of step with itself. +Try that before building windowing. + +--- + +## The editor contract + +Rich text on top, Markdown underneath, and a source view that shows the file. + +The supported subset — deliberately small, because everything in it has to survive +the round trip: + +| Block | Markdown | +|---|---| +| Title | `# ` | +| Heading | `## ` | +| Subheading | `### ` | +| Quote | `> ` | +| Code block | ```` ``` ```` with an optional language | +| Bullet and numbered lists | `- `, `1. `, nested | + +Inline: `**bold**`, `*italic*`, `~~strikethrough~~`, `` `code` ``, `underline`, +and images as `![alt](url)`. + +Five rules that were each learned the hard way: + +1. **Every line becomes exactly one block.** Markdown normally joins consecutive + lines into a paragraph. Here it must not: the person typed a line and expects + to get it back, and an editor that silently reflows what somebody wrote is an + editor they stop trusting. +2. **Load on id change, never while typing.** Writing to the editor's HTML puts + the caret back at the start, which mid-sentence is unusable. +3. **One definition of what a code block contains.** A copy button inside the + `
` and a syntax highlighter reading `textContent` will disagree, and the
+   difference ends up in the file — a click on "Copy" wrote the word *Copied* into
+   the code and the next save put it on disk. Have one function that says what a
+   block holds, used by the highlighter and the Markdown serialiser both.
+4. **A note that cannot be rendered must not take the pane with it.** One note
+   arriving in an unexpected shape threw inside the editor's reactive effect, and
+   from then on *no* note would open — the previous one just stayed on screen. Put
+   a guard around the render, fall back to showing the raw text, and let the rest
+   of the pane keep working.
+5. **Escaping between tags and escaping inside an attribute are different jobs.**
+   An escaper written for text between two tags handles `&`, `<`, `>` — and not
+   quotes, because between tags they are harmless. Inside an attribute they close
+   it. `![" onerror="alert(1)](…)` is the whole exploit.
+
+---
+
+## Attachments
+
+Images pasted or dragged into a note, stored content-addressed by hash, referenced
+from the body as ordinary Markdown images.
+
+The security shape, which is the part worth copying:
+
+- **Served as `application/octet-stream` with `nosniff` by default.** Attachments
+  are the one place a user supplies content another user opens. An uploaded SVG or
+  HTML file rendered inline runs its own script on your origin with the session
+  cookie attached.
+- **One exception, as a named allowlist:** a short list of raster image types
+  served inline as their exact type — `png`, `jpeg`, `gif`, `webp`, `avif`, `heic`.
+  A note carries its pictures in the text, and served as octet-stream every one of
+  them is a broken image.
+- **The list is named one type at a time, never matched on an `image/` prefix.**
+  The prefix lets `image/svg+xml` through, and an SVG is a document that can run
+  script.
+- **The stored path is checked against the files directory again at read time.** A
+  traversal that somehow got into the database must not become one that reads
+  `/etc/passwd`.
+- **Reaching an attachment follows the note, which follows its project.** Same
+  chain as everything else. Do not let a client hand you the parent id and take it
+  on trust.
+
+---
+
+## Import and export
+
+The same format both ways: **a zip of Markdown files**, one per note, with a YAML
+front matter block.
+
+```markdown
+---
+created: 2019-04-02T10:15:00Z
+modified: 2024-11-30T21:04:00Z
+---
+
+# Møde med Anders
+
+Han vil gerne have kaffe hver uge.
+```
+
+- `created` / `created_at` / `date` and `modified` / `updated` / `updated_at` are
+  all accepted on the way in. Obsidian and Bear write this block; so should you.
+- Images travel in the zip beside the notes, and the links in the text are
+  relative — **a note and its pictures must end up in the same archive**, which
+  matters the moment an export is split into parts.
+- File names collide: two notes called "Møde" are two notes, and a zip with one
+  entry twice is a zip that loses one.
+
+Limits that a real import taught, all of them cheap to add and expensive to omit:
+
+- A ceiling on the upload, on the number of entries, and on the **unpacked** size.
+  64 MB of zip expanded to 66 GB is a normal shape for an attack.
+- A per-file limit as well as a whole-archive budget: one enormous photograph and
+  ten thousand small ones are different attacks with the same ending.
+- The entry name is never used as a path. Store by hash; the name is only a key
+  for rewriting links.
+- **Count what you skip, and say so.** A note whose name contained an ellipsis was
+  dropped silently by a filter looking for `..` — one out of 1141, found only by
+  counting at both ends.
+
+### Timestamps on the way in
+
+An importer needs to write `created_at` and `updated_at` from the file, and the
+normal save path must refuse to let it.
+
+`updated_at` means *when did this program last write this note*. Let any caller set
+it and the column means whatever the last writer felt like. Give the importer its
+own narrow entry point instead, and say in the comment why it exists.
+
+Without those dates, a move of twelve hundred notes is a pile that was all written
+the same evening — and the order they were written in is half of what an archive
+is.
+
+---
+
+## What is still open here
+
+Written down because a specification that only lists what works is a specification
+that lies by omission.
+
+- **Note bodies are plaintext at rest.** They hold API keys and passwords, and
+  they are in every backup anybody can download. Encrypting them is bigger than it
+  sounds: the body is in the full-text index and the folded column is generated
+  from it, so both are a second plaintext copy in the same database. Encrypt and
+  both have to go, and search has to be rewritten to decrypt in application code —
+  fine at a thousand notes, a full scan per keystroke at ten thousand, at which
+  point the answer is a keyed index (an HMAC per word).
+- **Import never merges.** Every run creates new notes. Say so out loud in the UI.
+- **There is no sync.** Export and the API are the way out.
+
+---
+
+## A short list of the traps, for the person in a hurry
+
+1. A `body` field holding an excerpt will be saved over the real one.
+2. Selected row and loaded note are two different pieces of state.
+3. The title must be re-derived on every save or it goes stale.
+4. Diacritic stripping is not folding, and folding is language-specific.
+5. A tokenizer cannot see a word inside a word; decide what you do about it.
+6. An attachment served inline is a script on your origin unless you allowlist by
+   exact type and send `nosniff`.
+7. Escaping for text and escaping for attributes are different functions.
+8. One render failure must not be allowed to freeze the whole pane.
+9. An unpacked-size limit is not optional on an archive import.
+10. Let the importer set timestamps; let nobody else.