MAGI/docs/DOCUMENTATION.md

# Guildlib v2 — Complete Documentation

---

## Table of contents

1. [How it works — overview](#1-how-it-works)
2. [Unity attributes — complete reference](#2-unity-attributes)
3. [Creating your own shards and types](#3-creating-shards-and-types)
4. [Platform and tag system](#4-platform-and-tag-system)
5. [The sync cycle — step by step](#5-the-sync-cycle)
6. [Server API reference](#6-server-api-reference)
7. [Web editor guide](#7-web-editor-guide)
8. [Local testing (Windows / PowerShell)](#8-local-testing)
9. [Docker deployment](#9-docker-deployment)
10. [Changing things — common tasks](#10-changing-things)

---

## 1. How it works

```
Unity C# classes
      │
      │  [ShardTarget], [GuildLabel], [GuildPlatforms] …
      │  (attributes on your classes describe everything)
      ▼
SchemaExporter.Export()
      │
      │  produces schema.json
      ▼
POST /schema  ──────────────────────────► Bun server
                                                │
                                                │  stores schema.json
                                                │  auto-creates .sqlite per shard
                                                ▼
Web editor reads schema.json and renders type-safe forms
Users create / edit entries in the browser
Changes stored in weapons.sqlite, audio.sqlite, etc.

Unity Sync Panel clicks "Pull All Shards":
  1. GET /manifest/weapons  →  [{id, version, checksum}]  (lightweight!)
  2. diff against local cache
  3. POST /pull  { ids: [only what changed] }
  4. receive only changed rows
  5. upsert into local weapons.sqlite
```

Key design decisions inherited from the Perforce paper:
- **Manifest-first**: only checksums cross the wire on the first pass.
  No data is transferred if nothing changed.
- **Shard independence**: each shard is a completely separate .sqlite file.
  You can pull just `audio` without touching `weapons`.
- **Server-side filtering**: platform and tag filters are applied on the server
  before rows are returned, so a Switch build only downloads Switch data.

---

## 2. Unity attributes

All attributes live in `Guildlib.Runtime`. Import the namespace at the top of
your C# files.

### `[ShardTarget("name")]`  ← **REQUIRED**

Declares which shard database this type belongs to.

```csharp
[ShardTarget("characters")]
public class CharacterStats : DataEntry { ... }

[ShardTarget("ui")]
public class UIButtonConfig : DataEntry { ... }

[ShardTarget("vfx_particles")]    // underscores OK, lowercase
public class ParticlePreset : DataEntry { ... }
```

Rules:
- Lowercase, no spaces, no special characters except underscore
- The server auto-creates a new `<name>.sqlite` file the first time an entry
  of that type is pushed — you do not need to pre-register shards anywhere
- If two classes share a `[ShardTarget]` they live in the same database file
  (the shard is a file, not a type)

---

### `[GuildLabel("Display Name")]`  ← optional

Sets the human-readable name shown in the web editor's type list.
Defaults to the C# class name if omitted.

```csharp
[ShardTarget("characters")]
[GuildLabel("Character Stats")]
public class CharacterStats : DataEntry { ... }
// Shows as "Character Stats" in the web editor, not "CharacterStats"
```

---

### `[GuildDescription("...")]`  ← optional

A one-sentence description shown in the web editor's form panel under the type name.
Use it to explain what this entry type is for.

```csharp
[GuildDescription("Defines base stats for all playable and enemy characters.")]
public class CharacterStats : DataEntry { ... }
```

---

### `[GuildPlatforms("p1", "p2", ...)]`  ← optional

Sets the **default** platform tags for newly created entries of this type.
Can be overridden per-entry in the web editor.

```csharp
// This entry type ships on all platforms by default
[GuildPlatforms("all")]
public class CharacterStats : DataEntry { ... }

// PC and console only — Switch gets different data
[GuildPlatforms("pc", "ps5", "xbox_series")]
public class HighResTextureConfig : DataEntry { ... }

// Switch-specific preset
[GuildPlatforms("switch", "mobile_ios", "mobile_android")]
public class LowResTextureConfig : DataEntry { ... }
```

If omitted, defaults to `["all"]` — meaning every platform gets this data.

---

### `[GuildTags("t1", "t2", ...)]`  ← optional

Sets the **default** content/build tags for new entries of this type.
Can be overridden per-entry in the web editor.

```csharp
[GuildTags("base_game", "release")]
public class CharacterStats : DataEntry { ... }

[GuildTags("dlc_1")]
public class DLC1Character : CharacterStats { ... }

[GuildTags("debug")]
public class DebugTestCharacter : CharacterStats { ... }
```

---

### `[GuildDisplayName]`  ← optional, on a field

Marks one field as the "display name" for the entry list in the web editor.
Without this, the entry ID is shown. Put it on whichever field is most useful
as a human-readable label.

```csharp
public class CharacterStats : DataEntry
{
    [GuildDisplayName]
    public string characterName;    // web editor shows this in the list

    public float maxHealth;
}
```

---

### `[GuildEnum("A", "B", "C")]`  ← optional, on a string field

Constrains a string field to a fixed set of options.
The web editor renders a dropdown instead of a plain text input.

```csharp
[GuildEnum("Warrior", "Mage", "Rogue", "Ranger")]
public string characterClass;

[GuildEnum("Common", "Uncommon", "Rare", "Epic", "Legendary")]
public string rarity;

[GuildEnum("Fire", "Ice", "Lightning", "Physical", "Poison")]
public string damageType;
```

---

### `[GuildRef("shard_name")]`  ← optional, on a string field

Marks a string field as a foreign key reference to an entry in another shard.
The web editor shows an annotated hint (full picker UI is a planned enhancement).
The actual stored value is still the entryId string.

```csharp
public class CharacterStats : DataEntry
{
    [GuildRef("audio")]
    public string deathSoundId;     // points to an entry in audio.sqlite

    [GuildRef("models")]
    public string modelId;           // points to an entry in models.sqlite
}
```

---

### `[GuildMultiline]`  ← optional, on a string field

Makes the web editor render a textarea instead of a single-line input.
Use for long descriptions or multi-line text content.

```csharp
[GuildMultiline]
public string loreText;

[GuildMultiline]
public string debugNotes;
```

---

### `[GuildExclude]`  ← optional, on a field

Excludes a field from schema export and sync entirely.
Use for Unity-only fields (Texture2D refs, EditorGUILayout state, etc.)
that cannot be serialised to the database.

```csharp
[GuildExclude]
public Texture2D previewTexture;    // Unity asset ref — never synced

[GuildExclude]
public bool editorFoldout;          // editor UI state — never synced
```

---

## 3. Creating shards and types

### Step 1 — Write a C# class

Create a new .cs file anywhere under `Assets/` in your Unity project.

```csharp
using UnityEngine;
using Guildlib.Runtime;

// ── A brand new shard called "abilities" ─────────────────────────────────────

[ShardTarget("abilities")]
[GuildLabel("Ability Definition")]
[GuildDescription("Defines a single player or enemy ability.")]
[GuildPlatforms("all")]
[GuildTags("base_game", "release")]
[CreateAssetMenu(menuName = "Guildlib/Ability")]
public class AbilityDefinition : DataEntry
{
    [GuildDisplayName]
    public string abilityName;

    [GuildMultiline]
    public string description;

    [GuildEnum("Active", "Passive", "Toggle")]
    public string abilityType;

    public float  cooldown;
    public float  manaCost;
    public int    maxLevel;

    [GuildRef("audio")]
    public string castSoundId;

    [GuildRef("vfx_particles")]
    public string castVFXId;
}

// ── A child type — inherits all AbilityDefinition fields ─────────────────────

[ShardTarget("abilities")]
[GuildLabel("Area Ability")]
[GuildDescription("An ability that affects an area rather than a single target.")]
public class AreaAbility : AbilityDefinition
{
    public float  radius;
    public bool   falloffDamage;

    [GuildEnum("Circle", "Cone", "Rectangle")]
    public string areaShape;
}
```

### Step 2 — Upload the schema

Open **Window → Guildlib → Sync Panel**, then click **Upload Schema**.

This sends all your C# type definitions to the server. The server stores
the schema as `config/schema.json`. The web editor reads this file to know
what types exist, what their fields are, and what options are available.

The `abilities` shard now exists on the server. Its `.sqlite` file is created
automatically the first time an entry is pushed.

### Step 3 — Use in the web editor

Refresh the web editor. You will see the `abilities` shard appear in the left
column. Click it to see `Ability Definition` and `Area Ability` in the type tree.

### Inheritance rules

- A child type appears nested under its parent in the web editor
- The form shows inherited fields first (marked "inherited"), then own fields
- Child types go into the **same shard** as their parent
- You can nest arbitrarily deep: `DataEntry → Ability → AreaAbility → AOEBomb`
- Multiple children of the same parent are all shown in the tree

---

## 4. Platform and tag system

### What they are

**Platforms** describe which shipping targets an entry is relevant for.
**Tags** describe which content group, build configuration, or DLC an entry belongs to.

Both are stored per-entry (not per-type) and can be edited in the web editor.
Type-level `[GuildPlatforms]` and `[GuildTags]` attributes only set the default
for newly created entries — users can change them freely in the web editor.

### Defining your vocabulary

Both lists live in `DataEntry.cs`:

```csharp
public static class GuildPlatformRegistry
{
    public static readonly string[] All = new[]
    {
        "pc",
        "ps5",
        "ps4",
        "xbox_series",
        "xbox_one",
        "switch",
        "mobile_ios",
        "mobile_android",
        "vr",
    };
}

public static class GuildTagRegistry
{
    public static readonly string[] All = new[]
    {
        "release",
        "debug",
        "demo",
        "base_game",
        "dlc_1",
        "dlc_2",
        "high",
        "medium",
        "low",
        "global",
        "region_jp",
        "region_eu",
        "region_na",
    };
}
```

**To add a platform or tag**: add a string to the appropriate array and run
**Upload Schema** from the Sync Panel. The web editor will show the new option
immediately.

**To remove one**: remove it from the array and re-upload the schema. Existing
entries that had the old value keep it stored in the database — the web editor
won't show the removed value as a selectable checkbox, but the data is preserved.

### The special "all" platform

An entry tagged `"all"` is returned for every platform filter. This means if
you add a new platform later, entries tagged "all" are automatically included
without any data migration.

### Using platform/tag filters in Unity

In the Sync Panel:

1. Toggle on **Filter by platform**
2. Enter comma-separated platforms: `pc,ps5`
3. Toggle on **Filter by tag** (optional)
4. Enter tags: `release,base_game`
5. Click **Pull All Shards**

Only rows matching the filters come down the wire.

Programmatically from C#:

```csharp
var client = new ShardClient(config, "abilities");

// Pull only PC/PS5 release content
await client.PullAsync(
    platformFilter: new List<string> { "pc", "ps5" },
    tagFilter:      new List<string> { "release", "base_game" }
);

// Pull everything (no filter)
await client.PullAsync();
```

### Using filters on the server API directly

```
GET /entries/abilities?platforms=pc,ps5&tags=release,base_game
POST /pull  { "shard":"abilities", "ids":[...], "filterPlatforms":["switch"], "filterTags":["base_game"] }
```

---

## 5. The sync cycle

### Full cycle (first time or clean slate)

```
1. GET /manifest/weapons
   ← { shard:"weapons", rows: [] }        ← empty, nothing on server yet

2. (nothing to diff — skip pull)

3. (user creates entries via web editor)

4. GET /manifest/weapons
   ← { rows: [{id:"abc", version:1, checksum:"d41d..."}, ...] }

5. Diff: local has nothing → need all IDs

6. POST /pull  { shard:"weapons", ids:["abc", ...] }
   ← { rows: [{id:"abc", weaponType:"Sword", damage:45, ...}, ...] }

7. Upsert into local weapons.sqlite
8. Save manifest cache
```

### Incremental sync (normal operation)

```
1. GET /manifest/weapons
   ← 10,000 rows of {id, version, checksum}

2. Compare with local manifest:
   - 9,995 rows unchanged  → skip
   - 5 rows have newer version → add to needIds

3. POST /pull  { ids: ["id1","id2","id3","id4","id5"] }
   ← only those 5 rows

4. Upsert 5 rows locally
5. Update manifest cache
```

This means a sync where nothing changed transfers only the manifest
(a flat JSON list of id/version/checksum — a few KB even for large datasets)
and then completes immediately.

---

## 6. Server API reference

All endpoints accept and return `application/json`.
If `GUILDLIB_API_KEY` is set, include `X-Api-Key: <key>` on every request
except `/health`.

| Method | Path | Description |
|--------|------|-------------|
| GET | `/health` | Server status, version, shard list |
| GET | `/manifest/:shard` | Lightweight manifest for one shard |
| POST | `/pull` | Fetch rows by ID list with optional filters |
| POST | `/push` | Upsert one entry from Unity |
| POST | `/schema` | Upload schema.json from Unity |
| GET | `/schema` | Retrieve current schema.json |
| GET | `/entries/:shard` | List entries (supports query params) |
| POST | `/entries/:shard` | Create entry |
| GET | `/entries/:shard/:id` | Single entry |
| PUT | `/entries/:shard/:id` | Update entry |
| DELETE | `/entries/:shard/:id` | Soft-delete entry |

### `/manifest/:shard`

```
GET /manifest/weapons
→ {
    "shard": "weapons",
    "rows": [
      { "id": "abc123", "version": 3, "checksum": "d41d8cd9..." },
      ...
    ]
  }
```

### `/pull`

```
POST /pull
{
  "shard": "weapons",
  "ids": ["abc123", "def456"],
  "filterPlatforms": ["pc", "ps5"],   // optional
  "filterTags": ["release"]           // optional
}
→ {
    "shard": "weapons",
    "rows": [
      { "id":"abc123", "type_name":"...", "platforms":["all"],
        "tags":["release","base_game"], "weaponType":"Sword", ... },
      ...
    ]
  }
```

### `/entries/:shard` (list)

Query params:
- `type` — filter by C# type name
- `platforms` — comma-separated platform filter
- `tags` — comma-separated tag filter
- `parent` — filter by parent entry ID
- `limit` — default 100
- `offset` — default 0

```
GET /entries/weapons?type=Guildlib.Runtime.SwordData&platforms=pc,ps5&tags=release
```

### `/entries/:shard` (create)

```
POST /entries/weapons
{
  "typeName": "MyGame.AbilityDefinition",
  "platforms": ["all"],
  "tags": ["base_game", "release"],
  "data": {
    "abilityName": "Fireball",
    "cooldown": 3.5,
    "manaCost": 25
  }
}
→ { "id": "generated-uuid", "version": 1 }
```

### `/entries/:shard/:id` (update)

```
PUT /entries/weapons/abc123
{
  "platforms": ["pc", "ps5"],
  "tags": ["release"],
  "data": { "cooldown": 4.0 }
}
```

Only fields present in `data` are merged. Other fields are preserved.

---

## 7. Web editor guide

Open `web/index.html` in Chrome or Edge (no build step, no server needed for the file itself).

### Connecting

Enter your server URL (default `http://localhost:3000`) and click **Connect**.
The editor fetches `/health` to verify, then loads `/schema` to build the UI.

### Left column — shard and type tree

Lists all shards the schema describes. Each shard has a colour dot.
Click a shard to expand its type tree. Types are shown as root + children:

```
weapons
  ├─ WeaponData
  │    └─ SwordData    [child]
  │    └─ DaggerData   [child]
  └─ ...
abilities
  ├─ AbilityDefinition
  │    └─ AreaAbility  [child]
```

Click any type to load its entries.

### Middle column — entry list

Shows all entries of the selected type. Each row shows:
- The display name (from `[GuildDisplayName]` field, or entry ID)
- A short ID chip
- Row version
- Up to 3 platform tags (blue)
- Up to 2 content tags (yellow)

Use the search box to filter by display name.

### Right column — entry form

**Platform tags**: checkbox grid of all defined platforms. Click to toggle.
Click "all" to mark as platform-agnostic.

**Content tags**: checkbox grid of all defined tags. Click to toggle.

**Fields**: rendered based on the type's field schema:
- `[GuildEnum]` → dropdown
- `bool` → toggle switch
- `int`/`float` → number input
- `string[]` → textarea (one value per line)
- `[GuildMultiline]` → textarea
- `[GuildRef]` → text input with shard hint
- Everything else → text input

Inherited fields appear first with an "inherited" badge.

Click **Create entry** or **Save changes** to persist. Click **Delete** to soft-delete
(the row is marked `deleted=1` and version-bumped so Unity clients know to remove it).

---

## 8. Local testing (Windows / PowerShell)

### Install Bun (once)

```powershell
# Run as Administrator
powershell -c "irm bun.sh/install.ps1 | iex"
# Close and reopen PowerShell, then verify:
bun --version
```

### Start the server

```powershell
cd C:\path\to\guildlib-v2\server

bun install   # first time only

New-Item -ItemType Directory -Force -Path ".\shards",".\config"

$env:GUILDLIB_API_KEY = ""        # blank = auth disabled for local dev
$env:SHARD_DIR        = ".\shards"
$env:SCHEMA_PATH      = ".\config\schema.json"

bun run dev
```

Expected output:
```
[Guildlib] :3000  shards:   auth: OFF
[ShardManager] Opened: weapons.sqlite
[ShardManager] Opened: audio.sqlite
...
```

### Verify it works

```powershell
Invoke-RestMethod http://localhost:3000/health
```

### Push a test schema (without Unity)

```powershell
$schema = Get-Content -Raw "C:\path\to\guildlib-v2\docs\example-schema.json"
Invoke-RestMethod -Method POST `
  -Uri http://localhost:3000/schema `
  -ContentType "application/json" `
  -Body $schema
```

Or use the example at the bottom of this document.

### Create an entry via PowerShell

```powershell
$body = '{"typeName":"MyGame.AbilityDefinition","platforms":["all"],"tags":["base_game"],"data":{"abilityName":"Fireball","cooldown":3.5}}'
Invoke-RestMethod -Method POST `
  -Uri http://localhost:3000/entries/abilities `
  -ContentType "application/json" `
  -Body $body
```

### Test the delta sync

```powershell
# 1. Get manifest
Invoke-RestMethod http://localhost:3000/manifest/abilities

# 2. Pull specific IDs
$pull = '{"shard":"abilities","ids":["<paste-id-here>"]}'
Invoke-RestMethod -Method POST `
  -Uri http://localhost:3000/pull `
  -ContentType "application/json" `
  -Body $pull

# 3. Test platform filter
$pull = '{"shard":"abilities","ids":["<id>"],"filterPlatforms":["switch"]}'
Invoke-RestMethod -Method POST `
  -Uri http://localhost:3000/pull `
  -ContentType "application/json" `
  -Body $pull
```

---

## 9. Docker deployment

```bash
cd guildlib-v2/docker

# 1. Copy env template
cp .env.example .env
# Edit .env — set GUILDLIB_API_KEY to a real secret

# 2. Build and start
docker compose up -d

# 3. Check it's running
curl http://localhost:3000/health
```

The `guildlib-shards` and `guildlib-config` Docker volumes persist your data
across container restarts and updates.

To update the server:
```bash
docker compose pull
docker compose up -d
```

For HTTPS in production, uncomment the `caddy` block in `docker-compose.yml`
and set `DOMAIN=yourdomain.com` in `.env`.

---

## 10. Changing things — common tasks

### Add a new shard

Just write a class with `[ShardTarget("new_shard_name")]` and upload the schema.
No server config changes needed. The `.sqlite` file is created automatically
on first push.

### Add a new platform

1. Open `DataEntry.cs`
2. Add your platform string to `GuildPlatformRegistry.All`
3. Click **Upload Schema** in the Sync Panel
4. The web editor now shows the new platform as a checkbox

### Add a new tag

Same as adding a platform, but edit `GuildTagRegistry.All`.

### Add a field to an existing type

1. Add the `public` field to your C# class
2. Click **Upload Schema**
3. Existing entries in the database keep all their old data — the new field
   will be empty/default until entries are edited and re-saved

### Rename a field

Rename it in C# and re-upload the schema. The old field name remains in existing
database rows (the data is stored as JSON internally). Use the web editor to
open and re-save affected entries so they write the new field name.

### Remove a field

Remove it from C# and re-upload. Existing data with the old field name is
preserved in the database JSON blob but won't appear in the web editor forms.
It won't be synced to new Unity clients.

### Change a shard name

This is a breaking change. Rename the `[ShardTarget]` value, re-upload schema,
then manually copy the old `.sqlite` file on the server to the new name.
The server will open it automatically on restart.

### Add a child type

Subclass an existing DataEntry subclass, give it the same `[ShardTarget]`,
add its own fields, and upload the schema. It appears nested in the web editor
under its parent type immediately.

---

## Example schema JSON (for testing without Unity)

Save this as `example-schema.json` and POST it to `/schema`:

```json
{
  "version": "2",
  "exportedAt": 1710000000,
  "unityProjectId": "test",
  "shards": ["abilities", "characters"],
  "platformConfig": {
    "platforms": ["pc","ps5","xbox_series","switch","mobile_ios"],
    "tags":      ["release","debug","demo","base_game","dlc_1","high","low"]
  },
  "types": [
    {
      "typeName": "MyGame.AbilityDefinition",
      "displayName": "Ability Definition",
      "description": "Defines a single player or enemy ability.",
      "shard": "abilities",
      "parentType": null,
      "childTypes": ["MyGame.AreaAbility"],
      "defaultPlatforms": ["all"],
      "defaultTags": ["base_game", "release"],
      "fields": [
        {"name":"abilityName","type":"string","isInherited":false,"isDisplayName":true},
        {"name":"description","type":"string","isInherited":false,"isMultiline":true},
        {"name":"abilityType","type":"string","isInherited":false,"enumOptions":["Active","Passive","Toggle"]},
        {"name":"cooldown","type":"float","isInherited":false},
        {"name":"manaCost","type":"float","isInherited":false},
        {"name":"maxLevel","type":"int","isInherited":false},
        {"name":"castSoundId","type":"string","isInherited":false,"refShard":"audio"}
      ]
    },
    {
      "typeName": "MyGame.AreaAbility",
      "displayName": "Area Ability",
      "description": "An ability that affects an area.",
      "shard": "abilities",
      "parentType": "MyGame.AbilityDefinition",
      "childTypes": [],
      "defaultPlatforms": ["all"],
      "defaultTags": ["base_game", "release"],
      "fields": [
        {"name":"abilityName","type":"string","isInherited":true,"isDisplayName":true},
        {"name":"cooldown","type":"float","isInherited":true},
        {"name":"radius","type":"float","isInherited":false},
        {"name":"falloffDamage","type":"bool","isInherited":false},
        {"name":"areaShape","type":"string","isInherited":false,"enumOptions":["Circle","Cone","Rectangle"]}
      ]
    },
    {
      "typeName": "MyGame.CharacterStats",
      "displayName": "Character Stats",
      "description": "Base stats for all characters.",
      "shard": "characters",
      "parentType": null,
      "childTypes": [],
      "defaultPlatforms": ["all"],
      "defaultTags": ["base_game", "release"],
      "fields": [
        {"name":"characterName","type":"string","isInherited":false,"isDisplayName":true},
        {"name":"maxHealth","type":"float","isInherited":false},
        {"name":"moveSpeed","type":"float","isInherited":false},
        {"name":"characterClass","type":"string","isInherited":false,"enumOptions":["Warrior","Mage","Rogue","Ranger"]},
        {"name":"loreText","type":"string","isInherited":false,"isMultiline":true}
      ]
    }
  ]
}
```