vuk/MAGI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
Vuk Savić
2026-03-16 21:38:49 +01:00
commit d1eeccbefc
21 changed files with 3178 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

870
docs/DOCUMENTATION.md Normal file
View File

@@ -0,0 +1,870 @@
# Guildlib v2 — Complete Documentation
---
## Table of contents
1. [How it works — overview](#1-how-it-works)
2. [Unity attributes — complete reference](#2-unity-attributes)
3. [Creating your own shards and types](#3-creating-shards-and-types)
4. [Platform and tag system](#4-platform-and-tag-system)
5. [The sync cycle — step by step](#5-the-sync-cycle)
6. [Server API reference](#6-server-api-reference)
7. [Web editor guide](#7-web-editor-guide)
8. [Local testing (Windows / PowerShell)](#8-local-testing)
9. [Docker deployment](#9-docker-deployment)
10. [Changing things — common tasks](#10-changing-things)
---
## 1. How it works
```
Unity C# classes
│
│ [ShardTarget], [GuildLabel], [GuildPlatforms] …
│ (attributes on your classes describe everything)
▼
SchemaExporter.Export()
│
│ produces schema.json
▼
POST /schema ──────────────────────────► Bun server
│
│ stores schema.json
│ auto-creates .sqlite per shard
▼
Web editor reads schema.json and renders type-safe forms
Users create / edit entries in the browser
Changes stored in weapons.sqlite, audio.sqlite, etc.
Unity Sync Panel clicks "Pull All Shards":
1. GET /manifest/weapons → [{id, version, checksum}] (lightweight!)
2. diff against local cache
3. POST /pull { ids: [only what changed] }
4. receive only changed rows
5. upsert into local weapons.sqlite
```
Key design decisions inherited from the Perforce paper:
- **Manifest-first**: only checksums cross the wire on the first pass.
No data is transferred if nothing changed.
- **Shard independence**: each shard is a completely separate .sqlite file.
You can pull just `audio` without touching `weapons`.
- **Server-side filtering**: platform and tag filters are applied on the server
before rows are returned, so a Switch build only downloads Switch data.
---
## 2. Unity attributes
All attributes live in `Guildlib.Runtime`. Import the namespace at the top of
your C# files.
### `[ShardTarget("name")]` ← **REQUIRED**
Declares which shard database this type belongs to.
```csharp
[ShardTarget("characters")]
public class CharacterStats : DataEntry { ... }
[ShardTarget("ui")]
public class UIButtonConfig : DataEntry { ... }
[ShardTarget("vfx_particles")] // underscores OK, lowercase
public class ParticlePreset : DataEntry { ... }
```
Rules:
- Lowercase, no spaces, no special characters except underscore
- The server auto-creates a new `<name>.sqlite` file the first time an entry
of that type is pushed — you do not need to pre-register shards anywhere
- If two classes share a `[ShardTarget]` they live in the same database file
(the shard is a file, not a type)
---
### `[GuildLabel("Display Name")]` ← optional
Sets the human-readable name shown in the web editor's type list.
Defaults to the C# class name if omitted.
```csharp
[ShardTarget("characters")]
[GuildLabel("Character Stats")]
public class CharacterStats : DataEntry { ... }
// Shows as "Character Stats" in the web editor, not "CharacterStats"
```
---
### `[GuildDescription("...")]` ← optional
A one-sentence description shown in the web editor's form panel under the type name.
Use it to explain what this entry type is for.
```csharp
[GuildDescription("Defines base stats for all playable and enemy characters.")]
public class CharacterStats : DataEntry { ... }
```
---
### `[GuildPlatforms("p1", "p2", ...)]` ← optional
Sets the **default** platform tags for newly created entries of this type.
Can be overridden per-entry in the web editor.
```csharp
// This entry type ships on all platforms by default
[GuildPlatforms("all")]
public class CharacterStats : DataEntry { ... }
// PC and console only — Switch gets different data
[GuildPlatforms("pc", "ps5", "xbox_series")]
public class HighResTextureConfig : DataEntry { ... }
// Switch-specific preset
[GuildPlatforms("switch", "mobile_ios", "mobile_android")]
public class LowResTextureConfig : DataEntry { ... }
```
If omitted, defaults to `["all"]` — meaning every platform gets this data.
---
### `[GuildTags("t1", "t2", ...)]` ← optional
Sets the **default** content/build tags for new entries of this type.
Can be overridden per-entry in the web editor.
```csharp
[GuildTags("base_game", "release")]
public class CharacterStats : DataEntry { ... }
[GuildTags("dlc_1")]
public class DLC1Character : CharacterStats { ... }
[GuildTags("debug")]
public class DebugTestCharacter : CharacterStats { ... }
```
---
### `[GuildDisplayName]` ← optional, on a field
Marks one field as the "display name" for the entry list in the web editor.
Without this, the entry ID is shown. Put it on whichever field is most useful
as a human-readable label.
```csharp
public class CharacterStats : DataEntry
{
[GuildDisplayName]
public string characterName; // web editor shows this in the list
public float maxHealth;
}
```
---
### `[GuildEnum("A", "B", "C")]` ← optional, on a string field
Constrains a string field to a fixed set of options.
The web editor renders a dropdown instead of a plain text input.
```csharp
[GuildEnum("Warrior", "Mage", "Rogue", "Ranger")]
public string characterClass;
[GuildEnum("Common", "Uncommon", "Rare", "Epic", "Legendary")]
public string rarity;
[GuildEnum("Fire", "Ice", "Lightning", "Physical", "Poison")]
public string damageType;
```
---
### `[GuildRef("shard_name")]` ← optional, on a string field
Marks a string field as a foreign key reference to an entry in another shard.
The web editor shows an annotated hint (full picker UI is a planned enhancement).
The actual stored value is still the entryId string.
```csharp
public class CharacterStats : DataEntry
{
[GuildRef("audio")]
public string deathSoundId; // points to an entry in audio.sqlite
[GuildRef("models")]
public string modelId; // points to an entry in models.sqlite
}
```
---
### `[GuildMultiline]` ← optional, on a string field
Makes the web editor render a textarea instead of a single-line input.
Use for long descriptions or multi-line text content.
```csharp
[GuildMultiline]
public string loreText;
[GuildMultiline]
public string debugNotes;
```
---
### `[GuildExclude]` ← optional, on a field
Excludes a field from schema export and sync entirely.
Use for Unity-only fields (Texture2D refs, EditorGUILayout state, etc.)
that cannot be serialised to the database.
```csharp
[GuildExclude]
public Texture2D previewTexture; // Unity asset ref — never synced
[GuildExclude]
public bool editorFoldout; // editor UI state — never synced
```
---
## 3. Creating shards and types
### Step 1 — Write a C# class
Create a new .cs file anywhere under `Assets/` in your Unity project.
```csharp
using UnityEngine;
using Guildlib.Runtime;
// ── A brand new shard called "abilities" ─────────────────────────────────────
[ShardTarget("abilities")]
[GuildLabel("Ability Definition")]
[GuildDescription("Defines a single player or enemy ability.")]
[GuildPlatforms("all")]
[GuildTags("base_game", "release")]
[CreateAssetMenu(menuName = "Guildlib/Ability")]
public class AbilityDefinition : DataEntry
{
[GuildDisplayName]
public string abilityName;
[GuildMultiline]
public string description;
[GuildEnum("Active", "Passive", "Toggle")]
public string abilityType;
public float cooldown;
public float manaCost;
public int maxLevel;
[GuildRef("audio")]
public string castSoundId;
[GuildRef("vfx_particles")]
public string castVFXId;
}
// ── A child type — inherits all AbilityDefinition fields ─────────────────────
[ShardTarget("abilities")]
[GuildLabel("Area Ability")]
[GuildDescription("An ability that affects an area rather than a single target.")]
public class AreaAbility : AbilityDefinition
{
public float radius;
public bool falloffDamage;
[GuildEnum("Circle", "Cone", "Rectangle")]
public string areaShape;
}
```
### Step 2 — Upload the schema
Open **Window → Guildlib → Sync Panel**, then click **Upload Schema**.
This sends all your C# type definitions to the server. The server stores
the schema as `config/schema.json`. The web editor reads this file to know
what types exist, what their fields are, and what options are available.
The `abilities` shard now exists on the server. Its `.sqlite` file is created
automatically the first time an entry is pushed.
### Step 3 — Use in the web editor
Refresh the web editor. You will see the `abilities` shard appear in the left
column. Click it to see `Ability Definition` and `Area Ability` in the type tree.
### Inheritance rules
- A child type appears nested under its parent in the web editor
- The form shows inherited fields first (marked "inherited"), then own fields
- Child types go into the **same shard** as their parent
- You can nest arbitrarily deep: `DataEntry → Ability → AreaAbility → AOEBomb`
- Multiple children of the same parent are all shown in the tree
---
## 4. Platform and tag system
### What they are
**Platforms** describe which shipping targets an entry is relevant for.
**Tags** describe which content group, build configuration, or DLC an entry belongs to.
Both are stored per-entry (not per-type) and can be edited in the web editor.
Type-level `[GuildPlatforms]` and `[GuildTags]` attributes only set the default
for newly created entries — users can change them freely in the web editor.
### Defining your vocabulary
Both lists live in `DataEntry.cs`:
```csharp
public static class GuildPlatformRegistry
{
public static readonly string[] All = new[]
{
"pc",
"ps5",
"ps4",
"xbox_series",
"xbox_one",
"switch",
"mobile_ios",
"mobile_android",
"vr",
};
}
public static class GuildTagRegistry
{
public static readonly string[] All = new[]
{
"release",
"debug",
"demo",
"base_game",
"dlc_1",
"dlc_2",
"high",
"medium",
"low",
"global",
"region_jp",
"region_eu",
"region_na",
};
}
```
**To add a platform or tag**: add a string to the appropriate array and run
**Upload Schema** from the Sync Panel. The web editor will show the new option
immediately.
**To remove one**: remove it from the array and re-upload the schema. Existing
entries that had the old value keep it stored in the database — the web editor
won't show the removed value as a selectable checkbox, but the data is preserved.
### The special "all" platform
An entry tagged `"all"` is returned for every platform filter. This means if
you add a new platform later, entries tagged "all" are automatically included
without any data migration.
### Using platform/tag filters in Unity
In the Sync Panel:
1. Toggle on **Filter by platform**
2. Enter comma-separated platforms: `pc,ps5`
3. Toggle on **Filter by tag** (optional)
4. Enter tags: `release,base_game`
5. Click **Pull All Shards**
Only rows matching the filters come down the wire.
Programmatically from C#:
```csharp
var client = new ShardClient(config, "abilities");
// Pull only PC/PS5 release content
await client.PullAsync(
platformFilter: new List<string> { "pc", "ps5" },
tagFilter: new List<string> { "release", "base_game" }
);
// Pull everything (no filter)
await client.PullAsync();
```
### Using filters on the server API directly
```
GET /entries/abilities?platforms=pc,ps5&tags=release,base_game
POST /pull { "shard":"abilities", "ids":[...], "filterPlatforms":["switch"], "filterTags":["base_game"] }
```
---
## 5. The sync cycle
### Full cycle (first time or clean slate)
```
1. GET /manifest/weapons
← { shard:"weapons", rows: [] } ← empty, nothing on server yet
2. (nothing to diff — skip pull)
3. (user creates entries via web editor)
4. GET /manifest/weapons
← { rows: [{id:"abc", version:1, checksum:"d41d..."}, ...] }
5. Diff: local has nothing → need all IDs
6. POST /pull { shard:"weapons", ids:["abc", ...] }
← { rows: [{id:"abc", weaponType:"Sword", damage:45, ...}, ...] }
7. Upsert into local weapons.sqlite
8. Save manifest cache
```
### Incremental sync (normal operation)
```
1. GET /manifest/weapons
← 10,000 rows of {id, version, checksum}
2. Compare with local manifest:
- 9,995 rows unchanged → skip
- 5 rows have newer version → add to needIds
3. POST /pull { ids: ["id1","id2","id3","id4","id5"] }
← only those 5 rows
4. Upsert 5 rows locally
5. Update manifest cache
```
This means a sync where nothing changed transfers only the manifest
(a flat JSON list of id/version/checksum — a few KB even for large datasets)
and then completes immediately.
---
## 6. Server API reference
All endpoints accept and return `application/json`.
If `GUILDLIB_API_KEY` is set, include `X-Api-Key: <key>` on every request
except `/health`.
| Method | Path | Description |
|--------|------|-------------|
| GET | `/health` | Server status, version, shard list |
| GET | `/manifest/:shard` | Lightweight manifest for one shard |
| POST | `/pull` | Fetch rows by ID list with optional filters |
| POST | `/push` | Upsert one entry from Unity |
| POST | `/schema` | Upload schema.json from Unity |
| GET | `/schema` | Retrieve current schema.json |
| GET | `/entries/:shard` | List entries (supports query params) |
| POST | `/entries/:shard` | Create entry |
| GET | `/entries/:shard/:id` | Single entry |
| PUT | `/entries/:shard/:id` | Update entry |
| DELETE | `/entries/:shard/:id` | Soft-delete entry |
### `/manifest/:shard`
```
GET /manifest/weapons
→ {
"shard": "weapons",
"rows": [
{ "id": "abc123", "version": 3, "checksum": "d41d8cd9..." },
...
]
}
```
### `/pull`
```
POST /pull
{
"shard": "weapons",
"ids": ["abc123", "def456"],
"filterPlatforms": ["pc", "ps5"], // optional
"filterTags": ["release"] // optional
}
→ {
"shard": "weapons",
"rows": [
{ "id":"abc123", "type_name":"...", "platforms":["all"],
"tags":["release","base_game"], "weaponType":"Sword", ... },
...
]
}
```
### `/entries/:shard` (list)
Query params:
- `type` — filter by C# type name
- `platforms` — comma-separated platform filter
- `tags` — comma-separated tag filter
- `parent` — filter by parent entry ID
- `limit` — default 100
- `offset` — default 0
```
GET /entries/weapons?type=Guildlib.Runtime.SwordData&platforms=pc,ps5&tags=release
```
### `/entries/:shard` (create)
```
POST /entries/weapons
{
"typeName": "MyGame.AbilityDefinition",
"platforms": ["all"],
"tags": ["base_game", "release"],
"data": {
"abilityName": "Fireball",
"cooldown": 3.5,
"manaCost": 25
}
}
→ { "id": "generated-uuid", "version": 1 }
```
### `/entries/:shard/:id` (update)
```
PUT /entries/weapons/abc123
{
"platforms": ["pc", "ps5"],
"tags": ["release"],
"data": { "cooldown": 4.0 }
}
```
Only fields present in `data` are merged. Other fields are preserved.
---
## 7. Web editor guide
Open `web/index.html` in Chrome or Edge (no build step, no server needed for the file itself).
### Connecting
Enter your server URL (default `http://localhost:3000`) and click **Connect**.
The editor fetches `/health` to verify, then loads `/schema` to build the UI.
### Left column — shard and type tree
Lists all shards the schema describes. Each shard has a colour dot.
Click a shard to expand its type tree. Types are shown as root + children:
```
weapons
├─ WeaponData
│ └─ SwordData [child]
│ └─ DaggerData [child]
└─ ...
abilities
├─ AbilityDefinition
│ └─ AreaAbility [child]
```
Click any type to load its entries.
### Middle column — entry list
Shows all entries of the selected type. Each row shows:
- The display name (from `[GuildDisplayName]` field, or entry ID)
- A short ID chip
- Row version
- Up to 3 platform tags (blue)
- Up to 2 content tags (yellow)
Use the search box to filter by display name.
### Right column — entry form
**Platform tags**: checkbox grid of all defined platforms. Click to toggle.
Click "all" to mark as platform-agnostic.
**Content tags**: checkbox grid of all defined tags. Click to toggle.
**Fields**: rendered based on the type's field schema:
- `[GuildEnum]` → dropdown
- `bool` → toggle switch
- `int`/`float` → number input
- `string[]` → textarea (one value per line)
- `[GuildMultiline]` → textarea
- `[GuildRef]` → text input with shard hint
- Everything else → text input
Inherited fields appear first with an "inherited" badge.
Click **Create entry** or **Save changes** to persist. Click **Delete** to soft-delete
(the row is marked `deleted=1` and version-bumped so Unity clients know to remove it).
---
## 8. Local testing (Windows / PowerShell)
### Install Bun (once)
```powershell
# Run as Administrator
powershell -c "irm bun.sh/install.ps1 | iex"
# Close and reopen PowerShell, then verify:
bun --version
```
### Start the server
```powershell
cd C:\path\to\guildlib-v2\server
bun install # first time only
New-Item -ItemType Directory -Force -Path ".\shards",".\config"
$env:GUILDLIB_API_KEY = "" # blank = auth disabled for local dev
$env:SHARD_DIR = ".\shards"
$env:SCHEMA_PATH = ".\config\schema.json"
bun run dev
```
Expected output:
```
[Guildlib] :3000 shards: auth: OFF
[ShardManager] Opened: weapons.sqlite
[ShardManager] Opened: audio.sqlite
...
```
### Verify it works
```powershell
Invoke-RestMethod http://localhost:3000/health
```
### Push a test schema (without Unity)
```powershell
$schema = Get-Content -Raw "C:\path\to\guildlib-v2\docs\example-schema.json"
Invoke-RestMethod -Method POST `
-Uri http://localhost:3000/schema `
-ContentType "application/json" `
-Body $schema
```
Or use the example at the bottom of this document.
### Create an entry via PowerShell
```powershell
$body = '{"typeName":"MyGame.AbilityDefinition","platforms":["all"],"tags":["base_game"],"data":{"abilityName":"Fireball","cooldown":3.5}}'
Invoke-RestMethod -Method POST `
-Uri http://localhost:3000/entries/abilities `
-ContentType "application/json" `
-Body $body
```
### Test the delta sync
```powershell
# 1. Get manifest
Invoke-RestMethod http://localhost:3000/manifest/abilities
# 2. Pull specific IDs
$pull = '{"shard":"abilities","ids":["<paste-id-here>"]}'
Invoke-RestMethod -Method POST `
-Uri http://localhost:3000/pull `
-ContentType "application/json" `
-Body $pull
# 3. Test platform filter
$pull = '{"shard":"abilities","ids":["<id>"],"filterPlatforms":["switch"]}'
Invoke-RestMethod -Method POST `
-Uri http://localhost:3000/pull `
-ContentType "application/json" `
-Body $pull
```
---
## 9. Docker deployment
```bash
cd guildlib-v2/docker
# 1. Copy env template
cp .env.example .env
# Edit .env — set GUILDLIB_API_KEY to a real secret
# 2. Build and start
docker compose up -d
# 3. Check it's running
curl http://localhost:3000/health
```
The `guildlib-shards` and `guildlib-config` Docker volumes persist your data
across container restarts and updates.
To update the server:
```bash
docker compose pull
docker compose up -d
```
For HTTPS in production, uncomment the `caddy` block in `docker-compose.yml`
and set `DOMAIN=yourdomain.com` in `.env`.
---
## 10. Changing things — common tasks
### Add a new shard
Just write a class with `[ShardTarget("new_shard_name")]` and upload the schema.
No server config changes needed. The `.sqlite` file is created automatically
on first push.
### Add a new platform
1. Open `DataEntry.cs`
2. Add your platform string to `GuildPlatformRegistry.All`
3. Click **Upload Schema** in the Sync Panel
4. The web editor now shows the new platform as a checkbox
### Add a new tag
Same as adding a platform, but edit `GuildTagRegistry.All`.
### Add a field to an existing type
1. Add the `public` field to your C# class
2. Click **Upload Schema**
3. Existing entries in the database keep all their old data — the new field
will be empty/default until entries are edited and re-saved
### Rename a field
Rename it in C# and re-upload the schema. The old field name remains in existing
database rows (the data is stored as JSON internally). Use the web editor to
open and re-save affected entries so they write the new field name.
### Remove a field
Remove it from C# and re-upload. Existing data with the old field name is
preserved in the database JSON blob but won't appear in the web editor forms.
It won't be synced to new Unity clients.
### Change a shard name
This is a breaking change. Rename the `[ShardTarget]` value, re-upload schema,
then manually copy the old `.sqlite` file on the server to the new name.
The server will open it automatically on restart.
### Add a child type
Subclass an existing DataEntry subclass, give it the same `[ShardTarget]`,
add its own fields, and upload the schema. It appears nested in the web editor
under its parent type immediately.
---
## Example schema JSON (for testing without Unity)
Save this as `example-schema.json` and POST it to `/schema`:
```json
{
"version": "2",
"exportedAt": 1710000000,
"unityProjectId": "test",
"shards": ["abilities", "characters"],
"platformConfig": {
"platforms": ["pc","ps5","xbox_series","switch","mobile_ios"],
"tags": ["release","debug","demo","base_game","dlc_1","high","low"]
},
"types": [
{
"typeName": "MyGame.AbilityDefinition",
"displayName": "Ability Definition",
"description": "Defines a single player or enemy ability.",
"shard": "abilities",
"parentType": null,
"childTypes": ["MyGame.AreaAbility"],
"defaultPlatforms": ["all"],
"defaultTags": ["base_game", "release"],
"fields": [
{"name":"abilityName","type":"string","isInherited":false,"isDisplayName":true},
{"name":"description","type":"string","isInherited":false,"isMultiline":true},
{"name":"abilityType","type":"string","isInherited":false,"enumOptions":["Active","Passive","Toggle"]},
{"name":"cooldown","type":"float","isInherited":false},
{"name":"manaCost","type":"float","isInherited":false},
{"name":"maxLevel","type":"int","isInherited":false},
{"name":"castSoundId","type":"string","isInherited":false,"refShard":"audio"}
]
},
{
"typeName": "MyGame.AreaAbility",
"displayName": "Area Ability",
"description": "An ability that affects an area.",
"shard": "abilities",
"parentType": "MyGame.AbilityDefinition",
"childTypes": [],
"defaultPlatforms": ["all"],
"defaultTags": ["base_game", "release"],
"fields": [
{"name":"abilityName","type":"string","isInherited":true,"isDisplayName":true},
{"name":"cooldown","type":"float","isInherited":true},
{"name":"radius","type":"float","isInherited":false},
{"name":"falloffDamage","type":"bool","isInherited":false},
{"name":"areaShape","type":"string","isInherited":false,"enumOptions":["Circle","Cone","Rectangle"]}
]
},
{
"typeName": "MyGame.CharacterStats",
"displayName": "Character Stats",
"description": "Base stats for all characters.",
"shard": "characters",
"parentType": null,
"childTypes": [],
"defaultPlatforms": ["all"],
"defaultTags": ["base_game", "release"],
"fields": [
{"name":"characterName","type":"string","isInherited":false,"isDisplayName":true},
{"name":"maxHealth","type":"float","isInherited":false},
{"name":"moveSpeed","type":"float","isInherited":false},
{"name":"characterClass","type":"string","isInherited":false,"enumOptions":["Warrior","Mage","Rogue","Ranger"]},
{"name":"loreText","type":"string","isInherited":false,"isMultiline":true}
]
}
]
}
```