project-vesper/vesper/documentation/project-vesper-plan.md

# Project Vesper — Manufacturing Automation Master Plan

> **How to use this document:** Work through each Phase in order. Each Phase has self-contained tasks you can hand directly to Claude Code. Phases 1–3 are foundational; don't skip ahead. Phases 4–6 build on top of them.

---

## Current Stack (Reference)

| Layer | Technology |
|---|---|
| Microcontroller | ESP32 (ESP8266 on older models, STM32 possible future) |
| MCU Firmware | Arduino / C++ (moving to PlatformIO) |
| Tablet App | Flutter / FlutterFlow (Android) |
| Phone App | Flutter / FlutterFlow (Android + iOS) |
| Admin Console Backend | Python 3.11+ / FastAPI |
| Admin Console Frontend | React + Tailwind CSS |
| Primary Database | Firebase Firestore (via Admin SDK) |
| Secondary Database | Local SQLite (MQTT logs, etc.) |
| File Storage | Firebase Storage |
| MQTT Broker | Mosquitto on VPS |
| Web Server | NGINX on VPS (OTA files, reverse proxy) |
| Deployment | Gitea → git pull on VPS, Docker Compose locally |

---

## Serial Number Format (Locked In)

```
PV-YYMMM-BBTTR-XXXXX

PV       = Project Vesper prefix
YY       = 2-digit year         (e.g. 26)
MMM      = 3-char date block    = 2-digit month letter + 2-digit day (e.g. A18 = Jan 18)
BB       = Board Type code      (e.g. BC = BellCore, BP = BellPRO)
TT       = Board Type revision  (e.g. 02)
R        = Literal "R"
XXXXX    = 5-char random suffix (A-Z, 0-9, excluding 0/O/1/I for label clarity)

Example: PV-26A18-BC02R-X7KQA
```

**Month Letter Codes:**
A=Jan, B=Feb, C=Mar, D=Apr, E=May, F=Jun, G=Jul, H=Aug, I=Sep, J=Oct, K=Nov, L=Dec

---

## Phase 1 — PlatformIO Migration (Firmware Side)

> **Goal:** Replace Arduino IDE with PlatformIO. All future firmware work happens here. This unlocks scripted builds and the ability to produce `.bin` files on demand.
>
> **Tell Claude Code:** *"Help me migrate my existing Arduino IDE ESP32 project to PlatformIO with multiple board environments."*

### Tasks

- [ ] Install PlatformIO extension in VS Code
- [ ] Create `platformio.ini` with one `[env]` block per hardware variant. Example:

```ini
[env:bellcore-v2]
platform = espressif32
board = esp32dev
board_build.partitions = partitions/custom_4mb.csv
board_build.flash_mode = dio
board_upload.flash_size = 4MB
build_flags =
    -DBOARD_TYPE="BC"
    -DBOARD_VERSION="02"
    -DPSRAM_ENABLED=1

[env:bellcore-v1]
platform = espressif32
board = esp32dev
board_build.partitions = partitions/custom_2mb.csv
board_build.flash_mode = dout
board_upload.flash_size = 2MB
build_flags =
    -DBOARD_TYPE="BC"
    -DBOARD_VERSION="01"
    -DPSRAM_ENABLED=0
```

- [ ] Move all `#include` library dependencies into `lib_deps` in `platformio.ini` (no more manual library manager)
- [ ] Verify `pio run -e bellcore-v2` compiles clean
- [ ] Confirm `.pio/build/bellcore-v2/firmware.bin` is produced
- [ ] Create a `/firmware` directory structure on the server (NGINX already serves this):

```
/srv/ota/
  bellcore-v1/
    latest.bin
    v1.0.0.bin
    v1.0.1.bin
  bellcore-v2/
    latest.bin
    ...
```

- [ ] Update your OTA logic in firmware to pull from `/ota/{board_type}/latest.bin`
- [ ] Add a `scripts/build_and_upload.sh` that compiles + copies the new `.bin` to the right NGINX folder (run this after every release)

### NVS Partition Generator Setup

- [ ] Install `esptool` and `esp-idf` NVS tool: `pip install esptool`
- [ ] Grab `nvs_partition_gen.py` from ESP-IDF tools (or install via `idf-component-manager`)
- [ ] Test generating a `.bin` from a CSV manually:

```csv
key,type,encoding,value
serial_number,data,string,PV-26A18-BC02R-X7KQA
hw_type,data,string,BC
hw_version,data,string,02
```

```bash
python nvs_partition_gen.py generate nvs_data.csv nvs_data.bin 0x6000
```

- [ ] Confirm the ESP32 reads NVS values correctly on boot with this pre-flashed partition
- [ ] Note the NVS partition address for your board (check your partition table CSV — typically `0x9000`)

---

## Phase 2 — MQTT Dynamic Auth (Backend Side)

> **Goal:** Replace `mosquitto_passwd` manual SSH with automatic credential management. New devices are live on MQTT the moment they exist in your database. Per-device topic isolation enforced automatically.
>
> **Tell Claude Code:** *"Help me set up mosquitto-go-auth on my VPS with a FastAPI backend for dynamic MQTT authentication and ACL enforcement."*

### Tasks

#### 2a. Install mosquitto-go-auth on VPS

- [ ] Install Go on VPS (required to build the plugin)
- [ ] Clone and build `mosquitto-go-auth`:
  ```bash
  git clone https://github.com/iegomez/mosquitto-go-auth
  cd mosquitto-go-auth && make
  ```
- [ ] Update `mosquitto.conf` to load the plugin:
  ```
  auth_plugin /path/to/go-auth.so
  auth_opt_backends http
  auth_opt_http_host localhost
  auth_opt_http_port 8000
  auth_opt_http_getuser_uri /mqtt/auth/user
  auth_opt_http_aclcheck_uri /mqtt/auth/acl
  auth_opt_cache true
  auth_opt_cache_host localhost
  auth_opt_cache_reset true
  auth_opt_auth_cache_seconds 300
  auth_opt_acl_cache_seconds 300
  ```

#### 2b. Add MQTT Auth Endpoints to FastAPI

- [ ] Create `/mqtt/auth/user` endpoint — Mosquitto calls this on CONNECT:
  - Receives: `username` (= device SN), `password`
  - Checks Firestore/SQLite for device record + hashed password
  - Returns: `200` (allow) or `403` (deny)

- [ ] Create `/mqtt/auth/acl` endpoint — Mosquitto calls this on SUBSCRIBE/PUBLISH:
  - Receives: `username`, `topic`, `acc` (1=sub, 2=pub)
  - Rule: username must match the SN segment in the topic
    - Topic pattern: `/vesper/{SN}/data` or `/vesper/{SN}/control`
    - Extract `{SN}` from topic, compare to `username`
  - Returns: `200` or `403`

- [ ] **For user phone app clients:** Add a separate user auth flow
  - Users authenticate with their Firebase UID as MQTT username
  - ACL check: look up which devices this UID owns in Firestore, permit only those SNs in topics

#### 2c. MQTT Password Strategy

Use **HMAC-derived passwords** so you never have to store or manually set them:

```python
import hmac, hashlib

MQTT_SECRET = os.getenv("MQTT_SECRET")  # Keep in .env, never commit

def derive_mqtt_password(serial_number: str) -> str:
    return hmac.new(
        MQTT_SECRET.encode(),
        serial_number.encode(),
        hashlib.sha256
    ).hexdigest()[:32]
```

- Device SN is known → password is deterministic → firmware can compute it at boot
- No password storage needed in DB (just re-derive on auth check)
- Changing `MQTT_SECRET` rotates all passwords at once if ever needed

- [ ] Add `MQTT_SECRET` to your `.env` and Docker Compose secrets
- [ ] Update firmware to derive its own MQTT password using the same HMAC logic (port to C++)
- [ ] Remove all existing `mosquitto_passwd` file entries and disable static auth

#### 2d. Test

- [ ] New device connects with correct SN + derived password → allowed
- [ ] Device tries to sub/pub on another device's topic → denied (403)
- [ ] Wrong password → denied
- [ ] Confirm cache is working (check logs, only 1-2 auth calls per session)

---

## Phase 3 — Serial Number & Batch Management in Admin Console

> **Goal:** SN generation, DB registration, and MQTT credential provisioning all happen in one flow in the React Console. The Flutter admin app is retired.
>
> **Tell Claude Code:** *"Add a Manufacturing / Batch Management section to our React+FastAPI admin console with the following features..."*

### Tasks

#### 3a. Backend — New API Routes in FastAPI

- [ ] `POST /manufacturing/batch` — Create a new batch:
  - Input: `board_type`, `board_version`, `quantity`, `subscription_plan`, `available_outputs`
  - Generate N serial numbers using the `PV-YYMMM-BBTTR-XXXXX` format
  - Check Firestore for collisions, regenerate if collision found
  - Write N device documents to Firestore collection `devices`:
    ```json
    {
      "serial_number": "PV-26A18-BC02R-X7KQA",
      "hw_type": "BC",
      "hw_version": "02",
      "status": "manufactured",
      "subscription_plan": "standard",
      "available_outputs": 8,
      "created_at": "...",
      "owner": null,
      "users_list": []
    }
    ```
  - Returns: list of created SNs

- [ ] `GET /manufacturing/batch/{batch_id}` — List devices in a batch with status
- [ ] `GET /manufacturing/devices` — List all devices with filters (status, hw_type, date range)
- [ ] `POST /manufacturing/devices/{sn}/assign` — Pre-assign device to a customer email
- [ ] `GET /manufacturing/firmware/{hw_type}/{hw_version}` — Return download URL for the correct `.bin`

#### 3b. SN Generator Utility (Python)

```python
# utils/serial_number.py
import random, string
from datetime import datetime

MONTH_CODES = "ABCDEFGHIJKL"
SAFE_CHARS = "ABCDEFGHJKLMNPQRSTUVWXYZ23456789"  # No 0,O,1,I

def generate_serial(board_type: str, board_version: str) -> str:
    now = datetime.utcnow()
    year = now.strftime("%y")
    month = MONTH_CODES[now.month - 1]
    day = now.strftime("%d")
    random_suffix = "".join(random.choices(SAFE_CHARS, k=5))
    return f"PV-{year}{month}{day}-{board_type}{board_version}R-{random_suffix}"
```

#### 3c. NVS Binary Generation in FastAPI

- [ ] Copy `nvs_partition_gen.py` into your FastAPI project
- [ ] Add endpoint `GET /manufacturing/devices/{sn}/nvs.bin`:
  - Generates a temp CSV for this SN
  - Runs `nvs_partition_gen.py` to produce the `.bin`
  - Returns the binary file as a download
- [ ] Add endpoint `GET /manufacturing/devices/{sn}/firmware.bin`:
  - Looks up device's `hw_type` and `hw_version` from Firestore
  - Returns the correct firmware `.bin` from the NGINX folder (or redirects to NGINX URL)

#### 3d. Label Sheet Generation

- [ ] Add `POST /manufacturing/batch/{batch_id}/labels` endpoint
  - Returns a PDF with one label per device
  - Each label contains: SN (human readable), QR code of SN, HW Type, HW Version
  - Use `reportlab` or `fpdf2` Python library for PDF generation
  - QR code: use `qrcode` Python library

#### 3e. Frontend — Manufacturing Section in React Console

- [ ] New route: `/manufacturing`
- [ ] **Batch Creator:** Form with board type selector, quantity, subscription plan → calls `POST /manufacturing/batch` → shows created SNs + download label PDF button
- [ ] **Device List:** Filterable table of all devices with status badges (manufactured / sold / claimed / active)
- [ ] **Device Detail Page:** Shows all fields, allows status update, shows assignment history

---

## Phase 4 — Browser-Based Flashing (The Provisioning Wizard)

> **Goal:** A single browser tab handles the entire provisioning flow. Plug in ESP32, click through wizard, done. No Arduino IDE, no esptool CLI, no SSH.
>
> **Tell Claude Code:** *"Add a Device Provisioning Wizard to our React admin console using esptool-js and the Web Serial API."*
>
> **Browser requirement:** Chrome or Edge only (Web Serial API). Firefox not supported. This is fine for an internal manufacturing tool.

### Tasks

#### 4a. Add esptool-js to React Console

- [ ] `npm install esptool-js` (or use the CDN build)
- [ ] Confirm Chrome is used on the manufacturing bench laptop

#### 4b. Provisioning Wizard UI (React Component)

Build a step-by-step wizard. Steps:

**Step 1 — Select or Create Device**
- Search existing unprovisioned device by SN, OR
- Quick-create single device (calls `POST /manufacturing/batch` with qty=1)
- Displays SN, HW Type, HW Version

**Step 2 — Flash Device**
- "Connect Device" button → triggers Web Serial port picker
- Fetches `nvs.bin` and `firmware.bin` from your FastAPI backend for this SN
- Shows two progress bars: NVS partition flash + Firmware flash
- Flash addresses (example for standard ESP32):
  - NVS: `0x9000` (verify against your partition table)
  - Firmware: `0x10000`
- On completion: updates device status to `flashed` in Firestore via API call

**Step 3 — Verify**
- Prompt: "Power cycle device and wait for it to connect"
- Poll Firestore (or MQTT) for first heartbeat/connection from this SN
- Show green checkmark when device phone home
- Updates status to `provisioned`

**Step 4 — Done**
- Show summary
- Option: "Provision next device" (loops back to Step 1 with same batch settings)
- Option: "Print label" (downloads single-device PDF label)

#### 4c. esptool-js Flash Logic (Skeleton)

```javascript
import { ESPLoader, Transport } from "esptool-js";

async function flashDevice(serialPort, nvsArrayBuffer, firmwareArrayBuffer) {
  const transport = new Transport(serialPort);
  const loader = new ESPLoader({ transport, baudrate: 460800 });

  await loader.main_fn();
  await loader.flash_id();

  await loader.write_flash({
    fileArray: [
      { data: nvsArrayBuffer,      address: 0x9000  },
      { data: firmwareArrayBuffer, address: 0x10000 },
    ],
    flashSize: "keep",
    flashMode: "keep",
    flashFreq: "keep",
    eraseAll: false,
    compress: true,
  });

  await transport.disconnect();
}
```

#### 4d. NGINX CORS Headers

Add to your NGINX config so the browser can fetch `.bin` files:

```nginx
location /ota/ {
    add_header Access-Control-Allow-Origin "https://your-console-domain.com";
    add_header Access-Control-Allow-Methods "GET";
}
```

---

## Phase 5 — Email Notifications

> **Goal:** Admin Console can send transactional emails (device assignment invites, alerts, etc.)
>
> **Tell Claude Code:** *"Add email sending capability to our FastAPI backend using Resend (or SMTP)."*

### Tasks

- [ ] Sign up for [Resend](https://resend.com) (free tier: 3000 emails/month, 100/day)
- [ ] Add `RESEND_API_KEY` to `.env`
- [ ] Install: `pip install resend`
- [ ] Create `utils/email.py`:

```python
import resend
import os

resend.api_key = os.getenv("RESEND_API_KEY")

def send_device_invite(customer_email: str, serial_number: str, customer_name: str = None):
    resend.Emails.send({
        "from": "noreply@yourcompany.com",
        "to": customer_email,
        "subject": "Your Vesper device is ready",
        "html": f"""
            <h2>Your device has been registered</h2>
            <p>Serial Number: <strong>{serial_number}</strong></p>
            <p>Open the Vesper app and enter this serial number to get started.</p>
        """
    })
```

- [ ] Hook into `POST /manufacturing/devices/{sn}/assign` to send invite automatically
- [ ] Add basic email templates for: device assignment, welcome, error alerts

---

## Phase 6 — Polish & Retire Legacy Tools

> **Goal:** Clean up. Everything lives in the Console. Nothing is done manually.

### Tasks

- [ ] **Retire Flutter admin app** — confirm every function it had is now in the React Console
- [ ] **Remove static mosquitto password file** — all auth is dynamic now
- [ ] **Add device status dashboard** to Console home: counts by status, recent provisioning activity
- [ ] **Add audit log** — every manufacturing action (batch created, device flashed, device assigned) logged to SQLite with timestamp and admin user
- [ ] **Document your `platformio.ini` environments** — add a `FIRMWARE_VARIANTS.md` to the firmware repo
- [ ] **Set up Gitea webhook** → on push to `main`, VPS auto-pulls and restarts Docker containers (replaces manual `git pull`)

---

## Gitea / Docker Compose Deployment Note

Your local Docker Compose setup and VPS production setup are the same codebase — this is correct and will continue to work fine. A few tips:

- Use a `.env.production` and `.env.development` file, never commit either
- Your `docker-compose.yml` should reference `${ENV_VAR}` from the env file
- The Gitea webhook for auto-deploy is a simple shell script triggered by the webhook:
  ```bash
  #!/bin/bash
  cd /path/to/project
  git pull origin main
  docker compose up -d --build
  ```
- Protect this webhook endpoint with a secret token

---

## Summary — What Gets Killed

| Old Way | Replaced By |
|---|---|
| Arduino IDE | PlatformIO (VS Code) |
| Manual `mosquitto_passwd` via SSH | FastAPI dynamic auth endpoints |
| Flutter admin app | React Admin Console |
| Manual SN generation | Console batch creator |
| Manual DB entry per device | Auto-provisioned on batch creation |
| Manual firmware flash + config page | Browser provisioning wizard (esptool-js) |
| Manual NVS entry via HTTP config page | Pre-flashed NVS partition |

## Estimated Time Per Device (After All Phases Complete)

| Task | Time |
|---|---|
| Generate 15-device batch + print labels | ~2 min |
| Flash each device (plug in, click Flash, done) | ~3 min each (parallelizable) |
| Devices self-verify on lab WiFi | passive, ~1 min each |
| **Total for 15 devices** | **~20-25 min** |

vs. current ~20 min per device = ~5 hours for 15.