feat: Migrate firmware to PlatformIO (ESP32-S3, vesper-v1 env)
Replaces Arduino IDE with PlatformIO as the build system. Entry point moved from vesper.ino to src/main.cpp. All library dependencies are now declared in platformio.ini and downloaded automatically via lib_deps. Board: Kincony KC868-A6 (ESP32-S3, 4MB flash) → env:vesper-v1 Future variants Vesper+ and Vesper Pro are pre-configured but commented out. Compatibility fixes applied for this framework version: - Removed ETH.h dependency (Ethernet was disabled in v138) - Watchdog init updated to IDF v4 API (esp_task_wdt_init signature) - ETH.linkUp() check removed from OTAManager Also adds .pio/ to .gitignore and commits the manufacturing plan docs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
463
vesper/documentation/project-vesper-plan.md
Normal file
463
vesper/documentation/project-vesper-plan.md
Normal file
@@ -0,0 +1,463 @@
|
||||
# Project Vesper — Manufacturing Automation Master Plan
|
||||
|
||||
> **How to use this document:** Work through each Phase in order. Each Phase has self-contained tasks you can hand directly to Claude Code. Phases 1–3 are foundational; don't skip ahead. Phases 4–6 build on top of them.
|
||||
|
||||
---
|
||||
|
||||
## Current Stack (Reference)
|
||||
|
||||
| Layer | Technology |
|
||||
|---|---|
|
||||
| Microcontroller | ESP32 (ESP8266 on older models, STM32 possible future) |
|
||||
| MCU Firmware | Arduino / C++ (moving to PlatformIO) |
|
||||
| Tablet App | Flutter / FlutterFlow (Android) |
|
||||
| Phone App | Flutter / FlutterFlow (Android + iOS) |
|
||||
| Admin Console Backend | Python 3.11+ / FastAPI |
|
||||
| Admin Console Frontend | React + Tailwind CSS |
|
||||
| Primary Database | Firebase Firestore (via Admin SDK) |
|
||||
| Secondary Database | Local SQLite (MQTT logs, etc.) |
|
||||
| File Storage | Firebase Storage |
|
||||
| MQTT Broker | Mosquitto on VPS |
|
||||
| Web Server | NGINX on VPS (OTA files, reverse proxy) |
|
||||
| Deployment | Gitea → git pull on VPS, Docker Compose locally |
|
||||
|
||||
---
|
||||
|
||||
## Serial Number Format (Locked In)
|
||||
|
||||
```
|
||||
PV-YYMMM-BBTTR-XXXXX
|
||||
|
||||
PV = Project Vesper prefix
|
||||
YY = 2-digit year (e.g. 26)
|
||||
MMM = 3-char date block = 2-digit month letter + 2-digit day (e.g. A18 = Jan 18)
|
||||
BB = Board Type code (e.g. BC = BellCore, BP = BellPRO)
|
||||
TT = Board Type revision (e.g. 02)
|
||||
R = Literal "R"
|
||||
XXXXX = 5-char random suffix (A-Z, 0-9, excluding 0/O/1/I for label clarity)
|
||||
|
||||
Example: PV-26A18-BC02R-X7KQA
|
||||
```
|
||||
|
||||
**Month Letter Codes:**
|
||||
A=Jan, B=Feb, C=Mar, D=Apr, E=May, F=Jun, G=Jul, H=Aug, I=Sep, J=Oct, K=Nov, L=Dec
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — PlatformIO Migration (Firmware Side)
|
||||
|
||||
> **Goal:** Replace Arduino IDE with PlatformIO. All future firmware work happens here. This unlocks scripted builds and the ability to produce `.bin` files on demand.
|
||||
>
|
||||
> **Tell Claude Code:** *"Help me migrate my existing Arduino IDE ESP32 project to PlatformIO with multiple board environments."*
|
||||
|
||||
### Tasks
|
||||
|
||||
- [ ] Install PlatformIO extension in VS Code
|
||||
- [ ] Create `platformio.ini` with one `[env]` block per hardware variant. Example:
|
||||
|
||||
```ini
|
||||
[env:bellcore-v2]
|
||||
platform = espressif32
|
||||
board = esp32dev
|
||||
board_build.partitions = partitions/custom_4mb.csv
|
||||
board_build.flash_mode = dio
|
||||
board_upload.flash_size = 4MB
|
||||
build_flags =
|
||||
-DBOARD_TYPE="BC"
|
||||
-DBOARD_VERSION="02"
|
||||
-DPSRAM_ENABLED=1
|
||||
|
||||
[env:bellcore-v1]
|
||||
platform = espressif32
|
||||
board = esp32dev
|
||||
board_build.partitions = partitions/custom_2mb.csv
|
||||
board_build.flash_mode = dout
|
||||
board_upload.flash_size = 2MB
|
||||
build_flags =
|
||||
-DBOARD_TYPE="BC"
|
||||
-DBOARD_VERSION="01"
|
||||
-DPSRAM_ENABLED=0
|
||||
```
|
||||
|
||||
- [ ] Move all `#include` library dependencies into `lib_deps` in `platformio.ini` (no more manual library manager)
|
||||
- [ ] Verify `pio run -e bellcore-v2` compiles clean
|
||||
- [ ] Confirm `.pio/build/bellcore-v2/firmware.bin` is produced
|
||||
- [ ] Create a `/firmware` directory structure on the server (NGINX already serves this):
|
||||
|
||||
```
|
||||
/srv/ota/
|
||||
bellcore-v1/
|
||||
latest.bin
|
||||
v1.0.0.bin
|
||||
v1.0.1.bin
|
||||
bellcore-v2/
|
||||
latest.bin
|
||||
...
|
||||
```
|
||||
|
||||
- [ ] Update your OTA logic in firmware to pull from `/ota/{board_type}/latest.bin`
|
||||
- [ ] Add a `scripts/build_and_upload.sh` that compiles + copies the new `.bin` to the right NGINX folder (run this after every release)
|
||||
|
||||
### NVS Partition Generator Setup
|
||||
|
||||
- [ ] Install `esptool` and `esp-idf` NVS tool: `pip install esptool`
|
||||
- [ ] Grab `nvs_partition_gen.py` from ESP-IDF tools (or install via `idf-component-manager`)
|
||||
- [ ] Test generating a `.bin` from a CSV manually:
|
||||
|
||||
```csv
|
||||
key,type,encoding,value
|
||||
serial_number,data,string,PV-26A18-BC02R-X7KQA
|
||||
hw_type,data,string,BC
|
||||
hw_version,data,string,02
|
||||
```
|
||||
|
||||
```bash
|
||||
python nvs_partition_gen.py generate nvs_data.csv nvs_data.bin 0x6000
|
||||
```
|
||||
|
||||
- [ ] Confirm the ESP32 reads NVS values correctly on boot with this pre-flashed partition
|
||||
- [ ] Note the NVS partition address for your board (check your partition table CSV — typically `0x9000`)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — MQTT Dynamic Auth (Backend Side)
|
||||
|
||||
> **Goal:** Replace `mosquitto_passwd` manual SSH with automatic credential management. New devices are live on MQTT the moment they exist in your database. Per-device topic isolation enforced automatically.
|
||||
>
|
||||
> **Tell Claude Code:** *"Help me set up mosquitto-go-auth on my VPS with a FastAPI backend for dynamic MQTT authentication and ACL enforcement."*
|
||||
|
||||
### Tasks
|
||||
|
||||
#### 2a. Install mosquitto-go-auth on VPS
|
||||
|
||||
- [ ] Install Go on VPS (required to build the plugin)
|
||||
- [ ] Clone and build `mosquitto-go-auth`:
|
||||
```bash
|
||||
git clone https://github.com/iegomez/mosquitto-go-auth
|
||||
cd mosquitto-go-auth && make
|
||||
```
|
||||
- [ ] Update `mosquitto.conf` to load the plugin:
|
||||
```
|
||||
auth_plugin /path/to/go-auth.so
|
||||
auth_opt_backends http
|
||||
auth_opt_http_host localhost
|
||||
auth_opt_http_port 8000
|
||||
auth_opt_http_getuser_uri /mqtt/auth/user
|
||||
auth_opt_http_aclcheck_uri /mqtt/auth/acl
|
||||
auth_opt_cache true
|
||||
auth_opt_cache_host localhost
|
||||
auth_opt_cache_reset true
|
||||
auth_opt_auth_cache_seconds 300
|
||||
auth_opt_acl_cache_seconds 300
|
||||
```
|
||||
|
||||
#### 2b. Add MQTT Auth Endpoints to FastAPI
|
||||
|
||||
- [ ] Create `/mqtt/auth/user` endpoint — Mosquitto calls this on CONNECT:
|
||||
- Receives: `username` (= device SN), `password`
|
||||
- Checks Firestore/SQLite for device record + hashed password
|
||||
- Returns: `200` (allow) or `403` (deny)
|
||||
|
||||
- [ ] Create `/mqtt/auth/acl` endpoint — Mosquitto calls this on SUBSCRIBE/PUBLISH:
|
||||
- Receives: `username`, `topic`, `acc` (1=sub, 2=pub)
|
||||
- Rule: username must match the SN segment in the topic
|
||||
- Topic pattern: `/vesper/{SN}/data` or `/vesper/{SN}/control`
|
||||
- Extract `{SN}` from topic, compare to `username`
|
||||
- Returns: `200` or `403`
|
||||
|
||||
- [ ] **For user phone app clients:** Add a separate user auth flow
|
||||
- Users authenticate with their Firebase UID as MQTT username
|
||||
- ACL check: look up which devices this UID owns in Firestore, permit only those SNs in topics
|
||||
|
||||
#### 2c. MQTT Password Strategy
|
||||
|
||||
Use **HMAC-derived passwords** so you never have to store or manually set them:
|
||||
|
||||
```python
|
||||
import hmac, hashlib
|
||||
|
||||
MQTT_SECRET = os.getenv("MQTT_SECRET") # Keep in .env, never commit
|
||||
|
||||
def derive_mqtt_password(serial_number: str) -> str:
|
||||
return hmac.new(
|
||||
MQTT_SECRET.encode(),
|
||||
serial_number.encode(),
|
||||
hashlib.sha256
|
||||
).hexdigest()[:32]
|
||||
```
|
||||
|
||||
- Device SN is known → password is deterministic → firmware can compute it at boot
|
||||
- No password storage needed in DB (just re-derive on auth check)
|
||||
- Changing `MQTT_SECRET` rotates all passwords at once if ever needed
|
||||
|
||||
- [ ] Add `MQTT_SECRET` to your `.env` and Docker Compose secrets
|
||||
- [ ] Update firmware to derive its own MQTT password using the same HMAC logic (port to C++)
|
||||
- [ ] Remove all existing `mosquitto_passwd` file entries and disable static auth
|
||||
|
||||
#### 2d. Test
|
||||
|
||||
- [ ] New device connects with correct SN + derived password → allowed
|
||||
- [ ] Device tries to sub/pub on another device's topic → denied (403)
|
||||
- [ ] Wrong password → denied
|
||||
- [ ] Confirm cache is working (check logs, only 1-2 auth calls per session)
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Serial Number & Batch Management in Admin Console
|
||||
|
||||
> **Goal:** SN generation, DB registration, and MQTT credential provisioning all happen in one flow in the React Console. The Flutter admin app is retired.
|
||||
>
|
||||
> **Tell Claude Code:** *"Add a Manufacturing / Batch Management section to our React+FastAPI admin console with the following features..."*
|
||||
|
||||
### Tasks
|
||||
|
||||
#### 3a. Backend — New API Routes in FastAPI
|
||||
|
||||
- [ ] `POST /manufacturing/batch` — Create a new batch:
|
||||
- Input: `board_type`, `board_version`, `quantity`, `subscription_plan`, `available_outputs`
|
||||
- Generate N serial numbers using the `PV-YYMMM-BBTTR-XXXXX` format
|
||||
- Check Firestore for collisions, regenerate if collision found
|
||||
- Write N device documents to Firestore collection `devices`:
|
||||
```json
|
||||
{
|
||||
"serial_number": "PV-26A18-BC02R-X7KQA",
|
||||
"hw_type": "BC",
|
||||
"hw_version": "02",
|
||||
"status": "manufactured",
|
||||
"subscription_plan": "standard",
|
||||
"available_outputs": 8,
|
||||
"created_at": "...",
|
||||
"owner": null,
|
||||
"users_list": []
|
||||
}
|
||||
```
|
||||
- Returns: list of created SNs
|
||||
|
||||
- [ ] `GET /manufacturing/batch/{batch_id}` — List devices in a batch with status
|
||||
- [ ] `GET /manufacturing/devices` — List all devices with filters (status, hw_type, date range)
|
||||
- [ ] `POST /manufacturing/devices/{sn}/assign` — Pre-assign device to a customer email
|
||||
- [ ] `GET /manufacturing/firmware/{hw_type}/{hw_version}` — Return download URL for the correct `.bin`
|
||||
|
||||
#### 3b. SN Generator Utility (Python)
|
||||
|
||||
```python
|
||||
# utils/serial_number.py
|
||||
import random, string
|
||||
from datetime import datetime
|
||||
|
||||
MONTH_CODES = "ABCDEFGHIJKL"
|
||||
SAFE_CHARS = "ABCDEFGHJKLMNPQRSTUVWXYZ23456789" # No 0,O,1,I
|
||||
|
||||
def generate_serial(board_type: str, board_version: str) -> str:
|
||||
now = datetime.utcnow()
|
||||
year = now.strftime("%y")
|
||||
month = MONTH_CODES[now.month - 1]
|
||||
day = now.strftime("%d")
|
||||
random_suffix = "".join(random.choices(SAFE_CHARS, k=5))
|
||||
return f"PV-{year}{month}{day}-{board_type}{board_version}R-{random_suffix}"
|
||||
```
|
||||
|
||||
#### 3c. NVS Binary Generation in FastAPI
|
||||
|
||||
- [ ] Copy `nvs_partition_gen.py` into your FastAPI project
|
||||
- [ ] Add endpoint `GET /manufacturing/devices/{sn}/nvs.bin`:
|
||||
- Generates a temp CSV for this SN
|
||||
- Runs `nvs_partition_gen.py` to produce the `.bin`
|
||||
- Returns the binary file as a download
|
||||
- [ ] Add endpoint `GET /manufacturing/devices/{sn}/firmware.bin`:
|
||||
- Looks up device's `hw_type` and `hw_version` from Firestore
|
||||
- Returns the correct firmware `.bin` from the NGINX folder (or redirects to NGINX URL)
|
||||
|
||||
#### 3d. Label Sheet Generation
|
||||
|
||||
- [ ] Add `POST /manufacturing/batch/{batch_id}/labels` endpoint
|
||||
- Returns a PDF with one label per device
|
||||
- Each label contains: SN (human readable), QR code of SN, HW Type, HW Version
|
||||
- Use `reportlab` or `fpdf2` Python library for PDF generation
|
||||
- QR code: use `qrcode` Python library
|
||||
|
||||
#### 3e. Frontend — Manufacturing Section in React Console
|
||||
|
||||
- [ ] New route: `/manufacturing`
|
||||
- [ ] **Batch Creator:** Form with board type selector, quantity, subscription plan → calls `POST /manufacturing/batch` → shows created SNs + download label PDF button
|
||||
- [ ] **Device List:** Filterable table of all devices with status badges (manufactured / sold / claimed / active)
|
||||
- [ ] **Device Detail Page:** Shows all fields, allows status update, shows assignment history
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Browser-Based Flashing (The Provisioning Wizard)
|
||||
|
||||
> **Goal:** A single browser tab handles the entire provisioning flow. Plug in ESP32, click through wizard, done. No Arduino IDE, no esptool CLI, no SSH.
|
||||
>
|
||||
> **Tell Claude Code:** *"Add a Device Provisioning Wizard to our React admin console using esptool-js and the Web Serial API."*
|
||||
>
|
||||
> **Browser requirement:** Chrome or Edge only (Web Serial API). Firefox not supported. This is fine for an internal manufacturing tool.
|
||||
|
||||
### Tasks
|
||||
|
||||
#### 4a. Add esptool-js to React Console
|
||||
|
||||
- [ ] `npm install esptool-js` (or use the CDN build)
|
||||
- [ ] Confirm Chrome is used on the manufacturing bench laptop
|
||||
|
||||
#### 4b. Provisioning Wizard UI (React Component)
|
||||
|
||||
Build a step-by-step wizard. Steps:
|
||||
|
||||
**Step 1 — Select or Create Device**
|
||||
- Search existing unprovisioned device by SN, OR
|
||||
- Quick-create single device (calls `POST /manufacturing/batch` with qty=1)
|
||||
- Displays SN, HW Type, HW Version
|
||||
|
||||
**Step 2 — Flash Device**
|
||||
- "Connect Device" button → triggers Web Serial port picker
|
||||
- Fetches `nvs.bin` and `firmware.bin` from your FastAPI backend for this SN
|
||||
- Shows two progress bars: NVS partition flash + Firmware flash
|
||||
- Flash addresses (example for standard ESP32):
|
||||
- NVS: `0x9000` (verify against your partition table)
|
||||
- Firmware: `0x10000`
|
||||
- On completion: updates device status to `flashed` in Firestore via API call
|
||||
|
||||
**Step 3 — Verify**
|
||||
- Prompt: "Power cycle device and wait for it to connect"
|
||||
- Poll Firestore (or MQTT) for first heartbeat/connection from this SN
|
||||
- Show green checkmark when device phone home
|
||||
- Updates status to `provisioned`
|
||||
|
||||
**Step 4 — Done**
|
||||
- Show summary
|
||||
- Option: "Provision next device" (loops back to Step 1 with same batch settings)
|
||||
- Option: "Print label" (downloads single-device PDF label)
|
||||
|
||||
#### 4c. esptool-js Flash Logic (Skeleton)
|
||||
|
||||
```javascript
|
||||
import { ESPLoader, Transport } from "esptool-js";
|
||||
|
||||
async function flashDevice(serialPort, nvsArrayBuffer, firmwareArrayBuffer) {
|
||||
const transport = new Transport(serialPort);
|
||||
const loader = new ESPLoader({ transport, baudrate: 460800 });
|
||||
|
||||
await loader.main_fn();
|
||||
await loader.flash_id();
|
||||
|
||||
await loader.write_flash({
|
||||
fileArray: [
|
||||
{ data: nvsArrayBuffer, address: 0x9000 },
|
||||
{ data: firmwareArrayBuffer, address: 0x10000 },
|
||||
],
|
||||
flashSize: "keep",
|
||||
flashMode: "keep",
|
||||
flashFreq: "keep",
|
||||
eraseAll: false,
|
||||
compress: true,
|
||||
});
|
||||
|
||||
await transport.disconnect();
|
||||
}
|
||||
```
|
||||
|
||||
#### 4d. NGINX CORS Headers
|
||||
|
||||
Add to your NGINX config so the browser can fetch `.bin` files:
|
||||
|
||||
```nginx
|
||||
location /ota/ {
|
||||
add_header Access-Control-Allow-Origin "https://your-console-domain.com";
|
||||
add_header Access-Control-Allow-Methods "GET";
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Email Notifications
|
||||
|
||||
> **Goal:** Admin Console can send transactional emails (device assignment invites, alerts, etc.)
|
||||
>
|
||||
> **Tell Claude Code:** *"Add email sending capability to our FastAPI backend using Resend (or SMTP)."*
|
||||
|
||||
### Tasks
|
||||
|
||||
- [ ] Sign up for [Resend](https://resend.com) (free tier: 3000 emails/month, 100/day)
|
||||
- [ ] Add `RESEND_API_KEY` to `.env`
|
||||
- [ ] Install: `pip install resend`
|
||||
- [ ] Create `utils/email.py`:
|
||||
|
||||
```python
|
||||
import resend
|
||||
import os
|
||||
|
||||
resend.api_key = os.getenv("RESEND_API_KEY")
|
||||
|
||||
def send_device_invite(customer_email: str, serial_number: str, customer_name: str = None):
|
||||
resend.Emails.send({
|
||||
"from": "noreply@yourcompany.com",
|
||||
"to": customer_email,
|
||||
"subject": "Your Vesper device is ready",
|
||||
"html": f"""
|
||||
<h2>Your device has been registered</h2>
|
||||
<p>Serial Number: <strong>{serial_number}</strong></p>
|
||||
<p>Open the Vesper app and enter this serial number to get started.</p>
|
||||
"""
|
||||
})
|
||||
```
|
||||
|
||||
- [ ] Hook into `POST /manufacturing/devices/{sn}/assign` to send invite automatically
|
||||
- [ ] Add basic email templates for: device assignment, welcome, error alerts
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Polish & Retire Legacy Tools
|
||||
|
||||
> **Goal:** Clean up. Everything lives in the Console. Nothing is done manually.
|
||||
|
||||
### Tasks
|
||||
|
||||
- [ ] **Retire Flutter admin app** — confirm every function it had is now in the React Console
|
||||
- [ ] **Remove static mosquitto password file** — all auth is dynamic now
|
||||
- [ ] **Add device status dashboard** to Console home: counts by status, recent provisioning activity
|
||||
- [ ] **Add audit log** — every manufacturing action (batch created, device flashed, device assigned) logged to SQLite with timestamp and admin user
|
||||
- [ ] **Document your `platformio.ini` environments** — add a `FIRMWARE_VARIANTS.md` to the firmware repo
|
||||
- [ ] **Set up Gitea webhook** → on push to `main`, VPS auto-pulls and restarts Docker containers (replaces manual `git pull`)
|
||||
|
||||
---
|
||||
|
||||
## Gitea / Docker Compose Deployment Note
|
||||
|
||||
Your local Docker Compose setup and VPS production setup are the same codebase — this is correct and will continue to work fine. A few tips:
|
||||
|
||||
- Use a `.env.production` and `.env.development` file, never commit either
|
||||
- Your `docker-compose.yml` should reference `${ENV_VAR}` from the env file
|
||||
- The Gitea webhook for auto-deploy is a simple shell script triggered by the webhook:
|
||||
```bash
|
||||
#!/bin/bash
|
||||
cd /path/to/project
|
||||
git pull origin main
|
||||
docker compose up -d --build
|
||||
```
|
||||
- Protect this webhook endpoint with a secret token
|
||||
|
||||
---
|
||||
|
||||
## Summary — What Gets Killed
|
||||
|
||||
| Old Way | Replaced By |
|
||||
|---|---|
|
||||
| Arduino IDE | PlatformIO (VS Code) |
|
||||
| Manual `mosquitto_passwd` via SSH | FastAPI dynamic auth endpoints |
|
||||
| Flutter admin app | React Admin Console |
|
||||
| Manual SN generation | Console batch creator |
|
||||
| Manual DB entry per device | Auto-provisioned on batch creation |
|
||||
| Manual firmware flash + config page | Browser provisioning wizard (esptool-js) |
|
||||
| Manual NVS entry via HTTP config page | Pre-flashed NVS partition |
|
||||
|
||||
## Estimated Time Per Device (After All Phases Complete)
|
||||
|
||||
| Task | Time |
|
||||
|---|---|
|
||||
| Generate 15-device batch + print labels | ~2 min |
|
||||
| Flash each device (plug in, click Flash, done) | ~3 min each (parallelizable) |
|
||||
| Devices self-verify on lab WiFi | passive, ~1 min each |
|
||||
| **Total for 15 devices** | **~20-25 min** |
|
||||
|
||||
vs. current ~20 min per device = ~5 hours for 15.
|
||||
Reference in New Issue
Block a user