bonamin/bellsystems-cp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: bonamin:main
Branches Tags
bonamin:main
..
compare: bonamin:b280d62ee52477769a8fec28ba5c8efbe6dcdf5f
Branches Tags
bonamin:main

1 Commits
main ... b280d62ee5

Author SHA1 Message Date
bonamin
b280d62ee5 update: Major Overhault to all subsystems 2026-03-07 11:32:18 +02:00
635 changed files with 7484 additions and 102557 deletions
Expand all files Collapse all files

153
.claude/backend-mqtt-alerts-prompt.md Normal file
View File

@@ -0,0 +1,153 @@
# Backend Task: Subscribe to Vesper MQTT Alert Topics
> Use this document as a prompt / task brief for implementing the backend side
> of the Vesper MQTT alert system. The firmware changes are complete.
> Full topic spec: `docs/reference/mqtt-events.md`
---
## What the firmware now publishes
The Vesper firmware (v155+) publishes on three status topics:
### 1. `vesper/{device_id}/status/heartbeat` (unchanged)
- Every 30 seconds, retained, QoS 1
- You already handle this — **no change needed** except: suppress any log entry / display update triggered by heartbeat arrival. Update `last_seen` silently. Only surface an event when the device goes *silent* (no heartbeat for 90s).
### 2. `vesper/{device_id}/status/alerts` (NEW)
- Published only when a subsystem state changes (HEALTHY → WARNING, WARNING → CRITICAL, etc.)
- QoS 1, not retained
- One message per state transition — not repeated until state changes again
**Alert payload:**
```json
{ "subsystem": "FileManager", "state": "WARNING", "msg": "ConfigManager health check failed" }
```
**Cleared payload (recovery):**
```json
{ "subsystem": "FileManager", "state": "CLEARED" }
```
### 3. `vesper/{device_id}/status/info` (NEW)
- Published on significant device state changes (playback start/stop, etc.)
- QoS 0, not retained
```json
{ "type": "playback_started", "payload": { "melody_uid": "ABC123" } }
```
---
## What to implement in the backend (FastAPI + MQTT)
### Subscribe to new topics
Add to your MQTT subscription list:
```python
client.subscribe("vesper/+/status/alerts", qos=1)
client.subscribe("vesper/+/status/info", qos=0)
```
### Database model — active alerts per device
Create a table (or document) to store the current alert state per device:
```sql
CREATE TABLE device_alerts (
device_id TEXT NOT NULL,
subsystem TEXT NOT NULL,
state TEXT NOT NULL, -- WARNING | CRITICAL | FAILED
message TEXT,
updated_at TIMESTAMP NOT NULL,
PRIMARY KEY (device_id, subsystem)
);
```
Or equivalent in your ORM / MongoDB / Redis structure.
### MQTT message handler — alerts topic
```python
def on_alerts_message(device_id: str, payload: dict):
subsystem = payload["subsystem"]
state = payload["state"]
message = payload.get("msg", "")
if state == "CLEARED":
# Remove alert from active set
db.device_alerts.delete(device_id=device_id, subsystem=subsystem)
else:
# Upsert — create or update
db.device_alerts.upsert(
device_id = device_id,
subsystem = subsystem,
state = state,
message = message,
updated_at = now()
)
# Optionally push a WebSocket event to the console UI
ws_broadcast(device_id, {"event": "alert_update", "subsystem": subsystem, "state": state})
```
### MQTT message handler — info topic
```python
def on_info_message(device_id: str, payload: dict):
event_type = payload["type"]
data = payload.get("payload", {})
# Store or forward as needed — e.g. update device playback state
if event_type == "playback_started":
db.devices.update(device_id, playback_active=True, melody_uid=data.get("melody_uid"))
elif event_type == "playback_stopped":
db.devices.update(device_id, playback_active=False, melody_uid=None)
```
### API endpoint — get active alerts for a device
```
GET /api/devices/{device_id}/alerts
```
Returns the current active alert set (the upserted rows from the table above):
```json
[
{ "subsystem": "FileManager", "state": "WARNING", "message": "SD mount failed", "updated_at": "..." },
{ "subsystem": "TimeKeeper", "state": "WARNING", "message": "NTP sync failed", "updated_at": "..." }
]
```
An empty array means the device is fully healthy (no active alerts).
### Console UI guidance
- Device list: show a coloured dot next to each device (green = no alerts, yellow = warnings, red = critical/failed). Update via WebSocket push.
- Device detail page: show an "Active Alerts" section that renders the alert set statically. Do not render a scrolling alert log — just the current state.
- When a `CLEARED` event arrives, remove the entry from the UI immediately.
---
## What NOT to do
- **Do not log every heartbeat** as a visible event. Heartbeats are internal housekeeping.
- **Do not poll the device** for health status — the device pushes on change.
- **Do not store alerts as an append-only log** — upsert by `(device_id, subsystem)`. The server holds the current state, not a history.
---
## Testing
1. Flash a device with firmware v155+
2. Subscribe manually:
```bash
mosquitto_sub -h <broker> -t "vesper/+/status/alerts" -v
mosquitto_sub -h <broker> -t "vesper/+/status/info" -v
```
3. Remove the SD card from the device — expect a `FileManager` `WARNING` alert within 5 minutes (next health check cycle), or trigger it immediately via:
```json
{ "v": 2, "cmd": "system.health" }
```
sent to `vesper/{device_id}/control`
4. Reinsert the SD card — expect a `FileManager` `CLEARED` alert on the next health check

243
.claude/crm-build-plan.md Normal file
View File

@@ -0,0 +1,243 @@
# BellSystems CRM — Build Plan & Step Prompts
## Overview
A bespoke CRM module built directly into the existing BellSystems web console.
Stack: FastAPI backend (Firestore), React + Vite frontend.
No new auth — uses the existing JWT + permission system.
No file storage on VPS — all media lives on Nextcloud via WebDAV.
---
## Architecture Summary
### Backend
- New module: `backend/crm/` with `models.py`, `service.py`, `router.py`
- Firestore collections: `crm_customers`, `crm_orders`, `crm_products`
- SQLite (existing `mqtt_data.db`) for comms_log (high-write, queryable)
- Router registered in `backend/main.py` as `/api/crm`
### Frontend
- New section: `frontend/src/crm/`
- Routes added to `frontend/src/App.jsx`
- Nav entries added to `frontend/src/layout/Sidebar.jsx`
### Integrations (later steps)
- Nextcloud: WebDAV via `httpx` in backend
- Email: IMAP (read) + SMTP (send) via `imaplib` / `smtplib`
- WhatsApp: Meta Cloud API webhook
- FreePBX: Asterisk AMI socket listener
---
## Data Model Reference
### `crm_customers` (Firestore)
```json
{
"id": "auto",
"name": "Στέλιος Μπιμπης",
"organization": "Ενορία Αγ. Παρασκευής",
"contacts": [
{ "type": "email", "label": "personal", "value": "...", "primary": true },
{ "type": "phone", "label": "mobile", "value": "...", "primary": true }
],
"notes": [
{ "text": "...", "by": "user_name", "at": "ISO datetime" }
],
"location": { "city": "", "country": "", "region": "" },
"language": "el",
"tags": [],
"owned_items": [
{ "type": "console_device", "device_id": "UID", "label": "..." },
{ "type": "product", "product_id": "pid", "product_name": "...", "quantity": 1, "serial_numbers": [] },
{ "type": "freetext", "description": "...", "serial_number": "", "notes": "" }
],
"linked_user_ids": [],
"nextcloud_folder": "05_Customers/FOLDER_NAME",
"created_at": "ISO",
"updated_at": "ISO"
}
```
### `crm_orders` (Firestore)
```json
{
"id": "auto",
"customer_id": "ref",
"order_number": "ORD-2026-001",
"status": "draft",
"items": [
{
"type": "console_device|product|freetext",
"product_id": "",
"product_name": "",
"description": "",
"quantity": 1,
"unit_price": 0.0,
"serial_numbers": []
}
],
"subtotal": 0.0,
"discount": { "type": "percentage|fixed", "value": 0, "reason": "" },
"total_price": 0.0,
"currency": "EUR",
"shipping": {
"method": "",
"tracking_number": "",
"carrier": "",
"shipped_at": null,
"delivered_at": null,
"destination": ""
},
"payment_status": "pending",
"invoice_path": "",
"notes": "",
"created_at": "ISO",
"updated_at": "ISO"
}
```
### `crm_products` (Firestore)
```json
{
"id": "auto",
"name": "Vesper Plus",
"sku": "VSP-001",
"category": "controller|striker|clock|part|repair_service",
"description": "",
"price": 0.0,
"currency": "EUR",
"costs": {
"pcb": 0.0, "components": 0.0, "enclosure": 0.0,
"labor_hours": 0, "labor_rate": 0.0, "shipping_in": 0.0,
"total": 0.0
},
"stock": { "on_hand": 0, "reserved": 0, "available": 0 },
"nextcloud_folder": "02_Products/FOLDER",
"linked_device_type": "",
"active": true,
"created_at": "ISO",
"updated_at": "ISO"
}
```
### `crm_comms_log` (SQLite table — existing mqtt_data.db)
```sql
CREATE TABLE crm_comms_log (
id TEXT PRIMARY KEY,
customer_id TEXT NOT NULL,
type TEXT NOT NULL, -- email|whatsapp|call|sms|note|in_person
direction TEXT NOT NULL, -- inbound|outbound|internal
subject TEXT,
body TEXT,
attachments TEXT, -- JSON array of {filename, nextcloud_path}
ext_message_id TEXT, -- IMAP uid, WhatsApp msg id, AMI call id
logged_by TEXT,
occurred_at TEXT NOT NULL,
created_at TEXT NOT NULL
);
```
### `crm_media` (SQLite table — existing mqtt_data.db)
```sql
CREATE TABLE crm_media (
id TEXT PRIMARY KEY,
customer_id TEXT,
order_id TEXT,
filename TEXT NOT NULL,
nextcloud_path TEXT NOT NULL,
mime_type TEXT,
direction TEXT, -- received|sent|internal
tags TEXT, -- JSON array
uploaded_by TEXT,
created_at TEXT NOT NULL
);
```
---
## IMPORTANT NOTES FOR ALL STEPS
- **Backend location**: `c:\development\bellsystems-cp\backend\`
- **Frontend location**: `c:\development\bellsystems-cp\frontend\`
- **Auth pattern**: All routes use `Depends(require_permission("crm", "view"))` or `"edit"`. Import from `auth.dependencies`.
- **Firestore pattern**: Use `from shared.firebase import get_db`. See `backend/devices/service.py` for reference patterns.
- **SQLite pattern**: Use `from mqtt import database as mqtt_db` — `mqtt_db.db` is the aiosqlite connection. See `backend/mqtt/database.py`.
- **Frontend auth**: `getAuthHeaders()` from `../api/auth` gives Bearer token headers. See any existing page for pattern.
- **Frontend routing**: Routes live in `frontend/src/App.jsx`. Sidebar nav in `frontend/src/layout/Sidebar.jsx`.
- **Token**: localStorage key is `"access_token"`.
- **UI pattern**: Use existing component style — `SectionCard`, `FieldRow`, inline styles for grids. See `frontend/src/devices/` for reference.
- **No new dependencies unless absolutely necessary.**
---
## Step 1 — Backend: CRM Module Scaffold + Products CRUD
**File**: `.claude/crm-step-01.md`
---
## Step 2 — Backend: Customers CRUD
**File**: `.claude/crm-step-02.md`
---
## Step 3 — Backend: Orders CRUD
**File**: `.claude/crm-step-03.md`
---
## Step 4 — Backend: Comms Log + Media (SQLite)
**File**: `.claude/crm-step-04.md`
---
## Step 5 — Frontend: Products Module
**File**: `.claude/crm-step-05.md`
---
## Step 6 — Frontend: Customers List + Detail Page
**File**: `.claude/crm-step-06.md`
---
## Step 7 — Frontend: Orders Module
**File**: `.claude/crm-step-07.md`
---
## Step 8 — Frontend: Comms Log + Media Tab (manual entry)
**File**: `.claude/crm-step-08.md`
---
## Step 9 — Integration: Nextcloud WebDAV
**File**: `.claude/crm-step-09.md`
---
## Step 10 — Integration: IMAP/SMTP Email
**File**: `.claude/crm-step-10.md`
---
## Step 11 — Integration: WhatsApp Business API
**File**: `.claude/crm-step-11.md`
---
## Step 12 — Integration: FreePBX AMI Call Logging
**File**: `.claude/crm-step-12.md`

49
.claude/crm-step-01.md Normal file
View File

@@ -0,0 +1,49 @@
# CRM Step 01 — Backend: Module Scaffold + Products CRUD
## Context
Read `.claude/crm-build-plan.md` first for full data models, conventions, and IMPORTANT NOTES.
## Task
Create the `backend/crm/` module with Products CRUD. This is the first CRM backend step.
## What to build
### 1. `backend/crm/__init__.py` — empty
### 2. `backend/crm/models.py`
Pydantic models for Products:
- `ProductCosts` — pcb, components, enclosure, labor_hours, labor_rate, shipping_in, total (all float/int, all optional)
- `ProductStock` — on_hand, reserved, available (int, defaults 0)
- `ProductCategory` enum — controller, striker, clock, part, repair_service
- `ProductCreate` — name, sku (optional), category, description (optional), price (float), currency (default "EUR"), costs (ProductCosts optional), stock (ProductStock optional), nextcloud_folder (optional), linked_device_type (optional), active (bool default True)
- `ProductUpdate` — all fields Optional
- `ProductInDB` — extends ProductCreate + id (str), created_at (str), updated_at (str)
- `ProductListResponse` — products: List[ProductInDB], total: int
### 3. `backend/crm/service.py`
Firestore collection: `crm_products`
Functions:
- `list_products(search=None, category=None, active_only=False) -> List[ProductInDB]`
- `get_product(product_id) -> ProductInDB` — raises HTTPException 404 if not found
- `create_product(data: ProductCreate) -> ProductInDB` — generates UUID id, sets created_at/updated_at to ISO now
- `update_product(product_id, data: ProductUpdate) -> ProductInDB` — partial update (only set fields), updates updated_at
- `delete_product(product_id) -> None` — raises 404 if not found
### 4. `backend/crm/router.py`
Prefix: `/api/crm/products`, tag: `crm-products`
All routes require `require_permission("crm", "view")` for GET, `require_permission("crm", "edit")` for POST/PUT/DELETE.
- `GET /` → list_products (query params: search, category, active_only)
- `GET /{product_id}` → get_product
- `POST /` → create_product
- `PUT /{product_id}` → update_product
- `DELETE /{product_id}` → delete_product
### 5. Register in `backend/main.py`
Add: `from crm.router import router as crm_products_router`
Add: `app.include_router(crm_products_router)` (after existing routers)
## Notes
- Use `uuid.uuid4()` for IDs
- Use `datetime.utcnow().isoformat()` for timestamps
- Follow exact Firestore pattern from `backend/devices/service.py`
- No new pip dependencies needed

61
.claude/crm-step-02.md Normal file
View File

@@ -0,0 +1,61 @@
# CRM Step 02 — Backend: Customers CRUD
## Context
Read `.claude/crm-build-plan.md` first for full data models, conventions, and IMPORTANT NOTES.
Step 01 must be complete (`backend/crm/` module exists).
## Task
Add Customers models, service, and router to `backend/crm/`.
## What to build
### 1. Add to `backend/crm/models.py`
**Contact entry:**
- `ContactType` enum — email, phone, whatsapp, other
- `CustomerContact` — type (ContactType), label (str, e.g. "personal"/"church"), value (str), primary (bool default False)
**Note entry:**
- `CustomerNote` — text (str), by (str), at (str ISO datetime)
**Owned items (3 tiers):**
- `OwnedItemType` enum — console_device, product, freetext
- `OwnedItem`:
- type: OwnedItemType
- For console_device: device_id (Optional[str]), label (Optional[str])
- For product: product_id (Optional[str]), product_name (Optional[str]), quantity (Optional[int]), serial_numbers (Optional[List[str]])
- For freetext: description (Optional[str]), serial_number (Optional[str]), notes (Optional[str])
**Location:**
- `CustomerLocation` — city (Optional[str]), country (Optional[str]), region (Optional[str])
**Customer models:**
- `CustomerCreate` — name (str), organization (Optional[str]), contacts (List[CustomerContact] default []), notes (List[CustomerNote] default []), location (Optional[CustomerLocation]), language (str default "el"), tags (List[str] default []), owned_items (List[OwnedItem] default []), linked_user_ids (List[str] default []), nextcloud_folder (Optional[str])
- `CustomerUpdate` — all fields Optional
- `CustomerInDB` — extends CustomerCreate + id, created_at, updated_at
- `CustomerListResponse` — customers: List[CustomerInDB], total: int
### 2. Add to `backend/crm/service.py`
Firestore collection: `crm_customers`
Functions:
- `list_customers(search=None, tag=None) -> List[CustomerInDB]`
- search matches against name, organization, and any contact value
- `get_customer(customer_id) -> CustomerInDB` — 404 if not found
- `create_customer(data: CustomerCreate) -> CustomerInDB`
- `update_customer(customer_id, data: CustomerUpdate) -> CustomerInDB`
- `delete_customer(customer_id) -> None`
### 3. Add to `backend/crm/router.py`
Add a second router or extend existing file with prefix `/api/crm/customers`:
- `GET /` — list_customers (query: search, tag)
- `GET /{customer_id}` — get_customer
- `POST /` — create_customer
- `PUT /{customer_id}` — update_customer
- `DELETE /{customer_id}` — delete_customer
Register this router in `backend/main.py` alongside the products router.
## Notes
- OwnedItem is a flexible struct — store all fields, service doesn't validate which fields are relevant per type (frontend handles that)
- linked_user_ids are Firebase Auth UIDs (strings) — no validation needed here, just store them
- Search in list_customers: do client-side filter after fetching all (small dataset)

60
.claude/crm-step-03.md Normal file
View File

@@ -0,0 +1,60 @@
# CRM Step 03 — Backend: Orders CRUD
## Context
Read `.claude/crm-build-plan.md` first for full data models, conventions, and IMPORTANT NOTES.
Steps 01 and 02 must be complete.
## Task
Add Orders models, service, and router to `backend/crm/`.
## What to build
### 1. Add to `backend/crm/models.py`
**Enums:**
- `OrderStatus` — draft, confirmed, in_production, shipped, delivered, cancelled
- `PaymentStatus` — pending, partial, paid
**Structs:**
- `OrderDiscount` — type (str: "percentage" | "fixed"), value (float default 0), reason (Optional[str])
- `OrderShipping` — method (Optional[str]), tracking_number (Optional[str]), carrier (Optional[str]), shipped_at (Optional[str]), delivered_at (Optional[str]), destination (Optional[str])
- `OrderItem`:
- type: str (console_device | product | freetext)
- product_id: Optional[str]
- product_name: Optional[str]
- description: Optional[str] ← for freetext items
- quantity: int default 1
- unit_price: float default 0.0
- serial_numbers: List[str] default []
**Order models:**
- `OrderCreate` — customer_id (str), order_number (Optional[str] — auto-generated if not provided), status (OrderStatus default draft), items (List[OrderItem] default []), subtotal (float default 0), discount (Optional[OrderDiscount]), total_price (float default 0), currency (str default "EUR"), shipping (Optional[OrderShipping]), payment_status (PaymentStatus default pending), invoice_path (Optional[str]), notes (Optional[str])
- `OrderUpdate` — all fields Optional
- `OrderInDB` — extends OrderCreate + id, created_at, updated_at
- `OrderListResponse` — orders: List[OrderInDB], total: int
### 2. Add to `backend/crm/service.py`
Firestore collection: `crm_orders`
Auto order number generation: `ORD-{YYYY}-{NNN}` — query existing orders for current year, increment max.
Functions:
- `list_orders(customer_id=None, status=None, payment_status=None) -> List[OrderInDB]`
- `get_order(order_id) -> OrderInDB` — 404 if not found
- `create_order(data: OrderCreate) -> OrderInDB` — auto-generate order_number if not set
- `update_order(order_id, data: OrderUpdate) -> OrderInDB`
- `delete_order(order_id) -> None`
### 3. Add to `backend/crm/router.py`
Prefix `/api/crm/orders`:
- `GET /` — list_orders (query: customer_id, status, payment_status)
- `GET /{order_id}` — get_order
- `POST /` — create_order
- `PUT /{order_id}` — update_order
- `DELETE /{order_id}` — delete_order
Register in `backend/main.py`.
## Notes
- subtotal and total_price are stored as-is (calculated by frontend before POST/PUT). Backend does not recalculate.
- Order number generation doesn't need to be atomic/perfect — just a best-effort sequential label.

96
.claude/crm-step-04.md Normal file
View File

@@ -0,0 +1,96 @@
# CRM Step 04 — Backend: Comms Log + Media (SQLite)
## Context
Read `.claude/crm-build-plan.md` for full schema, conventions, and IMPORTANT NOTES.
Steps 01–03 must be complete.
## Task
Add `crm_comms_log` and `crm_media` tables to the existing SQLite DB, plus CRUD endpoints.
## What to build
### 1. Add tables to `backend/mqtt/database.py`
Inside `init_db()`, add these CREATE TABLE IF NOT EXISTS statements alongside existing tables:
```sql
CREATE TABLE IF NOT EXISTS crm_comms_log (
id TEXT PRIMARY KEY,
customer_id TEXT NOT NULL,
type TEXT NOT NULL,
direction TEXT NOT NULL,
subject TEXT,
body TEXT,
attachments TEXT DEFAULT '[]',
ext_message_id TEXT,
logged_by TEXT,
occurred_at TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS crm_media (
id TEXT PRIMARY KEY,
customer_id TEXT,
order_id TEXT,
filename TEXT NOT NULL,
nextcloud_path TEXT NOT NULL,
mime_type TEXT,
direction TEXT,
tags TEXT DEFAULT '[]',
uploaded_by TEXT,
created_at TEXT NOT NULL
);
```
### 2. Add to `backend/crm/models.py`
**Comms:**
- `CommType` enum — email, whatsapp, call, sms, note, in_person
- `CommDirection` enum — inbound, outbound, internal
- `CommAttachment` — filename (str), nextcloud_path (str)
- `CommCreate` — customer_id, type (CommType), direction (CommDirection), subject (Optional[str]), body (Optional[str]), attachments (List[CommAttachment] default []), ext_message_id (Optional[str]), logged_by (Optional[str]), occurred_at (str ISO — default to now if not provided)
- `CommUpdate` — subject, body, occurred_at all Optional
- `CommInDB` — all fields + id, created_at
- `CommListResponse` — entries: List[CommInDB], total: int
**Media:**
- `MediaDirection` enum — received, sent, internal
- `MediaCreate` — customer_id (Optional[str]), order_id (Optional[str]), filename, nextcloud_path, mime_type (Optional), direction (MediaDirection optional), tags (List[str] default []), uploaded_by (Optional[str])
- `MediaInDB` — all fields + id, created_at
- `MediaListResponse` — items: List[MediaInDB], total: int
### 3. Add to `backend/crm/service.py`
Import `from mqtt import database as mqtt_db` for aiosqlite access.
**Comms functions (all async):**
- `list_comms(customer_id, type=None, direction=None, limit=100) -> List[CommInDB]`
— SELECT ... WHERE customer_id=? ORDER BY occurred_at DESC
- `get_comm(comm_id) -> CommInDB` — 404 if not found
- `create_comm(data: CommCreate) -> CommInDB` — uuid id, created_at now, store attachments as JSON string
- `update_comm(comm_id, data: CommUpdate) -> CommInDB`
- `delete_comm(comm_id) -> None`
**Media functions (all async):**
- `list_media(customer_id=None, order_id=None) -> List[MediaInDB]`
- `create_media(data: MediaCreate) -> MediaInDB`
- `delete_media(media_id) -> None`
Parse `attachments` and `tags` JSON strings back to lists when returning models.
### 4. Add to `backend/crm/router.py`
Prefix `/api/crm/comms`:
- `GET /` — list_comms (query: customer_id required, type, direction)
- `POST /` — create_comm
- `PUT /{comm_id}` — update_comm
- `DELETE /{comm_id}` — delete_comm
Prefix `/api/crm/media`:
- `GET /` — list_media (query: customer_id or order_id)
- `POST /` — create_media (metadata only — no file upload here, that's Step 9)
- `DELETE /{media_id}` — delete_media
Register both in `backend/main.py`.
## Notes
- Use `mqtt_db.db` — it is an aiosqlite connection, use `async with mqtt_db.db.execute(...)` pattern
- Look at `backend/mqtt/database.py` for exact aiosqlite usage pattern
- attachments and tags are stored as JSON strings in SQLite, deserialized to lists in the Pydantic model

55
.claude/crm-step-05.md Normal file
View File

@@ -0,0 +1,55 @@
# CRM Step 05 — Frontend: Products Module
## Context
Read `.claude/crm-build-plan.md` for full context and IMPORTANT NOTES.
Backend Steps 01–04 must be complete and running.
## Task
Build the Products section of the CRM frontend.
## Files to create
### `frontend/src/crm/products/ProductList.jsx`
- Fetch `GET /api/crm/products` with auth headers
- Show a table/list: Name, SKU, Category, Price, Stock (available), Active badge
- Search input (client-side filter on name/sku)
- Filter dropdown for category
- "New Product" button → navigate to `/crm/products/new`
- Row click → navigate to `/crm/products/:id`
### `frontend/src/crm/products/ProductForm.jsx`
Used for both create and edit. Receives `productId` prop (null = create mode).
Fields:
- name (required), sku, category (dropdown from enum), description (textarea)
- price (number), currency (default EUR)
- Costs section (collapsible): pcb, components, enclosure, labor_hours, labor_rate, shipping_in — show computed total
- Stock section: on_hand, reserved — show available = on_hand - reserved (readonly)
- nextcloud_folder, linked_device_type, active (toggle)
- Save / Cancel buttons
- In edit mode: show Delete button with confirmation
On save: POST `/api/crm/products` or PUT `/api/crm/products/:id`
On delete: DELETE `/api/crm/products/:id` then navigate back to list
### `frontend/src/crm/products/index.js`
Export both components.
## Routing
In `frontend/src/App.jsx` add:
```jsx
<Route path="/crm/products" element={<ProductList />} />
<Route path="/crm/products/new" element={<ProductForm />} />
<Route path="/crm/products/:id" element={<ProductForm />} />
```
## Sidebar
In `frontend/src/layout/Sidebar.jsx` add a "CRM" section with:
- Products → `/crm/products`
(Customers and Orders will be added in later steps)
## Notes
- Use existing UI patterns: SectionCard wrapper, inline styles for layout grid
- Follow the same auth header pattern as other frontend modules (getAuthHeaders from `../api/auth` or equivalent)
- Currency is always EUR for now — no need for a selector
- Computed costs total = pcb + components + enclosure + (labor_hours * labor_rate) + shipping_in, shown live as user types
- Category values: controller, striker, clock, part, repair_service — display as human-readable labels

84
.claude/crm-step-06.md Normal file
View File

@@ -0,0 +1,84 @@
# CRM Step 06 — Frontend: Customers List + Detail Page
## Context
Read `.claude/crm-build-plan.md` for full context, data models, and IMPORTANT NOTES.
Backend Steps 01–04 and Frontend Step 05 must be complete.
## Task
Build the Customers section — the core of the CRM.
## Files to create
### `frontend/src/crm/customers/CustomerList.jsx`
- Fetch `GET /api/crm/customers` (query: search, tag)
- Show cards or table rows: Name, Organization, Location, Tags, primary contact
- Search input → query param `search`
- "New Customer" button → `/crm/customers/new`
- Row/card click → `/crm/customers/:id`
### `frontend/src/crm/customers/CustomerForm.jsx`
Create/edit form. Receives `customerId` prop (null = create).
**Sections:**
1. **Basic Info** — name, organization, language, tags (pill input), nextcloud_folder
2. **Location** — city, country, region
3. **Contacts** — dynamic list of `{ type, label, value, primary }` entries. Add/remove rows. Radio to set primary per type group.
4. **Notes** — dynamic list of `{ text, by, at }`. Add new note button. Existing notes shown as read-only with author/date. `by` auto-filled from current user name.
5. **Owned Items** — dynamic list with type selector:
- `console_device`: device_id text input + label
- `product`: product selector (fetch `/api/crm/products` for dropdown) + quantity + serial_numbers (comma-separated input)
- `freetext`: description + serial_number + notes
Add/remove rows.
6. **Linked App Accounts** — list of Firebase UIDs (simple text inputs, add/remove). Label: "Linked App User IDs"
Save: POST or PUT. Delete with confirmation.
### `frontend/src/crm/customers/CustomerDetail.jsx`
The main customer page. Fetches customer by ID. Tab layout:
**Tab 1: Overview**
- Show all info from CustomerForm fields in read-only view
- "Edit" button → opens CustomerForm in a modal or navigates to edit route
**Tab 2: Orders**
- Fetch `GET /api/crm/orders?customer_id=:id`
- List orders: order_number, status badge, total_price, date
- "New Order" button → navigate to `/crm/orders/new?customer_id=:id`
- Row click → `/crm/orders/:id`
**Tab 3: Comms**
- Fetch `GET /api/crm/comms?customer_id=:id`
- Timeline view sorted by occurred_at descending
- Each entry shows: type icon, direction indicator, subject/body preview, date
- "Log Entry" button → inline form to create a new comms entry (type, direction, subject, body, occurred_at)
**Tab 4: Media**
- Fetch `GET /api/crm/media?customer_id=:id`
- Grid of files: filename, direction badge (Received/Sent/Internal), date
- "Add Media Record" button → form with filename, nextcloud_path, direction, tags (manual entry for now — Nextcloud integration comes in Step 9)
**Tab 5: Devices** (read-only summary)
- Display `owned_items` grouped by type
- For console_device items: link to `/devices/:device_id` in a new tab
### `frontend/src/crm/customers/index.js`
Export all components.
## Routing in `frontend/src/App.jsx`
```jsx
<Route path="/crm/customers" element={<CustomerList />} />
<Route path="/crm/customers/new" element={<CustomerForm />} />
<Route path="/crm/customers/:id" element={<CustomerDetail />} />
<Route path="/crm/customers/:id/edit" element={<CustomerForm />} />
```
## Sidebar update
Add to CRM section:
- Customers → `/crm/customers`
## Notes
- ALL hooks in CustomerDetail must be before any early returns (loading/error states)
- Tag input: comma or enter to add, click pill to remove
- Contact type icons: use simple text labels or emoji (📧 📞 💬) — keep it simple
- Comms type icons: simple colored badges per type (email=blue, whatsapp=green, call=yellow, note=grey)
- No file upload UI yet in Media tab — just nextcloud_path text field for now (Step 9 adds real upload)

71
.claude/crm-step-07.md Normal file
View File

@@ -0,0 +1,71 @@
# CRM Step 07 — Frontend: Orders Module
## Context
Read `.claude/crm-build-plan.md` for full context, data models, and IMPORTANT NOTES.
Steps 01–06 must be complete.
## Task
Build the Orders section.
## Files to create
### `frontend/src/crm/orders/OrderList.jsx`
- Fetch `GET /api/crm/orders` (query: status, payment_status)
- Table: Order #, Customer name (resolve from customer_id via separate fetch or denormalize), Status badge, Total, Payment status, Date
- Filter dropdowns: Status, Payment Status
- "New Order" button → `/crm/orders/new`
- Row click → `/crm/orders/:id`
### `frontend/src/crm/orders/OrderForm.jsx`
Create/edit. Receives `orderId` prop and optional `customerId` from query param.
**Sections:**
1. **Customer** — searchable dropdown (fetch `/api/crm/customers`). Shows name + organization.
2. **Order Info** — order_number (auto, editable), status (dropdown), currency
3. **Items** — dynamic list. Each item:
- type selector: console_device | product | freetext
- product: dropdown from `/api/crm/products` (auto-fills product_name, unit_price)
- console_device: text input for device_id + label
- freetext: description text input
- quantity (number), unit_price (number), serial_numbers (comma-separated)
- Remove row button
- Add Item button
4. **Pricing** — show computed subtotal (sum of qty * unit_price). Discount: type toggle (% or fixed) + value input + reason. Show computed total = subtotal - discount. These values are sent to backend as-is.
5. **Payment** — payment_status dropdown, invoice_path (nextcloud path text input)
6. **Shipping** — method, carrier, tracking_number, destination, shipped_at (date), delivered_at (date)
7. **Notes** — textarea
Save → POST or PUT. Delete with confirmation.
### `frontend/src/crm/orders/OrderDetail.jsx`
Read-only view of a single order.
- Header: order number, status badge, customer name (link to customer)
- Items table: product/description, qty, unit price, line total
- Pricing summary: subtotal, discount, total
- Shipping card: all shipping fields
- Payment card: status, invoice path (if set, show as link)
- Notes
- Edit button → OrderForm
- Back to customer button
### `frontend/src/crm/orders/index.js`
Export all components.
## Routing in `frontend/src/App.jsx`
```jsx
<Route path="/crm/orders" element={<OrderList />} />
<Route path="/crm/orders/new" element={<OrderForm />} />
<Route path="/crm/orders/:id" element={<OrderDetail />} />
<Route path="/crm/orders/:id/edit" element={<OrderForm />} />
```
## Sidebar update
Add to CRM section:
- Orders → `/crm/orders`
## Notes
- Status badge colors: draft=grey, confirmed=blue, in_production=orange, shipped=purple, delivered=green, cancelled=red
- Payment status: pending=yellow, partial=orange, paid=green
- Discount calculation: if type=percentage → total = subtotal * (1 - value/100). if type=fixed → total = subtotal - value
- When a product is selected from dropdown in item row, auto-fill unit_price from product.price (user can override)
- Order list needs customer names — either fetch all customers once and build a map, or add customer_name as a denormalized field when creating/updating orders (simpler: fetch customer list once)

53
.claude/crm-step-08.md Normal file
View File

@@ -0,0 +1,53 @@
# CRM Step 08 — Frontend: Comms Log + Media (Manual Entry Polish)
## Context
Read `.claude/crm-build-plan.md` for full context and IMPORTANT NOTES.
Steps 01–07 must be complete.
## Task
Two things:
1. A standalone **Inbox** page — unified comms view across all customers
2. Polish the Comms and Media tabs on CustomerDetail (from Step 06) — improve the UI
## Files to create/update
### `frontend/src/crm/inbox/InboxPage.jsx`
- Fetch `GET /api/crm/comms?customer_id=ALL` — wait, this doesn't exist yet.
→ Instead, fetch all customers, then fetch comms for each? No — too many requests.
→ Add a new backend endpoint first (see below).
- Show a unified timeline of all comms entries across all customers, sorted by occurred_at desc
- Each entry shows: customer name (link), type badge, direction, subject/body preview, date
- Filter by type (email/whatsapp/call/note/etc), direction, customer (dropdown)
- Pagination or virtual scroll (limit to last 100 entries)
### Backend addition needed — add to `backend/crm/router.py` and `service.py`:
`GET /api/crm/comms/all` — fetch all comms (no customer_id filter), sorted by occurred_at DESC, limit 200.
`list_all_comms(type=None, direction=None, limit=200) -> List[CommInDB]` in service.
### Comms tab improvements (update CustomerDetail.jsx)
- Full timeline view with visual connector line between entries
- Each entry is expandable — click to see full body
- Entry form as an inline slide-down panel (not a modal)
- Form fields: type (icons + labels), direction, subject, body (textarea), occurred_at (datetime-local input, defaults to now), attachments (add nextcloud_path manually for now)
- After save, refresh comms list
### Media tab improvements (update CustomerDetail.jsx)
- Group media by direction: "Received" section, "Sent" section, "Internal" section
- Show filename, tags as pills, date
- "Add Media" inline form: filename (required), nextcloud_path (required), direction (dropdown), tags (pill input)
- Delete button per item with confirmation
## Routing in `frontend/src/App.jsx`
```jsx
<Route path="/crm/inbox" element={<InboxPage />} />
```
## Sidebar update
Add to CRM section (at top of CRM group):
- Inbox → `/crm/inbox`
## Notes
- This step is mostly UI polish + the inbox page. No new integrations.
- The inbox page is the "central comms view" from the original requirements — all messages in one place
- Keep the comms entry form simple: only show attachment fields if user clicks "Add attachment"
- Type badges: email=blue, whatsapp=green, call=amber, sms=teal, note=grey, in_person=purple

92
.claude/crm-step-09.md Normal file
View File

@@ -0,0 +1,92 @@
# CRM Step 09 — Integration: Nextcloud WebDAV
## Context
Read `.claude/crm-build-plan.md` for full context and IMPORTANT NOTES.
Steps 01–08 must be complete.
## Task
Connect the console to Nextcloud via WebDAV so that:
1. Files in a customer's Nextcloud folder are listed in the Media tab automatically
2. Uploading a file from the console sends it to Nextcloud
3. Files can be downloaded/previewed via a backend proxy
## Backend changes
### 1. Add Nextcloud settings to `backend/config.py`
```python
nextcloud_url: str = "https://nextcloud.bonamin.gr" # e.g. https://cloud.example.com
nextcloud_email: str = "bellsystems.gr@gmail.com"
nextcloud_username: str = "bellsystems-console"
nextcloud_password: str = "ydE916VdaQdbP2CQGhD!"
nextcloud_app_password: str = "rtLCp-NCy3y-gNZdg-38MtN-r8D2N"
nextcloud_base_path: str = "BellSystems" # root folder inside Nextcloud
```
### 2. Create `backend/crm/nextcloud.py`
WebDAV client using `httpx` (already available). Functions:
```python
async def list_folder(nextcloud_path: str) -> List[dict]:
"""
PROPFIND request to Nextcloud WebDAV.
Returns list of {filename, path, mime_type, size, last_modified, is_dir}
Parse the XML response (use xml.etree.ElementTree).
URL: {nextcloud_url}/remote.php/dav/files/{username}/{nextcloud_base_path}/{nextcloud_path}
"""
async def upload_file(nextcloud_path: str, filename: str, content: bytes, mime_type: str) -> str:
"""
PUT request to upload file.
Returns the full nextcloud_path of the uploaded file.
"""
async def download_file(nextcloud_path: str) -> tuple[bytes, str]:
"""
GET request. Returns (content_bytes, mime_type).
"""
async def delete_file(nextcloud_path: str) -> None:
"""
DELETE request.
"""
```
Use HTTP Basic Auth with nextcloud_username/nextcloud_password.
If nextcloud_url is empty string, raise HTTPException 503 "Nextcloud not configured".
### 3. Add to `backend/crm/router.py`
**Media/Nextcloud endpoints:**
`GET /api/crm/nextcloud/browse?path=05_Customers/FOLDER`
→ calls `list_folder(path)`, returns file list
`GET /api/crm/nextcloud/file?path=05_Customers/FOLDER/photo.jpg`
→ calls `download_file(path)`, returns `Response(content=bytes, media_type=mime_type)`
→ This is the proxy endpoint — frontend uses this to display images
`POST /api/crm/nextcloud/upload`
→ accepts `UploadFile` + form field `nextcloud_path` (destination folder)
→ calls `upload_file(...)`, then calls `create_media(...)` to save the metadata record
→ returns the created `MediaInDB`
`DELETE /api/crm/nextcloud/file?path=...`
→ calls `delete_file(path)`, also deletes the matching `crm_media` record if found
## Frontend changes
### Update Media tab in `CustomerDetail.jsx`
- On load: if `customer.nextcloud_folder` is set, fetch `GET /api/crm/nextcloud/browse?path={customer.nextcloud_folder}` and merge results with existing `crm_media` records. Show files from both sources — deduplicate by nextcloud_path.
- Image files: render as `<img src="/api/crm/nextcloud/file?path=..." />` via the proxy endpoint
- Other files: show as a download link hitting the same proxy endpoint
- Upload button: file picker → POST to `/api/crm/nextcloud/upload` with file + destination path (default to customer's Sent Media subfolder)
- Show upload progress indicator
### Update Media tab in `CustomerDetail.jsx` — subfolder selector
When uploading, let user choose subfolder: "Sent Media" | "Received Media" | "Internal" (maps to direction field too)
## Notes
- `httpx` is likely already in requirements. If not, add it: `httpx>=0.27.0`
- PROPFIND response is XML (DAV namespace). Parse `D:response` elements, extract `D:href` and `D:prop` children.
- The proxy approach means the VPS never stores files — it just streams them through from Nextcloud
- nextcloud_base_path in config allows the root to be `BellSystems` so paths in DB are relative to that root

102
.claude/crm-step-10.md Normal file
View File

@@ -0,0 +1,102 @@
# CRM Step 10 — Integration: IMAP/SMTP Email
## Context
Read `.claude/crm-build-plan.md` for full context and IMPORTANT NOTES.
Steps 01–09 must be complete.
## Task
Integrate the company email mailbox so that:
1. Emails from/to a customer's email addresses appear in their Comms tab
2. New emails can be composed and sent from the console
3. A background sync runs periodically to pull new emails
## Backend changes
### 1. Add email settings to `backend/config.py`
```python
imap_host: str = ""
imap_port: int = 993
imap_username: str = ""
imap_password: str = ""
imap_use_ssl: bool = True
smtp_host: str = ""
smtp_port: int = 587
smtp_username: str = ""
smtp_password: str = ""
smtp_use_tls: bool = True
email_sync_interval_minutes: int = 15
```
### 2. Create `backend/crm/email_sync.py`
Using standard library `imaplib` and `email` (no new deps).
```python
async def sync_emails():
"""
Connect to IMAP. Search UNSEEN or since last sync date.
For each email:
- Parse from/to/subject/body (text/plain preferred, fallback to stripped HTML)
- Check if from-address or to-address matches any customer contact (search crm_customers)
- If match found: create crm_comms_log entry with type=email, ext_message_id=message-id header
- Skip if ext_message_id already exists in crm_comms_log (dedup)
Store last sync time in a simple SQLite table crm_sync_state:
CREATE TABLE IF NOT EXISTS crm_sync_state (key TEXT PRIMARY KEY, value TEXT)
"""
async def send_email(to: str, subject: str, body: str, cc: List[str] = []) -> str:
"""
Send email via SMTP. Returns message-id.
After sending, create a crm_comms_log entry: type=email, direction=outbound.
"""
```
### 3. Add SQLite table to `backend/mqtt/database.py`
```sql
CREATE TABLE IF NOT EXISTS crm_sync_state (
key TEXT PRIMARY KEY,
value TEXT
);
```
### 4. Add email endpoints to `backend/crm/router.py`
`POST /api/crm/email/send`
Body: `{ customer_id, to, subject, body, cc (optional) }`
→ calls `send_email(...)`, links to customer in comms_log
`POST /api/crm/email/sync`
→ manually trigger `sync_emails()` (for testing / on-demand)
→ returns count of new emails found
### 5. Add background sync to `backend/main.py`
In the `startup` event, add a periodic task:
```python
async def email_sync_loop():
while True:
await asyncio.sleep(settings.email_sync_interval_minutes * 60)
try:
from crm.email_sync import sync_emails
await sync_emails()
except Exception as e:
print(f"[EMAIL SYNC] Error: {e}")
asyncio.create_task(email_sync_loop())
```
Only start if `settings.imap_host` is set (non-empty).
## Frontend changes
### Update Comms tab in `CustomerDetail.jsx`
- Email entries show: from/to, subject, body (truncated with expand)
- "Compose Email" button → modal with: to (pre-filled from customer primary email), subject, body (textarea), CC
- On send: POST `/api/crm/email/send`, add new entry to comms list
### Update `InboxPage.jsx`
- Add "Sync Now" button → POST `/api/crm/email/sync`, show result count toast
## Notes
- `imaplib` is synchronous — wrap in `asyncio.run_in_executor(None, sync_fn)` for the async context
- For HTML emails: strip tags with a simple regex or `html.parser` — no need for an HTML renderer
- Email body matching: compare email From/To headers against ALL customer contacts where type=email
- Don't sync attachments yet — just text content. Attachment handling can be a future step.
- If imap_host is empty string, the sync loop doesn't start and the send endpoint returns 503

81
.claude/crm-step-11.md Normal file
View File

@@ -0,0 +1,81 @@
# CRM Step 11 — Integration: WhatsApp Business API
## Context
Read `.claude/crm-build-plan.md` for full context and IMPORTANT NOTES.
Steps 01–10 must be complete.
## Prerequisites (manual setup required before this step)
- A Meta Business account with WhatsApp Business API enabled
- A dedicated phone number registered to WhatsApp Business API (NOT a personal number)
- A Meta App with webhook configured to point to: `https://yourdomain.com/api/crm/whatsapp/webhook`
- The following values ready: `WHATSAPP_PHONE_NUMBER_ID`, `WHATSAPP_ACCESS_TOKEN`, `WHATSAPP_VERIFY_TOKEN`
## Task
Receive inbound WhatsApp messages via webhook and send outbound messages, all logged to crm_comms_log.
## Backend changes
### 1. Add to `backend/config.py`
```python
whatsapp_phone_number_id: str = ""
whatsapp_access_token: str = ""
whatsapp_verify_token: str = "change-me" # you set this in Meta webhook config
```
### 2. Create `backend/crm/whatsapp.py`
```python
async def send_whatsapp(to_phone: str, message: str) -> str:
"""
POST to https://graph.facebook.com/v19.0/{phone_number_id}/messages
Headers: Authorization: Bearer {access_token}
Body: { messaging_product: "whatsapp", to: to_phone, type: "text", text: { body: message } }
Returns the wamid (WhatsApp message ID).
"""
```
### 3. Add webhook + send endpoints to `backend/crm/router.py`
`GET /api/crm/whatsapp/webhook`
— Meta webhook verification. Check `hub.verify_token` == settings.whatsapp_verify_token.
Return `hub.challenge` if valid, else 403.
**No auth required on this endpoint.**
`POST /api/crm/whatsapp/webhook`
— Receive inbound message events from Meta.
**No auth required on this endpoint.**
Parse payload:
```
entry[0].changes[0].value.messages[0]
.from → sender phone number (e.g. "306974015758")
.id → wamid
.type → "text"
.text.body → message content
.timestamp → unix timestamp
```
For each message:
1. Look up customer by phone number in crm_customers contacts (where type=phone or whatsapp)
2. If found: create crm_comms_log entry (type=whatsapp, direction=inbound, ext_message_id=wamid)
3. If not found: still log it but with customer_id="unknown:{phone}"
`POST /api/crm/whatsapp/send`
Body: `{ customer_id, to_phone, message }`
Requires auth.
→ calls `send_whatsapp(...)`, creates outbound comms_log entry
## Frontend changes
### Update Comms tab in `CustomerDetail.jsx`
- WhatsApp entries: green background, WhatsApp icon
- "Send WhatsApp" button → modal with: to_phone (pre-filled from customer's whatsapp/phone contacts), message textarea
- On send: POST `/api/crm/whatsapp/send`
### Update `InboxPage.jsx`
- WhatsApp entries are already included (from crm_comms_log)
- Add type filter option for "WhatsApp"
## Notes
- Phone number format: Meta sends numbers without `+` (e.g. "306974015758"). Normalize when matching against customer contacts (strip `+` and spaces).
- Webhook payload can contain multiple entries and messages — iterate and handle each
- Rate limits: Meta free tier = 1000 conversations/month (a conversation = 24h window with a customer). More than enough.
- If whatsapp_phone_number_id is empty, the send endpoint returns 503. The webhook endpoint must always be available (it's a public endpoint).
- Media messages (images, docs): in this step, just log "Media message received" as body text. Full media download is a future enhancement.

97
.claude/crm-step-12.md Normal file
View File

@@ -0,0 +1,97 @@
# CRM Step 12 — Integration: FreePBX AMI Call Logging
## Context
Read `.claude/crm-build-plan.md` for full context and IMPORTANT NOTES.
Steps 01–11 must be complete.
## Prerequisites (manual setup before this step)
- FreePBX server with AMI (Asterisk Manager Interface) enabled
- An AMI user created in FreePBX: Admin → Asterisk Manager Users
- Username + password (set these in config)
- Permissions needed: read = "call,cdr" (minimum)
- Network access from VPS to FreePBX AMI port (default: 5038)
- Values ready: `AMI_HOST`, `AMI_PORT` (5038), `AMI_USERNAME`, `AMI_PASSWORD`
## Task
Connect to FreePBX AMI over TCP, listen for call events, and auto-log them to crm_comms_log matched against customer phone numbers.
## Backend changes
### 1. Add to `backend/config.py`
```python
ami_host: str = ""
ami_port: int = 5038
ami_username: str = ""
ami_password: str = ""
```
### 2. Create `backend/crm/ami_listener.py`
AMI uses a plain TCP socket with a text protocol (key: value\r\n pairs, events separated by \r\n\r\n).
```python
import asyncio
from config import settings
from mqtt import database as mqtt_db
async def ami_connect_and_listen():
"""
1. Open TCP connection to ami_host:ami_port
2. Read the banner line
3. Send login action:
Action: Login\r\n
Username: {ami_username}\r\n
Secret: {ami_password}\r\n\r\n
4. Read response — check for "Response: Success"
5. Loop reading events. Parse each event block into a dict.
6. Handle Event: Hangup:
- CallerID: the phone number (field: CallerIDNum)
- Duration: call duration seconds (field: Duration, may not always be present)
- Channel direction: inbound if DestChannel starts with "PJSIP/" or "SIP/",
outbound if Channel starts with "PJSIP/" or "SIP/"
- Normalize CallerIDNum: strip leading + and spaces
- Look up customer by normalized phone
- Create crm_comms_log entry: type=call, direction=inbound|outbound,
body=f"Call duration: {duration}s", ext_message_id=Uniqueid field
7. On disconnect: wait 30s, reconnect. Infinite retry loop.
"""
async def start_ami_listener():
"""Entry point — only starts if ami_host is set."""
if not settings.ami_host:
return
asyncio.create_task(ami_connect_and_listen())
```
### 3. Add to `backend/main.py` startup
```python
from crm.ami_listener import start_ami_listener
# in startup():
await start_ami_listener()
```
### 4. Add manual log endpoint to `backend/crm/router.py`
`POST /api/crm/calls/log`
Body: `{ customer_id, direction, duration_seconds, notes, occurred_at }`
Requires auth.
→ create crm_comms_log entry (type=call) manually
→ useful if auto-logging misses a call or for logging calls made outside the office
## Frontend changes
### Update Comms tab in `CustomerDetail.jsx`
- Call entries: amber/yellow color, phone icon
- Show duration if available (parse from body)
- "Log Call" button → quick modal with: direction (inbound/outbound), duration (minutes + seconds), notes, occurred_at
- On save: POST `/api/crm/calls/log`
### Update `InboxPage.jsx`
- Add "Call" to type filter options
- Call entries show customer name, direction arrow, duration
## Notes
- AMI protocol reference: each event/response is a block of `Key: Value` lines terminated by `\r\n\r\n`
- The `Hangup` event fires at end of call and includes Duration in seconds
- CallerIDNum for inbound calls is the caller's number. For outbound it's typically the extension — may need to use `DestCallerIDNum` instead. Test against your FreePBX setup.
- Phone matching uses the same normalization as WhatsApp step (strip `+`, spaces, leading zeros if needed)
- If AMI connection drops (FreePBX restart, network blip), the reconnect loop handles it silently
- This gives you: auto-logged inbound calls matched to customers, duration recorded, plus a manual log option for anything missed

16
.claude/settings.local.json Normal file
View File

@@ -0,0 +1,16 @@
{
"permissions": {
"allow": [
"Bash(npm create:*)",
"Bash(npm install:*)",
"Bash(npm run build:*)",
"Bash(python -c:*)",
"Bash(npx vite build:*)",
"Bash(wc:*)",
"Bash(ls:*)",
"Bash(node -c:*)",
"Bash(npm run lint:*)",
"Bash(python:*)"
]
}
}

2
.env.example
View File

@@ -25,7 +25,7 @@ DEBUG=true
NGINX_PORT=80 NGINX_PORT=80
# Local file storage (override if you want to store data elsewhere) # Local file storage (override if you want to store data elsewhere)
SQLITE_DB_PATH=./data/database.db SQLITE_DB_PATH=./mqtt_data.db
BUILT_MELODIES_STORAGE_PATH=./storage/built_melodies BUILT_MELODIES_STORAGE_PATH=./storage/built_melodies
FIRMWARE_STORAGE_PATH=./storage/firmware FIRMWARE_STORAGE_PATH=./storage/firmware

10
.gitignore vendored
View File

@@ -12,11 +12,6 @@ firebase-service-account.json
!/data/.gitkeep !/data/.gitkeep
!/data/built_melodies/.gitkeep !/data/built_melodies/.gitkeep
# SQLite databases
*.db
*.db-shm
*.db-wal
# Python # Python
__pycache__/ __pycache__/
*.pyc *.pyc
@@ -37,7 +32,4 @@ Thumbs.db
.MAIN-APP-REFERENCE/ .MAIN-APP-REFERENCE/
.project-vesper-plan.md .project-vesper-plan.md
# claude
.claude/

133
.stitch/DESIGN.md
View File

@@ -1,133 +0,0 @@
# Design System: BellSystems Console Design
**Project ID:** 18406618574074411899
---
## 1. Visual Theme & Atmosphere
**"The Digital Observatory"** — A high-fidelity, immersive enterprise command center aesthetic that rejects the typical boxed-in SaaS feel. The mood is best described as **Atmospheric Depth**: dark, spacious, and precision-focused. Inspired by high-end editorial design and command center interfaces, data is treated as a premium asset surfaced through tonal layering rather than structural lines.
The base is built on **Midnight Navy** — a deep blue-black that evokes an infinite canvas — with UI elements that appear to float and glow rather than sit flat. Hierarchy is established exclusively through surface tone shifts; hard borders are forbidden. The result feels like peering through a high-resolution window into enterprise data, where clarity comes from contrast in depth rather than weight.
Overall density is **medium-high** — information-rich layouts with generous vertical whitespace between elements, but no wasted screen real estate.
---
## 2. Color Palette & Roles
### Core Surfaces (darkest to lightest)
| Name | Hex | Role |
|---|---|---|
| **Abyss** | `#0a0e14` | Deepest well; nested content backgrounds |
| **Midnight** | `#10141a` | Main application viewport / page background |
| **Void Navy** | `#181c22` | Sidebar, header, and secondary navigation surfaces |
| **Deep Slate** | `#1c2026` | Default card and information module background |
| **Elevated Slate** | `#262a31` | High cards, hovered table rows |
| **Island** | `#31353c` | Active states, selected rows, top-layer containers |
| **Frosted Glass** | `#353940` | Floating modals and dropdowns (at 80% opacity with blur) |
### Accent & Brand Colors
| Name | Hex | Role |
|---|---|---|
| **Indigo Glow** | `#c0c1ff` | Primary accent; CTA text, key metrics, active nav indicator |
| **Lavender Soft** | `#8083ff` | Primary container fills |
| **Violet Pulse** | `#d2bbff` | Secondary accent; gradient pair to Indigo Glow |
| **Deep Indigo** | `#6001d1` | Secondary container fills |
| **Royal Indigo** | `#494bd6` | Inverse primary; used on light-over-dark contexts |
| **Aqua Sky** | `#7bd0ff` | Tertiary accent; data visualization, device status indicators |
| **Ocean** | `#009bd1` | Tertiary container |
### Semantic / State Colors
| Name | Hex | Role |
|---|---|---|
| **Coral Error** | `#ffb4ab` | Error text and destructive action labels |
| **Crimson** | `#93000a` | Error container backgrounds |
| **Emerald** (Tailwind) | `emerald-*` | Online / active device status badges |
| **Amber** (Tailwind) | `amber-*` | Warning / pending device status badges |
### Text Colors
| Name | Hex | Role |
|---|---|---|
| **Cloud** | `#dfe2eb` | Primary text; body copy, data values, headings |
| **Mist** | `#c7c4d7` | Secondary / muted text; labels, metadata |
| **Ghost** | `#908fa0` | Placeholder text, disabled states, divider fill |
| **Boundary** | `#464554` | Ghost border fallback for inputs (used at low opacity) |
---
## 3. Typography Rules
**Single font family throughout: Inter** (geometric sans-serif). Used for headlines, body, and labels alike — unity is achieved through weight and tracking variation rather than font switching.
| Level | Size | Weight | Tracking | Usage |
|---|---|---|---|---|
| **Display** | 56px / 3.5rem | Bold (700) | Tight (negative) | Hero KPI metrics, total counts |
| **Headline** | 24px / 1.5rem | SemiBold (600) | Normal | Page titles, major section headings |
| **Title** | 16px / 1.0rem | Medium (500) | Normal | Card titles, module headers, tab labels |
| **Body** | 14px / 0.875rem | Regular (400) | Normal | Default text, table rows, descriptions |
| **Label** | 11px / 0.6875rem | SemiBold (600) | Wide (+0.1em) | Sidebar category headers, metadata chips — All Caps |
**The Editorial Rule:** Sidebar category headers use the Label style in all-caps with wide letter-spacing (+0.1em). This creates a "system-level" authority that contrasts with fluid body text and signals structural navigation. Body text must never be all-caps.
---
## 4. Component Stylings
### Buttons
- **Primary CTA:** Gradient fill from Indigo Glow (`#c0c1ff`) to Violet Pulse (`#d2bbff`). White text. Softly rounded corners (matching `lg` radius — just barely rounded, not pill). On hover, a high-glow indigo shadow emanates beneath the button.
- **Secondary / Outline:** Island background (`#31353c`) with an extremely faint ghost border — outline-variant (`#464554`) at 20–30% opacity. Mist text (`#c7c4d7`).
- **Tertiary / Ghost:** Fully transparent background. Mist text. On hover: Cloud text with elevated background tint.
- **Destructive:** Coral Error text (`#ffb4ab`) on a Crimson container (`#93000a` at low opacity). Used for delete and irreversible actions. Never bright red.
### Cards & Containers
- **Default Card:** Deep Slate background (`#1c2026`). No visible border — separation is purely tonal. Features a **subtle top-edge inner glow**: `inset 0px 1px 0px rgba(192, 193, 255, 0.05)` — mimics a ceiling light reflecting off the card's glass surface.
- **Corner Rounding:** Minimal — just enough to soften, not enough to feel playful. Approximately 4px (the `lg` token = 0.25rem). Cards feel rectangular and purposeful, not bubbly.
- **Elevation principle:** No drop shadows on standard cards. Hierarchy comes from the surface color step (Void Navy → Deep Slate → Elevated Slate).
- **Floating Modals / Dropdowns:** Frosted Glass background (`rgba(53, 57, 64, 0.8)`) with `backdrop-filter: blur(12px)`. This "Glassmorphism" effect keeps the user contextually anchored to the underlying page while the modal floats above. Shadow: `0px 8px 24px rgba(13, 17, 23, 0.6)` — navy-tinted, never pure black.
### Inputs & Forms
- **Default state:** Deep Slate background. Ghost border at very low opacity (near invisible). Text in Cloud color.
- **Focus state:** Indigo Glow (`#c0c1ff`) border at 40% alpha creates a soft halo glow — not a hard ring. The inner glow on the input also subtly strengthens.
- **Placeholder text:** Ghost color (`#908fa0`).
- **Select / Dropdown:** Same surface as inputs; opens as a Frosted Glass panel with blur.
- **No hard outlines at rest** — inputs feel embedded in the surface until interacted with.
### Status Badges
- **Shape:** Fully pill-shaped (maximum border-radius / `full` = 0.75rem).
- **Style:** Functional color (Emerald, Amber, Coral) at approximately 15% background opacity, with the same color at full 100% opacity for the label text. Creates a "glowing ink" effect — the badge appears illuminated from within.
- **Examples:** Online → soft emerald glow; Warning → soft amber glow; Error/Offline → soft coral glow.
### Navigation Sidebar (224px wide)
- **Background:** Void Navy (`#181c22`) — one step lighter than the main viewport.
- **Active item indicator:** A `3px` vertical bar on the far-left edge using Indigo Glow (`#c0c1ff`). The text weight also increases slightly. No background highlight on active items — the light bar IS the indicator.
- **Inactive items:** Mist text (`#c7c4d7`) at normal weight.
- **Category headers:** Label-style — all-caps, 11px, SemiBold, wide tracking. Ghost color (`#908fa0`).
- **Item padding:** Comfortable vertical padding (~9.6px / 0.6rem) for breathing room between items.
- **No dividers** between nav sections — spacing does the work.
### Data Tables
- **Row separation:** No horizontal dividers. Alternating subtle tonal rows (Island `#31353c` on hover) and consistent vertical gap rhythm.
- **Header row:** Mist text (`#c7c4d7`), Label-style capitalization, slightly smaller than body.
- **Hovered row:** Elevated Slate (`#262a31`) or Island (`#31353c`) background.
- **Selected row:** Island background with Indigo Glow left border accent.
### Scrollbars
- Slim — 4px wide track.
- Thumb: Boundary color (`#464554`), with 2px border-radius.
- Track: Transparent.
- Overall feel: Nearly invisible unless sought out.
---
## 5. Layout Principles
**Whitespace is structure.** The design never uses lines or dividers to separate sections — space does that job. When content feels disconnected, the solution is always to add vertical breathing room, never to draw a border.
- **Page content area:** Fills the viewport to the right of the 224px sidebar. Padding inside the content area is generous — approximately 24–32px on all sides.
- **Section spacing:** Major sections within a page are separated by approximately 24px of vertical space. Sub-sections by 16px.
- **Card grid:** Cards sit in fluid grids with 16px gaps. Cards never touch each other.
- **Alignment:** Strong left-edge alignment for all content. Data tables, card headers, and page titles all share the same left origin point.
- **No horizontal rules / `<hr>` elements:** Surface color transitions and whitespace define the visual structure entirely.
- **Modals:** Centered in the viewport, overlaid on a dark scrim. The page content behind is still readable through the frosted glass effect, maintaining spatial context.
- **The "No Raw Border" rule:** Any element requiring a visible boundary for accessibility (e.g., an active input) must use the ghost border approach — Boundary color (`#464554`) at 20% opacity maximum. Full-opacity borders are strictly prohibited.
- **Mobile / responsive:** The sidebar collapses to a drawer on narrow viewports. Cards reflow to single-column. The design's depth relies on background layers, so it translates naturally to smaller screens.

0
strategies/BellSystems_AdminPanel_Strategy.md → BellSystems_AdminPanel_Strategy.md
View File

383
CLAUDE.md
View File

@@ -1,383 +0,0 @@
# CLAUDE.md — BellSystems Control Panel v2
# Instructions for Claude Code
> Read this file at the start of every session.
> This is the v2 project — a clean rebuild. The old v1 code lives in `frontend/src/_archive/` for reference only.
> Also read `DESIGN.md` before writing any UI code.
---
## Project Structure
```
C:\development\bellsystems-cp-v2\
│ CLAUDE.md ← you are here
│ DESIGN.md ← design rules, component contracts, page layout spec
│ docker-compose.yml
│
├── backend/ ← FastAPI backend — DO NOT MODIFY
│
└── frontend/
├── src/
│ ├── _archive/ ← v1 reference code — READ ONLY, never import from here except auth
│ ├── assets/
│ │ ├── global-icons/ ← action SVGs (edit, delete, download, etc.)
│ │ ├── side-menu-icons/ ← sidebar navigation SVGs
│ │ ├── comms/ ← communication type SVGs
│ │ ├── other-icons/ ← misc SVGs
│ │ └── customer-status/ ← CRM status SVGs
│ ├── components/
│ │ ├── ui/ ← design system components (the ONLY place to source UI)
│ │ ├── layout/ ← Sidebar, Header, MainLayout
│ │ └── shared/
│ ├── hooks/
│ ├── lib/
│ ├── modals/ ← all modal components live here, grouped by domain
│ ├── pages/ ← one file per page, grouped by domain
│ ├── providers/
│ ├── router/
│ │ └── index.jsx ← all routes defined here
│ ├── styles/
│ │ ├── tokens.css ← ALL design tokens (colors, fonts, spacing, shadows)
│ │ ├── components.css ← ALL component-level styles
│ │ └── global.css ← base resets, typography, scrollbar, .page-wrapper
│ └── main.jsx ← app entry point
└── vite.config.js
```
---
## Project Overview
Bespoke SaaS Admin Console for BellSystems. Manages Devices, Customers (CRM),
Manufacturing, Firmware, MQTT, Melodies, Staff, and more.
- **Backend:** FastAPI at `backend/` — never modify
- **Archive:** `frontend/src/_archive/` — v1 reference, read-only
- **Active code:** `frontend/src/` (everything except `_archive/`)
- **Design rules:** `DESIGN.md` at project root — read before writing any UI
- **Style Guide:** live at `/dev/styleguide` — shows every component with every variant
- **API client:** `frontend/src/lib/api.js` wraps `_archive/api/client.js`
---
## Import Alias
`@/` maps to `frontend/src/`:
```js
import Button from '@/components/ui/Button' ✅
import Select from '@/components/ui/Select' ✅
import { useAuth } from '@/hooks/useAuth' ✅
import MainLayout from '@/components/layout/MainLayout' ✅
```
Never use relative `../` paths except inside `providers/` and `hooks/` when referencing `_archive/`.
---
## The Golden Rules
- **Never modify `_archive/`** — it is read-only reference material
- **All new code goes in `frontend/src/`** — no exceptions
- **No `/v2/` prefix anywhere** — routes start from `/`, imports start from `@/`
- **Read `DESIGN.md` before writing any UI code**
- **Source every UI element from `@/components/ui/`** — no raw HTML elements for styled things
- **Use only CSS tokens** — never raw hex, rgb, or pixel values in component or page files
- **Use `.masonry-grid` for all content pages with multiple variable-height sections** — never `display: grid` with fixed columns for card layouts. See DESIGN.md §11.
---
## Available UI Components
Every component lives in `frontend/src/components/ui/`. These are the ONLY components to use.
Check the live Style Guide at `/dev/styleguide` to see all variants and states.
| Component | Import path | Purpose |
|-----------------|--------------------------------------|----------------------------------------------|
| `Button` | `@/components/ui/Button` | All interactive actions |
| `StatusBadge` | `@/components/ui/StatusBadge` | Coloured status pills |
| `FormField` | `@/components/ui/FormField` | Every text/email/password/textarea input |
| `Select` | `@/components/ui/Select` | Custom dropdown (used inside FormField type="select") |
| `Modal` | `@/components/ui/Modal` | All overlay dialogs |
| `ConfirmDialog` | `@/components/ui/ConfirmDialog` | Destructive / confirmation prompts |
| `DataTable` | `@/components/ui/DataTable` | All tabular data with sorting/selection |
| `Pagination` | `@/components/ui/Pagination` | Page controls beneath DataTable |
| `Spinner` | `@/components/ui/Spinner` | Loading indicators |
| `PageHeader` | `@/components/ui/PageHeader` | Page title block — every page starts with this |
| `Card` | `@/components/ui/Card` | Contained content sections |
| `Tabs` | `@/components/ui/Tabs` | Tabbed navigation within a page |
| `Toast` | `@/components/ui/Toast` | Transient notifications (via `useToast`) |
| `SearchBar` | `@/components/ui/SearchBar` | Search inputs with debounce |
| `Breadcrumbs` | `@/components/ui/Breadcrumbs` | Navigation trail on detail pages |
| `Icon` | `@/components/ui/Icon` | Inline SVG icons by name |
---
## Folder Structure — Pages & Modals
Folders mirror the sidebar section hierarchy exactly.
```
frontend/src/pages/
├── auth/ ← unauthenticated routes (login)
├── dashboard/ ← General section
├── bellcloud/ ← Bell Cloud section
│ ├── devices/
│ │ └── notes/
│ ├── users/
│ ├── melodies/
│ │ └── archetypes/
│ └── mqtt/
├── crm/ ← Headquarters section
│ ├── comms/
│ │ └── mail/
│ ├── customers/
│ │ └── tabs/
│ ├── orders/
│ ├── quotations/
│ └── products/
├── engineering/ ← Engineering section
│ ├── manufacturing/
│ ├── firmware/
│ └── developer/
├── public/ ← Public / unauthenticated pages
│ ├── cloudflash/
│ └── serial/
├── settings/ ← Console Settings section
│ └── staff/
└── dev/ ← Internal dev tools (StyleGuide)
```
```
frontend/src/modals/
├── bellcloud/
│ ├── devices/
│ ├── melodies/
│ └── users/
├── crm/
│ └── products/
├── engineering/
│ └── manufacturing/
└── shared/
```
---
## Page Layout — How Every Page Is Structured
Every authenticated page is rendered inside `MainLayout`, which provides:
- **Sidebar** — fixed left, `224px` wide (`--sidebar-width`)
- **Header** — fixed top, `56px` tall (`--header-height`)
- **Content area** — the remaining viewport space
Inside the content area, every page uses the `.page-wrapper` class (defined in `global.css`):
```css
.page-wrapper {
flex: 1;
display: flex;
flex-direction: column;
padding: var(--space-12); /* 48px on all sides — desktop */
gap: var(--space-6); /* 24px between top-level sections */
min-width: 0;
}
@media (max-width: 768px) {
.page-wrapper {
padding: var(--space-8); /* 32px on mobile */
gap: var(--space-4);
}
}
```
**This is the consistency guarantee**: because every page uses `.page-wrapper`, the `<PageHeader>` title on every page starts at exactly the same position — 48px from the top and 48px from the left edge of the content area. Never override these paddings. Never add extra wrappers around `page-wrapper` that introduce additional offset.
### Content width modes
Pages fall into two modes — choose based on how much content the page has:
**Full-width** (default — lists, tables, dashboards):
```jsx
<div className="page-wrapper">
```
**Centered** (forms, settings, pages with very few items):
```jsx
<div className="page-wrapper page-wrapper--centered">
```
Centered mode caps each direct child at `--content-max-width-sm` (640px) by default and centers it horizontally. Override when needed:
```jsx
<div className="page-wrapper page-wrapper--centered"
style={{ '--page-content-max-width': 'var(--content-max-width-md)' }}>
```
Available tokens: `--content-max-width-xs` (480px), `--content-max-width-sm` (640px), `--content-max-width-md` (800px), `--content-max-width-lg` (1024px).
---
## Rules for Every Page
### Before writing any code
1. Read `DESIGN.md` — confirm tokens and components to use
2. Check `frontend/src/components/ui/` — use existing components only
3. Check the Style Guide at `/dev/styleguide` for the correct variant/props
4. Check `frontend/src/_archive/` for the equivalent v1 page — copy API calls and data shape only, never styling
### Page template
```jsx
// frontend/src/pages/[domain]/PageName.jsx
import PageHeader from '@/components/ui/PageHeader'
import { useAuth } from '@/hooks/useAuth'
// Import ONLY from @/components/ui/ — never raw HTML elements for styled things
export default function PageName() {
// 1. Auth
const { user } = useAuth()
// 2. State & data fetching
// 3. Event handlers
// 4. Render — always handle: loading, error, empty, data states
return (
<div className="page-wrapper">
<PageHeader title="Page Title" subtitle="Optional description">
{/* Action buttons — use <Button variant="primary"> etc. */}
</PageHeader>
{/* Page content — use Card, DataTable, Tabs, etc. */}
</div>
)
}
```
### Styling rules
- **Tailwind** for layout only: `flex`, `grid`, `items-center`, `min-w-0`, etc.
- **CSS token variables** for ALL colors, spacing, typography — `var(--token-name)`
- **No** `.module.css` or per-page scoped CSS files
- **No** `style={{ }}` inline styles except for genuinely dynamic values (e.g. calculated widths)
- **No** raw hex, rgb, or pixel values anywhere
### Toolbar buttons — matching SearchBar height
`.btn` has `line-height: 1` while `.searchbar-input` has `line-height: var(--line-height-base)` (1.5). This makes buttons shorter than the search bar by default. Whenever a `<Button>`, `<SegmentedControl>`, or `<IconButtonGroup>` sits in the same toolbar row as a `<SearchBar>`, all buttons must use `padding-top/bottom: var(--space-3)` and `line-height: var(--line-height-base)`.
**Already handled automatically (no extra props needed):**
- `SegmentedControl` — `.seg-ctrl__btn` in `components.css` enforces `--space-3` padding and `var(--line-height-base)` globally.
- `IconButtonGroup` — `.icon-btn-group__btn` in `components.css` enforces `--space-3` padding globally.
**Must be overridden manually — standalone `<Button>` in the same row as a `<SearchBar>`:**
```jsx
// Option A — inline style prop on the button:
<Button
size="md"
style={{ paddingTop: 'var(--space-3)', paddingBottom: 'var(--space-3)', lineHeight: 'var(--line-height-base)' }}
>
Label
</Button>
// Option B (preferred when multiple buttons share a toolbar) — scoped CSS block:
<>
<style>{`
.my-toolbar .btn {
padding-top: var(--space-3) !important;
padding-bottom: var(--space-3) !important;
line-height: var(--line-height-base) !important;
}
`}</style>
<div className="my-toolbar" style={{ display: 'flex', alignItems: 'center', gap: 'var(--space-2)' }}>
<IconButtonGroup … />
<Button variant="primary" size="md">Compose</Button>
</div>
</>
```
### Date & time formatting
- **Greek/European date style everywhere** — DD/MM/YYYY, never US-style MM/DD/YYYY
- **All date/time formatting must use `@/lib/formatters`** — never use raw `toLocaleDateString()`, `Intl.DateTimeFormat`, `toLocaleString()`, or `toISOString().slice()` in pages or modals
- **For `datetime-local` input values** — use `toDatetimeLocal(iso)` and `nowLocal()` from formatters. Never use `new Date(x).toISOString().slice(0, 16)` — it converts to UTC and shifts the time (timezone bug)
- **Currency** — use `fmtEuro(n)` from formatters (Greek locale: `1.250,00 €`)
Available formatters (`import { ... } from '@/lib/formatters'`):
| Function | Output example | Use for |
|---------------------|------------------------------------|--------------------------------------|
| `fmtDate` | `05/03/2026` | Short numeric dates (tables, lists) |
| `fmtDateMedium` | `5 Mar 2026` | Medium dates (cards, details) |
| `fmtDateLong` | `5 March 2026` | Long dates (headings, summaries) |
| `fmtDateFull` | `Wednesday, 5 March 2026` | Dashboard, full context |
| `fmtDateTime` | `5 March 2026, 2:30 pm` | Date + 12h time |
| `fmtDateTimeMedium` | `5 Mar 2026, 14:30` | Date + 24h time (compact) |
| `fmtDateTimeFull` | `Wed, 5 Mar 2026, 2:30 pm` | Emails, comms |
| `fmtTime24` | `14:30:05` | Time with seconds |
| `fmtRelative` | `5 minutes ago` | Relative timestamps |
| `toDatetimeLocal` | `2026-03-05T14:30` | `datetime-local` input values |
| `nowLocal` | `2026-03-05T14:30` | Current time for form defaults |
| `toDateInput` | `2026-03-05` | `date` input values |
| `fmtEuro` | `1.250,00 €` | Euro currency |
### Data fetching
- Use `frontend/src/lib/api.js` (wraps `_archive/api/client.js`)
- Every data-fetching component must handle **loading**, **error**, and **empty** states
### Modals
- Never defined inside page files
- Live in `frontend/src/modals/[sidebar-section]/[domain]/ModalName.jsx` — mirror the pages folder structure
- Pass data via props, actions via callbacks
---
## Missing Components — Stop and Ask
If building a page requires a UI component that does not exist in
frontend/src/components/ui/, Claude Code must STOP and say:
"I need a [ComponentName] component which doesn't exist yet.
Please build it and add it to the StyleGuide before I continue."
Do NOT:
- Invent an inline one-off component inside a page file
- Use raw HTML elements styled with inline CSS as a substitute
- Proceed and leave a placeholder
The StyleGuide at frontend/src/pages/dev/StyleGuide.jsx is the source
of truth for what components exist and how they look. Every component
used in a page must have a visible example there first.
## Building a New Page — Checklist
- [ ] File in correct `frontend/src/pages/[domain]/` folder
- [ ] Root element is `<div className="page-wrapper">` — nothing else, nothing wrapping it
- [ ] First child inside `page-wrapper` is `<PageHeader title="...">`
- [ ] Only components from `frontend/src/components/ui/` used
- [ ] No raw hex colors or pixel spacing values anywhere
- [ ] Loading state implemented
- [ ] Error state implemented
- [ ] Empty state implemented
- [ ] Mobile responsive (375px minimum)
- [ ] Modals in `frontend/src/modals/`
- [ ] Route added to `frontend/src/router/index.jsx`
---
## API Client
```js
// frontend/src/lib/api.js
export { default } from '../_archive/api/client'
```
All pages import from `@/lib/api`, never directly from `_archive`.
---
## Auth
`frontend/src/hooks/useAuth.js` re-exports from the archive AuthContext.
`frontend/src/providers/AuthProvider.jsx` re-exports the AuthProvider.
These are the ONLY two files permitted to import from `_archive/auth/`.
All other files use `@/hooks/useAuth`.

627
DESIGN.md
View File

@@ -1,627 +0,0 @@
# DESIGN SYSTEM — BellSystems Control Panel v2
> Single source of truth for all UI/UX decisions.
> Read before writing any page, component, or modal.
> Never override these rules inline. Change the rule here first, then propagate.
>
> Live reference: `/dev/styleguide` — every component, every variant, every state.
---
## 1. Core Philosophy
- **Consistency over creativity.** Every page must feel like it belongs to the same product.
- **Tokens over hardcoded values.** Never write a raw color, spacing value, or font size. Always `var(--token)`.
- **Components over repetition.** If you write the same pattern twice, it becomes a shared component.
- **Page layout is global.** All pages share the same padding and spacing anchors. Content always starts at the same position.
- **Accessible by default.** ARIA labels, keyboard navigation, visible focus states on all interactive elements.
---
## 2. Page Layout Anatomy
Every authenticated page lives inside `MainLayout`, which provides:
```
┌─────────────┬────────────────────────────────────────────┐
│ │ HEADER (height: 56px / --header-height) │
│ │────────────────────────────────────────────┤
│ │ │
│ SIDEBAR │ CONTENT AREA │
│ (224px / │ ┌──────────────────────────────────────┐ │
│ --sidebar- │ │ .page-wrapper │ │
│ width) │ │ padding: 48px (--space-12) │ │
│ │ │ gap: 24px between sections │ │
│ │ │ │ │
│ │ │ <PageHeader> ← always first │ │
│ │ │ <content...> │ │
│ │ └──────────────────────────────────────┘ │
└─────────────┴────────────────────────────────────────────┘
```
### The consistency guarantee
`.page-wrapper` is defined once in `global.css`. It is the only wrapper used on every page:
```css
.page-wrapper {
flex: 1;
display: flex;
flex-direction: column;
padding: var(--space-12); /* 48px — desktop */
gap: var(--space-6); /* 24px between direct children */
min-width: 0;
}
/* Mobile: padding drops to --space-8 (32px), gap to --space-4 (16px) */
```
**Rules:**
- Every page's root element is `<div className="page-wrapper">` — no exceptions
- Never add extra padding, margin, or wrapper divs that shift content relative to `.page-wrapper`
- Never override `.page-wrapper` padding per-page
- The `<PageHeader>` is always the first child inside `.page-wrapper`
- This ensures that on every page, the title starts at exactly 48px from the top-left corner of the content area
### Content width modes
Every page falls into one of two modes:
#### 1. Full-width (default)
Content expands to fill the entire available area. Use this for all data-heavy pages: lists, tables, dashboards, detail views.
```jsx
<div className="page-wrapper">
{/* content fills the full content area */}
</div>
```
#### 2. Centered (narrow content)
For pages with a small number of elements that would look lost spanning the full viewport — e.g. settings forms, auth pages, single-entity configuration screens.
```jsx
<div className="page-wrapper page-wrapper--centered">
{/* every direct child is capped at --content-max-width-sm (640px) and centered */}
</div>
```
Default max-width is `--content-max-width-sm` (640px). Override per-page only when necessary:
```jsx
<div
className="page-wrapper page-wrapper--centered"
style={{ '--page-content-max-width': 'var(--content-max-width-md)' }}
>
```
Available width tokens:
| Token | Value | Use |
|-----------------------------|--------|-----------------------------------------|
| `--content-max-width-xs` | 480px | Tiny forms, login, auth |
| `--content-max-width-sm` | 640px | Small forms, simple settings (default) |
| `--content-max-width-md` | 800px | Medium forms, detail-light pages |
| `--content-max-width-lg` | 1024px | Moderate-width constrained pages |
**Rules:**
- Never use `page-wrapper--centered` on a list page, data table page, or any page where content should grow with the viewport
- Never hardcode a pixel `max-width` in a page file — always use a token
---
## 3. Color Tokens
All colors are CSS custom properties defined in `frontend/src/styles/tokens.css`.
The system is **dark-first**: `:root` = dark theme. `[data-theme="light"]` overrides exist as a placeholder.
### Rule: never write a raw color value in any component or page file. Always `var(--token)`.
### Background Surfaces (7-step tonal ladder)
| Token | Value | Use |
|------------------------|--------------|--------------------------------------------------------|
| `--color-bg-abyss` | `#0a0e14` | Deepest well: code blocks, input backgrounds |
| `--color-bg-base` | `#10141a` | Page background (viewport fill) |
| `--color-bg-void` | `#181c22` | Sidebar, header |
| `--color-bg-surface` | `#1c2026` | Default card / panel background |
| `--color-bg-elevated` | `#262a31` | Raised cards, hovered rows, dropdowns |
| `--color-bg-island` | `#31353c` | Active states, selected rows, pressed buttons |
| `--color-bg-float` | `rgba(53,57,64,0.80)` | Glassmorphism: modals, floating panels |
### Brand / Primary (Indigo Glow)
| Token | Value | Use |
|----------------------------|------------------------------|--------------------------------------|
| `--color-primary` | `#c0c1ff` | CTAs, active nav, key accent |
| `--color-primary-hover` | `#d2bbff` | Hover, gradient endpoint |
| `--color-primary-container`| `#8083ff` | Container fills |
| `--color-primary-subtle` | `rgba(128,131,255,0.12)` | Hover backgrounds, tinted areas |
| `--gradient-primary` | `linear-gradient(135deg, #c0c1ff, #d2bbff)` | Primary button fill |
### Semantic / State Colors
| Token | Value | Use |
|------------------------|---------------------------|------------------------------------------------|
| `--color-success` | `#4ade80` | Online, active, confirmed |
| `--color-success-bg` | `rgba(74,222,128,0.12)` | Success badge / button resting background |
| `--color-warning` | `#fbbf24` | Pending, needs attention |
| `--color-warning-bg` | `rgba(251,191,36,0.12)` | Warning badge / button resting background |
| `--color-danger` | `#ff5c5c` | Error text, destructive actions |
| `--color-danger-bg` | `rgba(255,92,92,0.12)` | Danger badge / button resting background |
| `--color-info` | `#7bd0ff` | Informational, aqua-sky accent |
| `--color-info-bg` | `rgba(123,208,255,0.12)` | Info badge background |
### Text Colors (4-step hierarchy)
| Token | Value | Use |
|---------------------------|-------------|---------------------------------------------------|
| `--color-text-primary` | `#dfe2eb` | Body copy, data values, headings |
| `--color-text-secondary` | `#c7c4d7` | Labels, metadata, inactive nav |
| `--color-text-muted` | `#908fa0` | Placeholders, disabled, category headers |
| `--color-text-inverse` | `#10141a` | Text on primary/accent backgrounds (dark on light)|
| `--color-text-accent` | `#c0c1ff` | Active nav items, links |
### Borders
| Token | Value | Use |
|-------------------------|----------------------------|--------------------------------------------|
| `--color-border` | `rgba(70,69,84,0.20)` | Resting inputs, card outlines |
| `--color-border-strong` | `rgba(70,69,84,0.45)` | Secondary buttons, stronger dividers |
| `--color-border-focus` | `rgba(192,193,255,0.40)` | Focus ring halo on inputs |
---
## 4. Typography
Two-font system. Three families total.
### Font Families
| Token | Font | Role |
|--------------------------|---------------------|---------------------------------------------------|
| `--font-family-display` | `Barlow Condensed` | H1, H2, page titles, modal titles |
| `--font-family-base` | `Onest` | All UI text, body, labels, buttons, table rows |
| `--font-family-mono` | `JetBrains Mono` | Serial numbers, IDs, code, API keys, terminal |
**Why this pairing:**
- `Barlow Condensed` has an industrial/engineering quality — feels like instrument panel labelling. Makes page titles immediately distinctive.
- `Onest` is a Ukrainian geometric grotesque with slightly unusual proportions and excellent numerics. Clean at 14px. Not the overused Inter/Space Grotesk.
- `JetBrains Mono` is the standard for developer-facing data.
### Font Sizes
| Token | Value | Use |
|--------------------|------------|----------------------------------------------|
| `--font-size-xs` | `0.6875rem` (11px) | Labels, sidebar category headers, chips |
| `--font-size-sm` | `0.75rem` (12px) | Captions, helper text, table headers |
| `--font-size-base` | `0.875rem` (14px) | Body text, table rows (default) |
| `--font-size-md` | `1rem` (16px) | Card titles, module headers |
| `--font-size-lg` | `1.125rem` (18px) | Section subheadings |
| `--font-size-xl` | `1.5rem` (24px) | Page headings (h1/h2) |
| `--font-size-2xl` | `3.5rem` (56px) | Hero KPI numbers, dashboard metrics |
### Font Weights
| Token | Value | Use |
|---------------------------|-------|----------------------------------------|
| `--font-weight-normal` | 400 | Body copy |
| `--font-weight-medium` | 500 | Emphasized body, table values |
| `--font-weight-semibold` | 600 | Headings, button labels, field labels |
| `--font-weight-bold` | 700 | Strong emphasis, hero metrics |
### Typography Usage Rules
- **Page titles (`<PageHeader>`):** `Barlow Condensed`, `1.75rem`, weight 600 (handled by `.v2-page-header-title`)
- **Modal titles:** `Barlow Condensed`, `1.125rem`, weight 600 (handled by `.v2-modal-title`)
- **H1, H2 globally:** `Barlow Condensed`, `var(--font-size-xl)`, weight 600 — set in `global.css`
- **H3–H6:** `Onest` (body font), normal heading weights
- **Card titles:** `Onest`, `--font-size-base`, weight 600
- **Table headers:** `Onest`, `--font-size-sm`, weight 600, uppercase, `--tracking-wide`
- **Body / cell text:** `Onest`, `--font-size-base`, weight 400
- **Muted / helper text:** `Onest`, `--font-size-sm`, `--color-text-muted`
- **Serials, IDs, codes:** `JetBrains Mono`, `--font-size-sm`
### Letter Spacing
| Token | Value | Use |
|---------------------|-----------|--------------------------------------------|
| `--tracking-normal` | `0em` | Default |
| `--tracking-tight` | `-0.01em` | Barlow Condensed headings |
| `--tracking-wide` | `0.08em` | Uppercase labels, sidebar category headers |
| `--tracking-display`| `-0.02em` | Hero KPI numbers at 56px |
---
## 4b. Date, Time & Currency Formatting
All dates use **Greek/European style** (day-first). Never use US-style MM/DD/YYYY anywhere in the app.
All formatting is centralized in `frontend/src/lib/formatters.js`. Never use raw `toLocaleDateString()`, `Intl.DateTimeFormat`, `toLocaleString()`, or `toISOString().slice()` in pages or modals — always import from `@/lib/formatters`.
### Available formatters
| Function | Output example | Use for |
|---------------------|------------------------------------|--------------------------------------|
| `fmtDate` | `05/03/2026` | Short numeric dates (tables, lists) |
| `fmtDateMedium` | `5 Mar 2026` | Medium dates (cards, details) |
| `fmtDateLong` | `5 March 2026` | Long dates (headings, summaries) |
| `fmtDateFull` | `Wednesday, 5 March 2026` | Dashboard, full context |
| `fmtDateTime` | `5 March 2026, 2:30 pm` | Date + 12h time |
| `fmtDateTimeMedium` | `5 Mar 2026, 14:30` | Date + 24h time (compact) |
| `fmtDateTimeFull` | `Wed, 5 Mar 2026, 2:30 pm` | Emails, comms |
| `fmtRelative` | `5 minutes ago` | Relative timestamps |
| `fmtEuro` | `1.250,00 €` | Euro currency (Greek locale) |
### Form input helpers
| Function | Output example | Use for |
|---------------------|-------------------------|--------------------------------------------------|
| `toDatetimeLocal` | `2026-03-05T14:30` | Populating `datetime-local` inputs (local time) |
| `nowLocal` | `2026-03-05T14:30` | Current time for form defaults |
| `toDateInput` | `2026-03-05` | Populating `date` inputs |
### Critical rule: no `toISOString().slice()` for form inputs
`toISOString()` converts to **UTC**, which shifts the time by the user's timezone offset (e.g. 3 hours for Greece). Always use `toDatetimeLocal()` or `nowLocal()` instead.
---
## 5. Spacing System
4px base unit. All spacing must use tokens — no arbitrary pixel values.
| Token | Value | Common use |
|--------------|--------|--------------------------------------------------|
| `--space-1` | 4px | Tight gaps, icon padding |
| `--space-2` | 8px | Between label and input, inline gaps |
| `--space-3` | 12px | Table cell padding, compact button padding |
| `--space-4` | 16px | Between form fields, mobile page padding |
| `--space-5` | 20px | Tab item spacing |
| `--space-6` | 24px | **Page padding**, card padding, section gap |
| `--space-8` | 32px | Between major sections |
| `--space-10` | 40px | Large section gap |
| `--space-12` | 48px | Extra large spacing |
| `--space-16` | 64px | Maximum spacing, hero sections |
---
## 6. Border Radius & Shadows
### Border Radius
| Token | Value | Use |
|----------------|----------|-------------------------------------------|
| `--radius-sm` | 4px | Tags, small chips, select option rows |
| `--radius-md` | 6px | Buttons, inputs, table badges |
| `--radius-lg` | 8px | Cards, panels, dropdown menus |
| `--radius-xl` | 12px | Modals, large containers |
| `--radius-full`| 9999px | Status badge pills, avatars |
### Shadows
| Token | Value | Use |
|-------------------------|------------------------------------------|------------------------------------|
| `--shadow-card` | `inset 0 1px 0 rgba(192,193,255,0.05)` | Card top-edge glass reflection |
| `--shadow-sm` | `0 2px 8px rgba(10,14,20,0.40)` | Subtle lift |
| `--shadow-md` | `0 4px 16px rgba(10,14,20,0.50)` | Elevated cards |
| `--shadow-lg` | `0 8px 24px rgba(13,17,23,0.60)` | Modals, dropdowns |
| `--shadow-focus` | `0 0 0 3px rgba(192,193,255,0.20)` | Focus ring glow |
| `--shadow-primary-glow` | `0 4px 16px rgba(192,193,255,0.28)` | Primary button hover halo |
| `--shadow-danger-glow` | `0 4px 16px rgba(255,92,92,0.40)` | Danger button hover halo |
| `--shadow-success-glow` | `0 4px 16px rgba(74,222,128,0.35)` | Success button hover halo |
---
## 7. Component Rules
### Button
Import: `@/components/ui/Button`
**Variants:**
| Variant | Resting state | Hover state |
|------------------|---------------------------------------|----------------------------------------------------------|
| `primary` | Indigo→violet gradient, dark text | `+brightness(1.06)` + `--shadow-primary-glow` halo |
| `secondary` | Island bg, ghost border | Elevated bg + focus border + subtle indigo glow |
| `ghost` | Transparent | Elevated bg + whisper indigo glow |
| `danger` | Coral tint bg, coral text | **Solid coral fill**, dark text + `--shadow-danger-glow` |
| `success` | Emerald tint bg, emerald text | **Solid emerald fill**, dark text + `--shadow-success-glow` |
| `table-actions` | Fully transparent, muted text | Island bg + strong border (identical to `secondary`) — also activates on `tr:hover` |
**Sizes:** `sm`, `md` (default), `lg`
**Rules:**
- Never use a raw `<button>` element for a styled action
- Always pass `loading` prop for async actions (shows spinner, disables interaction)
- Icon-only buttons must have `aria-label`
- Active/press state: `filter: brightness(0.94)`, glow removed
---
### FormField
Import: `@/components/ui/FormField`
Wraps every form control: label + input/textarea/select + hint + error message.
Never place a raw `<input>` on a page.
**Types:** `text`, `email`, `password`, `number`, `tel`, `url`, `textarea`, `select`
**For `type="select"`**: pass `<option>` elements as children. FormField uses the custom `Select` component internally — the native `<select>` is never rendered.
```jsx
<FormField label="Status" name="status" type="select" value={val} onChange={handleChange}>
<option value="">Choose…</option>
<option value="active">Active</option>
<option value="inactive">Inactive</option>
</FormField>
```
**Input appearance:** "cutout" inset-shadow treatment — the field appears recessed into the surface. Background: `--color-bg-abyss`. Focus: `--color-border-focus` ring.
---
### Select (standalone)
Import: `@/components/ui/Select`
Fully custom dropdown replacing native `<select>`. Floating menu via portal, keyboard navigation, checkmark on selected item. Usually consumed via `FormField type="select"`. Use directly when you need a select outside a form label context.
---
### DataTable
Import: `@/components/ui/DataTable`
**Always include:** column headers, loading skeleton, empty state, pagination.
**Rows:** alternate tint via `--color-tint-row` (`rgba(192,193,255,0.015)`). Hover: `--color-bg-island`.
**Status columns:** always `<StatusBadge>` — never raw text.
**Row actions:** last column, right-aligned, use portal-based action menu.
---
### Modal
Import: `@/components/ui/Modal`
Sizes: `sm` (480px), `md` (640px — default), `lg` (800px), `xl` (60vw/60vh), `xxl` (85vw/85vh), `full` (calc(100vw/100vh − 64px)).
**Rules:**
- Always has: title, close (×) button, footer action buttons
- Closes on Escape + backdrop click unless `persistent={true}`
- Destructive prompts use `<ConfirmDialog>` instead
- Modal JSX never lives inside a page file — always in `frontend/src/modals/[domain]/`
---
### ConfirmDialog
Import: `@/components/ui/ConfirmDialog`
Wraps `<Modal size="sm">` with a centred icon + message. Use for any action that is destructive or hard to reverse.
Variants: `danger` (coral circle + triangle icon), `primary` (indigo circle + info icon).
---
### PageHeader
Import: `@/components/ui/PageHeader`
**Always the first element inside `.page-wrapper`.** Creates the page title block.
Props: `title` (required), `subtitle`, `breadcrumbs`, `children` (action buttons slot).
The title renders as `<h1>` with class `.v2-page-header-title` — uses `Barlow Condensed` at `1.75rem` / weight 600.
```jsx
<PageHeader title="Device Inventory" subtitle="All registered Bell units">
<Button variant="primary">Add Device</Button>
</PageHeader>
```
---
### Card
Import: `@/components/ui/Card`
Variants: `flat` (default), `elevated`, `outlined`.
Props: `title`, `subtitle`, `footer`, `padding` (bool, default true), `children`.
Card header has a faint indigo gradient ceiling (`linear-gradient` from top).
---
### Tabs
Import: `@/components/ui/Tabs`
Variants: `line` (default — underline indicator), `pill` (filled background).
Props: `tabs` (array of `{key, label, icon?, count?}`), `active`, `onChange`, `variant`.
Line variant uses a sliding indicator measured with `useLayoutEffect`. Pill variant uses filled backgrounds.
Spacing: line tabs have `gap: --space-5` between items, pill tabs `gap: --space-4`.
---
### Toast
Import: `@/components/ui/Toast` → `{ ToastProvider, useToast }`
Setup: wrap the app (or router) with `<ToastProvider>`. Then in any component:
```jsx
const toast = useToast()
toast.success('Saved', 'Device updated successfully.')
toast.danger('Error', 'Failed to connect.')
toast.warning('Warning', 'Firmware is outdated.')
toast.info('Info', 'Sync in progress.')
```
Toasts auto-dismiss after 4000ms. Hover pauses the timer. Stack appears in the bottom-right corner.
---
### SearchBar
Import: `@/components/ui/SearchBar`
Supports controlled (`value` + `onChange`) or uncontrolled mode.
Debounced by default (300ms). Clear button appears when text is present.
Appearance matches the `FormField` cutout treatment.
---
### Breadcrumbs
Import: `@/components/ui/Breadcrumbs`
Use on detail pages only (not list pages). Items: array of `{ label, href? }`. Last item has no href — it is the current page.
---
### Spinner
Import: `@/components/ui/Spinner`
Props: `size` (`sm`, `md`, `lg`), `color` (defaults to `--color-primary`).
Use inside loading states. Buttons show their own spinner via `loading` prop — do not add a separate `<Spinner>` inside buttons.
---
### StatusBadge
Import: `@/components/ui/StatusBadge`
Never use a raw `<span>` with a background color for status. Always `<StatusBadge>`.
Variants: `success`, `warning`, `danger`, `info`, `neutral`.
---
### Icon
Import: `@/components/ui/Icon`
Renders an inline SVG by name. 35 named icons available (see Style Guide `/dev/styleguide` → Icon section for the full list).
```jsx
<Icon name="edit" size={16} />
<Icon name="delete" size={20} color="var(--color-danger)" />
```
**Asset SVGs** (from `/assets/` folders) are displayed via `<img>` tags in the Style Guide, not via `<Icon>`. These are pre-rendered SVG files used for sidebar icons, comms icons, customer status icons, etc. Use them as image sources, not as Icon component names.
---
## 8. Icons
Three sources:
| Source | Use case | How to render |
|-------------------------------------|---------------------------------------|-----------------------------|
| `<Icon name="..." />` | Action icons, UI chrome | `@/components/ui/Icon` |
| `assets/side-menu-icons/*.svg` | Sidebar navigation | `<img src={...} />` |
| `assets/comms/*.svg` | Communication type indicators | `<img src={...} />` |
| `assets/customer-status/*.svg` | CRM status icons | `<img src={...} />` |
| `assets/global-icons/*.svg` | Legacy action icons (prefer `<Icon>`) | `<img src={...} />` |
| `assets/other-icons/*.svg` | Misc UI icons | `<img src={...} />` |
Never add a new icon library (e.g. heroicons, lucide). Use the existing sources.
---
## 9. Theming Rules
- Theme is controlled by `data-theme` attribute on `<html>`
- Default is dark (`:root` = dark theme)
- **Never** use Tailwind's `dark:` prefix — theming is handled entirely via CSS tokens
- `[data-theme="light"]` overrides exist in `tokens.css` as a future placeholder
---
## 10. Responsive Breakpoints
| Token | Value | Behaviour |
|--------------------|--------|--------------------------------------------------------|
| `--breakpoint-sm` | 640px | |
| `--breakpoint-md` | 768px | Sidebar collapses; page padding drops to `--space-4` |
| `--breakpoint-lg` | 1024px | Full sidebar shown |
| `--breakpoint-xl` | 1280px | |
Mobile (`< 768px`): single column, sidebar hidden (drawer), tables may become card lists.
---
## 11. Section Layout — Masonry Grid
**Default layout for ALL content pages with multiple variable-height sections.**
Sections on a content page must flow like physical objects stacked in columns — the next section always drops into the shortest column. This is CSS column masonry.
### How it works
```
Column 1 | Column 2 | Column 3
────────────┼─────────────┼────────────
Section A | Section B | Section C
(300px) | (250px) | (350px)
│ │
Section E | Section D |
(200px) | (200px) |
```
Sections fill left-to-right across the top, then each new section drops into whichever column is currently shortest. This is automatic — the browser handles placement via CSS `columns`.
### Usage
```jsx
{/* 2 columns */}
<div className="masonry-grid masonry-grid--2">
<Card title="Account Info">…</Card>
<Card title="Profile">…</Card>
<Card title="Security">…</Card> {/* auto-drops into shortest column */}
</div>
{/* 3 columns */}
<div className="masonry-grid masonry-grid--3">
{sections.map(s => <Card key={s.id}>…</Card>)}
</div>
```
Available variants: `masonry-grid--2`, `masonry-grid--3`, `masonry-grid--4`
Responsive behaviour:
- `--4` collapses to 3 cols at 1024px, 1 col at 768px
- `--3` collapses to 2 cols at 1024px, 1 col at 768px
- `--2` collapses to 1 col at 768px
### Rules
- **Use `.masonry-grid` by default** on all content pages with 2+ variable-height sections
- **Do NOT** use `display: grid` with `gridTemplateColumns` for variable-height card layouts — this creates uneven whitespace when cards differ in height
- **Do NOT** use `.masonry-grid` for DataTable pages — tables span full width on their own
- **Do NOT** use `.masonry-grid` when sections must align horizontally (e.g. two fields that are semantically paired side-by-side within a card) — that's an internal card layout, not page-level masonry
- The `Card` component already has `break-inside: avoid` so it will never be split across columns
---
## 12. What Claude Code Must NEVER Do
- ❌ Write a hex color, `rgb()`, or `hsl()` value directly in any component or page file
- ❌ Write a pixel spacing or size value that isn't a `--space-*` token
- ❌ Use a raw `<button>`, `<input>`, or `<select>` for anything styled — always use the wrapper component
- ❌ Create a `.module.css` or any per-page CSS file
- ❌ Use Tailwind's `dark:` prefix — theming is via CSS tokens only
- ❌ Place modal JSX inside a page file — modals live in `frontend/src/modals/`
- ❌ Wrap `.page-wrapper` in additional divs that shift content alignment
- ❌ Override `.page-wrapper`'s padding to make a single page "different"
- ❌ Skip loading, error, and empty states on any data-fetching component
- ❌ Import from `_archive/` anywhere except `@/lib/api.js`, `@/hooks/useAuth.js`, and `@/providers/AuthProvider.jsx`
- ❌ Install a new icon library or introduce new SVG icons outside of `assets/`
- ❌ Invent new color values not in `tokens.css`

4
backend/Dockerfile
View File

@@ -1,6 +1,6 @@
FROM python:3.11-slim FROM python:3.11-slim
# System dependencies: WeasyPrint (pango/cairo), ffmpeg (video thumbs), poppler (pdf2image) # WeasyPrint system dependencies (libpango, libcairo, etc.)
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
libpango-1.0-0 \ libpango-1.0-0 \
libpangocairo-1.0-0 \ libpangocairo-1.0-0 \
@@ -8,8 +8,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libffi-dev \ libffi-dev \
shared-mime-info \ shared-mime-info \
fonts-dejavu-core \ fonts-dejavu-core \
ffmpeg \
poppler-utils \
&& apt-get clean && rm -rf /var/lib/apt/lists/* && apt-get clean && rm -rf /var/lib/apt/lists/*
WORKDIR /app WORKDIR /app

113
backend/alembic.ini
View File

@@ -1,113 +0,0 @@
# A generic, single database configuration.
[alembic]
# path to migration scripts
script_location = alembic
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
# Uncomment the line below if you want the files to be prepended with date and time
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
# sys.path path, will be prepended to sys.path if present.
# defaults to the current working directory.
prepend_sys_path = .
# timezone to use when rendering the date within the migration file
# as well as the filename.
# If specified, requires the python>=3.9 or backports.zoneinfo library.
# Any required deps can installed by adding `tzdata` to the `[alembic]` section
# of pyproject.toml.
# timezone =
# max length of characters to apply to the "slug" field
# truncate_slug_length = 40
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# set to 'true' to allow .pyc and .pyo files without
# a source .py file to be detected as revisions in the
# versions/ directory
# sourceless = false
# version location specification; This defaults
# to alembic/versions. When using multiple version
# directories, initial revisions must be specified with --version-path.
# The path separator used here should be the separator specified by "version_path_separator" below.
# version_locations = %(here)s/bar:%(here)s/bat:alembic/versions
# version path separator; As mentioned above, this is the character used to split
# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
# Valid values for version_path_separator are:
#
# version_path_separator = :
# version_path_separator = ;
# version_path_separator = space
version_path_separator = os # Use os.pathsep. Default configuration used for new projects.
# set to 'true' to search source files recursively
# in "version_locations" directory
# New in Alembic version 1.10
# recursive_version_locations = false
# the output encoding used when revision files
# are written from script.py.mako
# output_encoding = utf-8
# NOTE: The database URL is set programmatically in env.py from settings.
# Do not set sqlalchemy.url here.
[post_write_hooks]
# post_write_hooks defines scripts or Python functions that are run
# on newly generated revision scripts. See the documentation for further
# detail and examples
# format using "black" - use the console_scripts runner, against the "black" entrypoint
# hooks = black
# black.type = console_scripts
# black.entrypoint = black
# black.options = -l 79 REVISION_SCRIPT_FILENAME
# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
# hooks = ruff
# ruff.type = exec
# ruff.executable = %(here)s/.venv/bin/ruff
# ruff.options = --fix REVISION_SCRIPT_FILENAME
# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

44
backend/alembic/env.py
View File

@@ -1,44 +0,0 @@
import asyncio
from logging.config import fileConfig
from sqlalchemy.ext.asyncio import create_async_engine
from alembic import context
from config import settings
# Import all models so Alembic can see them
from database.models import Base # noqa: F401 — triggers all ORM imports
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
target_metadata = Base.metadata
def run_migrations_offline() -> None:
url = settings.database_url
context.configure(url=url, target_metadata=target_metadata, literal_binds=True)
with context.begin_transaction():
context.run_migrations()
def do_run_migrations(connection):
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations() -> None:
engine = create_async_engine(settings.database_url)
async with engine.begin() as conn:
await conn.run_sync(do_run_migrations)
await engine.dispose()
def run_migrations_online() -> None:
asyncio.run(run_async_migrations())
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()

26
backend/alembic/script.py.mako
View File

@@ -1,26 +0,0 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, None] = ${repr(down_revision)}
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}

0
backend/alembic/versions/.gitkeep
View File

83
backend/alembic/versions/244a0b0f35be_rename_entries_to_crm_entries.py
View File

@@ -1,83 +0,0 @@
"""rename_entries_to_crm_entries
Revision ID: 244a0b0f35be
Revises: 485d40e86e4b
Create Date: 2026-04-15 20:05:20.835281
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
# revision identifiers, used by Alembic.
revision: str = '244a0b0f35be'
down_revision: Union[str, None] = '485d40e86e4b'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# 1. Drop FK from support_tickets -> entries before touching the entries table
op.drop_constraint('support_tickets_linked_entry_id_fkey', 'support_tickets', type_='foreignkey')
# 2. Drop dependent table first, then parent
op.drop_table('entry_links')
op.drop_table('entries')
# 3. Create new tables with crm_ prefix
op.create_table('crm_entries',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('type', sa.String(length=10), nullable=False),
sa.Column('title', sa.String(length=500), nullable=False),
sa.Column('body', sa.Text(), nullable=True),
sa.Column('status', sa.String(length=20), nullable=True),
sa.Column('severity', sa.String(length=10), nullable=True),
sa.Column('author_id', sa.String(length=128), nullable=False),
sa.Column('author_name', sa.String(length=255), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_table('crm_entry_links',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('entry_id', sa.UUID(), nullable=False),
sa.Column('entity_type', sa.String(length=20), nullable=False),
sa.Column('entity_id', sa.String(length=128), nullable=False),
sa.ForeignKeyConstraint(['entry_id'], ['crm_entries.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('entry_id', 'entity_type', 'entity_id')
)
# 4. Recreate FK on support_tickets pointing at new table
op.create_foreign_key(None, 'support_tickets', 'crm_entries', ['linked_entry_id'], ['id'], ondelete='SET NULL')
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_constraint(None, 'support_tickets', type_='foreignkey')
op.create_foreign_key('support_tickets_linked_entry_id_fkey', 'support_tickets', 'entries', ['linked_entry_id'], ['id'], ondelete='SET NULL')
op.create_table('entries',
sa.Column('id', sa.UUID(), autoincrement=False, nullable=False),
sa.Column('type', sa.VARCHAR(length=10), autoincrement=False, nullable=False),
sa.Column('title', sa.VARCHAR(length=500), autoincrement=False, nullable=False),
sa.Column('body', sa.TEXT(), autoincrement=False, nullable=True),
sa.Column('status', sa.VARCHAR(length=20), autoincrement=False, nullable=True),
sa.Column('severity', sa.VARCHAR(length=10), autoincrement=False, nullable=True),
sa.Column('author_id', sa.VARCHAR(length=128), autoincrement=False, nullable=False),
sa.Column('author_name', sa.VARCHAR(length=255), autoincrement=False, nullable=True),
sa.Column('created_at', postgresql.TIMESTAMP(timezone=True), autoincrement=False, nullable=False),
sa.Column('updated_at', postgresql.TIMESTAMP(timezone=True), autoincrement=False, nullable=False),
sa.PrimaryKeyConstraint('id', name='entries_pkey'),
postgresql_ignore_search_path=False
)
op.create_table('entry_links',
sa.Column('id', sa.UUID(), autoincrement=False, nullable=False),
sa.Column('entry_id', sa.UUID(), autoincrement=False, nullable=False),
sa.Column('entity_type', sa.VARCHAR(length=20), autoincrement=False, nullable=False),
sa.Column('entity_id', sa.VARCHAR(length=128), autoincrement=False, nullable=False),
sa.ForeignKeyConstraint(['entry_id'], ['entries.id'], name='entry_links_entry_id_fkey', ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id', name='entry_links_pkey'),
sa.UniqueConstraint('entry_id', 'entity_type', 'entity_id', name='entry_links_entry_id_entity_type_entity_id_key')
)
op.drop_table('crm_entry_links')
op.drop_table('crm_entries')
# ### end Alembic commands ###

82
backend/alembic/versions/485d40e86e4b_initial_notes_and_tickets.py
View File

@@ -1,82 +0,0 @@
"""initial_notes_and_tickets
Revision ID: 485d40e86e4b
Revises:
Create Date: 2026-04-15 20:01:04.225959
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '485d40e86e4b'
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('entries',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('type', sa.String(length=10), nullable=False),
sa.Column('title', sa.String(length=500), nullable=False),
sa.Column('body', sa.Text(), nullable=True),
sa.Column('status', sa.String(length=20), nullable=True),
sa.Column('severity', sa.String(length=10), nullable=True),
sa.Column('author_id', sa.String(length=128), nullable=False),
sa.Column('author_name', sa.String(length=255), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_table('entry_links',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('entry_id', sa.UUID(), nullable=False),
sa.Column('entity_type', sa.String(length=20), nullable=False),
sa.Column('entity_id', sa.String(length=128), nullable=False),
sa.ForeignKeyConstraint(['entry_id'], ['entries.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('entry_id', 'entity_type', 'entity_id')
)
op.create_table('support_tickets',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('customer_id', sa.String(length=128), nullable=False),
sa.Column('customer_name', sa.String(length=255), nullable=True),
sa.Column('device_id', sa.String(length=128), nullable=True),
sa.Column('device_serial', sa.String(length=64), nullable=True),
sa.Column('subject', sa.String(length=500), nullable=False),
sa.Column('status', sa.String(length=30), nullable=False),
sa.Column('priority', sa.String(length=10), nullable=True),
sa.Column('opened_via', sa.String(length=20), nullable=True),
sa.Column('linked_entry_id', sa.UUID(), nullable=True),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['linked_entry_id'], ['entries.id'], ondelete='SET NULL'),
sa.PrimaryKeyConstraint('id')
)
op.create_table('ticket_messages',
sa.Column('id', sa.UUID(), nullable=False),
sa.Column('ticket_id', sa.UUID(), nullable=False),
sa.Column('sender_type', sa.String(length=10), nullable=False),
sa.Column('sender_id', sa.String(length=128), nullable=False),
sa.Column('sender_name', sa.String(length=255), nullable=True),
sa.Column('body', sa.Text(), nullable=False),
sa.Column('is_internal', sa.Boolean(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['ticket_id'], ['support_tickets.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('ticket_messages')
op.drop_table('support_tickets')
op.drop_table('entry_links')
op.drop_table('entries')
# ### end Alembic commands ###

23
backend/alembic/versions/a1b2c3d4e5f6_add_category_to_crm_entries.py
View File

@@ -1,23 +0,0 @@
"""add_category_to_crm_entries
Revision ID: a1b2c3d4e5f6
Revises: 244a0b0f35be
Create Date: 2026-04-16 09:00:00.000000
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = 'a1b2c3d4e5f6'
down_revision: Union[str, None] = '244a0b0f35be'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column('crm_entries', sa.Column('category', sa.String(length=30), nullable=True))
def downgrade() -> None:
op.drop_column('crm_entries', 'category')

507
backend/alembic/versions/b1c2d3e4f5a6_phase_0_schema_foundation.py
View File

@@ -1,507 +0,0 @@
"""phase_0_schema_foundation
Adds all Phase 0 tables:
- _migration_runs (migration tracking)
- audit_log (staff action audit trail)
- crm_products
- crm_customers
- crm_orders
- crm_comms_log
- crm_media
- crm_sync_state
- crm_quotations
- crm_quotation_items
- staff
- console_settings
- public_features
- melody_drafts
- built_melodies
- mfg_audit_log
- device_alerts
- commands (raw SQL — no ORM model)
- heartbeats (raw SQL — no ORM model)
- device_logs (partitioned by month — raw SQL)
- device_logs_2025_01 … device_logs_2026_06 (initial partitions)
Revision ID: b1c2d3e4f5a6
Revises: a1b2c3d4e5f6
Create Date: 2026-04-17 00:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import ARRAY, JSONB
# revision identifiers
revision: str = "b1c2d3e4f5a6"
down_revision: Union[str, None] = "a1b2c3d4e5f6"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
# ------------------------------------------------------------------ #
# _migration_runs
# ------------------------------------------------------------------ #
op.create_table(
"_migration_runs",
sa.Column("id", sa.BigInteger(), primary_key=True, autoincrement=True),
sa.Column("script_name", sa.String(256), nullable=False),
sa.Column("ran_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
sa.Column("source_rows", sa.BigInteger(), nullable=False, server_default="0"),
sa.Column("dest_rows", sa.BigInteger(), nullable=False, server_default="0"),
sa.Column("success", sa.String(8), nullable=False, server_default="ok"),
sa.Column("notes", sa.Text(), nullable=True),
)
# ------------------------------------------------------------------ #
# audit_log
# ------------------------------------------------------------------ #
op.create_table(
"audit_log",
sa.Column("id", sa.BigInteger(), primary_key=True, autoincrement=True),
sa.Column("occurred_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
sa.Column("actor_id", sa.String(128), nullable=False),
sa.Column("actor_name", sa.String(255), nullable=False),
sa.Column("action", sa.String(64), nullable=False),
sa.Column("entity_type", sa.String(64), nullable=False),
sa.Column("entity_id", sa.String(128), nullable=False),
sa.Column("entity_label", sa.String(500), nullable=True),
sa.Column("changes", JSONB, nullable=True),
sa.Column("meta", JSONB, nullable=True),
)
op.create_index("idx_audit_actor", "audit_log", ["actor_id", "occurred_at"])
op.create_index("idx_audit_entity", "audit_log", ["entity_type","entity_id", "occurred_at"])
op.create_index("idx_audit_action", "audit_log", ["action", "occurred_at"])
op.create_index("idx_audit_occurred", "audit_log", ["occurred_at"])
# ------------------------------------------------------------------ #
# staff
# ------------------------------------------------------------------ #
op.create_table(
"staff",
sa.Column("id", sa.String(128), primary_key=True),
sa.Column("firestore_id", sa.String(128), nullable=True, unique=True),
sa.Column("email", sa.String(256), nullable=False, unique=True),
sa.Column("name", sa.String(255), nullable=False),
sa.Column("role", sa.String(64), nullable=False, server_default="staff"),
sa.Column("permissions", JSONB, nullable=False, server_default="{}"),
sa.Column("hashed_password", sa.String(256), nullable=False),
sa.Column("is_active", sa.Boolean(), nullable=False, server_default="true"),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
)
# ------------------------------------------------------------------ #
# console_settings & public_features
# ------------------------------------------------------------------ #
op.create_table(
"console_settings",
sa.Column("key", sa.String(128), primary_key=True),
sa.Column("value", JSONB, nullable=True),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
)
op.create_table(
"public_features",
sa.Column("key", sa.String(128), primary_key=True),
sa.Column("value", JSONB, nullable=True),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
)
# ------------------------------------------------------------------ #
# crm_products
# ------------------------------------------------------------------ #
op.create_table(
"crm_products",
sa.Column("id", sa.String(128), primary_key=True),
sa.Column("firestore_id", sa.String(128), nullable=True, unique=True),
sa.Column("name", sa.String(500), nullable=False),
sa.Column("sku", sa.String(128), nullable=True),
sa.Column("category", sa.String(128), nullable=True),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("unit_cost", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("currency", sa.String(10), nullable=False, server_default="EUR"),
sa.Column("unit_type", sa.String(32), nullable=False, server_default="pcs"),
sa.Column("is_active", sa.Boolean(), nullable=False, server_default="true"),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
)
# ------------------------------------------------------------------ #
# crm_customers
# ------------------------------------------------------------------ #
op.create_table(
"crm_customers",
sa.Column("id", sa.String(128), primary_key=True),
sa.Column("firestore_id", sa.String(128), nullable=True, unique=True),
sa.Column("title", sa.String(32), nullable=True),
sa.Column("name", sa.String(255), nullable=False),
sa.Column("surname", sa.String(255), nullable=True),
sa.Column("organization", sa.String(500), nullable=True),
sa.Column("religion", sa.String(64), nullable=True),
sa.Column("language", sa.String(10), nullable=False, server_default="el"),
sa.Column("folder_id", sa.String(128), nullable=False, unique=True),
sa.Column("relationship_status", sa.String(64), nullable=False, server_default="lead"),
sa.Column("nextcloud_folder", sa.String(500), nullable=True),
sa.Column("contacts", JSONB, nullable=False, server_default="[]"),
sa.Column("notes", JSONB, nullable=False, server_default="[]"),
sa.Column("location", JSONB, nullable=True),
sa.Column("tags", ARRAY(sa.String()), nullable=False, server_default="{}"),
sa.Column("owned_items", JSONB, nullable=False, server_default="[]"),
sa.Column("linked_user_ids", ARRAY(sa.String()), nullable=False, server_default="{}"),
sa.Column("technical_issues", JSONB, nullable=False, server_default="[]"),
sa.Column("install_support", JSONB, nullable=False, server_default="[]"),
sa.Column("transaction_history", JSONB, nullable=False, server_default="[]"),
sa.Column("crm_summary", JSONB, nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
)
op.create_index("idx_crm_customers_rel_status", "crm_customers", ["relationship_status"])
op.create_index("idx_crm_customers_name", "crm_customers", ["name", "surname"])
op.create_index("idx_crm_customers_tags", "crm_customers", ["tags"],
postgresql_using="gin")
# ------------------------------------------------------------------ #
# crm_orders
# ------------------------------------------------------------------ #
op.create_table(
"crm_orders",
sa.Column("id", sa.String(128), primary_key=True),
sa.Column("customer_id", sa.String(128),
sa.ForeignKey("crm_customers.id", ondelete="CASCADE"), nullable=False),
sa.Column("order_number", sa.String(64), nullable=False, unique=True),
sa.Column("title", sa.String(500), nullable=True),
sa.Column("created_by", sa.String(128), nullable=True),
sa.Column("status", sa.String(64), nullable=False,
server_default="negotiating"),
sa.Column("status_updated_date", sa.DateTime(timezone=True), nullable=True),
sa.Column("status_updated_by", sa.String(128), nullable=True),
sa.Column("items", JSONB, nullable=False, server_default="[]"),
sa.Column("subtotal", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("discount", JSONB, nullable=True),
sa.Column("total_price", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("currency", sa.String(10), nullable=False, server_default="EUR"),
sa.Column("shipping", JSONB, nullable=True),
sa.Column("payment_status", JSONB, nullable=False, server_default="{}"),
sa.Column("invoice_path", sa.String(500), nullable=True),
sa.Column("notes", sa.Text(), nullable=True),
sa.Column("timeline", JSONB, nullable=False, server_default="[]"),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
)
op.create_index("idx_crm_orders_customer", "crm_orders", ["customer_id"])
op.create_index("idx_crm_orders_status", "crm_orders", ["status"])
# ------------------------------------------------------------------ #
# crm_comms_log
# ------------------------------------------------------------------ #
op.create_table(
"crm_comms_log",
sa.Column("id", sa.String(128), primary_key=True),
sa.Column("customer_id", sa.String(128),
sa.ForeignKey("crm_customers.id", ondelete="SET NULL"), nullable=True),
sa.Column("type", sa.String(32), nullable=False),
sa.Column("mail_account", sa.String(256), nullable=True),
sa.Column("direction", sa.String(16), nullable=False),
sa.Column("subject", sa.String(500), nullable=True),
sa.Column("body", sa.Text(), nullable=True),
sa.Column("body_html", sa.Text(), nullable=True),
sa.Column("attachments", JSONB, nullable=False, server_default="[]"),
sa.Column("ext_message_id", sa.String(500), nullable=True),
sa.Column("from_addr", sa.String(500), nullable=True),
sa.Column("to_addrs", sa.Text(), nullable=True),
sa.Column("logged_by", sa.String(128), nullable=True),
sa.Column("is_important", sa.Boolean(), nullable=False, server_default="false"),
sa.Column("is_read", sa.Boolean(), nullable=False, server_default="true"),
sa.Column("occurred_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
)
op.create_index("idx_crm_comms_customer", "crm_comms_log", ["customer_id", "occurred_at"])
# ------------------------------------------------------------------ #
# crm_media
# ------------------------------------------------------------------ #
op.create_table(
"crm_media",
sa.Column("id", sa.String(128), primary_key=True),
sa.Column("customer_id", sa.String(128),
sa.ForeignKey("crm_customers.id", ondelete="SET NULL"), nullable=True),
sa.Column("order_id", sa.String(128), nullable=True),
sa.Column("filename", sa.String(500), nullable=False),
sa.Column("nextcloud_path", sa.String(1000), nullable=False),
sa.Column("thumbnail_path", sa.String(1000), nullable=True),
sa.Column("mime_type", sa.String(128), nullable=True),
sa.Column("direction", sa.String(16), nullable=True),
sa.Column("tags", JSONB, nullable=False, server_default="[]"),
sa.Column("uploaded_by", sa.String(128), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
)
op.create_index("idx_crm_media_customer", "crm_media", ["customer_id"])
op.create_index("idx_crm_media_order", "crm_media", ["order_id"])
# ------------------------------------------------------------------ #
# crm_sync_state
# ------------------------------------------------------------------ #
op.create_table(
"crm_sync_state",
sa.Column("key", sa.String(128), primary_key=True),
sa.Column("value", sa.Text(), nullable=True),
)
# ------------------------------------------------------------------ #
# crm_quotations
# ------------------------------------------------------------------ #
op.create_table(
"crm_quotations",
sa.Column("id", sa.String(128), primary_key=True),
sa.Column("quotation_number", sa.String(64), nullable=False, unique=True),
sa.Column("title", sa.String(500), nullable=True),
sa.Column("subtitle", sa.String(500), nullable=True),
sa.Column("customer_id", sa.String(128),
sa.ForeignKey("crm_customers.id", ondelete="CASCADE"), nullable=False),
sa.Column("language", sa.String(10), nullable=False, server_default="en"),
sa.Column("status", sa.String(32), nullable=False, server_default="draft"),
sa.Column("order_type", sa.String(64), nullable=True),
sa.Column("shipping_method", sa.String(64), nullable=True),
sa.Column("estimated_shipping_date", sa.String(32), nullable=True),
sa.Column("global_discount_label", sa.String(128), nullable=True),
sa.Column("global_discount_percent", sa.Numeric(8, 4), nullable=False, server_default="0"),
sa.Column("vat_percent", sa.Numeric(8, 4), nullable=False, server_default="24"),
sa.Column("global_vat_percent", sa.Numeric(8, 4), nullable=False, server_default="24"),
sa.Column("shipping_cost", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("shipping_cost_discount", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("install_cost", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("install_cost_discount", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("extras_label", sa.String(256), nullable=True),
sa.Column("extras_cost", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("comments", JSONB, nullable=False, server_default="[]"),
sa.Column("quick_notes", JSONB, nullable=False, server_default="{}"),
sa.Column("subtotal_before_discount", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("global_discount_amount", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("new_subtotal", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("vat_amount", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("final_total", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("nextcloud_pdf_path", sa.String(1000), nullable=True),
sa.Column("nextcloud_pdf_url", sa.String(1000), nullable=True),
sa.Column("client_org", sa.String(500), nullable=True),
sa.Column("client_name", sa.String(500), nullable=True),
sa.Column("client_location", sa.String(500), nullable=True),
sa.Column("client_phone", sa.String(64), nullable=True),
sa.Column("client_email", sa.String(256), nullable=True),
sa.Column("is_legacy", sa.Boolean(), nullable=False, server_default="false"),
sa.Column("legacy_date", sa.String(32), nullable=True),
sa.Column("legacy_pdf_path", sa.String(1000), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
)
op.create_index("idx_crm_quotations_customer", "crm_quotations", ["customer_id"])
# ------------------------------------------------------------------ #
# crm_quotation_items
# ------------------------------------------------------------------ #
op.create_table(
"crm_quotation_items",
sa.Column("id", sa.String(128), primary_key=True),
sa.Column("quotation_id", sa.String(128),
sa.ForeignKey("crm_quotations.id", ondelete="CASCADE"), nullable=False),
sa.Column("product_id", sa.String(128), nullable=True),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("description_en", sa.Text(), nullable=True),
sa.Column("description_gr", sa.Text(), nullable=True),
sa.Column("unit_type", sa.String(32), nullable=False, server_default="pcs"),
sa.Column("unit_cost", sa.Numeric(12, 4), nullable=False, server_default="0"),
sa.Column("discount_percent", sa.Numeric(8, 4), nullable=False, server_default="0"),
sa.Column("vat_percent", sa.Numeric(8, 4), nullable=False, server_default="24"),
sa.Column("quantity", sa.Numeric(12, 4), nullable=False, server_default="1"),
sa.Column("line_total", sa.Numeric(12, 2), nullable=False, server_default="0"),
sa.Column("sort_order", sa.Integer(), nullable=False, server_default="0"),
)
op.create_index("idx_crm_quotation_items_quotation", "crm_quotation_items",
["quotation_id", "sort_order"])
# ------------------------------------------------------------------ #
# melody_drafts
# ------------------------------------------------------------------ #
op.create_table(
"melody_drafts",
sa.Column("id", sa.String(128), primary_key=True),
sa.Column("status", sa.String(32), nullable=False, server_default="draft"),
sa.Column("data", JSONB, nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
)
op.create_index("idx_melody_drafts_status", "melody_drafts", ["status"])
# ------------------------------------------------------------------ #
# built_melodies
# ------------------------------------------------------------------ #
op.create_table(
"built_melodies",
sa.Column("id", sa.String(128), primary_key=True),
sa.Column("name", sa.String(500), nullable=False),
sa.Column("pid", sa.String(128), nullable=False),
sa.Column("steps", JSONB, nullable=False),
sa.Column("binary_path", sa.String(1000), nullable=True),
sa.Column("progmem_code", sa.Text(), nullable=True),
sa.Column("assigned_melody_ids", JSONB, nullable=False, server_default="[]"),
sa.Column("is_builtin", sa.Boolean(), nullable=False, server_default="false"),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
)
# ------------------------------------------------------------------ #
# mfg_audit_log
# ------------------------------------------------------------------ #
op.create_table(
"mfg_audit_log",
sa.Column("id", sa.BigInteger(), primary_key=True, autoincrement=True),
sa.Column("timestamp", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
sa.Column("admin_user", sa.String(256), nullable=False),
sa.Column("action", sa.String(128), nullable=False),
sa.Column("serial_number", sa.String(128), nullable=True),
sa.Column("detail", sa.Text(), nullable=True),
)
op.create_index("idx_mfg_audit_time", "mfg_audit_log", ["timestamp"])
op.create_index("idx_mfg_audit_action", "mfg_audit_log", ["action"])
# ------------------------------------------------------------------ #
# device_alerts
# ------------------------------------------------------------------ #
op.create_table(
"device_alerts",
sa.Column("id", sa.BigInteger(), primary_key=True, autoincrement=True),
sa.Column("device_serial", sa.String(128), nullable=False),
sa.Column("subsystem", sa.String(128), nullable=False),
sa.Column("state", sa.String(64), nullable=False),
sa.Column("message", sa.Text(), nullable=True),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False,
server_default=sa.func.now()),
sa.UniqueConstraint("device_serial", "subsystem", name="uq_device_alerts_serial_subsystem"),
)
op.create_index("idx_device_alerts_serial", "device_alerts", ["device_serial"])
# ------------------------------------------------------------------ #
# commands (raw SQL — mirrors SQLite schema, no ORM model)
# ------------------------------------------------------------------ #
op.execute("""
CREATE TABLE commands (
id BIGSERIAL PRIMARY KEY,
device_serial TEXT NOT NULL,
command_name TEXT NOT NULL,
command_payload TEXT,
status TEXT NOT NULL DEFAULT 'pending',
response_payload TEXT,
sent_at TIMESTAMPTZ NOT NULL DEFAULT now(),
responded_at TIMESTAMPTZ
)
""")
op.execute("CREATE INDEX idx_commands_serial_time ON commands(device_serial, sent_at DESC)")
op.execute("CREATE INDEX idx_commands_status ON commands(status)")
# ------------------------------------------------------------------ #
# heartbeats (raw SQL — mirrors SQLite schema, no ORM model)
# ------------------------------------------------------------------ #
op.execute("""
CREATE TABLE heartbeats (
id BIGSERIAL PRIMARY KEY,
device_serial TEXT NOT NULL,
device_id TEXT,
firmware_version TEXT,
ip_address TEXT,
gateway TEXT,
uptime_ms BIGINT,
uptime_display TEXT,
received_at TIMESTAMPTZ NOT NULL DEFAULT now()
)
""")
op.execute("CREATE INDEX idx_heartbeats_serial_time ON heartbeats(device_serial, received_at DESC)")
# ------------------------------------------------------------------ #
# device_logs — partitioned by month on received_at
# ------------------------------------------------------------------ #
op.execute("""
CREATE TABLE device_logs (
id BIGSERIAL,
device_serial TEXT NOT NULL,
level TEXT NOT NULL,
message TEXT NOT NULL,
device_timestamp BIGINT,
received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (id, received_at)
) PARTITION BY RANGE (received_at)
""")
op.execute("""
CREATE INDEX idx_device_logs_serial_time
ON device_logs(device_serial, received_at DESC)
""")
op.execute("""
CREATE INDEX idx_device_logs_level
ON device_logs(level, received_at DESC)
""")
# Create partitions: 2025-01 through 2026-06 (covers all existing data + near future)
partitions = [
("2025_01", "2025-01-01", "2025-02-01"),
("2025_02", "2025-02-01", "2025-03-01"),
("2025_03", "2025-03-01", "2025-04-01"),
("2025_04", "2025-04-01", "2025-05-01"),
("2025_05", "2025-05-01", "2025-06-01"),
("2025_06", "2025-06-01", "2025-07-01"),
("2025_07", "2025-07-01", "2025-08-01"),
("2025_08", "2025-08-01", "2025-09-01"),
("2025_09", "2025-09-01", "2025-10-01"),
("2025_10", "2025-10-01", "2025-11-01"),
("2025_11", "2025-11-01", "2025-12-01"),
("2025_12", "2025-12-01", "2026-01-01"),
("2026_01", "2026-01-01", "2026-02-01"),
("2026_02", "2026-02-01", "2026-03-01"),
("2026_03", "2026-03-01", "2026-04-01"),
("2026_04", "2026-04-01", "2026-05-01"),
("2026_05", "2026-05-01", "2026-06-01"),
("2026_06", "2026-06-01", "2026-07-01"),
]
for suffix, start, end in partitions:
op.execute(f"""
CREATE TABLE device_logs_{suffix} PARTITION OF device_logs
FOR VALUES FROM ('{start}') TO ('{end}')
""")
def downgrade() -> None:
# Drop in reverse dependency order
op.execute("DROP TABLE IF EXISTS device_logs CASCADE") # drops all partitions too
op.execute("DROP TABLE IF EXISTS heartbeats CASCADE")
op.execute("DROP TABLE IF EXISTS commands CASCADE")
op.drop_table("device_alerts")
op.drop_table("mfg_audit_log")
op.drop_table("built_melodies")
op.drop_table("melody_drafts")
op.drop_table("crm_quotation_items")
op.drop_table("crm_quotations")
op.drop_table("crm_sync_state")
op.drop_table("crm_media")
op.drop_table("crm_comms_log")
op.drop_table("crm_orders")
op.drop_table("crm_customers")
op.drop_table("crm_products")
op.drop_table("public_features")
op.drop_table("console_settings")
op.drop_table("staff")
op.drop_table("audit_log")
op.drop_table("_migration_runs")

32
backend/alembic/versions/c3d4e5f6a7b8_phase_3_staff_ui_prefs.py
View File

@@ -1,32 +0,0 @@
"""phase_3_staff_ui_prefs
Adds ui_prefs JSONB column to the staff table (Phase 3 — staff auth cutover).
Also corrects permissions to be nullable (sysadmin/admin have NULL permissions).
Revision ID: c3d4e5f6a7b8
Revises: b1c2d3e4f5a6
Create Date: 2026-04-17 00:00:00.000000
"""
from typing import Sequence, Union
import sqlalchemy as sa
from sqlalchemy.dialects.postgresql import JSONB
from alembic import op
revision: str = "c3d4e5f6a7b8"
down_revision: Union[str, None] = "b1c2d3e4f5a6"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
op.add_column(
"staff",
sa.Column("ui_prefs", JSONB, nullable=False, server_default="{}"),
)
# permissions was NOT NULL DEFAULT '{}' — relax to nullable for sysadmin/admin
op.alter_column("staff", "permissions", nullable=True)
def downgrade() -> None:
op.drop_column("staff", "ui_prefs")
op.alter_column("staff", "permissions", nullable=False)

0
backend/audit/__init__.py
View File

74
backend/audit/router.py
View File

@@ -1,74 +0,0 @@
from datetime import datetime
from typing import Optional
from fastapi import APIRouter, Depends, Query
from sqlalchemy import select, and_
from sqlalchemy.ext.asyncio import AsyncSession
from database.postgres import get_pg_session
from shared.orm import AuditLog
from auth.dependencies import require_sysadmin
from auth.models import TokenPayload
router = APIRouter(prefix="/api/audit-log", tags=["audit-log"])
_MAX_LIMIT = 200
_DEFAULT_LIMIT = 50
@router.get("")
async def list_audit_log(
actor_id: Optional[str] = Query(None),
entity_type: Optional[str] = Query(None),
entity_id: Optional[str] = Query(None),
action: Optional[str] = Query(None),
from_date: Optional[datetime] = Query(None),
to_date: Optional[datetime] = Query(None),
limit: int = Query(_DEFAULT_LIMIT, ge=1, le=_MAX_LIMIT),
offset: int = Query(0, ge=0),
_user: TokenPayload = Depends(require_sysadmin),
db: AsyncSession = Depends(get_pg_session),
):
filters = []
if actor_id:
filters.append(AuditLog.actor_id == actor_id)
if entity_type:
filters.append(AuditLog.entity_type == entity_type)
if entity_id:
filters.append(AuditLog.entity_id == entity_id)
if action:
filters.append(AuditLog.action == action)
if from_date:
filters.append(AuditLog.occurred_at >= from_date)
if to_date:
filters.append(AuditLog.occurred_at <= to_date)
stmt = (
select(AuditLog)
.where(and_(*filters) if filters else True)
.order_by(AuditLog.occurred_at.desc())
.offset(offset)
.limit(limit)
)
result = await db.execute(stmt)
rows = result.scalars().all()
return {
"entries": [
{
"id": r.id,
"occurred_at": r.occurred_at.isoformat(),
"actor_id": r.actor_id,
"actor_name": r.actor_name,
"action": r.action,
"entity_type": r.entity_type,
"entity_id": r.entity_id,
"entity_label": r.entity_label,
"changes": r.changes,
"meta": r.meta,
}
for r in rows
],
"limit": limit,
"offset": offset,
}

31
backend/auth/dependencies.py
View File

@@ -1,14 +1,10 @@
from fastapi import Depends from fastapi import Depends
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from jose import JWTError from jose import JWTError
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from auth.utils import decode_access_token from auth.utils import decode_access_token
from auth.models import TokenPayload, Role from auth.models import TokenPayload, Role
from database.postgres import get_pg_session
from staff.orm import Staff
from shared.exceptions import AuthenticationError, AuthorizationError from shared.exceptions import AuthenticationError, AuthorizationError
from shared.firebase import get_db
security = HTTPBearer() security = HTTPBearer()
@@ -41,15 +37,18 @@ def require_roles(*allowed_roles: Role):
return role_checker return role_checker
async def _get_user_permissions(user: TokenPayload, db: AsyncSession) -> dict | None: async def _get_user_permissions(user: TokenPayload) -> dict:
"""Fetch permissions from Postgres for the given user.""" """Fetch permissions from Firestore for the given user."""
if user.role in (Role.sysadmin, Role.admin): if user.role in (Role.sysadmin, Role.admin):
return None # Full access return None # Full access
result = await db.execute(select(Staff).where(Staff.id == user.sub).limit(1)) db = get_db()
staff = result.scalar_one_or_none() if not db:
if staff is None:
raise AuthorizationError() raise AuthorizationError()
return staff.permissions doc = db.collection("admin_users").document(user.sub).get()
if not doc.exists:
raise AuthorizationError()
data = doc.to_dict()
return data.get("permissions")
def require_permission(section: str, action: str): def require_permission(section: str, action: str):
@@ -59,17 +58,17 @@ def require_permission(section: str, action: str):
""" """
async def permission_checker( async def permission_checker(
current_user: TokenPayload = Depends(get_current_user), current_user: TokenPayload = Depends(get_current_user),
db: AsyncSession = Depends(get_pg_session),
) -> TokenPayload: ) -> TokenPayload:
# sysadmin and admin have full access
if current_user.role in (Role.sysadmin, Role.admin): if current_user.role in (Role.sysadmin, Role.admin):
return current_user return current_user
permissions = await _get_user_permissions(current_user, db) permissions = await _get_user_permissions(current_user)
if not permissions: if not permissions:
raise AuthorizationError() raise AuthorizationError()
if section == "mqtt": if section == "mqtt":
if not permissions.get("mqtt", {}).get("access", False): if not permissions.get("mqtt", False):
raise AuthorizationError() raise AuthorizationError()
return current_user return current_user
@@ -90,7 +89,11 @@ def require_permission(section: str, action: str):
# Pre-built convenience dependencies # Pre-built convenience dependencies
require_sysadmin = require_roles(Role.sysadmin) require_sysadmin = require_roles(Role.sysadmin)
require_admin_or_above = require_roles(Role.sysadmin, Role.admin) require_admin_or_above = require_roles(Role.sysadmin, Role.admin)
# Staff management: only sysadmin and admin
require_staff_management = require_roles(Role.sysadmin, Role.admin) require_staff_management = require_roles(Role.sysadmin, Role.admin)
# Viewer-level: any authenticated user (actual permission check per-action)
require_any_authenticated = require_roles( require_any_authenticated = require_roles(
Role.sysadmin, Role.admin, Role.editor, Role.user, Role.sysadmin, Role.admin, Role.editor, Role.user,
) )

79
backend/auth/router.py
View File

@@ -1,74 +1,59 @@
from fastapi import APIRouter, Depends, Request from fastapi import APIRouter
from sqlalchemy import select from shared.firebase import get_db
from sqlalchemy.ext.asyncio import AsyncSession
from database.postgres import get_pg_session
from staff.orm import Staff
from auth.models import LoginRequest, TokenResponse from auth.models import LoginRequest, TokenResponse
from auth.utils import verify_password, create_access_token from auth.utils import verify_password, create_access_token
from shared.audit import log_action
from shared.exceptions import AuthenticationError from shared.exceptions import AuthenticationError
router = APIRouter(prefix="/api/auth", tags=["auth"]) router = APIRouter(prefix="/api/auth", tags=["auth"])
_ROLE_MAP = {
"superadmin": "sysadmin",
"melody_editor": "editor",
"device_manager": "editor",
"user_manager": "editor",
"viewer": "user",
"staff": "user",
}
@router.post("/login", response_model=TokenResponse) @router.post("/login", response_model=TokenResponse)
async def login( async def login(body: LoginRequest):
body: LoginRequest, db = get_db()
request: Request, if not db:
db: AsyncSession = Depends(get_pg_session), raise AuthenticationError("Service unavailable")
):
result = await db.execute(
select(Staff).where(Staff.email == body.email).limit(1)
)
staff = result.scalar_one_or_none()
if staff is None: users_ref = db.collection("admin_users")
query = users_ref.where("email", "==", body.email).limit(1).get()
if not query:
raise AuthenticationError("Invalid email or password") raise AuthenticationError("Invalid email or password")
if not staff.is_active: doc = query[0]
user_data = doc.to_dict()
if not user_data.get("is_active", True):
raise AuthenticationError("Account is disabled") raise AuthenticationError("Account is disabled")
if not verify_password(body.password, staff.hashed_password): if not verify_password(body.password, user_data["hashed_password"]):
raise AuthenticationError("Invalid email or password") raise AuthenticationError("Invalid email or password")
role = _ROLE_MAP.get(staff.role, staff.role) role = user_data["role"]
# Map legacy roles to new roles
role_mapping = {
"superadmin": "sysadmin",
"melody_editor": "editor",
"device_manager": "editor",
"user_manager": "editor",
"viewer": "user",
}
role = role_mapping.get(role, role)
token = create_access_token({ token = create_access_token({
"sub": staff.id, "sub": doc.id,
"email": staff.email, "email": user_data["email"],
"role": role, "role": role,
"name": staff.name, "name": user_data["name"],
}) })
# Get permissions for editor/user roles
permissions = None permissions = None
if role in ("editor", "user"): if role in ("editor", "user"):
permissions = staff.permissions permissions = user_data.get("permissions")
await log_action(
db,
actor_id=staff.id,
actor_name=staff.name,
action="LOGIN",
entity_type="staff",
entity_id=staff.id,
entity_label=staff.email,
meta={"ip": request.client.host if request.client else None},
)
await db.commit()
return TokenResponse( return TokenResponse(
access_token=token, access_token=token,
role=role, role=role,
name=staff.name, name=user_data["name"],
permissions=permissions, permissions=permissions,
) )

29
backend/builder/database.py
View File

@@ -1,38 +1,27 @@
import json import json
import logging import logging
from database import get_db from mqtt.database import get_db
logger = logging.getLogger("builder.database") logger = logging.getLogger("builder.database")
async def insert_built_melody(melody_id: str, name: str, pid: str, steps: str, is_builtin: bool = False) -> None: async def insert_built_melody(melody_id: str, name: str, pid: str, steps: str) -> None:
db = await get_db() db = await get_db()
await db.execute( await db.execute(
"""INSERT INTO built_melodies (id, name, pid, steps, assigned_melody_ids, is_builtin) """INSERT INTO built_melodies (id, name, pid, steps, assigned_melody_ids)
VALUES (?, ?, ?, ?, ?, ?)""", VALUES (?, ?, ?, ?, ?)""",
(melody_id, name, pid, steps, json.dumps([]), 1 if is_builtin else 0), (melody_id, name, pid, steps, json.dumps([])),
) )
await db.commit() await db.commit()
async def update_built_melody(melody_id: str, name: str, pid: str, steps: str, is_builtin: bool = False) -> None: async def update_built_melody(melody_id: str, name: str, pid: str, steps: str) -> None:
db = await get_db() db = await get_db()
await db.execute( await db.execute(
"""UPDATE built_melodies """UPDATE built_melodies
SET name = ?, pid = ?, steps = ?, is_builtin = ?, updated_at = datetime('now') SET name = ?, pid = ?, steps = ?, updated_at = datetime('now')
WHERE id = ?""", WHERE id = ?""",
(name, pid, steps, 1 if is_builtin else 0, melody_id), (name, pid, steps, melody_id),
)
await db.commit()
async def update_builtin_flag(melody_id: str, is_builtin: bool) -> None:
db = await get_db()
await db.execute(
"""UPDATE built_melodies
SET is_builtin = ?, updated_at = datetime('now')
WHERE id = ?""",
(1 if is_builtin else 0, melody_id),
) )
await db.commit() await db.commit()
@@ -79,7 +68,6 @@ async def get_built_melody(melody_id: str) -> dict | None:
return None return None
row = dict(rows[0]) row = dict(rows[0])
row["assigned_melody_ids"] = json.loads(row["assigned_melody_ids"] or "[]") row["assigned_melody_ids"] = json.loads(row["assigned_melody_ids"] or "[]")
row["is_builtin"] = bool(row.get("is_builtin", 0))
return row return row
@@ -92,7 +80,6 @@ async def list_built_melodies() -> list[dict]:
for row in rows: for row in rows:
r = dict(row) r = dict(row)
r["assigned_melody_ids"] = json.loads(r["assigned_melody_ids"] or "[]") r["assigned_melody_ids"] = json.loads(r["assigned_melody_ids"] or "[]")
r["is_builtin"] = bool(r.get("is_builtin", 0))
results.append(r) results.append(r)
return results return results

3
backend/builder/models.py
View File

@@ -6,14 +6,12 @@ class BuiltMelodyCreate(BaseModel):
name: str name: str
pid: str pid: str
steps: str # raw step string e.g. "1,2,2+1,1,2,3+1" steps: str # raw step string e.g. "1,2,2+1,1,2,3+1"
is_builtin: bool = False
class BuiltMelodyUpdate(BaseModel): class BuiltMelodyUpdate(BaseModel):
name: Optional[str] = None name: Optional[str] = None
pid: Optional[str] = None pid: Optional[str] = None
steps: Optional[str] = None steps: Optional[str] = None
is_builtin: Optional[bool] = None
class BuiltMelodyInDB(BaseModel): class BuiltMelodyInDB(BaseModel):
@@ -21,7 +19,6 @@ class BuiltMelodyInDB(BaseModel):
name: str name: str
pid: str pid: str
steps: str steps: str
is_builtin: bool = False
binary_path: Optional[str] = None binary_path: Optional[str] = None
binary_url: Optional[str] = None binary_url: Optional[str] = None
progmem_code: Optional[str] = None progmem_code: Optional[str] = None

46
backend/builder/router.py
View File

@@ -1,6 +1,5 @@
from fastapi import APIRouter, Depends, HTTPException from fastapi import APIRouter, Depends, HTTPException
from fastapi.responses import FileResponse, PlainTextResponse from fastapi.responses import FileResponse
from sqlalchemy.ext.asyncio import AsyncSession
from auth.models import TokenPayload from auth.models import TokenPayload
from auth.dependencies import require_permission from auth.dependencies import require_permission
from builder.models import ( from builder.models import (
@@ -10,8 +9,6 @@ from builder.models import (
BuiltMelodyListResponse, BuiltMelodyListResponse,
) )
from builder import service from builder import service
from database.postgres import get_pg_session
from shared.audit import log_action
router = APIRouter(prefix="/api/builder/melodies", tags=["builder"]) router = APIRouter(prefix="/api/builder/melodies", tags=["builder"])
@@ -23,7 +20,6 @@ async def list_built_melodies(
melodies = await service.list_built_melodies() melodies = await service.list_built_melodies()
return BuiltMelodyListResponse(melodies=melodies, total=len(melodies)) return BuiltMelodyListResponse(melodies=melodies, total=len(melodies))
@router.get("/for-melody/{firestore_melody_id}") @router.get("/for-melody/{firestore_melody_id}")
async def get_for_firestore_melody( async def get_for_firestore_melody(
firestore_melody_id: str, firestore_melody_id: str,
@@ -36,14 +32,6 @@ async def get_for_firestore_melody(
return result.model_dump() return result.model_dump()
@router.get("/generate-builtin-list")
async def generate_builtin_list(
_user: TokenPayload = Depends(require_permission("melodies", "view")),
):
"""Generate a C++ header with PROGMEM arrays for all is_builtin archetypes."""
code = await service.generate_builtin_list()
return PlainTextResponse(content=code, media_type="text/plain")
@router.get("/{melody_id}", response_model=BuiltMelodyInDB) @router.get("/{melody_id}", response_model=BuiltMelodyInDB)
async def get_built_melody( async def get_built_melody(
@@ -57,12 +45,8 @@ async def get_built_melody(
async def create_built_melody( async def create_built_melody(
body: BuiltMelodyCreate, body: BuiltMelodyCreate,
_user: TokenPayload = Depends(require_permission("melodies", "edit")), _user: TokenPayload = Depends(require_permission("melodies", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
melody = await service.create_built_melody(body) return await service.create_built_melody(body)
await log_action(db, _user.sub, _user.name or _user.email, "CREATE", "archetype",
str(melody.id), melody.name or str(melody.id))
return melody
@router.put("/{melody_id}", response_model=BuiltMelodyInDB) @router.put("/{melody_id}", response_model=BuiltMelodyInDB)
@@ -70,40 +54,16 @@ async def update_built_melody(
melody_id: str, melody_id: str,
body: BuiltMelodyUpdate, body: BuiltMelodyUpdate,
_user: TokenPayload = Depends(require_permission("melodies", "edit")), _user: TokenPayload = Depends(require_permission("melodies", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
old = await service.get_built_melody(melody_id) return await service.update_built_melody(melody_id, body)
melody = await service.update_built_melody(melody_id, body)
_SKIP = {"updated_at", "id", "steps", "builtin_code"}
changes = {
k: {"old": getattr(old, k, None), "new": getattr(melody, k, None)}
for k in body.model_fields_set
if k not in _SKIP and getattr(old, k, None) != getattr(melody, k, None)
}
await log_action(db, _user.sub, _user.name or _user.email, "UPDATE", "archetype",
melody_id, melody.name or melody_id, changes=changes or None)
return melody
@router.delete("/{melody_id}", status_code=204) @router.delete("/{melody_id}", status_code=204)
async def delete_built_melody( async def delete_built_melody(
melody_id: str, melody_id: str,
_user: TokenPayload = Depends(require_permission("melodies", "delete")), _user: TokenPayload = Depends(require_permission("melodies", "delete")),
db: AsyncSession = Depends(get_pg_session),
): ):
melody = await service.get_built_melody(melody_id)
await service.delete_built_melody(melody_id) await service.delete_built_melody(melody_id)
await log_action(db, _user.sub, _user.name or _user.email, "DELETE", "archetype",
melody_id, melody.name if melody else melody_id)
@router.post("/{melody_id}/toggle-builtin", response_model=BuiltMelodyInDB)
async def toggle_builtin(
melody_id: str,
_user: TokenPayload = Depends(require_permission("melodies", "edit")),
):
"""Toggle the is_builtin flag for an archetype."""
return await service.toggle_builtin(melody_id)
@router.post("/{melody_id}/build-binary", response_model=BuiltMelodyInDB) @router.post("/{melody_id}/build-binary", response_model=BuiltMelodyInDB)

137
backend/builder/service.py
View File

@@ -32,7 +32,6 @@ def _row_to_built_melody(row: dict) -> BuiltMelodyInDB:
name=row["name"], name=row["name"],
pid=row["pid"], pid=row["pid"],
steps=row["steps"], steps=row["steps"],
is_builtin=row.get("is_builtin", False),
binary_path=binary_path, binary_path=binary_path,
binary_url=binary_url, binary_url=binary_url,
progmem_code=row.get("progmem_code"), progmem_code=row.get("progmem_code"),
@@ -152,12 +151,8 @@ async def create_built_melody(data: BuiltMelodyCreate) -> BuiltMelodyInDB:
name=data.name, name=data.name,
pid=data.pid, pid=data.pid,
steps=data.steps, steps=data.steps,
is_builtin=data.is_builtin,
) )
# Auto-build binary and builtin code on creation return await get_built_melody(melody_id)
result = await get_built_melody(melody_id)
result = await _do_build(melody_id)
return result
async def update_built_melody(melody_id: str, data: BuiltMelodyUpdate) -> BuiltMelodyInDB: async def update_built_melody(melody_id: str, data: BuiltMelodyUpdate) -> BuiltMelodyInDB:
@@ -168,22 +163,11 @@ async def update_built_melody(melody_id: str, data: BuiltMelodyUpdate) -> BuiltM
new_name = data.name if data.name is not None else row["name"] new_name = data.name if data.name is not None else row["name"]
new_pid = data.pid if data.pid is not None else row["pid"] new_pid = data.pid if data.pid is not None else row["pid"]
new_steps = data.steps if data.steps is not None else row["steps"] new_steps = data.steps if data.steps is not None else row["steps"]
new_is_builtin = data.is_builtin if data.is_builtin is not None else row.get("is_builtin", False)
await _check_unique(new_name, new_pid or "", exclude_id=melody_id) await _check_unique(new_name, new_pid or "", exclude_id=melody_id)
steps_changed = (data.steps is not None) and (data.steps != row["steps"]) await db.update_built_melody(melody_id, name=new_name, pid=new_pid, steps=new_steps)
return await get_built_melody(melody_id)
await db.update_built_melody(melody_id, name=new_name, pid=new_pid, steps=new_steps, is_builtin=new_is_builtin)
# If steps changed, flag all assigned melodies as outdated, then rebuild
if steps_changed:
assigned_ids = row.get("assigned_melody_ids", [])
if assigned_ids:
await _flag_melodies_outdated(assigned_ids, True)
# Auto-rebuild binary and builtin code on every save
return await _do_build(melody_id)
async def delete_built_melody(melody_id: str) -> None: async def delete_built_melody(melody_id: str) -> None:
@@ -191,11 +175,6 @@ async def delete_built_melody(melody_id: str) -> None:
if not row: if not row:
raise HTTPException(status_code=404, detail=f"Built melody '{melody_id}' not found") raise HTTPException(status_code=404, detail=f"Built melody '{melody_id}' not found")
# Flag all assigned melodies as outdated before deleting
assigned_ids = row.get("assigned_melody_ids", [])
if assigned_ids:
await _flag_melodies_outdated(assigned_ids, True)
# Delete the .bsm file if it exists # Delete the .bsm file if it exists
if row.get("binary_path"): if row.get("binary_path"):
bsm_path = Path(row["binary_path"]) bsm_path = Path(row["binary_path"])
@@ -205,26 +184,10 @@ async def delete_built_melody(melody_id: str) -> None:
await db.delete_built_melody(melody_id) await db.delete_built_melody(melody_id)
async def toggle_builtin(melody_id: str) -> BuiltMelodyInDB:
"""Toggle the is_builtin flag for an archetype."""
row = await db.get_built_melody(melody_id)
if not row:
raise HTTPException(status_code=404, detail=f"Built melody '{melody_id}' not found")
new_value = not row.get("is_builtin", False)
await db.update_builtin_flag(melody_id, new_value)
return await get_built_melody(melody_id)
# ============================================================================ # ============================================================================
# Build Actions # Build Actions
# ============================================================================ # ============================================================================
async def _do_build(melody_id: str) -> BuiltMelodyInDB:
"""Internal: build both binary and PROGMEM code, return updated record."""
await build_binary(melody_id)
return await build_builtin_code(melody_id)
async def build_binary(melody_id: str) -> BuiltMelodyInDB: async def build_binary(melody_id: str) -> BuiltMelodyInDB:
"""Parse steps and write a .bsm binary file to storage.""" """Parse steps and write a .bsm binary file to storage."""
row = await db.get_built_melody(melody_id) row = await db.get_built_melody(melody_id)
@@ -273,48 +236,6 @@ async def get_binary_path(melody_id: str) -> Optional[Path]:
return path return path
async def generate_builtin_list() -> str:
"""Generate a C++ header with PROGMEM arrays for all is_builtin archetypes."""
rows = await db.list_built_melodies()
builtin_rows = [r for r in rows if r.get("is_builtin")]
if not builtin_rows:
return "// No built-in archetypes defined.\n"
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
parts = [
f"// Auto-generated Built-in Archetype List",
f"// Generated: {timestamp}",
f"// Total built-ins: {len(builtin_rows)}",
"",
"#pragma once",
"#include <avr/pgmspace.h>",
"",
]
entry_refs = []
for row in builtin_rows:
values = steps_string_to_values(row["steps"])
array_name = f"melody_builtin_{row['name'].lower().replace(' ', '_')}"
display_name = row["name"].replace("_", " ").title()
pid = row.get("pid") or f"builtin_{row['name'].lower()}"
parts.append(f"// {display_name} | PID: {pid} | Steps: {len(values)}")
parts.append(format_melody_array(row["name"].lower().replace(" ", "_"), values))
parts.append("")
entry_refs.append((display_name, pid, array_name, len(values)))
# Generate MELODY_LIBRARY array
parts.append("// --- MELODY_LIBRARY entries ---")
parts.append("// Add these to your firmware's MELODY_LIBRARY[] array:")
parts.append("// {")
for display_name, pid, array_name, step_count in entry_refs:
parts.append(f'// {{ "{display_name}", "{pid}", {array_name}, {step_count} }},')
parts.append("// };")
return "\n".join(parts)
# ============================================================================ # ============================================================================
# Assignment # Assignment
# ============================================================================ # ============================================================================
@@ -330,9 +251,6 @@ async def assign_to_melody(built_id: str, firestore_melody_id: str) -> BuiltMelo
assigned.append(firestore_melody_id) assigned.append(firestore_melody_id)
await db.update_assigned_melody_ids(built_id, assigned) await db.update_assigned_melody_ids(built_id, assigned)
# Clear outdated flag on the melody being assigned
await _flag_melodies_outdated([firestore_melody_id], False)
return await get_built_melody(built_id) return await get_built_melody(built_id)
@@ -344,10 +262,6 @@ async def unassign_from_melody(built_id: str, firestore_melody_id: str) -> Built
assigned = [mid for mid in row.get("assigned_melody_ids", []) if mid != firestore_melody_id] assigned = [mid for mid in row.get("assigned_melody_ids", []) if mid != firestore_melody_id]
await db.update_assigned_melody_ids(built_id, assigned) await db.update_assigned_melody_ids(built_id, assigned)
# Flag the melody as outdated since it no longer has an archetype
await _flag_melodies_outdated([firestore_melody_id], True)
return await get_built_melody(built_id) return await get_built_melody(built_id)
@@ -358,48 +272,3 @@ async def get_built_melody_for_firestore_id(firestore_melody_id: str) -> Optiona
if firestore_melody_id in row.get("assigned_melody_ids", []): if firestore_melody_id in row.get("assigned_melody_ids", []):
return _row_to_built_melody(row) return _row_to_built_melody(row)
return None return None
# ============================================================================
# Outdated Flag Helpers
# ============================================================================
async def _flag_melodies_outdated(melody_ids: List[str], outdated: bool) -> None:
"""Set or clear the outdated_archetype flag on a list of Firestore melody IDs.
This updates both SQLite (melody_drafts) and Firestore (published melodies).
We import inline to avoid circular imports.
"""
if not melody_ids:
return
try:
from melodies import database as melody_db
from shared.firebase import get_db as get_firestore
except ImportError:
logger.warning("Could not import melody/firebase modules — skipping outdated flag update")
return
firestore_db = get_firestore()
for melody_id in melody_ids:
try:
row = await melody_db.get_melody(melody_id)
if not row:
continue
data = row["data"]
info = dict(data.get("information", {}))
info["outdated_archetype"] = outdated
data["information"] = info
await melody_db.update_melody(melody_id, data)
# If published, also update Firestore
if row.get("status") == "published":
doc_ref = firestore_db.collection("melodies").document(melody_id)
doc_ref.update({"information.outdated_archetype": outdated})
logger.info(f"Set outdated_archetype={outdated} on melody {melody_id}")
except Exception as e:
logger.error(f"Failed to set outdated flag on melody {melody_id}: {e}")

8
backend/config.py
View File

@@ -22,17 +22,13 @@ class Settings(BaseSettings):
mosquitto_password_file: str = "/etc/mosquitto/passwd" mosquitto_password_file: str = "/etc/mosquitto/passwd"
mqtt_client_id: str = "bellsystems-admin-panel" mqtt_client_id: str = "bellsystems-admin-panel"
# SQLite (local application database) # SQLite (MQTT data storage)
sqlite_db_path: str = "./data/database.db" sqlite_db_path: str = "./mqtt_data.db"
mqtt_data_retention_days: int = 90 mqtt_data_retention_days: int = 90
# Postgres
database_url: str = "postgresql+asyncpg://bellsystems_user:password@postgres:5432/bellsystems_db"
# Local file storage # Local file storage
built_melodies_storage_path: str = "./storage/built_melodies" built_melodies_storage_path: str = "./storage/built_melodies"
firmware_storage_path: str = "./storage/firmware" firmware_storage_path: str = "./storage/firmware"
flash_assets_storage_path: str = "./storage/flash_assets"
# Email (Resend) # Email (Resend)
resend_api_key: str = "re_placeholder_change_me" resend_api_key: str = "re_placeholder_change_me"

10
backend/crm/comms_router.py
View File

@@ -28,16 +28,6 @@ class MailListResponse(BaseModel):
total: int total: int
@router.get("/latest-batch", response_model=dict)
async def latest_comm_batch(
ids: str = Query(..., description="Comma-separated customer IDs"),
_user: TokenPayload = Depends(require_permission("crm", "view")),
):
"""Return the latest comm summary (id, type, occurred_at) keyed by customer_id."""
customer_ids = [i.strip() for i in ids.split(",") if i.strip()]
return await service.get_latest_comm_batch(customer_ids)
@router.get("/all", response_model=CommListResponse) @router.get("/all", response_model=CommListResponse)
async def list_all_comms( async def list_all_comms(
type: Optional[str] = Query(None), type: Optional[str] = Query(None),

281
backend/crm/customers_router.py
View File

@@ -1,108 +1,28 @@
import asyncio import asyncio
import logging import logging
from fastapi import APIRouter, Depends, Query, BackgroundTasks, Body from fastapi import APIRouter, Depends, Query, BackgroundTasks
from typing import Optional from typing import Optional
from sqlalchemy.ext.asyncio import AsyncSession
from auth.models import TokenPayload from auth.models import TokenPayload
from auth.dependencies import require_permission from auth.dependencies import require_permission
from crm.models import CustomerCreate, CustomerUpdate, CustomerInDB, CustomerListResponse, TransactionEntry from crm.models import CustomerCreate, CustomerUpdate, CustomerInDB, CustomerListResponse
from crm import service, nextcloud from crm import service, nextcloud
from config import settings from config import settings
from database.postgres import get_pg_session
from shared.audit import log_action
router = APIRouter(prefix="/api/crm/customers", tags=["crm-customers"]) router = APIRouter(prefix="/api/crm/customers", tags=["crm-customers"])
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# ── Diff helpers ──────────────────────────────────────────────────────────────
_SCALAR_FIELDS = {
"name", "surname", "title", "organization", "religion", "language",
"relationship_status", "nextcloud_folder",
}
_SKIP_FIELDS = {"updated_at", "firestore_id", "id"}
def _scalar_diff(old, new) -> dict:
result = {}
for f in _SCALAR_FIELDS:
ov = getattr(old, f, None)
nv = getattr(new, f, None)
if ov != nv:
result[f] = {"old": ov, "new": nv}
return result
def _list_diff(field: str, old_list: list, new_list: list, label_fn) -> dict:
old_labels = {label_fn(i) for i in (old_list or [])}
new_labels = {label_fn(i) for i in (new_list or [])}
added = new_labels - old_labels
removed = old_labels - new_labels
result = {}
if added:
result[f"{field}.added"] = {"old": None, "new": sorted(added)}
if removed:
result[f"{field}.removed"] = {"old": sorted(removed), "new": None}
return result
def _customer_diff(old, new, changed_fields: set) -> dict:
changes = _scalar_diff(old, new)
# contacts — keyed by type+value string
if "contacts" in changed_fields:
changes.update(_list_diff(
"contacts",
old.contacts or [],
new.contacts or [],
lambda c: f"{c.get('type','?')}:{c.get('value','?')}" if isinstance(c, dict)
else f"{getattr(c,'type','?')}:{getattr(c,'value','?')}",
))
# location — flatten to individual sub-fields
if "location" in changed_fields:
old_loc = old.location or {}
new_loc = new.location or {}
if isinstance(old_loc, object) and not isinstance(old_loc, dict):
old_loc = old_loc.model_dump() if hasattr(old_loc, "model_dump") else {}
if isinstance(new_loc, object) and not isinstance(new_loc, dict):
new_loc = new_loc.model_dump() if hasattr(new_loc, "model_dump") else {}
for k in set(old_loc) | set(new_loc):
ov, nv = old_loc.get(k), new_loc.get(k)
if ov != nv:
changes[f"location.{k}"] = {"old": ov, "new": nv}
# tags
if "tags" in changed_fields:
old_tags = set(old.tags or [])
new_tags = set(new.tags or [])
if old_tags != new_tags:
changes["tags"] = {"old": sorted(old_tags), "new": sorted(new_tags)}
return changes
@router.get("", response_model=CustomerListResponse) @router.get("", response_model=CustomerListResponse)
async def list_customers( def list_customers(
search: Optional[str] = Query(None), search: Optional[str] = Query(None),
tag: Optional[str] = Query(None), tag: Optional[str] = Query(None),
sort: Optional[str] = Query(None),
_user: TokenPayload = Depends(require_permission("crm", "view")), _user: TokenPayload = Depends(require_permission("crm", "view")),
): ):
customers = service.list_customers(search=search, tag=tag, sort=sort) customers = service.list_customers(search=search, tag=tag)
if sort == "latest_comm":
customers = await service.list_customers_sorted_by_latest_comm(customers)
return CustomerListResponse(customers=customers, total=len(customers)) return CustomerListResponse(customers=customers, total=len(customers))
@router.get("/tags", response_model=list[str])
def list_tags(
_user: TokenPayload = Depends(require_permission("crm", "view")),
):
return service.list_all_tags()
@router.get("/{customer_id}", response_model=CustomerInDB) @router.get("/{customer_id}", response_model=CustomerInDB)
def get_customer( def get_customer(
customer_id: str, customer_id: str,
@@ -116,14 +36,10 @@ async def create_customer(
body: CustomerCreate, body: CustomerCreate,
background_tasks: BackgroundTasks, background_tasks: BackgroundTasks,
_user: TokenPayload = Depends(require_permission("crm", "edit")), _user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
customer = service.create_customer(body) customer = service.create_customer(body)
if settings.nextcloud_url: if settings.nextcloud_url:
background_tasks.add_task(_init_nextcloud_folder, customer) background_tasks.add_task(_init_nextcloud_folder, customer)
label = " ".join(filter(None, [customer.name, customer.surname])) or customer.organization or customer.id
await log_action(db, _user.sub, _user.name or _user.email, "CREATE", "customer",
customer.id, label)
return customer return customer
@@ -139,198 +55,17 @@ async def _init_nextcloud_folder(customer) -> None:
@router.put("/{customer_id}", response_model=CustomerInDB) @router.put("/{customer_id}", response_model=CustomerInDB)
async def update_customer( def update_customer(
customer_id: str, customer_id: str,
body: CustomerUpdate, body: CustomerUpdate,
_user: TokenPayload = Depends(require_permission("crm", "edit")), _user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
old = service.get_customer(customer_id) return service.update_customer(customer_id, body)
customer = service.update_customer(customer_id, body)
label = " ".join(filter(None, [customer.name, customer.surname])) or customer.organization or customer_id
changes = _customer_diff(old, customer, body.model_fields_set)
await log_action(db, _user.sub, _user.name or _user.email, "UPDATE", "customer",
customer_id, label, changes=changes or None)
return customer
@router.delete("/{customer_id}", status_code=204) @router.delete("/{customer_id}", status_code=204)
async def delete_customer( def delete_customer(
customer_id: str, customer_id: str,
wipe_comms: bool = Query(False),
wipe_files: bool = Query(False),
wipe_nextcloud: bool = Query(False),
_user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
customer = service.delete_customer(customer_id)
nc_path = service.get_customer_nc_path(customer)
if wipe_comms or wipe_nextcloud:
await service.delete_customer_comms(customer_id)
if wipe_files or wipe_nextcloud:
await service.delete_customer_media_entries(customer_id)
if settings.nextcloud_url:
folder = f"customers/{nc_path}"
if wipe_nextcloud:
try:
await nextcloud.delete_file(folder)
except Exception as e:
logger.warning("Could not delete NC folder for customer %s: %s", customer_id, e)
elif wipe_files:
stale_folder = f"customers/STALE_{nc_path}"
try:
await nextcloud.rename_folder(folder, stale_folder)
except Exception as e:
logger.warning("Could not rename NC folder for customer %s: %s", customer_id, e)
label = " ".join(filter(None, [customer.name, customer.surname])) or customer.organization or customer_id
await log_action(db, _user.sub, _user.name or _user.email, "DELETE", "customer",
customer_id, label)
@router.get("/{customer_id}/last-comm-direction")
async def get_last_comm_direction(
customer_id: str,
_user: TokenPayload = Depends(require_permission("crm", "view")),
):
result = await service.get_last_comm_direction(customer_id)
return result
# ── Relationship Status ───────────────────────────────────────────────────────
@router.patch("/{customer_id}/relationship-status", response_model=CustomerInDB)
async def update_relationship_status(
customer_id: str,
body: dict = Body(...),
_user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
customer = service.update_relationship_status(customer_id, body.get("status", ""))
label = " ".join(filter(None, [customer.name, customer.surname])) or customer.organization or customer_id
await log_action(db, _user.sub, _user.name or _user.email, "STATUS_CHANGE", "customer",
customer_id, label, meta={"status": body.get("status", "")})
return customer
# ── Technical Issues ──────────────────────────────────────────────────────────
@router.post("/{customer_id}/technical-issues", response_model=CustomerInDB)
def add_technical_issue(
customer_id: str,
body: dict = Body(...),
_user: TokenPayload = Depends(require_permission("crm", "edit")), _user: TokenPayload = Depends(require_permission("crm", "edit")),
): ):
return service.add_technical_issue( service.delete_customer(customer_id)
customer_id,
note=body.get("note", ""),
opened_by=body.get("opened_by", ""),
date=body.get("date"),
)
@router.patch("/{customer_id}/technical-issues/{index}/resolve", response_model=CustomerInDB)
def resolve_technical_issue(
customer_id: str,
index: int,
body: dict = Body(...),
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.resolve_technical_issue(customer_id, index, body.get("resolved_by", ""))
@router.patch("/{customer_id}/technical-issues/{index}", response_model=CustomerInDB)
def edit_technical_issue(
customer_id: str,
index: int,
body: dict = Body(...),
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.edit_technical_issue(customer_id, index, body.get("note", ""), body.get("opened_date"))
@router.delete("/{customer_id}/technical-issues/{index}", response_model=CustomerInDB)
def delete_technical_issue(
customer_id: str,
index: int,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.delete_technical_issue(customer_id, index)
# ── Install Support ───────────────────────────────────────────────────────────
@router.post("/{customer_id}/install-support", response_model=CustomerInDB)
def add_install_support(
customer_id: str,
body: dict = Body(...),
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.add_install_support(
customer_id,
note=body.get("note", ""),
opened_by=body.get("opened_by", ""),
date=body.get("date"),
)
@router.patch("/{customer_id}/install-support/{index}/resolve", response_model=CustomerInDB)
def resolve_install_support(
customer_id: str,
index: int,
body: dict = Body(...),
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.resolve_install_support(customer_id, index, body.get("resolved_by", ""))
@router.patch("/{customer_id}/install-support/{index}", response_model=CustomerInDB)
def edit_install_support(
customer_id: str,
index: int,
body: dict = Body(...),
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.edit_install_support(customer_id, index, body.get("note", ""), body.get("opened_date"))
@router.delete("/{customer_id}/install-support/{index}", response_model=CustomerInDB)
def delete_install_support(
customer_id: str,
index: int,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.delete_install_support(customer_id, index)
# ── Transactions ──────────────────────────────────────────────────────────────
@router.post("/{customer_id}/transactions", response_model=CustomerInDB)
def add_transaction(
customer_id: str,
body: TransactionEntry,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.add_transaction(customer_id, body)
@router.patch("/{customer_id}/transactions/{index}", response_model=CustomerInDB)
def update_transaction(
customer_id: str,
index: int,
body: TransactionEntry,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.update_transaction(customer_id, index, body)
@router.delete("/{customer_id}/transactions/{index}", response_model=CustomerInDB)
def delete_transaction(
customer_id: str,
index: int,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.delete_transaction(customer_id, index)

2
backend/crm/email_sync.py
View File

@@ -23,7 +23,7 @@ from email import encoders
from typing import List, Optional, Tuple from typing import List, Optional, Tuple
from config import settings from config import settings
import database as mqtt_db from mqtt import database as mqtt_db
from crm.mail_accounts import get_mail_accounts, account_by_key, account_by_email from crm.mail_accounts import get_mail_accounts, account_by_key, account_by_email
logger = logging.getLogger("crm.email_sync") logger = logging.getLogger("crm.email_sync")

122
backend/crm/models.py
View File

@@ -1,5 +1,5 @@
from enum import Enum from enum import Enum
from typing import Any, Dict, List, Optional from typing import List, Optional
from pydantic import BaseModel from pydantic import BaseModel
@@ -35,10 +35,6 @@ class ProductCreate(BaseModel):
sku: Optional[str] = None sku: Optional[str] = None
category: ProductCategory category: ProductCategory
description: Optional[str] = None description: Optional[str] = None
name_en: Optional[str] = None
name_gr: Optional[str] = None
description_en: Optional[str] = None
description_gr: Optional[str] = None
price: float price: float
currency: str = "EUR" currency: str = "EUR"
costs: Optional[ProductCosts] = None costs: Optional[ProductCosts] = None
@@ -53,10 +49,6 @@ class ProductUpdate(BaseModel):
sku: Optional[str] = None sku: Optional[str] = None
category: Optional[ProductCategory] = None category: Optional[ProductCategory] = None
description: Optional[str] = None description: Optional[str] = None
name_en: Optional[str] = None
name_gr: Optional[str] = None
description_en: Optional[str] = None
description_gr: Optional[str] = None
price: Optional[float] = None price: Optional[float] = None
currency: Optional[str] = None currency: Optional[str] = None
costs: Optional[ProductCosts] = None costs: Optional[ProductCosts] = None
@@ -122,55 +114,9 @@ class OwnedItem(BaseModel):
class CustomerLocation(BaseModel): class CustomerLocation(BaseModel):
address: Optional[str] = None
city: Optional[str] = None city: Optional[str] = None
postal_code: Optional[str] = None
region: Optional[str] = None
country: Optional[str] = None country: Optional[str] = None
region: Optional[str] = None
# ── New customer status models ────────────────────────────────────────────────
class TechnicalIssue(BaseModel):
active: bool = True
opened_date: str # ISO string
resolved_date: Optional[str] = None
note: str
opened_by: str
resolved_by: Optional[str] = None
class InstallSupportEntry(BaseModel):
active: bool = True
opened_date: str # ISO string
resolved_date: Optional[str] = None
note: str
opened_by: str
resolved_by: Optional[str] = None
class TransactionEntry(BaseModel):
date: str # ISO string
flow: str # "invoice" | "payment" | "refund" | "credit"
payment_type: Optional[str] = None # "cash" | "bank_transfer" | "card" | "paypal" — null for invoices
category: str # "full_payment" | "advance" | "installment"
amount: float
currency: str = "EUR"
invoice_ref: Optional[str] = None
order_ref: Optional[str] = None
recorded_by: str
note: str = ""
# Lightweight summary stored on customer doc for fast CustomerList expanded view
class CrmSummary(BaseModel):
active_order_status: Optional[str] = None
active_order_status_date: Optional[str] = None
active_order_title: Optional[str] = None
active_issues_count: int = 0
latest_issue_date: Optional[str] = None
active_support_count: int = 0
latest_support_date: Optional[str] = None
class CustomerCreate(BaseModel): class CustomerCreate(BaseModel):
@@ -178,7 +124,6 @@ class CustomerCreate(BaseModel):
name: str name: str
surname: Optional[str] = None surname: Optional[str] = None
organization: Optional[str] = None organization: Optional[str] = None
religion: Optional[str] = None
contacts: List[CustomerContact] = [] contacts: List[CustomerContact] = []
notes: List[CustomerNote] = [] notes: List[CustomerNote] = []
location: Optional[CustomerLocation] = None location: Optional[CustomerLocation] = None
@@ -187,12 +132,7 @@ class CustomerCreate(BaseModel):
owned_items: List[OwnedItem] = [] owned_items: List[OwnedItem] = []
linked_user_ids: List[str] = [] linked_user_ids: List[str] = []
nextcloud_folder: Optional[str] = None nextcloud_folder: Optional[str] = None
folder_id: Optional[str] = None folder_id: Optional[str] = None # Human-readable Nextcloud folder name, e.g. "saint-john-corfu"
relationship_status: str = "lead"
technical_issues: List[Dict[str, Any]] = []
install_support: List[Dict[str, Any]] = []
transaction_history: List[Dict[str, Any]] = []
crm_summary: Optional[Dict[str, Any]] = None
class CustomerUpdate(BaseModel): class CustomerUpdate(BaseModel):
@@ -200,7 +140,6 @@ class CustomerUpdate(BaseModel):
name: Optional[str] = None name: Optional[str] = None
surname: Optional[str] = None surname: Optional[str] = None
organization: Optional[str] = None organization: Optional[str] = None
religion: Optional[str] = None
contacts: Optional[List[CustomerContact]] = None contacts: Optional[List[CustomerContact]] = None
notes: Optional[List[CustomerNote]] = None notes: Optional[List[CustomerNote]] = None
location: Optional[CustomerLocation] = None location: Optional[CustomerLocation] = None
@@ -209,7 +148,6 @@ class CustomerUpdate(BaseModel):
owned_items: Optional[List[OwnedItem]] = None owned_items: Optional[List[OwnedItem]] = None
linked_user_ids: Optional[List[str]] = None linked_user_ids: Optional[List[str]] = None
nextcloud_folder: Optional[str] = None nextcloud_folder: Optional[str] = None
relationship_status: Optional[str] = None
# folder_id intentionally excluded from update — set once at creation # folder_id intentionally excluded from update — set once at creation
@@ -227,34 +165,18 @@ class CustomerListResponse(BaseModel):
# ── Orders ─────────────────────────────────────────────────────────────────── # ── Orders ───────────────────────────────────────────────────────────────────
class OrderStatus(str, Enum): class OrderStatus(str, Enum):
negotiating = "negotiating" draft = "draft"
awaiting_quotation = "awaiting_quotation" confirmed = "confirmed"
awaiting_customer_confirmation = "awaiting_customer_confirmation" in_production = "in_production"
awaiting_fulfilment = "awaiting_fulfilment"
awaiting_payment = "awaiting_payment"
manufacturing = "manufacturing"
shipped = "shipped" shipped = "shipped"
installed = "installed" delivered = "delivered"
declined = "declined" cancelled = "cancelled"
complete = "complete"
class OrderPaymentStatus(BaseModel): class PaymentStatus(str, Enum):
required_amount: float = 0 pending = "pending"
received_amount: float = 0 partial = "partial"
balance_due: float = 0 paid = "paid"
advance_required: bool = False
advance_amount: Optional[float] = None
payment_complete: bool = False
class OrderTimelineEvent(BaseModel):
date: str # ISO string
type: str # "quote_request" | "quote_sent" | "quote_accepted" | "quote_declined"
# | "mfg_started" | "mfg_complete" | "order_shipped" | "installed"
# | "payment_received" | "invoice_sent" | "note"
note: str = ""
updated_by: str
class OrderDiscount(BaseModel): class OrderDiscount(BaseModel):
@@ -285,36 +207,29 @@ class OrderItem(BaseModel):
class OrderCreate(BaseModel): class OrderCreate(BaseModel):
customer_id: str customer_id: str
order_number: Optional[str] = None order_number: Optional[str] = None
title: Optional[str] = None status: OrderStatus = OrderStatus.draft
created_by: Optional[str] = None
status: OrderStatus = OrderStatus.negotiating
status_updated_date: Optional[str] = None
status_updated_by: Optional[str] = None
items: List[OrderItem] = [] items: List[OrderItem] = []
subtotal: float = 0 subtotal: float = 0
discount: Optional[OrderDiscount] = None discount: Optional[OrderDiscount] = None
total_price: float = 0 total_price: float = 0
currency: str = "EUR" currency: str = "EUR"
shipping: Optional[OrderShipping] = None shipping: Optional[OrderShipping] = None
payment_status: Optional[Dict[str, Any]] = None payment_status: PaymentStatus = PaymentStatus.pending
invoice_path: Optional[str] = None invoice_path: Optional[str] = None
notes: Optional[str] = None notes: Optional[str] = None
timeline: List[Dict[str, Any]] = []
class OrderUpdate(BaseModel): class OrderUpdate(BaseModel):
customer_id: Optional[str] = None
order_number: Optional[str] = None order_number: Optional[str] = None
title: Optional[str] = None
status: Optional[OrderStatus] = None status: Optional[OrderStatus] = None
status_updated_date: Optional[str] = None
status_updated_by: Optional[str] = None
items: Optional[List[OrderItem]] = None items: Optional[List[OrderItem]] = None
subtotal: Optional[float] = None subtotal: Optional[float] = None
discount: Optional[OrderDiscount] = None discount: Optional[OrderDiscount] = None
total_price: Optional[float] = None total_price: Optional[float] = None
currency: Optional[str] = None currency: Optional[str] = None
shipping: Optional[OrderShipping] = None shipping: Optional[OrderShipping] = None
payment_status: Optional[Dict[str, Any]] = None payment_status: Optional[PaymentStatus] = None
invoice_path: Optional[str] = None invoice_path: Optional[str] = None
notes: Optional[str] = None notes: Optional[str] = None
@@ -371,11 +286,8 @@ class CommCreate(BaseModel):
class CommUpdate(BaseModel): class CommUpdate(BaseModel):
type: Optional[CommType] = None
direction: Optional[CommDirection] = None
subject: Optional[str] = None subject: Optional[str] = None
body: Optional[str] = None body: Optional[str] = None
logged_by: Optional[str] = None
occurred_at: Optional[str] = None occurred_at: Optional[str] = None
@@ -421,7 +333,6 @@ class MediaCreate(BaseModel):
direction: Optional[MediaDirection] = None direction: Optional[MediaDirection] = None
tags: List[str] = [] tags: List[str] = []
uploaded_by: Optional[str] = None uploaded_by: Optional[str] = None
thumbnail_path: Optional[str] = None
class MediaInDB(BaseModel): class MediaInDB(BaseModel):
@@ -435,7 +346,6 @@ class MediaInDB(BaseModel):
tags: List[str] = [] tags: List[str] = []
uploaded_by: Optional[str] = None uploaded_by: Optional[str] = None
created_at: str created_at: str
thumbnail_path: Optional[str] = None
class MediaListResponse(BaseModel): class MediaListResponse(BaseModel):

15
backend/crm/nextcloud.py
View File

@@ -312,18 +312,3 @@ async def delete_file(relative_path: str) -> None:
resp = await client.request("DELETE", url, auth=_auth()) resp = await client.request("DELETE", url, auth=_auth())
if resp.status_code not in (200, 204, 404): if resp.status_code not in (200, 204, 404):
raise HTTPException(status_code=502, detail=f"Nextcloud delete failed: {resp.status_code}") raise HTTPException(status_code=502, detail=f"Nextcloud delete failed: {resp.status_code}")
async def rename_folder(old_relative_path: str, new_relative_path: str) -> None:
"""Rename/move a folder in Nextcloud using WebDAV MOVE."""
url = _full_url(old_relative_path)
destination = _full_url(new_relative_path)
client = _get_client()
resp = await client.request(
"MOVE",
url,
auth=_auth(),
headers={"Destination": destination, "Overwrite": "F"},
)
if resp.status_code not in (201, 204):
raise HTTPException(status_code=502, detail=f"Nextcloud rename failed: {resp.status_code}")

239
backend/crm/nextcloud_router.py
View File

@@ -10,7 +10,6 @@ Folder convention (all paths relative to nextcloud_base_path = BellSystems/Conso
folder_id = customer.folder_id if set, else customer.id (legacy fallback). folder_id = customer.folder_id if set, else customer.id (legacy fallback).
""" """
from fastapi import APIRouter, Depends, Query, UploadFile, File, Form, Response, HTTPException, Request from fastapi import APIRouter, Depends, Query, UploadFile, File, Form, Response, HTTPException, Request
from fastapi.responses import StreamingResponse
from typing import Optional from typing import Optional
from jose import JWTError from jose import JWTError
@@ -18,9 +17,7 @@ from auth.models import TokenPayload
from auth.dependencies import require_permission from auth.dependencies import require_permission
from auth.utils import decode_access_token from auth.utils import decode_access_token
from crm import nextcloud, service from crm import nextcloud, service
from config import settings
from crm.models import MediaCreate, MediaDirection from crm.models import MediaCreate, MediaDirection
from crm.thumbnails import generate_thumbnail
router = APIRouter(prefix="/api/crm/nextcloud", tags=["crm-nextcloud"]) router = APIRouter(prefix="/api/crm/nextcloud", tags=["crm-nextcloud"])
@@ -33,29 +30,6 @@ DIRECTION_MAP = {
} }
@router.get("/web-url")
async def get_web_url(
path: str = Query(..., description="Path relative to nextcloud_base_path"),
_user: TokenPayload = Depends(require_permission("crm", "view")),
):
"""
Return the Nextcloud Files web-UI URL for a given file path.
Opens the parent folder with the file highlighted.
"""
if not settings.nextcloud_url:
raise HTTPException(status_code=503, detail="Nextcloud not configured")
base = settings.nextcloud_base_path.strip("/")
# path is relative to base, e.g. "customers/abc/media/photo.jpg"
parts = path.rsplit("/", 1)
folder_rel = parts[0] if len(parts) == 2 else ""
filename = parts[-1]
nc_dir = f"/{base}/{folder_rel}" if folder_rel else f"/{base}"
from urllib.parse import urlencode, quote
qs = urlencode({"dir": nc_dir, "scrollto": filename})
url = f"{settings.nextcloud_url.rstrip('/')}/index.php/apps/files/?{qs}"
return {"url": url}
@router.get("/browse") @router.get("/browse")
async def browse( async def browse(
path: str = Query(..., description="Path relative to nextcloud_base_path"), path: str = Query(..., description="Path relative to nextcloud_base_path"),
@@ -82,14 +56,6 @@ async def browse_all(
all_files = await nextcloud.list_folder_recursive(base) all_files = await nextcloud.list_folder_recursive(base)
# Exclude _info.txt stubs — human-readable only, should never appear in the UI.
# .thumbs/ files are kept: the frontend needs them to build the thumbnail map
# (it already filters them out of the visible file list itself).
all_files = [
f for f in all_files
if not f["path"].endswith("/_info.txt")
]
# Tag each file with the top-level subfolder it lives under # Tag each file with the top-level subfolder it lives under
for item in all_files: for item in all_files:
parts = item["path"].split("/") parts = item["path"].split("/")
@@ -118,54 +84,33 @@ async def proxy_file(
except (JWTError, KeyError): except (JWTError, KeyError):
raise HTTPException(status_code=403, detail="Invalid token") raise HTTPException(status_code=403, detail="Invalid token")
# Forward the Range header to Nextcloud so we get a true partial response content, mime_type = await nextcloud.download_file(path)
# without buffering the whole file into memory. total = len(content)
nc_url = nextcloud._full_url(path)
nc_auth = nextcloud._auth()
forward_headers = {}
range_header = request.headers.get("range") range_header = request.headers.get("range")
if range_header: if range_header and range_header.startswith("bytes="):
forward_headers["Range"] = range_header # Parse "bytes=start-end"
import httpx as _httpx
# Use a dedicated streaming client — httpx.stream() keeps the connection open
# for the lifetime of the generator, so we can't reuse the shared persistent client.
# We enter the stream context here to get headers immediately (no body buffering),
# then hand the body iterator to StreamingResponse.
stream_client = _httpx.AsyncClient(timeout=None, follow_redirects=True)
nc_resp_ctx = stream_client.stream("GET", nc_url, auth=nc_auth, headers=forward_headers)
nc_resp = await nc_resp_ctx.__aenter__()
if nc_resp.status_code == 404:
await nc_resp_ctx.__aexit__(None, None, None)
await stream_client.aclose()
raise HTTPException(status_code=404, detail="File not found in Nextcloud")
if nc_resp.status_code not in (200, 206):
await nc_resp_ctx.__aexit__(None, None, None)
await stream_client.aclose()
raise HTTPException(status_code=502, detail=f"Nextcloud returned {nc_resp.status_code}")
mime_type = nc_resp.headers.get("content-type", "application/octet-stream").split(";")[0].strip()
resp_headers = {"Accept-Ranges": "bytes"}
for h in ("content-range", "content-length"):
if h in nc_resp.headers:
resp_headers[h.title()] = nc_resp.headers[h]
async def _stream():
try: try:
async for chunk in nc_resp.aiter_bytes(chunk_size=64 * 1024): range_spec = range_header[6:]
yield chunk start_str, _, end_str = range_spec.partition("-")
finally: start = int(start_str) if start_str else 0
await nc_resp_ctx.__aexit__(None, None, None) end = int(end_str) if end_str else total - 1
await stream_client.aclose() end = min(end, total - 1)
chunk = content[start:end + 1]
headers = {
"Content-Range": f"bytes {start}-{end}/{total}",
"Accept-Ranges": "bytes",
"Content-Length": str(len(chunk)),
"Content-Type": mime_type,
}
return Response(content=chunk, status_code=206, headers=headers, media_type=mime_type)
except (ValueError, IndexError):
pass
return StreamingResponse( return Response(
_stream(), content=content,
status_code=nc_resp.status_code,
media_type=mime_type, media_type=mime_type,
headers=resp_headers, headers={"Accept-Ranges": "bytes", "Content-Length": str(total)},
) )
@@ -219,24 +164,6 @@ async def upload_file(
mime_type = file.content_type or "application/octet-stream" mime_type = file.content_type or "application/octet-stream"
await nextcloud.upload_file(file_path, content, mime_type) await nextcloud.upload_file(file_path, content, mime_type)
# Generate and upload thumbnail (best-effort, non-blocking)
# Always stored as {stem}.jpg regardless of source extension so the thumb
# filename is unambiguous and the existence check can never false-positive.
thumb_path = None
try:
thumb_bytes = generate_thumbnail(content, mime_type, file.filename)
if thumb_bytes:
thumb_folder = f"{target_folder}/.thumbs"
stem = file.filename.rsplit(".", 1)[0] if "." in file.filename else file.filename
thumb_filename = f"{stem}.jpg"
thumb_nc_path = f"{thumb_folder}/{thumb_filename}"
await nextcloud.ensure_folder(thumb_folder)
await nextcloud.upload_file(thumb_nc_path, thumb_bytes, "image/jpeg")
thumb_path = thumb_nc_path
except Exception as e:
import logging
logging.getLogger(__name__).warning("Thumbnail generation failed for %s: %s", file.filename, e)
# Resolve direction # Resolve direction
resolved_direction = None resolved_direction = None
if direction: if direction:
@@ -257,7 +184,6 @@ async def upload_file(
direction=resolved_direction, direction=resolved_direction,
tags=tag_list, tags=tag_list,
uploaded_by=_user.name, uploaded_by=_user.name,
thumbnail_path=thumb_path,
)) ))
return media_record return media_record
@@ -318,11 +244,6 @@ async def sync_nextcloud_files(
# Collect all NC files recursively (handles nested folders at any depth) # Collect all NC files recursively (handles nested folders at any depth)
all_nc_files = await nextcloud.list_folder_recursive(base) all_nc_files = await nextcloud.list_folder_recursive(base)
# Skip .thumbs/ folder contents and the _info.txt stub — these are internal
all_nc_files = [
f for f in all_nc_files
if "/.thumbs/" not in f["path"] and not f["path"].endswith("/_info.txt")
]
for item in all_nc_files: for item in all_nc_files:
parts = item["path"].split("/") parts = item["path"].split("/")
item["_subfolder"] = parts[2] if len(parts) > 2 else "media" item["_subfolder"] = parts[2] if len(parts) > 2 else "media"
@@ -353,105 +274,6 @@ async def sync_nextcloud_files(
return {"synced": synced, "skipped": skipped} return {"synced": synced, "skipped": skipped}
@router.post("/generate-thumbs")
async def generate_thumbs(
customer_id: str = Form(...),
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
"""
Scan all customer files in Nextcloud and generate thumbnails for any file
that doesn't already have one in the corresponding .thumbs/ sub-folder.
Skips files inside .thumbs/ itself and file types that can't be thumbnailed.
Returns counts of generated, skipped (already exists), and failed files.
"""
customer = service.get_customer(customer_id)
nc_path = service.get_customer_nc_path(customer)
base = f"customers/{nc_path}"
all_nc_files = await nextcloud.list_folder_recursive(base)
# Build a set of existing thumb paths for O(1) lookup
existing_thumbs = {
f["path"] for f in all_nc_files if "/.thumbs/" in f["path"]
}
# Only process real files (not thumbs themselves)
candidates = [f for f in all_nc_files if "/.thumbs/" not in f["path"]]
generated = 0
skipped = 0
failed = 0
for f in candidates:
# Derive where the thumb would live
path = f["path"] # e.g. customers/{nc_path}/{subfolder}/photo.jpg
parts = path.rsplit("/", 1)
if len(parts) != 2:
skipped += 1
continue
parent_folder, filename = parts
stem = filename.rsplit(".", 1)[0] if "." in filename else filename
thumb_filename = f"{stem}.jpg"
thumb_nc_path = f"{parent_folder}/.thumbs/{thumb_filename}"
if thumb_nc_path in existing_thumbs:
skipped += 1
continue
# Download the file, generate thumb, upload
try:
content, mime_type = await nextcloud.download_file(path)
thumb_bytes = generate_thumbnail(content, mime_type, filename)
if not thumb_bytes:
skipped += 1 # unsupported file type
continue
thumb_folder = f"{parent_folder}/.thumbs"
await nextcloud.ensure_folder(thumb_folder)
await nextcloud.upload_file(thumb_nc_path, thumb_bytes, "image/jpeg")
generated += 1
except Exception as e:
import logging
logging.getLogger(__name__).warning("Thumb gen failed for %s: %s", path, e)
failed += 1
return {"generated": generated, "skipped": skipped, "failed": failed}
@router.post("/clear-thumbs")
async def clear_thumbs(
customer_id: str = Form(...),
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
"""
Delete all .thumbs sub-folders for a customer across all subfolders.
This lets you regenerate thumbnails from scratch.
Returns count of .thumbs folders deleted.
"""
customer = service.get_customer(customer_id)
nc_path = service.get_customer_nc_path(customer)
base = f"customers/{nc_path}"
all_nc_files = await nextcloud.list_folder_recursive(base)
# Collect unique .thumbs folder paths
thumb_folders = set()
for f in all_nc_files:
if "/.thumbs/" in f["path"]:
folder = f["path"].split("/.thumbs/")[0] + "/.thumbs"
thumb_folders.add(folder)
deleted = 0
for folder in thumb_folders:
try:
await nextcloud.delete_file(folder)
deleted += 1
except Exception as e:
import logging
logging.getLogger(__name__).warning("Failed to delete .thumbs folder %s: %s", folder, e)
return {"deleted_folders": deleted}
@router.post("/untrack-deleted") @router.post("/untrack-deleted")
async def untrack_deleted_files( async def untrack_deleted_files(
customer_id: str = Form(...), customer_id: str = Form(...),
@@ -465,22 +287,15 @@ async def untrack_deleted_files(
nc_path = service.get_customer_nc_path(customer) nc_path = service.get_customer_nc_path(customer)
base = f"customers/{nc_path}" base = f"customers/{nc_path}"
# Collect all NC file paths recursively (excluding thumbs and info stub) # Collect all NC file paths recursively
all_nc_files = await nextcloud.list_folder_recursive(base) all_nc_files = await nextcloud.list_folder_recursive(base)
nc_paths = { nc_paths = {item["path"] for item in all_nc_files}
item["path"] for item in all_nc_files
if "/.thumbs/" not in item["path"] and not item["path"].endswith("/_info.txt")
}
# Find DB records whose NC path no longer exists, OR that are internal files # Find DB records whose NC path no longer exists
# (_info.txt / .thumbs/) which should never have been tracked in the first place.
existing = await service.list_media(customer_id=customer_id) existing = await service.list_media(customer_id=customer_id)
untracked = 0 untracked = 0
for m in existing: for m in existing:
is_internal = m.nextcloud_path and ( if m.nextcloud_path and m.nextcloud_path not in nc_paths:
"/.thumbs/" in m.nextcloud_path or m.nextcloud_path.endswith("/_info.txt")
)
if m.nextcloud_path and (is_internal or m.nextcloud_path not in nc_paths):
try: try:
await service.delete_media(m.id) await service.delete_media(m.id)
untracked += 1 untracked += 1

164
backend/crm/orders_router.py
View File

@@ -1,177 +1,57 @@
from fastapi import APIRouter, Depends, Query from fastapi import APIRouter, Depends, Query
from typing import Optional from typing import Optional
from sqlalchemy.ext.asyncio import AsyncSession
from auth.models import TokenPayload from auth.models import TokenPayload
from auth.dependencies import require_permission from auth.dependencies import require_permission
from crm.models import OrderCreate, OrderUpdate, OrderInDB, OrderListResponse from crm.models import OrderCreate, OrderUpdate, OrderInDB, OrderListResponse
from crm import service from crm import service
from database.postgres import get_pg_session
from shared.audit import log_action
router = APIRouter(prefix="/api/crm/customers/{customer_id}/orders", tags=["crm-orders"]) router = APIRouter(prefix="/api/crm/orders", tags=["crm-orders"])
@router.get("", response_model=OrderListResponse) @router.get("", response_model=OrderListResponse)
def list_orders( def list_orders(
customer_id: str, customer_id: Optional[str] = Query(None),
status: Optional[str] = Query(None),
payment_status: Optional[str] = Query(None),
_user: TokenPayload = Depends(require_permission("crm", "view")), _user: TokenPayload = Depends(require_permission("crm", "view")),
): ):
orders = service.list_orders(customer_id) orders = service.list_orders(
return OrderListResponse(orders=orders, total=len(orders))
# IMPORTANT: specific sub-paths must come before /{order_id}
@router.get("/next-order-number")
def get_next_order_number(
customer_id: str,
_user: TokenPayload = Depends(require_permission("crm", "view")),
):
"""Return the next globally unique order number (ORD-DDMMYY-NNN across all customers)."""
return {"order_number": service._generate_order_number(customer_id)}
@router.post("/init-negotiations", response_model=OrderInDB, status_code=201)
async def init_negotiations(
customer_id: str,
body: dict,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
order = service.init_negotiations(
customer_id=customer_id, customer_id=customer_id,
title=body.get("title", ""), status=status,
note=body.get("note", ""), payment_status=payment_status,
date=body.get("date"),
created_by=body.get("created_by", ""),
) )
await log_action(db, _user.sub, _user.name or _user.email, "CREATE", "order", return OrderListResponse(orders=orders, total=len(orders))
order.id, order.order_number or order.id, meta={"action_detail": "negotiations_started"})
return order
@router.post("", response_model=OrderInDB, status_code=201)
async def create_order(
customer_id: str,
body: OrderCreate,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
order = service.create_order(customer_id, body)
await log_action(db, _user.sub, _user.name or _user.email, "CREATE", "order",
order.id, order.order_number or order.id)
return order
@router.get("/{order_id}", response_model=OrderInDB) @router.get("/{order_id}", response_model=OrderInDB)
def get_order( def get_order(
customer_id: str,
order_id: str, order_id: str,
_user: TokenPayload = Depends(require_permission("crm", "view")), _user: TokenPayload = Depends(require_permission("crm", "view")),
): ):
return service.get_order(customer_id, order_id) return service.get_order(order_id)
@router.patch("/{order_id}", response_model=OrderInDB) @router.post("", response_model=OrderInDB, status_code=201)
async def update_order( def create_order(
customer_id: str, body: OrderCreate,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.create_order(body)
@router.put("/{order_id}", response_model=OrderInDB)
def update_order(
order_id: str, order_id: str,
body: OrderUpdate, body: OrderUpdate,
_user: TokenPayload = Depends(require_permission("crm", "edit")), _user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
old = service.get_order(customer_id, order_id) return service.update_order(order_id, body)
order = service.update_order(customer_id, order_id, body)
action = "STATUS_CHANGE" if body.status is not None else "UPDATE"
_SKIP = {"updated_at", "id", "customer_id", "items", "timeline", "discount", "shipping", "payment_status"}
changes = {
k: {"old": getattr(old, k, None), "new": getattr(order, k, None)}
for k in body.model_fields_set
if k not in _SKIP and getattr(old, k, None) != getattr(order, k, None)
}
await log_action(db, _user.sub, _user.name or _user.email, action, "order",
order_id, order.order_number or order_id, changes=changes or None)
return order
@router.delete("/{order_id}", status_code=204) @router.delete("/{order_id}", status_code=204)
async def delete_order( def delete_order(
customer_id: str,
order_id: str, order_id: str,
_user: TokenPayload = Depends(require_permission("crm", "edit")), _user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
service.delete_order(customer_id, order_id) service.delete_order(order_id)
await log_action(db, _user.sub, _user.name or _user.email, "DELETE", "order",
order_id, order_id)
@router.post("/{order_id}/timeline", response_model=OrderInDB)
def append_timeline_event(
customer_id: str,
order_id: str,
body: dict,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.append_timeline_event(customer_id, order_id, body)
@router.patch("/{order_id}/timeline/{index}", response_model=OrderInDB)
def update_timeline_event(
customer_id: str,
order_id: str,
index: int,
body: dict,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.update_timeline_event(customer_id, order_id, index, body)
@router.delete("/{order_id}/timeline/{index}", response_model=OrderInDB)
def delete_timeline_event(
customer_id: str,
order_id: str,
index: int,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.delete_timeline_event(customer_id, order_id, index)
@router.patch("/{order_id}/payment-status", response_model=OrderInDB)
def update_payment_status(
customer_id: str,
order_id: str,
body: dict,
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
return service.update_order_payment_status(customer_id, order_id, body)
# ── Global order list (collection group) ─────────────────────────────────────
# Separate router registered at /api/crm/orders for the global OrderList page
global_router = APIRouter(prefix="/api/crm/orders", tags=["crm-orders-global"])
@global_router.get("")
def list_all_orders(
status: Optional[str] = Query(None),
_user: TokenPayload = Depends(require_permission("crm", "view")),
):
orders = service.list_all_orders(status=status)
# Enrich with customer names
customer_ids = list({o.customer_id for o in orders if o.customer_id})
customer_names: dict[str, str] = {}
for cid in customer_ids:
try:
c = service.get_customer(cid)
parts = [c.name, c.organization] if c.organization else [c.name]
customer_names[cid] = " / ".join(filter(None, parts))
except Exception:
pass
enriched = []
for o in orders:
d = o.model_dump()
d["customer_name"] = customer_names.get(o.customer_id)
enriched.append(d)
return {"orders": enriched, "total": len(enriched)}

239
backend/crm/orm.py
View File

@@ -1,239 +0,0 @@
from datetime import datetime, timezone
from sqlalchemy import (
BigInteger, Boolean, Column, DateTime, ForeignKey, Index, Integer,
Numeric, String, Text, UniqueConstraint,
)
from sqlalchemy.dialects.postgresql import ARRAY, JSONB
from sqlalchemy.orm import relationship
from database.postgres import Base
def _now():
return datetime.now(timezone.utc)
class CrmProduct(Base):
__tablename__ = "crm_products"
id = Column(String(128), primary_key=True) # Firestore doc ID
firestore_id = Column(String(128), unique=True) # same as id during transition
name = Column(String(500), nullable=False)
sku = Column(String(128))
category = Column(String(128))
description = Column(Text)
unit_cost = Column(Numeric(12, 2), nullable=False, default=0)
currency = Column(String(10), nullable=False, default="EUR")
unit_type = Column(String(32), nullable=False, default="pcs")
is_active = Column(Boolean, nullable=False, default=True)
created_at = Column(DateTime(timezone=True), nullable=False, default=_now)
updated_at = Column(DateTime(timezone=True), nullable=False, default=_now, onupdate=_now)
class CrmCustomer(Base):
__tablename__ = "crm_customers"
__table_args__ = (
Index("idx_crm_customers_rel_status", "relationship_status"),
Index("idx_crm_customers_name", "name", "surname"),
Index("idx_crm_customers_tags", "tags", postgresql_using="gin"),
)
id = Column(String(128), primary_key=True) # Firestore doc ID
firestore_id = Column(String(128), unique=True)
title = Column(String(32))
name = Column(String(255), nullable=False)
surname = Column(String(255))
organization = Column(String(500))
religion = Column(String(64))
language = Column(String(10), nullable=False, default="el")
folder_id = Column(String(128), unique=True, nullable=False)
relationship_status = Column(String(64), nullable=False, default="lead")
nextcloud_folder = Column(String(500))
contacts = Column(JSONB, nullable=False, default=list)
notes = Column(JSONB, nullable=False, default=list)
location = Column(JSONB)
tags = Column(ARRAY(String), nullable=False, default=list)
owned_items = Column(JSONB, nullable=False, default=list)
linked_user_ids = Column(ARRAY(String), nullable=False, default=list)
technical_issues = Column(JSONB, nullable=False, default=list)
install_support = Column(JSONB, nullable=False, default=list)
transaction_history = Column(JSONB, nullable=False, default=list)
crm_summary = Column(JSONB)
created_at = Column(DateTime(timezone=True), nullable=False)
updated_at = Column(DateTime(timezone=True), nullable=False)
orders = relationship("CrmOrder", back_populates="customer",
cascade="all, delete-orphan", lazy="noload")
quotations = relationship("CrmQuotation", back_populates="customer",
cascade="all, delete-orphan", lazy="noload")
comms = relationship("CrmCommsLog", back_populates="customer",
cascade="all, delete-orphan", lazy="noload")
media = relationship("CrmMedia", back_populates="customer", lazy="noload")
class CrmOrder(Base):
__tablename__ = "crm_orders"
__table_args__ = (
Index("idx_crm_orders_customer", "customer_id"),
Index("idx_crm_orders_status", "status"),
)
id = Column(String(128), primary_key=True) # Firestore doc ID
customer_id = Column(String(128), ForeignKey("crm_customers.id", ondelete="CASCADE"),
nullable=False)
order_number = Column(String(64), unique=True, nullable=False)
title = Column(String(500))
created_by = Column(String(128))
status = Column(String(64), nullable=False, default="negotiating")
status_updated_date = Column(DateTime(timezone=True))
status_updated_by = Column(String(128))
items = Column(JSONB, nullable=False, default=list)
subtotal = Column(Numeric(12, 2), nullable=False, default=0)
discount = Column(JSONB)
total_price = Column(Numeric(12, 2), nullable=False, default=0)
currency = Column(String(10), nullable=False, default="EUR")
shipping = Column(JSONB)
payment_status = Column(JSONB, nullable=False, default=dict)
invoice_path = Column(String(500))
notes = Column(Text)
timeline = Column(JSONB, nullable=False, default=list)
created_at = Column(DateTime(timezone=True), nullable=False)
updated_at = Column(DateTime(timezone=True), nullable=False)
customer = relationship("CrmCustomer", back_populates="orders")
class CrmCommsLog(Base):
__tablename__ = "crm_comms_log"
__table_args__ = (
Index("idx_crm_comms_customer", "customer_id", "occurred_at"),
)
id = Column(String(128), primary_key=True)
customer_id = Column(String(128), ForeignKey("crm_customers.id", ondelete="SET NULL"),
nullable=True)
type = Column(String(32), nullable=False) # email | sms | call | note | ...
mail_account = Column(String(256))
direction = Column(String(16), nullable=False) # inbound | outbound
subject = Column(String(500))
body = Column(Text)
body_html = Column(Text)
attachments = Column(JSONB, nullable=False, default=list)
ext_message_id = Column(String(500))
from_addr = Column(String(500))
to_addrs = Column(Text) # JSON array as text or comma-sep
logged_by = Column(String(128))
is_important = Column(Boolean, nullable=False, default=False)
is_read = Column(Boolean, nullable=False, default=True)
occurred_at = Column(DateTime(timezone=True), nullable=False)
created_at = Column(DateTime(timezone=True), nullable=False, default=_now)
customer = relationship("CrmCustomer", back_populates="comms")
class CrmMedia(Base):
__tablename__ = "crm_media"
__table_args__ = (
Index("idx_crm_media_customer", "customer_id"),
Index("idx_crm_media_order", "order_id"),
)
id = Column(String(128), primary_key=True)
customer_id = Column(String(128), ForeignKey("crm_customers.id", ondelete="SET NULL"),
nullable=True)
order_id = Column(String(128))
filename = Column(String(500), nullable=False)
nextcloud_path = Column(String(1000), nullable=False)
thumbnail_path = Column(String(1000))
mime_type = Column(String(128))
direction = Column(String(16))
tags = Column(JSONB, nullable=False, default=list)
uploaded_by = Column(String(128))
created_at = Column(DateTime(timezone=True), nullable=False, default=_now)
customer = relationship("CrmCustomer", back_populates="media")
class CrmSyncState(Base):
__tablename__ = "crm_sync_state"
key = Column(String(128), primary_key=True)
value = Column(Text)
class CrmQuotation(Base):
__tablename__ = "crm_quotations"
__table_args__ = (
Index("idx_crm_quotations_customer", "customer_id"),
)
id = Column(String(128), primary_key=True)
quotation_number = Column(String(64), unique=True, nullable=False)
title = Column(String(500))
subtitle = Column(String(500))
customer_id = Column(String(128), ForeignKey("crm_customers.id", ondelete="CASCADE"),
nullable=False)
language = Column(String(10), nullable=False, default="en")
status = Column(String(32), nullable=False, default="draft")
order_type = Column(String(64))
shipping_method = Column(String(64))
estimated_shipping_date = Column(String(32)) # stored as DATE string
global_discount_label = Column(String(128))
global_discount_percent = Column(Numeric(8, 4), nullable=False, default=0)
vat_percent = Column(Numeric(8, 4), nullable=False, default=24)
global_vat_percent = Column(Numeric(8, 4), nullable=False, default=24)
shipping_cost = Column(Numeric(12, 2), nullable=False, default=0)
shipping_cost_discount = Column(Numeric(12, 2), nullable=False, default=0)
install_cost = Column(Numeric(12, 2), nullable=False, default=0)
install_cost_discount = Column(Numeric(12, 2), nullable=False, default=0)
extras_label = Column(String(256))
extras_cost = Column(Numeric(12, 2), nullable=False, default=0)
comments = Column(JSONB, nullable=False, default=list)
quick_notes = Column(JSONB, nullable=False, default=dict)
subtotal_before_discount = Column(Numeric(12, 2), nullable=False, default=0)
global_discount_amount = Column(Numeric(12, 2), nullable=False, default=0)
new_subtotal = Column(Numeric(12, 2), nullable=False, default=0)
vat_amount = Column(Numeric(12, 2), nullable=False, default=0)
final_total = Column(Numeric(12, 2), nullable=False, default=0)
nextcloud_pdf_path = Column(String(1000))
nextcloud_pdf_url = Column(String(1000))
# Client snapshot fields (denormalised for PDF generation)
client_org = Column(String(500))
client_name = Column(String(500))
client_location = Column(String(500))
client_phone = Column(String(64))
client_email = Column(String(256))
# Legacy quotation fields
is_legacy = Column(Boolean, nullable=False, default=False)
legacy_date = Column(String(32))
legacy_pdf_path = Column(String(1000))
created_at = Column(DateTime(timezone=True), nullable=False)
updated_at = Column(DateTime(timezone=True), nullable=False)
customer = relationship("CrmCustomer", back_populates="quotations")
items = relationship("CrmQuotationItem", back_populates="quotation",
cascade="all, delete-orphan",
order_by="CrmQuotationItem.sort_order", lazy="noload")
class CrmQuotationItem(Base):
__tablename__ = "crm_quotation_items"
__table_args__ = (
Index("idx_crm_quotation_items_quotation", "quotation_id", "sort_order"),
)
id = Column(String(128), primary_key=True)
quotation_id = Column(String(128), ForeignKey("crm_quotations.id", ondelete="CASCADE"),
nullable=False)
product_id = Column(String(128))
description = Column(Text)
description_en = Column(Text)
description_gr = Column(Text)
unit_type = Column(String(32), nullable=False, default="pcs")
unit_cost = Column(Numeric(12, 4), nullable=False, default=0)
discount_percent = Column(Numeric(8, 4), nullable=False, default=0)
vat_percent = Column(Numeric(8, 4), nullable=False, default=24)
quantity = Column(Numeric(12, 4), nullable=False, default=1)
line_total = Column(Numeric(12, 2), nullable=False, default=0)
sort_order = Column(Integer, nullable=False, default=0)
quotation = relationship("CrmQuotation", back_populates="items")

23
backend/crm/quotation_models.py
View File

@@ -5,17 +5,14 @@ from pydantic import BaseModel
class QuotationStatus(str, Enum): class QuotationStatus(str, Enum):
draft = "draft" draft = "draft"
built = "built"
sent = "sent" sent = "sent"
accepted = "accepted" accepted = "accepted"
declined = "declined" rejected = "rejected"
class QuotationItemCreate(BaseModel): class QuotationItemCreate(BaseModel):
product_id: Optional[str] = None product_id: Optional[str] = None
description: Optional[str] = None description: Optional[str] = None
description_en: Optional[str] = None
description_gr: Optional[str] = None
unit_type: str = "pcs" # pcs / kg / m unit_type: str = "pcs" # pcs / kg / m
unit_cost: float = 0.0 unit_cost: float = 0.0
discount_percent: float = 0.0 discount_percent: float = 0.0
@@ -40,7 +37,6 @@ class QuotationCreate(BaseModel):
estimated_shipping_date: Optional[str] = None estimated_shipping_date: Optional[str] = None
global_discount_label: Optional[str] = None global_discount_label: Optional[str] = None
global_discount_percent: float = 0.0 global_discount_percent: float = 0.0
global_vat_percent: float = 24.0
shipping_cost: float = 0.0 shipping_cost: float = 0.0
shipping_cost_discount: float = 0.0 shipping_cost_discount: float = 0.0
install_cost: float = 0.0 install_cost: float = 0.0
@@ -56,10 +52,6 @@ class QuotationCreate(BaseModel):
client_location: Optional[str] = None client_location: Optional[str] = None
client_phone: Optional[str] = None client_phone: Optional[str] = None
client_email: Optional[str] = None client_email: Optional[str] = None
# Legacy quotation fields
is_legacy: bool = False
legacy_date: Optional[str] = None # ISO date string, manually set
legacy_pdf_path: Optional[str] = None # Nextcloud path to uploaded PDF
class QuotationUpdate(BaseModel): class QuotationUpdate(BaseModel):
@@ -72,7 +64,6 @@ class QuotationUpdate(BaseModel):
estimated_shipping_date: Optional[str] = None estimated_shipping_date: Optional[str] = None
global_discount_label: Optional[str] = None global_discount_label: Optional[str] = None
global_discount_percent: Optional[float] = None global_discount_percent: Optional[float] = None
global_vat_percent: Optional[float] = None
shipping_cost: Optional[float] = None shipping_cost: Optional[float] = None
shipping_cost_discount: Optional[float] = None shipping_cost_discount: Optional[float] = None
install_cost: Optional[float] = None install_cost: Optional[float] = None
@@ -88,10 +79,6 @@ class QuotationUpdate(BaseModel):
client_location: Optional[str] = None client_location: Optional[str] = None
client_phone: Optional[str] = None client_phone: Optional[str] = None
client_email: Optional[str] = None client_email: Optional[str] = None
# Legacy quotation fields
is_legacy: Optional[bool] = None
legacy_date: Optional[str] = None
legacy_pdf_path: Optional[str] = None
class QuotationInDB(BaseModel): class QuotationInDB(BaseModel):
@@ -107,7 +94,6 @@ class QuotationInDB(BaseModel):
estimated_shipping_date: Optional[str] = None estimated_shipping_date: Optional[str] = None
global_discount_label: Optional[str] = None global_discount_label: Optional[str] = None
global_discount_percent: float = 0.0 global_discount_percent: float = 0.0
global_vat_percent: float = 24.0
shipping_cost: float = 0.0 shipping_cost: float = 0.0
shipping_cost_discount: float = 0.0 shipping_cost_discount: float = 0.0
install_cost: float = 0.0 install_cost: float = 0.0
@@ -132,10 +118,6 @@ class QuotationInDB(BaseModel):
client_location: Optional[str] = None client_location: Optional[str] = None
client_phone: Optional[str] = None client_phone: Optional[str] = None
client_email: Optional[str] = None client_email: Optional[str] = None
# Legacy quotation fields
is_legacy: bool = False
legacy_date: Optional[str] = None
legacy_pdf_path: Optional[str] = None
class QuotationListItem(BaseModel): class QuotationListItem(BaseModel):
@@ -148,9 +130,6 @@ class QuotationListItem(BaseModel):
created_at: str created_at: str
updated_at: str updated_at: str
nextcloud_pdf_url: Optional[str] = None nextcloud_pdf_url: Optional[str] = None
is_legacy: bool = False
legacy_date: Optional[str] = None
legacy_pdf_path: Optional[str] = None
class QuotationListResponse(BaseModel): class QuotationListResponse(BaseModel):

48
backend/crm/quotations_router.py
View File

@@ -1,8 +1,7 @@
from fastapi import APIRouter, Depends, Query, UploadFile, File from fastapi import APIRouter, Depends, Query
from fastapi.responses import StreamingResponse from fastapi.responses import StreamingResponse
from typing import Optional from typing import Optional
import io import io
from sqlalchemy.ext.asyncio import AsyncSession
from auth.dependencies import require_permission from auth.dependencies import require_permission
from auth.models import TokenPayload from auth.models import TokenPayload
@@ -14,8 +13,6 @@ from crm.quotation_models import (
QuotationUpdate, QuotationUpdate,
) )
from crm import quotations_service as svc from crm import quotations_service as svc
from database.postgres import get_pg_session
from shared.audit import log_action
router = APIRouter(prefix="/api/crm/quotations", tags=["crm-quotations"]) router = APIRouter(prefix="/api/crm/quotations", tags=["crm-quotations"])
@@ -31,14 +28,6 @@ async def get_next_number(
return NextNumberResponse(next_number=next_num) return NextNumberResponse(next_number=next_num)
@router.get("/all", response_model=list[dict])
async def list_all_quotations(
_user: TokenPayload = Depends(require_permission("crm", "view")),
):
"""Returns all quotations across all customers, each including customer_name."""
return await svc.list_all_quotations()
@router.get("/customer/{customer_id}", response_model=QuotationListResponse) @router.get("/customer/{customer_id}", response_model=QuotationListResponse)
async def list_quotations_for_customer( async def list_quotations_for_customer(
customer_id: str, customer_id: str,
@@ -75,15 +64,11 @@ async def create_quotation(
body: QuotationCreate, body: QuotationCreate,
generate_pdf: bool = Query(False), generate_pdf: bool = Query(False),
_user: TokenPayload = Depends(require_permission("crm", "edit")), _user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
""" """
Create a quotation. Pass ?generate_pdf=true to immediately generate and upload the PDF. Create a quotation. Pass ?generate_pdf=true to immediately generate and upload the PDF.
""" """
q = await svc.create_quotation(body, generate_pdf=generate_pdf) return await svc.create_quotation(body, generate_pdf=generate_pdf)
await log_action(db, _user.sub, _user.name or _user.email, "CREATE", "quotation",
str(q.id), q.quotation_number or str(q.id))
return q
@router.put("/{quotation_id}", response_model=QuotationInDB) @router.put("/{quotation_id}", response_model=QuotationInDB)
@@ -92,34 +77,19 @@ async def update_quotation(
body: QuotationUpdate, body: QuotationUpdate,
generate_pdf: bool = Query(False), generate_pdf: bool = Query(False),
_user: TokenPayload = Depends(require_permission("crm", "edit")), _user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
""" """
Update a quotation. Pass ?generate_pdf=true to regenerate the PDF. Update a quotation. Pass ?generate_pdf=true to regenerate the PDF.
""" """
old = await svc.get_quotation(quotation_id) return await svc.update_quotation(quotation_id, body, generate_pdf=generate_pdf)
q = await svc.update_quotation(quotation_id, body, generate_pdf=generate_pdf)
_SKIP = {"updated_at", "id", "items", "pdf_path"}
changes = {
k: {"old": getattr(old, k, None), "new": getattr(q, k, None)}
for k in body.model_fields_set
if k not in _SKIP and getattr(old, k, None) != getattr(q, k, None)
}
await log_action(db, _user.sub, _user.name or _user.email, "UPDATE", "quotation",
quotation_id, q.quotation_number or quotation_id, changes=changes or None)
return q
@router.delete("/{quotation_id}", status_code=204) @router.delete("/{quotation_id}", status_code=204)
async def delete_quotation( async def delete_quotation(
quotation_id: str, quotation_id: str,
_user: TokenPayload = Depends(require_permission("crm", "edit")), _user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
q = await svc.get_quotation(quotation_id)
await svc.delete_quotation(quotation_id) await svc.delete_quotation(quotation_id)
await log_action(db, _user.sub, _user.name or _user.email, "DELETE", "quotation",
quotation_id, q.quotation_number if q else quotation_id)
@router.post("/{quotation_id}/regenerate-pdf", response_model=QuotationInDB) @router.post("/{quotation_id}/regenerate-pdf", response_model=QuotationInDB)
@@ -129,15 +99,3 @@ async def regenerate_pdf(
): ):
"""Force PDF regeneration and re-upload to Nextcloud.""" """Force PDF regeneration and re-upload to Nextcloud."""
return await svc.regenerate_pdf(quotation_id) return await svc.regenerate_pdf(quotation_id)
@router.post("/{quotation_id}/legacy-pdf", response_model=QuotationInDB)
async def upload_legacy_pdf(
quotation_id: str,
file: UploadFile = File(...),
_user: TokenPayload = Depends(require_permission("crm", "edit")),
):
"""Upload a PDF file for a legacy quotation and store its Nextcloud path."""
pdf_bytes = await file.read()
filename = file.filename or f"legacy-{quotation_id}.pdf"
return await svc.upload_legacy_pdf(quotation_id, pdf_bytes, filename)

130
backend/crm/quotations_service.py
View File

@@ -19,7 +19,7 @@ from crm.quotation_models import (
QuotationUpdate, QuotationUpdate,
) )
from crm.service import get_customer from crm.service import get_customer
import database as mqtt_db from mqtt import database as mqtt_db
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -42,7 +42,6 @@ def _float(d: Decimal) -> float:
def _calculate_totals( def _calculate_totals(
items: list, items: list,
global_discount_percent: float, global_discount_percent: float,
global_vat_percent: float,
shipping_cost: float, shipping_cost: float,
shipping_cost_discount: float, shipping_cost_discount: float,
install_cost: float, install_cost: float,
@@ -51,20 +50,21 @@ def _calculate_totals(
) -> dict: ) -> dict:
""" """
Calculate all monetary totals using Decimal arithmetic (ROUND_HALF_UP). Calculate all monetary totals using Decimal arithmetic (ROUND_HALF_UP).
VAT is a single global rate applied to items only (not shipping or install). VAT is computed per-item from each item's vat_percent field.
Shipping and install costs carry 0% VAT. Shipping and install costs carry 0% VAT.
Returns a dict of floats ready for DB storage. Returns a dict of floats ready for DB storage.
""" """
# Per-line totals (items only) # Per-line totals and per-item VAT
item_totals = [] item_totals = []
item_vat = Decimal(0)
for item in items: for item in items:
cost = _d(item.get("unit_cost", 0)) cost = _d(item.get("unit_cost", 0))
qty = _d(item.get("quantity", 1)) qty = _d(item.get("quantity", 1))
disc = _d(item.get("discount_percent", 0)) disc = _d(item.get("discount_percent", 0))
net = cost * qty * (1 - disc / 100) net = cost * qty * (1 - disc / 100)
item_totals.append(net) item_totals.append(net)
vat_pct = _d(item.get("vat_percent", 24))
items_net = sum(item_totals, Decimal(0)) item_vat += net * (vat_pct / 100)
# Shipping net (VAT = 0%) # Shipping net (VAT = 0%)
ship_gross = _d(shipping_cost) ship_gross = _d(shipping_cost)
@@ -76,17 +76,16 @@ def _calculate_totals(
install_disc = _d(install_cost_discount) install_disc = _d(install_cost_discount)
install_net = install_gross * (1 - install_disc / 100) install_net = install_gross * (1 - install_disc / 100)
subtotal = items_net + ship_net + install_net subtotal = sum(item_totals, Decimal(0)) + ship_net + install_net
global_disc_pct = _d(global_discount_percent) global_disc_pct = _d(global_discount_percent)
global_disc_amount = subtotal * (global_disc_pct / 100) global_disc_amount = subtotal * (global_disc_pct / 100)
new_subtotal = subtotal - global_disc_amount new_subtotal = subtotal - global_disc_amount
# VAT applies only to items portion, scaled by the global discount ratio # Global discount proportionally reduces VAT too
vat_pct = _d(global_vat_percent) if subtotal > 0:
if subtotal > 0 and items_net > 0: disc_ratio = new_subtotal / subtotal
items_ratio = items_net / subtotal vat_amount = item_vat * disc_ratio
vat_amount = new_subtotal * items_ratio * (vat_pct / 100)
else: else:
vat_amount = Decimal(0) vat_amount = Decimal(0)
@@ -110,16 +109,14 @@ def _calc_line_total(item) -> float:
async def _generate_quotation_number(db) -> str: async def _generate_quotation_number(db) -> str:
now = datetime.utcnow() year = datetime.utcnow().year
yy = now.strftime("%y") prefix = f"QT-{year}-"
mm = now.strftime("%m")
prefix = f"QT-{yy}-{mm}-"
rows = await db.execute_fetchall( rows = await db.execute_fetchall(
"SELECT quotation_number FROM crm_quotations WHERE quotation_number LIKE ? ORDER BY quotation_number DESC LIMIT 1", "SELECT quotation_number FROM crm_quotations WHERE quotation_number LIKE ? ORDER BY quotation_number DESC LIMIT 1",
(f"{prefix}%",), (f"{prefix}%",),
) )
if rows: if rows:
last_num = rows[0][0] # e.g. "QT-26-04-012" last_num = rows[0][0] # e.g. "QT-2026-012"
try: try:
seq = int(last_num[len(prefix):]) + 1 seq = int(last_num[len(prefix):]) + 1
except ValueError: except ValueError:
@@ -156,45 +153,10 @@ async def get_next_number() -> str:
return await _generate_quotation_number(db) return await _generate_quotation_number(db)
async def list_all_quotations() -> list[dict]:
"""Return all quotations across all customers, with customer_name injected."""
from shared.firebase import get_db as get_firestore
db = await mqtt_db.get_db()
rows = await db.execute_fetchall(
"SELECT id, quotation_number, title, customer_id, status, final_total, created_at, updated_at, "
"nextcloud_pdf_url, is_legacy, legacy_date, legacy_pdf_path "
"FROM crm_quotations ORDER BY created_at DESC",
(),
)
items = [dict(r) for r in rows]
# Fetch unique customer names from Firestore in one pass
customer_ids = {i["customer_id"] for i in items if i.get("customer_id")}
customer_names: dict[str, str] = {}
if customer_ids:
fstore = get_firestore()
for cid in customer_ids:
try:
doc = fstore.collection("crm_customers").document(cid).get()
if doc.exists:
d = doc.to_dict()
name_parts = [d.get("name", ""), d.get("surname", "")]
full_name = " ".join(p for p in name_parts if p).strip()
org = (d.get("organization", "") or "").strip()
customer_names[cid] = {"name": full_name or cid, "org": org}
except Exception:
customer_names[cid] = {"name": cid, "org": ""}
for item in items:
info = customer_names.get(item["customer_id"], {"name": "", "org": ""})
item["customer_name"] = info["name"]
item["customer_org"] = info["org"]
return items
async def list_quotations(customer_id: str) -> list[QuotationListItem]: async def list_quotations(customer_id: str) -> list[QuotationListItem]:
db = await mqtt_db.get_db() db = await mqtt_db.get_db()
rows = await db.execute_fetchall( rows = await db.execute_fetchall(
"SELECT id, quotation_number, title, customer_id, status, final_total, created_at, updated_at, " "SELECT id, quotation_number, title, customer_id, status, final_total, created_at, updated_at, nextcloud_pdf_url "
"nextcloud_pdf_url, is_legacy, legacy_date, legacy_pdf_path "
"FROM crm_quotations WHERE customer_id = ? ORDER BY created_at DESC", "FROM crm_quotations WHERE customer_id = ? ORDER BY created_at DESC",
(customer_id,), (customer_id,),
) )
@@ -228,7 +190,6 @@ async def create_quotation(data: QuotationCreate, generate_pdf: bool = False) ->
totals = _calculate_totals( totals = _calculate_totals(
items_raw, items_raw,
data.global_discount_percent, data.global_discount_percent,
data.global_vat_percent,
data.shipping_cost, data.shipping_cost,
data.shipping_cost_discount, data.shipping_cost_discount,
data.install_cost, data.install_cost,
@@ -243,36 +204,33 @@ async def create_quotation(data: QuotationCreate, generate_pdf: bool = False) ->
"""INSERT INTO crm_quotations ( """INSERT INTO crm_quotations (
id, quotation_number, title, subtitle, customer_id, id, quotation_number, title, subtitle, customer_id,
language, status, order_type, shipping_method, estimated_shipping_date, language, status, order_type, shipping_method, estimated_shipping_date,
global_discount_label, global_discount_percent, global_vat_percent, global_discount_label, global_discount_percent,
shipping_cost, shipping_cost_discount, install_cost, install_cost_discount, shipping_cost, shipping_cost_discount, install_cost, install_cost_discount,
extras_label, extras_cost, comments, quick_notes, extras_label, extras_cost, comments, quick_notes,
subtotal_before_discount, global_discount_amount, new_subtotal, vat_amount, final_total, subtotal_before_discount, global_discount_amount, new_subtotal, vat_amount, final_total,
nextcloud_pdf_path, nextcloud_pdf_url, nextcloud_pdf_path, nextcloud_pdf_url,
client_org, client_name, client_location, client_phone, client_email, client_org, client_name, client_location, client_phone, client_email,
is_legacy, legacy_date, legacy_pdf_path,
created_at, updated_at created_at, updated_at
) VALUES ( ) VALUES (
?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, 'draft', ?, ?, ?, ?, 'draft', ?, ?, ?,
?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
NULL, NULL, NULL, NULL,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?,
?, ? ?, ?
)""", )""",
( (
qid, quotation_number, data.title, data.subtitle, data.customer_id, qid, quotation_number, data.title, data.subtitle, data.customer_id,
data.language, data.order_type, data.shipping_method, data.estimated_shipping_date, data.language, data.order_type, data.shipping_method, data.estimated_shipping_date,
data.global_discount_label, data.global_discount_percent, data.global_vat_percent, data.global_discount_label, data.global_discount_percent,
data.shipping_cost, data.shipping_cost_discount, data.install_cost, data.install_cost_discount, data.shipping_cost, data.shipping_cost_discount, data.install_cost, data.install_cost_discount,
data.extras_label, data.extras_cost, comments_json, quick_notes_json, data.extras_label, data.extras_cost, comments_json, quick_notes_json,
totals["subtotal_before_discount"], totals["global_discount_amount"], totals["subtotal_before_discount"], totals["global_discount_amount"],
totals["new_subtotal"], totals["vat_amount"], totals["final_total"], totals["new_subtotal"], totals["vat_amount"], totals["final_total"],
data.client_org, data.client_name, data.client_location, data.client_phone, data.client_email, data.client_org, data.client_name, data.client_location, data.client_phone, data.client_email,
1 if data.is_legacy else 0, data.legacy_date, data.legacy_pdf_path,
now, now, now, now,
), ),
) )
@@ -282,12 +240,11 @@ async def create_quotation(data: QuotationCreate, generate_pdf: bool = False) ->
item_id = str(uuid.uuid4()) item_id = str(uuid.uuid4())
await db.execute( await db.execute(
"""INSERT INTO crm_quotation_items """INSERT INTO crm_quotation_items
(id, quotation_id, product_id, description, description_en, description_gr, (id, quotation_id, product_id, description, unit_type, unit_cost,
unit_type, unit_cost, discount_percent, quantity, vat_percent, line_total, sort_order) discount_percent, quantity, vat_percent, line_total, sort_order)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""", VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
( (
item_id, qid, item.get("product_id"), item.get("description"), item_id, qid, item.get("product_id"), item.get("description"),
item.get("description_en"), item.get("description_gr"),
item.get("unit_type", "pcs"), item.get("unit_cost", 0), item.get("unit_type", "pcs"), item.get("unit_cost", 0),
item.get("discount_percent", 0), item.get("quantity", 1), item.get("discount_percent", 0), item.get("quantity", 1),
item.get("vat_percent", 24), item["line_total"], item.get("sort_order", i), item.get("vat_percent", 24), item["line_total"], item.get("sort_order", i),
@@ -298,7 +255,7 @@ async def create_quotation(data: QuotationCreate, generate_pdf: bool = False) ->
quotation = await get_quotation(qid) quotation = await get_quotation(qid)
if generate_pdf and not data.is_legacy: if generate_pdf:
quotation = await _do_generate_and_upload_pdf(quotation) quotation = await _do_generate_and_upload_pdf(quotation)
return quotation return quotation
@@ -324,11 +281,10 @@ async def update_quotation(quotation_id: str, data: QuotationUpdate, generate_pd
scalar_fields = [ scalar_fields = [
"title", "subtitle", "language", "status", "order_type", "shipping_method", "title", "subtitle", "language", "status", "order_type", "shipping_method",
"estimated_shipping_date", "global_discount_label", "global_discount_percent", "global_vat_percent", "estimated_shipping_date", "global_discount_label", "global_discount_percent",
"shipping_cost", "shipping_cost_discount", "install_cost", "shipping_cost", "shipping_cost_discount", "install_cost",
"install_cost_discount", "extras_label", "extras_cost", "install_cost_discount", "extras_label", "extras_cost",
"client_org", "client_name", "client_location", "client_phone", "client_email", "client_org", "client_name", "client_location", "client_phone", "client_email",
"legacy_date", "legacy_pdf_path",
] ]
for field in scalar_fields: for field in scalar_fields:
@@ -359,7 +315,6 @@ async def update_quotation(quotation_id: str, data: QuotationUpdate, generate_pd
totals = _calculate_totals( totals = _calculate_totals(
items_raw, items_raw,
float(merged.get("global_discount_percent", 0)), float(merged.get("global_discount_percent", 0)),
float(merged.get("global_vat_percent", 24)),
float(merged.get("shipping_cost", 0)), float(merged.get("shipping_cost", 0)),
float(merged.get("shipping_cost_discount", 0)), float(merged.get("shipping_cost_discount", 0)),
float(merged.get("install_cost", 0)), float(merged.get("install_cost", 0)),
@@ -388,12 +343,11 @@ async def update_quotation(quotation_id: str, data: QuotationUpdate, generate_pd
item_id = str(uuid.uuid4()) item_id = str(uuid.uuid4())
await db.execute( await db.execute(
"""INSERT INTO crm_quotation_items """INSERT INTO crm_quotation_items
(id, quotation_id, product_id, description, description_en, description_gr, (id, quotation_id, product_id, description, unit_type, unit_cost,
unit_type, unit_cost, discount_percent, quantity, vat_percent, line_total, sort_order) discount_percent, quantity, vat_percent, line_total, sort_order)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""", VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
( (
item_id, quotation_id, item.get("product_id"), item.get("description"), item_id, quotation_id, item.get("product_id"), item.get("description"),
item.get("description_en"), item.get("description_gr"),
item.get("unit_type", "pcs"), item.get("unit_cost", 0), item.get("unit_type", "pcs"), item.get("unit_cost", 0),
item.get("discount_percent", 0), item.get("quantity", 1), item.get("discount_percent", 0), item.get("quantity", 1),
item.get("vat_percent", 24), item["line_total"], item.get("sort_order", i), item.get("vat_percent", 24), item["line_total"], item.get("sort_order", i),
@@ -534,33 +488,7 @@ async def get_quotation_pdf_bytes(quotation_id: str) -> bytes:
"""Download the PDF for a quotation from Nextcloud and return raw bytes.""" """Download the PDF for a quotation from Nextcloud and return raw bytes."""
from fastapi import HTTPException from fastapi import HTTPException
quotation = await get_quotation(quotation_id) quotation = await get_quotation(quotation_id)
# For legacy quotations, the PDF is at legacy_pdf_path if not quotation.nextcloud_pdf_path:
path = quotation.legacy_pdf_path if quotation.is_legacy else quotation.nextcloud_pdf_path raise HTTPException(status_code=404, detail="No PDF generated for this quotation")
if not path: pdf_bytes, _ = await nextcloud.download_file(quotation.nextcloud_pdf_path)
raise HTTPException(status_code=404, detail="No PDF available for this quotation")
pdf_bytes, _ = await nextcloud.download_file(path)
return pdf_bytes return pdf_bytes
async def upload_legacy_pdf(quotation_id: str, pdf_bytes: bytes, filename: str) -> QuotationInDB:
"""Upload a legacy PDF to Nextcloud and store its path in the quotation record."""
quotation = await get_quotation(quotation_id)
if not quotation.is_legacy:
raise HTTPException(status_code=400, detail="This quotation is not a legacy quotation")
from crm.service import get_customer, get_customer_nc_path
customer = get_customer(quotation.customer_id)
nc_folder = get_customer_nc_path(customer)
await nextcloud.ensure_folder(f"customers/{nc_folder}/quotations")
rel_path = f"customers/{nc_folder}/quotations/{filename}"
await nextcloud.upload_file(rel_path, pdf_bytes, "application/pdf")
db = await mqtt_db.get_db()
now = datetime.utcnow().isoformat()
await db.execute(
"UPDATE crm_quotations SET legacy_pdf_path = ?, updated_at = ? WHERE id = ?",
(rel_path, now, quotation_id),
)
await db.commit()
return await get_quotation(quotation_id)

32
backend/crm/router.py
View File

@@ -3,14 +3,11 @@ from fastapi.responses import FileResponse
from typing import Optional from typing import Optional
import os import os
import shutil import shutil
from sqlalchemy.ext.asyncio import AsyncSession
from auth.models import TokenPayload from auth.models import TokenPayload
from auth.dependencies import require_permission from auth.dependencies import require_permission
from crm.models import ProductCreate, ProductUpdate, ProductInDB, ProductListResponse from crm.models import ProductCreate, ProductUpdate, ProductInDB, ProductListResponse
from crm import service from crm import service
from database.postgres import get_pg_session
from shared.audit import log_action
router = APIRouter(prefix="/api/crm/products", tags=["crm-products"]) router = APIRouter(prefix="/api/crm/products", tags=["crm-products"])
@@ -38,47 +35,28 @@ def get_product(
@router.post("", response_model=ProductInDB, status_code=201) @router.post("", response_model=ProductInDB, status_code=201)
async def create_product( def create_product(
body: ProductCreate, body: ProductCreate,
_user: TokenPayload = Depends(require_permission("crm", "edit")), _user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
product = service.create_product(body) return service.create_product(body)
await log_action(db, _user.sub, _user.name or _user.email, "CREATE", "product",
product.id, product.name)
return product
@router.put("/{product_id}", response_model=ProductInDB) @router.put("/{product_id}", response_model=ProductInDB)
async def update_product( def update_product(
product_id: str, product_id: str,
body: ProductUpdate, body: ProductUpdate,
_user: TokenPayload = Depends(require_permission("crm", "edit")), _user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
old = service.get_product(product_id) return service.update_product(product_id, body)
product = service.update_product(product_id, body)
_SKIP = {"updated_at", "id", "photo_url"}
changes = {
k: {"old": getattr(old, k, None), "new": getattr(product, k, None)}
for k in body.model_fields_set
if k not in _SKIP and getattr(old, k, None) != getattr(product, k, None)
}
await log_action(db, _user.sub, _user.name or _user.email, "UPDATE", "product",
product_id, product.name, changes=changes or None)
return product
@router.delete("/{product_id}", status_code=204) @router.delete("/{product_id}", status_code=204)
async def delete_product( def delete_product(
product_id: str, product_id: str,
_user: TokenPayload = Depends(require_permission("crm", "edit")), _user: TokenPayload = Depends(require_permission("crm", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
product = service.get_product(product_id)
service.delete_product(product_id) service.delete_product(product_id)
await log_action(db, _user.sub, _user.name or _user.email, "DELETE", "product",
product_id, product.name)
@router.post("/{product_id}/photo", response_model=ProductInDB) @router.post("/{product_id}/photo", response_model=ProductInDB)

748
backend/crm/service.py
View File

@@ -1,4 +1,3 @@
import asyncio
import json import json
import uuid import uuid
from datetime import datetime from datetime import datetime
@@ -7,14 +6,13 @@ from fastapi import HTTPException
from shared.firebase import get_db from shared.firebase import get_db
from shared.exceptions import NotFoundError from shared.exceptions import NotFoundError
import re as _re import re as _re
import database as mqtt_db from mqtt import database as mqtt_db
from crm.models import ( from crm.models import (
ProductCreate, ProductUpdate, ProductInDB, ProductCreate, ProductUpdate, ProductInDB,
CustomerCreate, CustomerUpdate, CustomerInDB, CustomerCreate, CustomerUpdate, CustomerInDB,
OrderCreate, OrderUpdate, OrderInDB, OrderCreate, OrderUpdate, OrderInDB,
CommCreate, CommUpdate, CommInDB, CommCreate, CommUpdate, CommInDB,
MediaCreate, MediaInDB, MediaCreate, MediaInDB,
TechnicalIssue, InstallSupportEntry, TransactionEntry,
) )
COLLECTION = "crm_products" COLLECTION = "crm_products"
@@ -22,11 +20,6 @@ COLLECTION = "crm_products"
def _doc_to_product(doc) -> ProductInDB: def _doc_to_product(doc) -> ProductInDB:
data = doc.to_dict() data = doc.to_dict()
# Backfill bilingual fields for existing products that predate the feature
if not data.get("name_en") and data.get("name"):
data["name_en"] = data["name"]
if not data.get("name_gr") and data.get("name"):
data["name_gr"] = data["name"]
return ProductInDB(id=doc.id, **data) return ProductInDB(id=doc.id, **data)
@@ -127,19 +120,14 @@ def delete_product(product_id: str) -> None:
CUSTOMERS_COLLECTION = "crm_customers" CUSTOMERS_COLLECTION = "crm_customers"
_LEGACY_CUSTOMER_FIELDS = {"negotiating", "has_problem"}
def _doc_to_customer(doc) -> CustomerInDB: def _doc_to_customer(doc) -> CustomerInDB:
data = doc.to_dict() data = doc.to_dict()
for f in _LEGACY_CUSTOMER_FIELDS:
data.pop(f, None)
return CustomerInDB(id=doc.id, **data) return CustomerInDB(id=doc.id, **data)
def list_customers( def list_customers(
search: str | None = None, search: str | None = None,
tag: str | None = None, tag: str | None = None,
sort: str | None = None,
) -> list[CustomerInDB]: ) -> list[CustomerInDB]:
db = get_db() db = get_db()
query = db.collection(CUSTOMERS_COLLECTION) query = db.collection(CUSTOMERS_COLLECTION)
@@ -153,64 +141,28 @@ def list_customers(
if search: if search:
s = search.lower() s = search.lower()
s_nospace = s.replace(" ", "")
name_match = s in (customer.name or "").lower() name_match = s in (customer.name or "").lower()
surname_match = s in (customer.surname or "").lower() surname_match = s in (customer.surname or "").lower()
org_match = s in (customer.organization or "").lower() org_match = s in (customer.organization or "").lower()
religion_match = s in (customer.religion or "").lower()
language_match = s in (customer.language or "").lower()
contact_match = any( contact_match = any(
s_nospace in (c.value or "").lower().replace(" ", "") s in (c.value or "").lower()
or s in (c.value or "").lower()
for c in (customer.contacts or []) for c in (customer.contacts or [])
) )
loc = customer.location loc = customer.location or {}
loc_match = bool(loc) and ( loc_match = (
s in (loc.address or "").lower() or s in (loc.get("city", "") or "").lower() or
s in (loc.city or "").lower() or s in (loc.get("country", "") or "").lower() or
s in (loc.postal_code or "").lower() or s in (loc.get("region", "") or "").lower()
s in (loc.region or "").lower() or
s in (loc.country or "").lower()
) )
tag_match = any(s in (t or "").lower() for t in (customer.tags or [])) tag_match = any(s in (t or "").lower() for t in (customer.tags or []))
if not (name_match or surname_match or org_match or religion_match or language_match or contact_match or loc_match or tag_match): if not (name_match or surname_match or org_match or contact_match or loc_match or tag_match):
continue continue
results.append(customer) results.append(customer)
# Sorting (non-latest_comm; latest_comm is handled by the async router wrapper)
_TITLES = {"fr.", "rev.", "archim.", "bp.", "abp.", "met.", "mr.", "mrs.", "ms.", "dr.", "prof."}
def _sort_name(c):
return (c.name or "").lower()
def _sort_surname(c):
return (c.surname or "").lower()
def _sort_default(c):
return c.created_at or ""
if sort == "name":
results.sort(key=_sort_name)
elif sort == "surname":
results.sort(key=_sort_surname)
elif sort == "default":
results.sort(key=_sort_default)
return results return results
def list_all_tags() -> list[str]:
db = get_db()
tags: set[str] = set()
for doc in db.collection(CUSTOMERS_COLLECTION).select(["tags"]).stream():
data = doc.to_dict()
for tag in (data.get("tags") or []):
if tag:
tags.add(tag)
return sorted(tags)
def get_customer(customer_id: str) -> CustomerInDB: def get_customer(customer_id: str) -> CustomerInDB:
db = get_db() db = get_db()
doc = db.collection(CUSTOMERS_COLLECTION).document(customer_id).get() doc = db.collection(CUSTOMERS_COLLECTION).document(customer_id).get()
@@ -254,7 +206,6 @@ def create_customer(data: CustomerCreate) -> CustomerInDB:
def update_customer(customer_id: str, data: CustomerUpdate) -> CustomerInDB: def update_customer(customer_id: str, data: CustomerUpdate) -> CustomerInDB:
from google.cloud.firestore_v1 import DELETE_FIELD
db = get_db() db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id) doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
doc = doc_ref.get() doc = doc_ref.get()
@@ -264,135 +215,35 @@ def update_customer(customer_id: str, data: CustomerUpdate) -> CustomerInDB:
update_data = data.model_dump(exclude_none=True) update_data = data.model_dump(exclude_none=True)
update_data["updated_at"] = datetime.utcnow().isoformat() update_data["updated_at"] = datetime.utcnow().isoformat()
# Fields that should be explicitly deleted from Firestore when set to None
# (exclude_none=True would just skip them, leaving the old value intact)
NULLABLE_FIELDS = {"title", "surname", "organization", "religion"}
set_fields = data.model_fields_set
for field in NULLABLE_FIELDS:
if field in set_fields and getattr(data, field) is None:
update_data[field] = DELETE_FIELD
doc_ref.update(update_data) doc_ref.update(update_data)
updated_doc = doc_ref.get() updated_doc = doc_ref.get()
return _doc_to_customer(updated_doc) return _doc_to_customer(updated_doc)
def delete_customer(customer_id: str) -> None:
async def get_last_comm_direction(customer_id: str) -> dict:
"""Return direction ('inbound'/'outbound') and timestamp of the most recent comm, or None."""
db = await mqtt_db.get_db()
rows = await db.execute_fetchall(
"SELECT direction, COALESCE(occurred_at, created_at) as ts FROM crm_comms_log WHERE customer_id = ? "
"AND direction IN ('inbound', 'outbound') "
"ORDER BY COALESCE(occurred_at, created_at) DESC, created_at DESC LIMIT 1",
(customer_id,),
)
if rows:
return {"direction": rows[0][0], "occurred_at": rows[0][1]}
return {"direction": None, "occurred_at": None}
async def get_last_comm_timestamp(customer_id: str) -> str | None:
"""Return the ISO timestamp of the most recent comm for this customer, or None."""
db = await mqtt_db.get_db()
rows = await db.execute_fetchall(
"SELECT COALESCE(occurred_at, created_at) as ts FROM crm_comms_log "
"WHERE customer_id = ? ORDER BY ts DESC LIMIT 1",
(customer_id,),
)
if rows:
return rows[0][0]
return None
async def get_latest_comm_batch(customer_ids: list[str]) -> dict[str, dict]:
"""Return a dict of customer_id → {id, type, occurred_at} for the latest comm per customer.
Uses a single SQL query — no N+1 regardless of list size.
"""
if not customer_ids:
return {}
db = await mqtt_db.get_db()
placeholders = ",".join("?" * len(customer_ids))
rows = await db.execute_fetchall(
f"""
SELECT customer_id, id, type, COALESCE(occurred_at, created_at) AS ts
FROM crm_comms_log
WHERE customer_id IN ({placeholders})
AND customer_id IS NOT NULL AND customer_id != ''
ORDER BY ts DESC
""",
customer_ids,
)
# Keep only the first (latest) row per customer
result: dict[str, dict] = {}
for row in rows:
cid = row[0]
if cid not in result:
result[cid] = {"id": row[1], "type": row[2], "occurred_at": row[3]}
return result
async def list_customers_sorted_by_latest_comm(customers: list[CustomerInDB]) -> list[CustomerInDB]:
"""Re-sort a list of customers so those with the most recent comm come first."""
timestamps = await asyncio.gather(
*[get_last_comm_timestamp(c.id) for c in customers]
)
paired = list(zip(customers, timestamps))
paired.sort(key=lambda x: x[1] or "", reverse=True)
return [c for c, _ in paired]
def delete_customer(customer_id: str) -> CustomerInDB:
"""Delete customer from Firestore. Returns the customer data (for NC path lookup)."""
db = get_db() db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id) doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
doc = doc_ref.get() doc = doc_ref.get()
if not doc.exists: if not doc.exists:
raise NotFoundError("Customer") raise NotFoundError("Customer")
customer = _doc_to_customer(doc)
doc_ref.delete() doc_ref.delete()
return customer
async def delete_customer_comms(customer_id: str) -> int: # ── Orders ───────────────────────────────────────────────────────────────────
"""Delete all comm log entries for a customer. Returns count deleted."""
db = await mqtt_db.get_db()
cursor = await db.execute(
"DELETE FROM crm_comms_log WHERE customer_id = ?", (customer_id,)
)
await db.commit()
return cursor.rowcount
ORDERS_COLLECTION = "crm_orders"
async def delete_customer_media_entries(customer_id: str) -> int:
"""Delete all media DB entries for a customer. Returns count deleted."""
db = await mqtt_db.get_db()
cursor = await db.execute(
"DELETE FROM crm_media WHERE customer_id = ?", (customer_id,)
)
await db.commit()
return cursor.rowcount
# ── Orders (subcollection under customers/{id}/orders) ────────────────────────
def _doc_to_order(doc) -> OrderInDB: def _doc_to_order(doc) -> OrderInDB:
data = doc.to_dict() data = doc.to_dict()
return OrderInDB(id=doc.id, **data) return OrderInDB(id=doc.id, **data)
def _order_collection(customer_id: str): def _generate_order_number(db) -> str:
db = get_db() year = datetime.utcnow().year
return db.collection(CUSTOMERS_COLLECTION).document(customer_id).collection("orders") prefix = f"ORD-{year}-"
def _generate_order_number(customer_id: str) -> str:
"""Generate next ORD-DDMMYY-NNN across all customers using collection group query."""
db = get_db()
now = datetime.utcnow()
prefix = f"ORD-{now.strftime('%d%m%y')}-"
max_n = 0 max_n = 0
for doc in db.collection_group("orders").stream(): for doc in db.collection(ORDERS_COLLECTION).stream():
data = doc.to_dict() data = doc.to_dict()
num = data.get("order_number", "") num = data.get("order_number", "")
if num and num.startswith(prefix): if num and num.startswith(prefix):
@@ -405,150 +256,50 @@ def _generate_order_number(customer_id: str) -> str:
return f"{prefix}{max_n + 1:03d}" return f"{prefix}{max_n + 1:03d}"
def _default_payment_status() -> dict: def list_orders(
return { customer_id: str | None = None,
"required_amount": 0, status: str | None = None,
"received_amount": 0, payment_status: str | None = None,
"balance_due": 0, ) -> list[OrderInDB]:
"advance_required": False,
"advance_amount": None,
"payment_complete": False,
}
def _recalculate_order_payment_status(customer_id: str, order_id: str) -> None:
"""Recompute an order's payment_status from transaction_history on the customer."""
db = get_db() db = get_db()
cust_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id) query = db.collection(ORDERS_COLLECTION)
cust_data = (cust_ref.get().to_dict()) or {}
txns = cust_data.get("transaction_history") or []
required = sum(float(t.get("amount") or 0) for t in txns
if t.get("order_ref") == order_id and t.get("flow") == "invoice")
received = sum(float(t.get("amount") or 0) for t in txns
if t.get("order_ref") == order_id and t.get("flow") == "payment")
balance_due = required - received
payment_complete = (required > 0 and balance_due <= 0)
order_ref = _order_collection(customer_id).document(order_id)
if not order_ref.get().exists:
return
order_ref.update({
"payment_status": {
"required_amount": required,
"received_amount": received,
"balance_due": balance_due,
"advance_required": False,
"advance_amount": None,
"payment_complete": payment_complete,
},
"updated_at": datetime.utcnow().isoformat(),
})
if customer_id:
def _update_crm_summary(customer_id: str) -> None: query = query.where("customer_id", "==", customer_id)
"""Recompute and store the crm_summary field on the customer document."""
db = get_db()
customer_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
# Load customer for issue/support arrays
customer_doc = customer_ref.get()
if not customer_doc.exists:
return
customer_data = customer_doc.to_dict() or {}
# Active issues
issues = customer_data.get("technical_issues") or []
active_issues = [i for i in issues if i.get("active")]
active_issues_count = len(active_issues)
latest_issue_date = None
if active_issues:
latest_issue_date = max((i.get("opened_date") or "") for i in active_issues) or None
# Active support
support = customer_data.get("install_support") or []
active_support = [s for s in support if s.get("active")]
active_support_count = len(active_support)
latest_support_date = None
if active_support:
latest_support_date = max((s.get("opened_date") or "") for s in active_support) or None
# Active order (most recent non-terminal status)
TERMINAL_STATUSES = {"declined", "complete"}
active_order_status = None
active_order_status_date = None
active_order_title = None
active_order_number = None
latest_order_date = ""
all_order_statuses = []
for doc in _order_collection(customer_id).stream():
data = doc.to_dict() or {}
status = data.get("status", "")
all_order_statuses.append(status)
if status not in TERMINAL_STATUSES:
upd = data.get("status_updated_date") or data.get("created_at") or ""
if upd > latest_order_date:
latest_order_date = upd
active_order_status = status
active_order_status_date = upd
active_order_title = data.get("title")
active_order_number = data.get("order_number")
summary = {
"active_order_status": active_order_status,
"active_order_status_date": active_order_status_date,
"active_order_title": active_order_title,
"active_order_number": active_order_number,
"all_orders_statuses": all_order_statuses,
"active_issues_count": active_issues_count,
"latest_issue_date": latest_issue_date,
"active_support_count": active_support_count,
"latest_support_date": latest_support_date,
}
customer_ref.update({"crm_summary": summary, "updated_at": datetime.utcnow().isoformat()})
def list_orders(customer_id: str) -> list[OrderInDB]:
return [_doc_to_order(doc) for doc in _order_collection(customer_id).stream()]
def list_all_orders(status: str | None = None) -> list[OrderInDB]:
"""Query across all customers using Firestore collection group."""
db = get_db()
query = db.collection_group("orders")
if status: if status:
query = query.where("status", "==", status) query = query.where("status", "==", status)
if payment_status:
query = query.where("payment_status", "==", payment_status)
return [_doc_to_order(doc) for doc in query.stream()] return [_doc_to_order(doc) for doc in query.stream()]
def get_order(customer_id: str, order_id: str) -> OrderInDB: def get_order(order_id: str) -> OrderInDB:
doc = _order_collection(customer_id).document(order_id).get() db = get_db()
doc = db.collection(ORDERS_COLLECTION).document(order_id).get()
if not doc.exists: if not doc.exists:
raise NotFoundError("Order") raise NotFoundError("Order")
return _doc_to_order(doc) return _doc_to_order(doc)
def create_order(customer_id: str, data: OrderCreate) -> OrderInDB: def create_order(data: OrderCreate) -> OrderInDB:
col = _order_collection(customer_id) db = get_db()
now = datetime.utcnow().isoformat() now = datetime.utcnow().isoformat()
order_id = str(uuid.uuid4()) order_id = str(uuid.uuid4())
doc_data = data.model_dump() doc_data = data.model_dump()
doc_data["customer_id"] = customer_id
if not doc_data.get("order_number"): if not doc_data.get("order_number"):
doc_data["order_number"] = _generate_order_number(customer_id) doc_data["order_number"] = _generate_order_number(db)
if not doc_data.get("payment_status"):
doc_data["payment_status"] = _default_payment_status()
if not doc_data.get("status_updated_date"):
doc_data["status_updated_date"] = now
doc_data["created_at"] = now doc_data["created_at"] = now
doc_data["updated_at"] = now doc_data["updated_at"] = now
col.document(order_id).set(doc_data) db.collection(ORDERS_COLLECTION).document(order_id).set(doc_data)
_update_crm_summary(customer_id)
return OrderInDB(id=order_id, **doc_data) return OrderInDB(id=order_id, **doc_data)
def update_order(customer_id: str, order_id: str, data: OrderUpdate) -> OrderInDB: def update_order(order_id: str, data: OrderUpdate) -> OrderInDB:
doc_ref = _order_collection(customer_id).document(order_id) db = get_db()
doc_ref = db.collection(ORDERS_COLLECTION).document(order_id)
doc = doc_ref.get() doc = doc_ref.get()
if not doc.exists: if not doc.exists:
raise NotFoundError("Order") raise NotFoundError("Order")
@@ -557,362 +308,17 @@ def update_order(customer_id: str, order_id: str, data: OrderUpdate) -> OrderInD
update_data["updated_at"] = datetime.utcnow().isoformat() update_data["updated_at"] = datetime.utcnow().isoformat()
doc_ref.update(update_data) doc_ref.update(update_data)
_update_crm_summary(customer_id) updated_doc = doc_ref.get()
result = _doc_to_order(doc_ref.get()) return _doc_to_order(updated_doc)
# Auto-mark customer as inactive when all orders are complete
if update_data.get("status") == "complete":
all_orders = list_orders(customer_id)
if all_orders and all(o.status == "complete" for o in all_orders):
db = get_db()
db.collection(CUSTOMERS_COLLECTION).document(customer_id).update({
"relationship_status": "inactive",
"updated_at": datetime.utcnow().isoformat(),
})
return result
def delete_order(customer_id: str, order_id: str) -> None: def delete_order(order_id: str) -> None:
doc_ref = _order_collection(customer_id).document(order_id) db = get_db()
if not doc_ref.get().exists: doc_ref = db.collection(ORDERS_COLLECTION).document(order_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Order") raise NotFoundError("Order")
doc_ref.delete() doc_ref.delete()
_update_crm_summary(customer_id)
def append_timeline_event(customer_id: str, order_id: str, event: dict) -> OrderInDB:
from google.cloud.firestore_v1 import ArrayUnion
doc_ref = _order_collection(customer_id).document(order_id)
if not doc_ref.get().exists:
raise NotFoundError("Order")
now = datetime.utcnow().isoformat()
doc_ref.update({
"timeline": ArrayUnion([event]),
"status_updated_date": event.get("date", now),
"status_updated_by": event.get("updated_by", ""),
"updated_at": now,
})
return _doc_to_order(doc_ref.get())
def update_timeline_event(customer_id: str, order_id: str, index: int, data: dict) -> OrderInDB:
doc_ref = _order_collection(customer_id).document(order_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Order")
timeline = list(doc.to_dict().get("timeline") or [])
if index < 0 or index >= len(timeline):
raise HTTPException(status_code=404, detail="Timeline index out of range")
timeline[index] = {**timeline[index], **data}
doc_ref.update({"timeline": timeline, "updated_at": datetime.utcnow().isoformat()})
return _doc_to_order(doc_ref.get())
def delete_timeline_event(customer_id: str, order_id: str, index: int) -> OrderInDB:
doc_ref = _order_collection(customer_id).document(order_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Order")
timeline = list(doc.to_dict().get("timeline") or [])
if index < 0 or index >= len(timeline):
raise HTTPException(status_code=404, detail="Timeline index out of range")
timeline.pop(index)
doc_ref.update({"timeline": timeline, "updated_at": datetime.utcnow().isoformat()})
return _doc_to_order(doc_ref.get())
def update_order_payment_status(customer_id: str, order_id: str, payment_data: dict) -> OrderInDB:
doc_ref = _order_collection(customer_id).document(order_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Order")
existing = doc.to_dict().get("payment_status") or _default_payment_status()
existing.update({k: v for k, v in payment_data.items() if v is not None})
doc_ref.update({
"payment_status": existing,
"updated_at": datetime.utcnow().isoformat(),
})
return _doc_to_order(doc_ref.get())
def init_negotiations(customer_id: str, title: str, note: str, date: str, created_by: str) -> OrderInDB:
"""Create a new order with status=negotiating and bump customer relationship_status if needed."""
db = get_db()
customer_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
customer_doc = customer_ref.get()
if not customer_doc.exists:
raise NotFoundError("Customer")
now = datetime.utcnow().isoformat()
order_id = str(uuid.uuid4())
timeline_event = {
"date": date or now,
"type": "note",
"note": note or "",
"updated_by": created_by,
}
doc_data = {
"customer_id": customer_id,
"order_number": _generate_order_number(customer_id),
"title": title,
"created_by": created_by,
"status": "negotiating",
"status_updated_date": date or now,
"status_updated_by": created_by,
"items": [],
"subtotal": 0,
"discount": None,
"total_price": 0,
"currency": "EUR",
"shipping": None,
"payment_status": _default_payment_status(),
"invoice_path": None,
"notes": note or "",
"timeline": [timeline_event],
"created_at": now,
"updated_at": now,
}
_order_collection(customer_id).document(order_id).set(doc_data)
# Upgrade relationship_status only if currently lead or prospect
current_data = customer_doc.to_dict() or {}
current_rel = current_data.get("relationship_status", "lead")
if current_rel in ("lead", "prospect"):
customer_ref.update({"relationship_status": "active", "updated_at": now})
_update_crm_summary(customer_id)
return OrderInDB(id=order_id, **doc_data)
# ── Technical Issues & Install Support ────────────────────────────────────────
def add_technical_issue(customer_id: str, note: str, opened_by: str, date: str | None = None) -> CustomerInDB:
from google.cloud.firestore_v1 import ArrayUnion
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
if not doc_ref.get().exists:
raise NotFoundError("Customer")
now = datetime.utcnow().isoformat()
issue = {
"active": True,
"opened_date": date or now,
"resolved_date": None,
"note": note,
"opened_by": opened_by,
"resolved_by": None,
}
doc_ref.update({"technical_issues": ArrayUnion([issue]), "updated_at": now})
_update_crm_summary(customer_id)
return _doc_to_customer(doc_ref.get())
def resolve_technical_issue(customer_id: str, index: int, resolved_by: str) -> CustomerInDB:
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Customer")
data = doc.to_dict() or {}
issues = list(data.get("technical_issues") or [])
if index < 0 or index >= len(issues):
raise HTTPException(status_code=404, detail="Issue index out of range")
now = datetime.utcnow().isoformat()
issues[index] = {**issues[index], "active": False, "resolved_date": now, "resolved_by": resolved_by}
doc_ref.update({"technical_issues": issues, "updated_at": now})
_update_crm_summary(customer_id)
return _doc_to_customer(doc_ref.get())
def add_install_support(customer_id: str, note: str, opened_by: str, date: str | None = None) -> CustomerInDB:
from google.cloud.firestore_v1 import ArrayUnion
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
if not doc_ref.get().exists:
raise NotFoundError("Customer")
now = datetime.utcnow().isoformat()
entry = {
"active": True,
"opened_date": date or now,
"resolved_date": None,
"note": note,
"opened_by": opened_by,
"resolved_by": None,
}
doc_ref.update({"install_support": ArrayUnion([entry]), "updated_at": now})
_update_crm_summary(customer_id)
return _doc_to_customer(doc_ref.get())
def resolve_install_support(customer_id: str, index: int, resolved_by: str) -> CustomerInDB:
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Customer")
data = doc.to_dict() or {}
entries = list(data.get("install_support") or [])
if index < 0 or index >= len(entries):
raise HTTPException(status_code=404, detail="Support index out of range")
now = datetime.utcnow().isoformat()
entries[index] = {**entries[index], "active": False, "resolved_date": now, "resolved_by": resolved_by}
doc_ref.update({"install_support": entries, "updated_at": now})
_update_crm_summary(customer_id)
return _doc_to_customer(doc_ref.get())
def edit_technical_issue(customer_id: str, index: int, note: str, opened_date: str | None = None) -> CustomerInDB:
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Customer")
data = doc.to_dict() or {}
issues = list(data.get("technical_issues") or [])
if index < 0 or index >= len(issues):
raise HTTPException(status_code=404, detail="Issue index out of range")
issues[index] = {**issues[index], "note": note}
if opened_date:
issues[index]["opened_date"] = opened_date
doc_ref.update({"technical_issues": issues, "updated_at": datetime.utcnow().isoformat()})
_update_crm_summary(customer_id)
return _doc_to_customer(doc_ref.get())
def delete_technical_issue(customer_id: str, index: int) -> CustomerInDB:
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Customer")
data = doc.to_dict() or {}
issues = list(data.get("technical_issues") or [])
if index < 0 or index >= len(issues):
raise HTTPException(status_code=404, detail="Issue index out of range")
issues.pop(index)
doc_ref.update({"technical_issues": issues, "updated_at": datetime.utcnow().isoformat()})
_update_crm_summary(customer_id)
return _doc_to_customer(doc_ref.get())
def edit_install_support(customer_id: str, index: int, note: str, opened_date: str | None = None) -> CustomerInDB:
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Customer")
data = doc.to_dict() or {}
entries = list(data.get("install_support") or [])
if index < 0 or index >= len(entries):
raise HTTPException(status_code=404, detail="Support index out of range")
entries[index] = {**entries[index], "note": note}
if opened_date:
entries[index]["opened_date"] = opened_date
doc_ref.update({"install_support": entries, "updated_at": datetime.utcnow().isoformat()})
_update_crm_summary(customer_id)
return _doc_to_customer(doc_ref.get())
def delete_install_support(customer_id: str, index: int) -> CustomerInDB:
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Customer")
data = doc.to_dict() or {}
entries = list(data.get("install_support") or [])
if index < 0 or index >= len(entries):
raise HTTPException(status_code=404, detail="Support index out of range")
entries.pop(index)
doc_ref.update({"install_support": entries, "updated_at": datetime.utcnow().isoformat()})
_update_crm_summary(customer_id)
return _doc_to_customer(doc_ref.get())
# ── Transactions ──────────────────────────────────────────────────────────────
def add_transaction(customer_id: str, entry: TransactionEntry) -> CustomerInDB:
from google.cloud.firestore_v1 import ArrayUnion
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
if not doc_ref.get().exists:
raise NotFoundError("Customer")
now = datetime.utcnow().isoformat()
doc_ref.update({"transaction_history": ArrayUnion([entry.model_dump()]), "updated_at": now})
if entry.order_ref:
_recalculate_order_payment_status(customer_id, entry.order_ref)
return _doc_to_customer(doc_ref.get())
def update_transaction(customer_id: str, index: int, entry: TransactionEntry) -> CustomerInDB:
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Customer")
data = doc.to_dict() or {}
txns = list(data.get("transaction_history") or [])
if index < 0 or index >= len(txns):
raise HTTPException(status_code=404, detail="Transaction index out of range")
txns[index] = entry.model_dump()
now = datetime.utcnow().isoformat()
doc_ref.update({"transaction_history": txns, "updated_at": now})
if entry.order_ref:
_recalculate_order_payment_status(customer_id, entry.order_ref)
return _doc_to_customer(doc_ref.get())
def delete_transaction(customer_id: str, index: int) -> CustomerInDB:
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Customer")
data = doc.to_dict() or {}
txns = list(data.get("transaction_history") or [])
if index < 0 or index >= len(txns):
raise HTTPException(status_code=404, detail="Transaction index out of range")
deleted_order_ref = txns[index].get("order_ref")
txns.pop(index)
now = datetime.utcnow().isoformat()
doc_ref.update({"transaction_history": txns, "updated_at": now})
if deleted_order_ref:
_recalculate_order_payment_status(customer_id, deleted_order_ref)
return _doc_to_customer(doc_ref.get())
# ── Relationship Status ───────────────────────────────────────────────────────
def update_relationship_status(customer_id: str, status: str) -> CustomerInDB:
VALID = {"lead", "prospect", "active", "inactive", "churned"}
if status not in VALID:
raise HTTPException(status_code=422, detail=f"Invalid relationship_status: {status}")
db = get_db()
doc_ref = db.collection(CUSTOMERS_COLLECTION).document(customer_id)
if not doc_ref.get().exists:
raise NotFoundError("Customer")
# Failsafe: cannot manually mark inactive if open (non-terminal) orders exist
if status == "inactive":
TERMINAL = {"declined", "complete"}
open_orders = [
doc for doc in _order_collection(customer_id).stream()
if (doc.to_dict() or {}).get("status", "") not in TERMINAL
]
if open_orders:
raise HTTPException(
status_code=409,
detail=(
f"Cannot mark as inactive: {len(open_orders)} open order(s) still exist. "
"Please resolve all orders before changing the status."
),
)
doc_ref.update({"relationship_status": status, "updated_at": datetime.utcnow().isoformat()})
return _doc_to_customer(doc_ref.get())
# ── Comms Log (SQLite, async) ───────────────────────────────────────────────── # ── Comms Log (SQLite, async) ─────────────────────────────────────────────────
@@ -1188,11 +594,11 @@ async def create_media(data: MediaCreate) -> MediaInDB:
await db.execute( await db.execute(
"""INSERT INTO crm_media """INSERT INTO crm_media
(id, customer_id, order_id, filename, nextcloud_path, mime_type, (id, customer_id, order_id, filename, nextcloud_path, mime_type,
direction, tags, uploaded_by, thumbnail_path, created_at) direction, tags, uploaded_by, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""", VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(media_id, data.customer_id, data.order_id, data.filename, (media_id, data.customer_id, data.order_id, data.filename,
data.nextcloud_path, data.mime_type, direction, data.nextcloud_path, data.mime_type, direction,
tags_json, data.uploaded_by, data.thumbnail_path, now), tags_json, data.uploaded_by, now),
) )
await db.commit() await db.commit()
@@ -1211,65 +617,3 @@ async def delete_media(media_id: str) -> None:
raise HTTPException(status_code=404, detail="Media entry not found") raise HTTPException(status_code=404, detail="Media entry not found")
await db.execute("DELETE FROM crm_media WHERE id = ?", (media_id,)) await db.execute("DELETE FROM crm_media WHERE id = ?", (media_id,))
await db.commit() await db.commit()
# ── Background polling ────────────────────────────────────────────────────────
PRE_MFG_STATUSES = {"negotiating", "awaiting_quotation", "awaiting_customer_confirmation", "awaiting_fulfilment", "awaiting_payment"}
TERMINAL_STATUSES = {"declined", "complete"}
def poll_crm_customer_statuses() -> None:
"""
Two checks run daily:
1. Active + open pre-mfg order + 12+ months since last comm → churn.
2. Inactive + has any open (non-terminal) order → flip back to active.
"""
db = get_db()
now = datetime.utcnow()
for doc in db.collection(CUSTOMERS_COLLECTION).stream():
try:
data = doc.to_dict() or {}
rel_status = data.get("relationship_status", "lead")
summary = data.get("crm_summary") or {}
all_statuses = summary.get("all_orders_statuses") or []
# ── Check 1: active + silent 12 months on a pre-mfg order → churned ──
if rel_status == "active":
has_open_pre_mfg = any(s in PRE_MFG_STATUSES for s in all_statuses)
if not has_open_pre_mfg:
continue
# Find last comm date from SQLite comms table
# (comms are stored in SQLite, keyed by customer_id)
# We rely on crm_summary not having this; use Firestore comms subcollection as fallback
# The last_comm_date is passed from the frontend; here we use the comms subcollection
comms = list(db.collection(CUSTOMERS_COLLECTION).document(doc.id).collection("comms").stream())
if not comms:
continue
latest_date_str = max((c.to_dict().get("date") or "") for c in comms)
if not latest_date_str:
continue
last_contact = datetime.fromisoformat(latest_date_str.rstrip("Z").split("+")[0])
days_since = (now - last_contact).days
if days_since >= 365:
db.collection(CUSTOMERS_COLLECTION).document(doc.id).update({
"relationship_status": "churned",
"updated_at": now.isoformat(),
})
print(f"[CRM POLL] {doc.id} → churned ({days_since}d silent, open pre-mfg order)")
# ── Check 2: inactive + open orders exist → flip back to active ──
elif rel_status == "inactive":
has_open = any(s not in TERMINAL_STATUSES for s in all_statuses)
if has_open:
db.collection(CUSTOMERS_COLLECTION).document(doc.id).update({
"relationship_status": "active",
"updated_at": now.isoformat(),
})
print(f"[CRM POLL] {doc.id} → active (inactive but has open orders)")
except Exception as e:
print(f"[CRM POLL] Error processing customer {doc.id}: {e}")

125
backend/crm/thumbnails.py
View File

@@ -1,125 +0,0 @@
"""
Thumbnail generation for uploaded media files.
Supports:
- Images (via Pillow): JPEG thumbnail at 300×300 max
- Videos (via ffmpeg subprocess): extract first frame as JPEG
- PDFs (via pdf2image + Poppler): render first page as JPEG
Returns None if the type is unsupported or if generation fails.
"""
import io
import logging
import subprocess
from pathlib import Path
logger = logging.getLogger(__name__)
THUMB_SIZE = (220, 220) # small enough for gallery tiles; keeps files ~4-6 KB
def _thumb_from_image(content: bytes) -> bytes | None:
try:
from PIL import Image, ImageOps
img = Image.open(io.BytesIO(content))
img = ImageOps.exif_transpose(img) # honour EXIF Orientation tag before resizing
img = img.convert("RGB")
img.thumbnail(THUMB_SIZE, Image.LANCZOS)
out = io.BytesIO()
# quality=55 + optimize=True + progressive encoding → ~4-6 KB for typical photos
img.save(out, format="JPEG", quality=65, optimize=True, progressive=True)
return out.getvalue()
except Exception as e:
logger.warning("Image thumbnail failed: %s", e)
return None
def _thumb_from_video(content: bytes) -> bytes | None:
"""
Extract the first frame of a video as a JPEG thumbnail.
We write the video to a temp file instead of piping it to ffmpeg because
most video containers (MP4, MOV, MKV …) store their index (moov atom) at
an arbitrary offset and ffmpeg cannot seek on a pipe — causing rc≠0 with
"moov atom not found" or similar errors when stdin is used.
"""
import tempfile
import os
try:
# Write to a temp file so ffmpeg can seek freely
with tempfile.NamedTemporaryFile(suffix=".video", delete=False) as tmp_in:
tmp_in.write(content)
tmp_in_path = tmp_in.name
with tempfile.NamedTemporaryFile(suffix=".jpg", delete=False) as tmp_out:
tmp_out_path = tmp_out.name
try:
result = subprocess.run(
[
"ffmpeg", "-y",
"-i", tmp_in_path,
"-vframes", "1",
"-vf", f"scale={THUMB_SIZE[0]}:-2",
"-q:v", "4", # JPEG quality 1-31 (lower = better); 4 ≈ ~80% quality
tmp_out_path,
],
capture_output=True,
timeout=60,
)
if result.returncode == 0 and os.path.getsize(tmp_out_path) > 0:
with open(tmp_out_path, "rb") as f:
return f.read()
logger.warning(
"ffmpeg video thumb failed (rc=%s): %s",
result.returncode,
result.stderr[-400:].decode(errors="replace") if result.stderr else "",
)
return None
finally:
os.unlink(tmp_in_path)
try:
os.unlink(tmp_out_path)
except FileNotFoundError:
pass
except FileNotFoundError:
logger.warning("ffmpeg not found — video thumbnails unavailable")
return None
except Exception as e:
logger.warning("Video thumbnail failed: %s", e)
return None
def _thumb_from_pdf(content: bytes) -> bytes | None:
try:
from pdf2image import convert_from_bytes
pages = convert_from_bytes(content, first_page=1, last_page=1, size=THUMB_SIZE)
if not pages:
return None
out = io.BytesIO()
pages[0].save(out, format="JPEG", quality=55, optimize=True, progressive=True)
return out.getvalue()
except ImportError:
logger.warning("pdf2image not installed — PDF thumbnails unavailable")
return None
except Exception as e:
logger.warning("PDF thumbnail failed: %s", e)
return None
def generate_thumbnail(content: bytes, mime_type: str, filename: str) -> bytes | None:
"""
Generate a small JPEG thumbnail for the given file content.
Returns JPEG bytes or None if unsupported / generation fails.
"""
mt = (mime_type or "").lower()
fn = (filename or "").lower()
if mt.startswith("image/"):
return _thumb_from_image(content)
if mt.startswith("video/"):
return _thumb_from_video(content)
if mt == "application/pdf" or fn.endswith(".pdf"):
return _thumb_from_pdf(content)
return None

47
backend/database/__init__.py
View File

@@ -1,47 +0,0 @@
# MQTT live data — Phase 5: all functions now backed by Postgres
from database.pg_mqtt import (
init_db,
close_db,
purge_loop,
purge_old_data,
insert_log,
insert_heartbeat,
insert_command,
update_command_response,
get_logs,
get_heartbeats,
get_commands,
get_latest_heartbeats,
get_pending_command,
upsert_alert,
delete_alert,
get_alerts,
partition_manager_loop,
ensure_current_partitions,
)
# SQLite connection — still used by melodies, builder, manufacturing, and crm
# modules that have not yet been cut over to Postgres.
from database.core import get_db
__all__ = [
"init_db",
"close_db",
"get_db",
"purge_loop",
"purge_old_data",
"insert_log",
"insert_heartbeat",
"insert_command",
"update_command_response",
"get_logs",
"get_heartbeats",
"get_commands",
"get_latest_heartbeats",
"get_pending_command",
"upsert_alert",
"delete_alert",
"get_alerts",
"partition_manager_loop",
"ensure_current_partitions",
]

23
backend/database/models.py
View File

@@ -1,23 +0,0 @@
from database.postgres import Base # noqa: F401 — Base must be imported for Alembic autogenerate
# Import all ORM models here so Alembic autogenerate detects them.
# Add each new model file as it is created.
# --- Existing ---
from notes.orm import Entry, EntryLink # noqa: F401
from tickets.orm import SupportTicket, TicketMessage # noqa: F401
# --- Phase 0 ---
from shared.orm import MigrationRun, AuditLog # noqa: F401
from crm.orm import ( # noqa: F401
CrmProduct, CrmCustomer, CrmOrder,
CrmCommsLog, CrmMedia, CrmSyncState,
CrmQuotation, CrmQuotationItem,
)
from staff.orm import Staff # noqa: F401
from settings.orm import ConsoleSetting, PublicFeature # noqa: F401
from melodies.orm import MelodyDraft, BuiltMelody # noqa: F401
from manufacturing.orm import MfgAuditLog # noqa: F401
from devices.orm import DeviceAlert # noqa: F401
# NOTE: device_logs, commands, heartbeats are partitioned/raw-SQL tables —
# they are NOT ORM models and are created via op.execute() in the migration.

411
backend/database/pg_mqtt.py
View File

@@ -1,411 +0,0 @@
"""
Phase 5 — MQTT live data functions backed by Postgres.
device_logs is a partitioned table; heartbeats and commands are plain tables.
All three are accessed via raw SQL (not ORM) because device_logs partitioning
does not play well with SQLAlchemy's declarative ORM.
device_alerts is an ORM model (devices/orm.py) and is handled here via raw SQL
to keep a single consistent interface for callers that used to import from database.core.
"""
import asyncio
import json
import logging
from datetime import date, datetime, timedelta, timezone
from sqlalchemy import text
from config import settings
from database.postgres import AsyncSessionLocal
logger = logging.getLogger("database.pg_mqtt")
# ---------------------------------------------------------------------------
# Insert operations
# ---------------------------------------------------------------------------
async def insert_log(device_serial: str, level: str, message: str,
device_timestamp: int | None = None) -> int:
async with AsyncSessionLocal() as session:
result = await session.execute(
text("""
INSERT INTO device_logs (device_serial, level, message, device_timestamp, received_at)
VALUES (:serial, :level, :message, :ts, now())
RETURNING id
"""),
{"serial": device_serial, "level": level, "message": message, "ts": device_timestamp},
)
row = result.fetchone()
await session.commit()
return row[0]
async def insert_heartbeat(device_serial: str, device_id: str,
firmware_version: str, ip_address: str,
gateway: str, uptime_ms: int, uptime_display: str) -> int:
async with AsyncSessionLocal() as session:
result = await session.execute(
text("""
INSERT INTO heartbeats
(device_serial, device_id, firmware_version, ip_address,
gateway, uptime_ms, uptime_display, received_at)
VALUES
(:serial, :device_id, :fw, :ip, :gw, :uptime_ms, :uptime_display, now())
RETURNING id
"""),
{
"serial": device_serial,
"device_id": device_id,
"fw": firmware_version,
"ip": ip_address,
"gw": gateway,
"uptime_ms": uptime_ms,
"uptime_display": uptime_display,
},
)
row = result.fetchone()
await session.commit()
return row[0]
async def insert_command(device_serial: str, command_name: str,
command_payload: dict) -> int:
async with AsyncSessionLocal() as session:
result = await session.execute(
text("""
INSERT INTO commands (device_serial, command_name, command_payload, sent_at)
VALUES (:serial, :name, :payload, now())
RETURNING id
"""),
{
"serial": device_serial,
"name": command_name,
"payload": json.dumps(command_payload),
},
)
row = result.fetchone()
await session.commit()
return row[0]
async def update_command_response(command_id: int, status: str,
response_payload: dict | None = None):
async with AsyncSessionLocal() as session:
await session.execute(
text("""
UPDATE commands
SET status = :status,
response_payload = :payload,
responded_at = now()
WHERE id = :id
"""),
{
"id": command_id,
"status": status,
"payload": json.dumps(response_payload) if response_payload else None,
},
)
await session.commit()
# ---------------------------------------------------------------------------
# Query operations
# ---------------------------------------------------------------------------
async def get_logs(device_serial: str, level: str | None = None,
search: str | None = None,
limit: int = 100, offset: int = 0) -> tuple[list, int]:
where = "device_serial = :serial"
params: dict = {"serial": device_serial, "limit": limit, "offset": offset}
if level:
where += " AND level = :level"
params["level"] = level
if search:
where += " AND message ILIKE :search"
params["search"] = f"%{search}%"
async with AsyncSessionLocal() as session:
count_result = await session.execute(
text(f"SELECT COUNT(*) FROM device_logs WHERE {where}"), params
)
total = count_result.scalar()
rows_result = await session.execute(
text(f"""
SELECT id, device_serial, level, message, device_timestamp,
received_at AT TIME ZONE 'UTC' AS received_at
FROM device_logs
WHERE {where}
ORDER BY received_at DESC
LIMIT :limit OFFSET :offset
"""),
params,
)
rows = rows_result.mappings().all()
return [_row_to_dict(r) for r in rows], total
async def get_heartbeats(device_serial: str, limit: int = 100,
offset: int = 0) -> tuple[list, int]:
async with AsyncSessionLocal() as session:
count_result = await session.execute(
text("SELECT COUNT(*) FROM heartbeats WHERE device_serial = :serial"),
{"serial": device_serial},
)
total = count_result.scalar()
rows_result = await session.execute(
text("""
SELECT id, device_serial, device_id, firmware_version, ip_address,
gateway, uptime_ms, uptime_display,
received_at AT TIME ZONE 'UTC' AS received_at
FROM heartbeats
WHERE device_serial = :serial
ORDER BY received_at DESC
LIMIT :limit OFFSET :offset
"""),
{"serial": device_serial, "limit": limit, "offset": offset},
)
rows = rows_result.mappings().all()
return [_row_to_dict(r) for r in rows], total
async def get_commands(device_serial: str, limit: int = 100,
offset: int = 0) -> tuple[list, int]:
async with AsyncSessionLocal() as session:
count_result = await session.execute(
text("SELECT COUNT(*) FROM commands WHERE device_serial = :serial"),
{"serial": device_serial},
)
total = count_result.scalar()
rows_result = await session.execute(
text("""
SELECT id, device_serial, command_name, command_payload, status,
response_payload,
sent_at AT TIME ZONE 'UTC' AS sent_at,
responded_at AT TIME ZONE 'UTC' AS responded_at
FROM commands
WHERE device_serial = :serial
ORDER BY sent_at DESC
LIMIT :limit OFFSET :offset
"""),
{"serial": device_serial, "limit": limit, "offset": offset},
)
rows = rows_result.mappings().all()
return [_row_to_dict(r) for r in rows], total
async def get_latest_heartbeats() -> list:
async with AsyncSessionLocal() as session:
rows_result = await session.execute(
text("""
SELECT DISTINCT ON (device_serial)
id, device_serial, device_id, firmware_version, ip_address,
gateway, uptime_ms, uptime_display,
received_at AT TIME ZONE 'UTC' AS received_at
FROM heartbeats
ORDER BY device_serial, received_at DESC
""")
)
rows = rows_result.mappings().all()
return [_row_to_dict(r) for r in rows]
async def get_pending_command(device_serial: str) -> dict | None:
async with AsyncSessionLocal() as session:
result = await session.execute(
text("""
SELECT id, device_serial, command_name, command_payload, status,
response_payload,
sent_at AT TIME ZONE 'UTC' AS sent_at,
responded_at AT TIME ZONE 'UTC' AS responded_at
FROM commands
WHERE device_serial = :serial AND status = 'pending'
ORDER BY sent_at DESC
LIMIT 1
"""),
{"serial": device_serial},
)
row = result.mappings().fetchone()
return _row_to_dict(row) if row else None
# ---------------------------------------------------------------------------
# Device alerts
# ---------------------------------------------------------------------------
async def upsert_alert(device_serial: str, subsystem: str, state: str,
message: str | None = None):
async with AsyncSessionLocal() as session:
await session.execute(
text("""
INSERT INTO device_alerts (device_serial, subsystem, state, message, updated_at)
VALUES (:serial, :subsystem, :state, :message, now())
ON CONFLICT (device_serial, subsystem)
DO UPDATE SET
state = EXCLUDED.state,
message = EXCLUDED.message,
updated_at = EXCLUDED.updated_at
"""),
{"serial": device_serial, "subsystem": subsystem, "state": state, "message": message},
)
await session.commit()
async def delete_alert(device_serial: str, subsystem: str):
async with AsyncSessionLocal() as session:
await session.execute(
text("DELETE FROM device_alerts WHERE device_serial = :serial AND subsystem = :subsystem"),
{"serial": device_serial, "subsystem": subsystem},
)
await session.commit()
async def get_alerts(device_serial: str) -> list:
async with AsyncSessionLocal() as session:
result = await session.execute(
text("""
SELECT id, device_serial, subsystem, state, message,
updated_at AT TIME ZONE 'UTC' AS updated_at
FROM device_alerts
WHERE device_serial = :serial
ORDER BY updated_at DESC
"""),
{"serial": device_serial},
)
rows = result.mappings().all()
return [_row_to_dict(r) for r in rows]
# ---------------------------------------------------------------------------
# Partition management
# ---------------------------------------------------------------------------
def _add_months(d: date, months: int) -> date:
month = d.month - 1 + months
year = d.year + month // 12
month = month % 12 + 1
return d.replace(year=year, month=month, day=1)
async def ensure_current_partitions():
"""Create device_logs partitions for the current and next month if missing."""
async with AsyncSessionLocal() as session:
for month_offset in (0, 1):
d = _add_months(date.today().replace(day=1), month_offset)
partition_name = f"device_logs_{d.strftime('%Y_%m')}"
start = d.isoformat()
end = _add_months(d, 1).isoformat()
await session.execute(text(f"""
CREATE TABLE IF NOT EXISTS {partition_name}
PARTITION OF device_logs
FOR VALUES FROM ('{start}') TO ('{end}')
"""))
await session.commit()
logger.info("Partition check complete")
async def drop_old_partitions(keep_months: int = 6):
"""Drop device_logs partitions older than keep_months."""
cutoff = _add_months(date.today().replace(day=1), -keep_months)
async with AsyncSessionLocal() as session:
result = await session.execute(text("""
SELECT tablename FROM pg_tables
WHERE schemaname = 'public'
AND tablename LIKE 'device_logs_%'
"""))
partitions = [r[0] for r in result.fetchall()]
for name in partitions:
# name format: device_logs_YYYY_MM
parts = name.split("_")
if len(parts) != 4:
continue
try:
partition_date = date(int(parts[2]), int(parts[3]), 1)
except ValueError:
continue
if partition_date < cutoff:
async with AsyncSessionLocal() as session:
await session.execute(text(f"DROP TABLE IF EXISTS {name}"))
await session.commit()
logger.info(f"Dropped old partition: {name}")
async def partition_manager_loop():
"""Runs once on startup, then monthly thereafter."""
await ensure_current_partitions()
while True:
# Sleep ~30 days, wake up and ensure next month's partition exists
await asyncio.sleep(30 * 24 * 3600)
try:
await ensure_current_partitions()
await drop_old_partitions()
except Exception as e:
logger.error(f"Partition manager error: {e}")
# ---------------------------------------------------------------------------
# Cleanup (replaces SQLite purge_loop — now a no-op since Postgres uses
# partition drops instead of row-by-row deletes for device_logs; heartbeats
# and commands are still purged by row deletion)
# ---------------------------------------------------------------------------
async def purge_old_data(retention_days: int | None = None):
days = retention_days or settings.mqtt_data_retention_days
cutoff = datetime.now(timezone.utc) - timedelta(days=days)
async with AsyncSessionLocal() as session:
await session.execute(
text("DELETE FROM heartbeats WHERE received_at < :cutoff"),
{"cutoff": cutoff},
)
await session.execute(
text("DELETE FROM commands WHERE sent_at < :cutoff"),
{"cutoff": cutoff},
)
await session.commit()
logger.info(f"Purged heartbeats and commands older than {days} days")
async def purge_loop():
while True:
await asyncio.sleep(86400)
try:
await purge_old_data()
except Exception as e:
logger.error(f"Purge failed: {e}")
# ---------------------------------------------------------------------------
# Stub — no longer needed but kept so nothing that imports init_db/close_db breaks
# ---------------------------------------------------------------------------
async def init_db():
"""No-op: Postgres schema is managed by Alembic, not runtime init."""
logger.info("Postgres MQTT backend active — no SQLite init needed")
async def close_db():
"""No-op: SQLAlchemy engine lifecycle is managed by the process."""
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _row_to_dict(row) -> dict:
"""Convert a SQLAlchemy RowMapping to a plain dict with ISO string timestamps."""
d = dict(row)
for key, val in d.items():
if isinstance(val, datetime):
d[key] = val.isoformat()
return d

16
backend/database/postgres.py
View File

@@ -1,16 +0,0 @@
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession, async_sessionmaker
from sqlalchemy.orm import DeclarativeBase
from config import settings
engine = create_async_engine(settings.database_url, pool_size=10, echo=False)
AsyncSessionLocal = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
class Base(DeclarativeBase):
pass
async def get_pg_session() -> AsyncSession:
"""FastAPI dependency — yields a DB session and closes it after the request."""
async with AsyncSessionLocal() as session:
yield session

35
backend/devices/models.py
View File

@@ -31,11 +31,11 @@ class DeviceTiers(str, Enum):
class DeviceNetworkSettings(BaseModel): class DeviceNetworkSettings(BaseModel):
hostname: str = "" hostname: str = ""
useStaticIP: bool = False useStaticIP: bool = False
ipAddress: Any = [] ipAddress: List[str] = []
gateway: Any = [] gateway: List[str] = []
subnet: Any = [] subnet: List[str] = []
dns1: Any = [] dns1: List[str] = []
dns2: Any = [] dns2: List[str] = []
class DeviceClockSettings(BaseModel): class DeviceClockSettings(BaseModel):
@@ -119,19 +119,13 @@ class DeviceCreate(BaseModel):
device_subscription: DeviceSubInformation = DeviceSubInformation() device_subscription: DeviceSubInformation = DeviceSubInformation()
device_stats: DeviceStatistics = DeviceStatistics() device_stats: DeviceStatistics = DeviceStatistics()
events_on: bool = False events_on: bool = False
device_location_coordinates: Any = None # GeoPoint dict {lat, lng} or legacy str device_location_coordinates: str = ""
device_melodies_all: List[MelodyMainItem] = [] device_melodies_all: List[MelodyMainItem] = []
device_melodies_favorites: List[str] = [] device_melodies_favorites: List[str] = []
user_list: List[str] = [] user_list: List[str] = []
websocket_url: str = "" websocket_url: str = ""
churchAssistantURL: str = "" churchAssistantURL: str = ""
staffNotes: str = "" staffNotes: str = ""
hw_family: str = ""
hw_revision: str = ""
tags: List[str] = []
serial_number: str = ""
customer_id: str = ""
mfg_status: str = ""
class DeviceUpdate(BaseModel): class DeviceUpdate(BaseModel):
@@ -144,23 +138,17 @@ class DeviceUpdate(BaseModel):
device_subscription: Optional[Dict[str, Any]] = None device_subscription: Optional[Dict[str, Any]] = None
device_stats: Optional[Dict[str, Any]] = None device_stats: Optional[Dict[str, Any]] = None
events_on: Optional[bool] = None events_on: Optional[bool] = None
device_location_coordinates: Optional[Any] = None # dict {lat, lng} or legacy str device_location_coordinates: Optional[str] = None
device_melodies_all: Optional[List[MelodyMainItem]] = None device_melodies_all: Optional[List[MelodyMainItem]] = None
device_melodies_favorites: Optional[List[str]] = None device_melodies_favorites: Optional[List[str]] = None
user_list: Optional[List[str]] = None user_list: Optional[List[str]] = None
websocket_url: Optional[str] = None websocket_url: Optional[str] = None
churchAssistantURL: Optional[str] = None churchAssistantURL: Optional[str] = None
staffNotes: Optional[str] = None staffNotes: Optional[str] = None
hw_family: Optional[str] = None
hw_revision: Optional[str] = None
tags: Optional[List[str]] = None
customer_id: Optional[str] = None
mfg_status: Optional[str] = None
class DeviceInDB(DeviceCreate): class DeviceInDB(DeviceCreate):
id: str id: str
# Legacy field — kept for backwards compat; new docs use serial_number
device_id: str = "" device_id: str = ""
@@ -169,15 +157,6 @@ class DeviceListResponse(BaseModel):
total: int total: int
class DeviceNoteCreate(BaseModel):
content: str
created_by: str = ""
class DeviceNoteUpdate(BaseModel):
content: str
class DeviceUserInfo(BaseModel): class DeviceUserInfo(BaseModel):
"""User info resolved from device_users sub-collection or user_list.""" """User info resolved from device_users sub-collection or user_list."""
user_id: str = "" user_id: str = ""

31
backend/devices/orm.py
View File

@@ -1,31 +0,0 @@
from datetime import datetime, timezone
from sqlalchemy import BigInteger, Column, DateTime, Index, String, Text, UniqueConstraint
from sqlalchemy.dialects.postgresql import JSONB
from database.postgres import Base
def _now():
return datetime.now(timezone.utc)
class DeviceAlert(Base):
"""Current alert state per device+subsystem (upserted, not appended)."""
__tablename__ = "device_alerts"
__table_args__ = (
UniqueConstraint("device_serial", "subsystem"),
Index("idx_device_alerts_serial", "device_serial"),
)
id = Column(BigInteger, primary_key=True, autoincrement=True)
device_serial = Column(String(128), nullable=False)
subsystem = Column(String(128), nullable=False)
state = Column(String(64), nullable=False)
message = Column(Text)
updated_at = Column(DateTime(timezone=True), nullable=False, default=_now, onupdate=_now)
# NOTE: device_logs, commands, and heartbeats are NOT declared as ORM models here.
# device_logs is a partitioned table — SQLAlchemy ORM does not support declarative
# partitioned tables cleanly. All three tables are created via raw SQL in the
# Alembic migration and accessed via raw queries in database/core.py (SQLite now)
# and will be accessed via raw async SQL after Phase 5 cutover.

436
backend/devices/router.py
View File

@@ -1,28 +1,17 @@
import uuid from fastapi import APIRouter, Depends, Query
from datetime import datetime from typing import Optional
from fastapi import APIRouter, Depends, Query, HTTPException
from typing import Optional, List
from pydantic import BaseModel
from sqlalchemy.ext.asyncio import AsyncSession
from auth.models import TokenPayload from auth.models import TokenPayload
from auth.dependencies import require_permission from auth.dependencies import require_permission
from devices.models import ( from devices.models import (
DeviceCreate, DeviceUpdate, DeviceInDB, DeviceListResponse, DeviceCreate, DeviceUpdate, DeviceInDB, DeviceListResponse,
DeviceUsersResponse, DeviceUserInfo, DeviceUsersResponse, DeviceUserInfo,
DeviceNoteCreate, DeviceNoteUpdate,
) )
from devices import service from devices import service
import database as mqtt_db from mqtt import database as mqtt_db
from mqtt.models import DeviceAlertEntry, DeviceAlertsResponse from mqtt.models import DeviceAlertEntry, DeviceAlertsResponse
from shared.firebase import get_db as get_firestore
from database.postgres import get_pg_session
from shared.audit import log_action
router = APIRouter(prefix="/api/devices", tags=["devices"]) router = APIRouter(prefix="/api/devices", tags=["devices"])
NOTES_COLLECTION = "notes"
CRM_COLLECTION = "crm_customers"
@router.get("", response_model=DeviceListResponse) @router.get("", response_model=DeviceListResponse)
async def list_devices( async def list_devices(
@@ -61,12 +50,8 @@ async def get_device_users(
async def create_device( async def create_device(
body: DeviceCreate, body: DeviceCreate,
_user: TokenPayload = Depends(require_permission("devices", "add")), _user: TokenPayload = Depends(require_permission("devices", "add")),
db: AsyncSession = Depends(get_pg_session),
): ):
device = service.create_device(body) return service.create_device(body)
await log_action(db, _user.sub, _user.name or _user.email, "CREATE", "device",
device.device_id, device.device_name or device.device_id)
return device
@router.put("/{device_id}", response_model=DeviceInDB) @router.put("/{device_id}", response_model=DeviceInDB)
@@ -74,32 +59,16 @@ async def update_device(
device_id: str, device_id: str,
body: DeviceUpdate, body: DeviceUpdate,
_user: TokenPayload = Depends(require_permission("devices", "edit")), _user: TokenPayload = Depends(require_permission("devices", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
old = service.get_device(device_id) return service.update_device(device_id, body)
device = service.update_device(device_id, body)
_SKIP = {"updated_at", "device_id", "tags", "user_list"}
changes = {
k: {"old": getattr(old, k, None), "new": getattr(device, k, None)}
for k in body.model_fields_set
if k not in _SKIP and getattr(old, k, None) != getattr(device, k, None)
}
if "tags" in body.model_fields_set and (old.tags or []) != (device.tags or []):
changes["tags"] = {"old": sorted(old.tags or []), "new": sorted(device.tags or [])}
await log_action(db, _user.sub, _user.name or _user.email, "UPDATE", "device",
device_id, device.device_name or device_id, changes=changes or None)
return device
@router.delete("/{device_id}", status_code=204) @router.delete("/{device_id}", status_code=204)
async def delete_device( async def delete_device(
device_id: str, device_id: str,
_user: TokenPayload = Depends(require_permission("devices", "delete")), _user: TokenPayload = Depends(require_permission("devices", "delete")),
db: AsyncSession = Depends(get_pg_session),
): ):
service.delete_device(device_id) service.delete_device(device_id)
await log_action(db, _user.sub, _user.name or _user.email, "DELETE", "device",
device_id, device_id)
@router.get("/{device_id}/alerts", response_model=DeviceAlertsResponse) @router.get("/{device_id}/alerts", response_model=DeviceAlertsResponse)
@@ -110,398 +79,3 @@ async def get_device_alerts(
"""Return the current active alert set for a device. Empty list means fully healthy.""" """Return the current active alert set for a device. Empty list means fully healthy."""
rows = await mqtt_db.get_alerts(device_id) rows = await mqtt_db.get_alerts(device_id)
return DeviceAlertsResponse(alerts=[DeviceAlertEntry(**r) for r in rows]) return DeviceAlertsResponse(alerts=[DeviceAlertEntry(**r) for r in rows])
# ─────────────────────────────────────────────────────────────────────────────
# Device Notes
# ─────────────────────────────────────────────────────────────────────────────
@router.get("/{device_id}/notes")
async def list_device_notes(
device_id: str,
_user: TokenPayload = Depends(require_permission("devices", "view")),
):
"""List all notes for a device."""
db = get_firestore()
docs = db.collection(NOTES_COLLECTION).where("device_id", "==", device_id).stream()
notes = []
for doc in docs:
note = doc.to_dict()
note["id"] = doc.id
for f in ("created_at", "updated_at"):
if hasattr(note.get(f), "isoformat"):
note[f] = note[f].isoformat()
notes.append(note)
notes.sort(key=lambda n: n.get("created_at") or "", reverse=False)
return {"notes": notes, "total": len(notes)}
@router.post("/{device_id}/notes", status_code=201)
async def create_device_note(
device_id: str,
body: DeviceNoteCreate,
_user: TokenPayload = Depends(require_permission("devices", "edit")),
):
"""Create a new note for a device."""
db = get_firestore()
now = datetime.utcnow()
note_id = str(uuid.uuid4())
note_data = {
"device_id": device_id,
"content": body.content,
"created_by": body.created_by or _user.name or "",
"created_at": now,
"updated_at": now,
}
db.collection(NOTES_COLLECTION).document(note_id).set(note_data)
note_data["id"] = note_id
note_data["created_at"] = now.isoformat()
note_data["updated_at"] = now.isoformat()
return note_data
@router.put("/{device_id}/notes/{note_id}")
async def update_device_note(
device_id: str,
note_id: str,
body: DeviceNoteUpdate,
_user: TokenPayload = Depends(require_permission("devices", "edit")),
):
"""Update an existing device note."""
db = get_firestore()
doc_ref = db.collection(NOTES_COLLECTION).document(note_id)
doc = doc_ref.get()
if not doc.exists or doc.to_dict().get("device_id") != device_id:
raise HTTPException(status_code=404, detail="Note not found")
now = datetime.utcnow()
doc_ref.update({"content": body.content, "updated_at": now})
updated = doc.to_dict()
updated["id"] = note_id
updated["content"] = body.content
updated["updated_at"] = now.isoformat()
if hasattr(updated.get("created_at"), "isoformat"):
updated["created_at"] = updated["created_at"].isoformat()
return updated
@router.delete("/{device_id}/notes/{note_id}", status_code=204)
async def delete_device_note(
device_id: str,
note_id: str,
_user: TokenPayload = Depends(require_permission("devices", "edit")),
):
"""Delete a device note."""
db = get_firestore()
doc_ref = db.collection(NOTES_COLLECTION).document(note_id)
doc = doc_ref.get()
if not doc.exists or doc.to_dict().get("device_id") != device_id:
raise HTTPException(status_code=404, detail="Note not found")
doc_ref.delete()
# ─────────────────────────────────────────────────────────────────────────────
# Device Tags
# ─────────────────────────────────────────────────────────────────────────────
class TagsUpdate(BaseModel):
tags: List[str]
@router.put("/{device_id}/tags", response_model=DeviceInDB)
async def update_device_tags(
device_id: str,
body: TagsUpdate,
_user: TokenPayload = Depends(require_permission("devices", "edit")),
):
"""Replace the tags list for a device."""
return service.update_device(device_id, DeviceUpdate(tags=body.tags))
# ─────────────────────────────────────────────────────────────────────────────
# Assign Device to Customer
# ─────────────────────────────────────────────────────────────────────────────
class CustomerSearchResult(BaseModel):
id: str
name: str
email: str
organization: str = ""
class AssignCustomerBody(BaseModel):
customer_id: str
label: str = ""
@router.get("/{device_id}/customer-search")
async def search_customers_for_device(
device_id: str,
q: str = Query(""),
_user: TokenPayload = Depends(require_permission("devices", "view")),
):
"""Search customers by name, email, phone, org, or tags, returning top 20 matches."""
db = get_firestore()
docs = db.collection(CRM_COLLECTION).stream()
results = []
q_lower = q.lower().strip()
for doc in docs:
data = doc.to_dict()
name = data.get("name", "") or ""
surname = data.get("surname", "") or ""
email = data.get("email", "") or ""
organization = data.get("organization", "") or ""
phone = data.get("phone", "") or ""
tags = " ".join(data.get("tags", []) or [])
location = data.get("location") or {}
city = location.get("city", "") or ""
searchable = f"{name} {surname} {email} {organization} {phone} {tags} {city}".lower()
if not q_lower or q_lower in searchable:
results.append({
"id": doc.id,
"name": name,
"surname": surname,
"email": email,
"organization": organization,
"city": city,
})
if len(results) >= 20:
break
return {"results": results}
@router.post("/{device_id}/assign-customer")
async def assign_device_to_customer(
device_id: str,
body: AssignCustomerBody,
_user: TokenPayload = Depends(require_permission("devices", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
"""Assign a device to a customer.
- Sets owner field on the device document.
- Adds a console_device entry to the customer's owned_items list.
"""
db = get_firestore()
# Verify device exists
device = service.get_device(device_id)
# Get customer
customer_ref = db.collection(CRM_COLLECTION).document(body.customer_id)
customer_doc = customer_ref.get()
if not customer_doc.exists:
raise HTTPException(status_code=404, detail="Customer not found")
customer_data = customer_doc.to_dict()
customer_email = customer_data.get("email", "")
# Update device: owner email + customer_id
device_ref = db.collection("devices").document(device_id)
device_ref.update({"owner": customer_email, "customer_id": body.customer_id})
# Add to customer owned_items (avoid duplicates)
owned_items = customer_data.get("owned_items", []) or []
already_assigned = any(
item.get("type") == "console_device" and item.get("console_device", {}).get("device_id") == device_id
for item in owned_items
)
if not already_assigned:
owned_items.append({
"type": "console_device",
"console_device": {
"device_id": device_id,
"label": body.label or device.device_name or device_id,
}
})
customer_ref.update({"owned_items": owned_items})
await log_action(db, _user.sub, _user.name or _user.email, "UPDATE", "device",
device_id, device_id, meta={"action_detail": "assigned_to_customer",
"customer_id": body.customer_id})
return {"status": "assigned", "device_id": device_id, "customer_id": body.customer_id}
@router.delete("/{device_id}/assign-customer", status_code=204)
async def unassign_device_from_customer(
device_id: str,
customer_id: str = Query(...),
_user: TokenPayload = Depends(require_permission("devices", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
"""Remove device assignment from a customer."""
db = get_firestore()
# Clear customer_id on device
device_ref = db.collection("devices").document(device_id)
device_ref.update({"customer_id": ""})
# Remove from customer owned_items
customer_ref = db.collection(CRM_COLLECTION).document(customer_id)
customer_doc = customer_ref.get()
if customer_doc.exists:
customer_data = customer_doc.to_dict()
owned_items = [
item for item in (customer_data.get("owned_items") or [])
if not (item.get("type") == "console_device" and item.get("console_device", {}).get("device_id") == device_id)
]
customer_ref.update({"owned_items": owned_items})
await log_action(db, _user.sub, _user.name or _user.email, "UPDATE", "device",
device_id, device_id, meta={"action_detail": "unassigned_from_customer",
"customer_id": customer_id})
# ─────────────────────────────────────────────────────────────────────────────
# Customer detail (for Owner display in fleet)
# ─────────────────────────────────────────────────────────────────────────────
@router.get("/{device_id}/customer")
async def get_device_customer(
device_id: str,
_user: TokenPayload = Depends(require_permission("devices", "view")),
):
"""Return basic customer details for a device's assigned customer_id."""
db = get_firestore()
device_ref = db.collection("devices").document(device_id)
device_doc = device_ref.get()
if not device_doc.exists:
raise HTTPException(status_code=404, detail="Device not found")
device_data = device_doc.to_dict() or {}
customer_id = device_data.get("customer_id")
if not customer_id:
return {"customer": None}
customer_doc = db.collection(CRM_COLLECTION).document(customer_id).get()
if not customer_doc.exists:
return {"customer": None}
cd = customer_doc.to_dict() or {}
return {
"customer": {
"id": customer_doc.id,
"name": cd.get("name") or "",
"email": cd.get("email") or "",
"organization": cd.get("organization") or "",
"phone": cd.get("phone") or "",
}
}
# ─────────────────────────────────────────────────────────────────────────────
# User list management (for Manage tab — assign/remove users from user_list)
# ─────────────────────────────────────────────────────────────────────────────
class UserSearchResult(BaseModel):
id: str
display_name: str = ""
email: str = ""
phone: str = ""
photo_url: str = ""
@router.get("/{device_id}/user-search")
async def search_users_for_device(
device_id: str,
q: str = Query(""),
_user: TokenPayload = Depends(require_permission("devices", "view")),
):
"""Search the users collection by name, email, or phone."""
db = get_firestore()
docs = db.collection("users").stream()
results = []
q_lower = q.lower().strip()
for doc in docs:
data = doc.to_dict() or {}
name = (data.get("display_name") or "").lower()
email = (data.get("email") or "").lower()
phone = (data.get("phone") or "").lower()
if not q_lower or q_lower in name or q_lower in email or q_lower in phone:
results.append({
"id": doc.id,
"display_name": data.get("display_name") or "",
"email": data.get("email") or "",
"phone": data.get("phone") or "",
"photo_url": data.get("photo_url") or "",
})
if len(results) >= 20:
break
return {"results": results}
class AddUserBody(BaseModel):
user_id: str
@router.post("/{device_id}/user-list", status_code=200)
async def add_user_to_device(
device_id: str,
body: AddUserBody,
_user: TokenPayload = Depends(require_permission("devices", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
"""Add a user reference to the device's user_list field."""
db = get_firestore()
device_ref = db.collection("devices").document(device_id)
device_doc = device_ref.get()
if not device_doc.exists:
raise HTTPException(status_code=404, detail="Device not found")
# Verify user exists
user_doc = db.collection("users").document(body.user_id).get()
if not user_doc.exists:
raise HTTPException(status_code=404, detail="User not found")
data = device_doc.to_dict() or {}
user_list = data.get("user_list", []) or []
# Avoid duplicates — check both string paths and DocumentReferences
from google.cloud.firestore_v1 import DocumentReference as DocRef
existing_ids = set()
for entry in user_list:
if isinstance(entry, DocRef):
existing_ids.add(entry.id)
elif isinstance(entry, str):
existing_ids.add(entry.split("/")[-1])
if body.user_id not in existing_ids:
user_ref = db.collection("users").document(body.user_id)
user_list.append(user_ref)
device_ref.update({"user_list": user_list})
await log_action(db, _user.sub, _user.name or _user.email, "UPDATE", "device",
device_id, device_id, meta={"action_detail": "user_added",
"user_id": body.user_id})
return {"status": "added", "user_id": body.user_id}
@router.delete("/{device_id}/user-list/{user_id}", status_code=200)
async def remove_user_from_device(
device_id: str,
user_id: str,
_user: TokenPayload = Depends(require_permission("devices", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
"""Remove a user reference from the device's user_list field."""
db = get_firestore()
device_ref = db.collection("devices").document(device_id)
device_doc = device_ref.get()
if not device_doc.exists:
raise HTTPException(status_code=404, detail="Device not found")
data = device_doc.to_dict() or {}
user_list = data.get("user_list", []) or []
from google.cloud.firestore_v1 import DocumentReference as DocRef
def resolves_to(entry, uid: str) -> bool:
if isinstance(entry, DocRef):
return entry.id == uid
if isinstance(entry, str):
return entry.split("/")[-1] == uid
return False
# Remove any entry that resolves to this user_id (handles both DocRef and string paths)
new_list = [entry for entry in user_list if not resolves_to(entry, user_id)]
device_ref.update({"user_list": new_list})
await log_action(db, _user.sub, _user.name or _user.email, "UPDATE", "device",
device_id, device_id, meta={"action_detail": "user_removed",
"user_id": user_id})
return {"status": "removed", "user_id": user_id}

52
backend/devices/service.py
View File

@@ -52,11 +52,10 @@ def _generate_serial_number() -> str:
def _ensure_unique_serial(db) -> str: def _ensure_unique_serial(db) -> str:
"""Generate a serial number and verify it doesn't already exist in Firestore.""" """Generate a serial number and verify it doesn't already exist in Firestore."""
existing_sns = set() existing_sns = set()
for doc in db.collection(COLLECTION).select(["serial_number"]).stream(): for doc in db.collection(COLLECTION).select(["device_id"]).stream():
data = doc.to_dict() data = doc.to_dict()
sn = data.get("serial_number") or data.get("device_id") if data.get("device_id"):
if sn: existing_sns.add(data["device_id"])
existing_sns.add(sn)
for _ in range(100): # safety limit for _ in range(100): # safety limit
sn = _generate_serial_number() sn = _generate_serial_number()
@@ -72,7 +71,7 @@ def _convert_firestore_value(val):
# Firestore DatetimeWithNanoseconds is a datetime subclass # Firestore DatetimeWithNanoseconds is a datetime subclass
return val.strftime("%d %B %Y at %H:%M:%S UTC%z") return val.strftime("%d %B %Y at %H:%M:%S UTC%z")
if isinstance(val, GeoPoint): if isinstance(val, GeoPoint):
return {"lat": val.latitude, "lng": val.longitude} return f"{val.latitude}° N, {val.longitude}° E"
if isinstance(val, DocumentReference): if isinstance(val, DocumentReference):
# Store the document path (e.g. "users/abc123") # Store the document path (e.g. "users/abc123")
return val.path return val.path
@@ -96,40 +95,18 @@ def _sanitize_dict(d: dict) -> dict:
return result return result
def _auto_upgrade_claimed(doc_ref, data: dict) -> dict:
"""If the device has entries in user_list and isn't already claimed/decommissioned,
upgrade mfg_status to 'claimed' automatically and return the updated data dict."""
current_status = data.get("mfg_status", "")
if current_status in ("claimed", "decommissioned"):
return data
user_list = data.get("user_list", []) or []
if user_list:
doc_ref.update({"mfg_status": "claimed"})
data = dict(data)
data["mfg_status"] = "claimed"
return data
def _doc_to_device(doc) -> DeviceInDB: def _doc_to_device(doc) -> DeviceInDB:
"""Convert a Firestore document snapshot to a DeviceInDB model. """Convert a Firestore document snapshot to a DeviceInDB model."""
data = _sanitize_dict(doc.to_dict())
Also auto-upgrades mfg_status to 'claimed' if user_list is non-empty.
"""
raw = doc.to_dict()
raw = _auto_upgrade_claimed(doc.reference, raw)
data = _sanitize_dict(raw)
return DeviceInDB(id=doc.id, **data) return DeviceInDB(id=doc.id, **data)
FLEET_STATUSES = {"sold", "claimed"}
def list_devices( def list_devices(
search: str | None = None, search: str | None = None,
online_only: bool | None = None, online_only: bool | None = None,
subscription_tier: str | None = None, subscription_tier: str | None = None,
) -> list[DeviceInDB]: ) -> list[DeviceInDB]:
"""List fleet devices (sold + claimed only) with optional filters.""" """List devices with optional filters."""
db = get_db() db = get_db()
ref = db.collection(COLLECTION) ref = db.collection(COLLECTION)
query = ref query = ref
@@ -141,14 +118,6 @@ def list_devices(
results = [] results = []
for doc in docs: for doc in docs:
raw = doc.to_dict() or {}
# Only include sold/claimed devices in the fleet view.
# Legacy devices without mfg_status are included to avoid breaking old data.
mfg_status = raw.get("mfg_status")
if mfg_status and mfg_status not in FLEET_STATUSES:
continue
device = _doc_to_device(doc) device = _doc_to_device(doc)
# Client-side filters # Client-side filters
@@ -159,7 +128,7 @@ def list_devices(
search_lower = search.lower() search_lower = search.lower()
name_match = search_lower in (device.device_name or "").lower() name_match = search_lower in (device.device_name or "").lower()
location_match = search_lower in (device.device_location or "").lower() location_match = search_lower in (device.device_location or "").lower()
sn_match = search_lower in (device.serial_number or "").lower() sn_match = search_lower in (device.device_id or "").lower()
if not (name_match or location_match or sn_match): if not (name_match or location_match or sn_match):
continue continue
@@ -213,11 +182,6 @@ def update_device(device_doc_id: str, data: DeviceUpdate) -> DeviceInDB:
update_data = data.model_dump(exclude_none=True) update_data = data.model_dump(exclude_none=True)
# Convert {lat, lng} dict to a Firestore GeoPoint
coords = update_data.get("device_location_coordinates")
if isinstance(coords, dict) and "lat" in coords and "lng" in coords:
update_data["device_location_coordinates"] = GeoPoint(coords["lat"], coords["lng"])
# Deep-merge nested structs so unmentioned sub-fields are preserved # Deep-merge nested structs so unmentioned sub-fields are preserved
existing = doc.to_dict() existing = doc.to_dict()
nested_keys = ( nested_keys = (

2
backend/equipment/service.py
View File

@@ -4,7 +4,7 @@ from shared.firebase import get_db
from shared.exceptions import NotFoundError from shared.exceptions import NotFoundError
from equipment.models import NoteCreate, NoteUpdate, NoteInDB from equipment.models import NoteCreate, NoteUpdate, NoteInDB
COLLECTION = "notes" COLLECTION = "equipment_notes"
VALID_CATEGORIES = {"general", "maintenance", "installation", "issue", "action_item", "other"} VALID_CATEGORIES = {"general", "maintenance", "installation", "issue", "action_item", "other"}

33
backend/firmware/models.py
View File

@@ -11,7 +11,7 @@ class UpdateType(str, Enum):
class FirmwareVersion(BaseModel): class FirmwareVersion(BaseModel):
id: str id: str
hw_type: str # e.g. "vesper", "vesper_plus", "vesper_pro", "bespoke" hw_type: str # e.g. "vesper", "vesper_plus", "vesper_pro"
channel: str # "stable", "beta", "alpha", "testing" channel: str # "stable", "beta", "alpha", "testing"
version: str # semver e.g. "1.5" version: str # semver e.g. "1.5"
filename: str filename: str
@@ -20,10 +20,8 @@ class FirmwareVersion(BaseModel):
update_type: UpdateType = UpdateType.mandatory update_type: UpdateType = UpdateType.mandatory
min_fw_version: Optional[str] = None # minimum fw version required to install this min_fw_version: Optional[str] = None # minimum fw version required to install this
uploaded_at: str uploaded_at: str
changelog: Optional[str] = None notes: Optional[str] = None
release_note: Optional[str] = None
is_latest: bool = False is_latest: bool = False
bespoke_uid: Optional[str] = None # only set when hw_type == "bespoke"
class FirmwareListResponse(BaseModel): class FirmwareListResponse(BaseModel):
@@ -32,34 +30,17 @@ class FirmwareListResponse(BaseModel):
class FirmwareMetadataResponse(BaseModel): class FirmwareMetadataResponse(BaseModel):
"""Returned by both /latest and /{version}/info endpoints. """Returned by both /latest and /{version}/info endpoints."""
Two orthogonal axes:
channel — the release track the device is subscribed to
("stable" | "beta" | "development")
Firmware validates this matches the channel it requested.
update_type — the urgency of THIS release, set by the publisher
("optional" | "mandatory" | "emergency")
Firmware reads mandatory/emergency booleans derived from this.
Additional firmware-compatible fields:
size — binary size in bytes (firmware reads "size", not "size_bytes")
mandatory — True when update_type is mandatory or emergency
emergency — True only when update_type is emergency
"""
hw_type: str hw_type: str
channel: str # release track — firmware validates this channel: str
version: str version: str
size: int # firmware reads "size" size_bytes: int
size_bytes: int # kept for admin-panel consumers
sha256: str sha256: str
update_type: UpdateType # urgency enum — for admin panel display update_type: UpdateType
mandatory: bool # derived: update_type in (mandatory, emergency)
emergency: bool # derived: update_type == emergency
min_fw_version: Optional[str] = None min_fw_version: Optional[str] = None
download_url: str download_url: str
uploaded_at: str uploaded_at: str
release_note: Optional[str] = None notes: Optional[str] = None
# Keep backwards-compatible alias # Keep backwards-compatible alias

134
backend/firmware/router.py
View File

@@ -1,21 +1,13 @@
from fastapi import APIRouter, Depends, Query, UploadFile, File, Form, HTTPException from fastapi import APIRouter, Depends, Query, UploadFile, File, Form
from fastapi.responses import FileResponse, PlainTextResponse from fastapi.responses import FileResponse
from pydantic import BaseModel
from typing import Optional from typing import Optional
import logging
from sqlalchemy.ext.asyncio import AsyncSession
from auth.models import TokenPayload from auth.models import TokenPayload
from auth.dependencies import require_permission from auth.dependencies import require_permission
from firmware.models import FirmwareVersion, FirmwareListResponse, FirmwareMetadataResponse, UpdateType from firmware.models import FirmwareVersion, FirmwareListResponse, FirmwareMetadataResponse, UpdateType
from firmware import service from firmware import service
from database.postgres import get_pg_session
from shared.audit import log_action
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/firmware", tags=["firmware"]) router = APIRouter(prefix="/api/firmware", tags=["firmware"])
ota_router = APIRouter(prefix="/api/ota", tags=["ota-telemetry"])
@router.post("/upload", response_model=FirmwareVersion, status_code=201) @router.post("/upload", response_model=FirmwareVersion, status_code=201)
@@ -25,28 +17,20 @@ async def upload_firmware(
version: str = Form(...), version: str = Form(...),
update_type: UpdateType = Form(UpdateType.mandatory), update_type: UpdateType = Form(UpdateType.mandatory),
min_fw_version: Optional[str] = Form(None), min_fw_version: Optional[str] = Form(None),
changelog: Optional[str] = Form(None), notes: Optional[str] = Form(None),
release_note: Optional[str] = Form(None),
bespoke_uid: Optional[str] = Form(None),
file: UploadFile = File(...), file: UploadFile = File(...),
_user: TokenPayload = Depends(require_permission("manufacturing", "add")), _user: TokenPayload = Depends(require_permission("manufacturing", "add")),
db: AsyncSession = Depends(get_pg_session),
): ):
file_bytes = await file.read() file_bytes = await file.read()
fw = service.upload_firmware( return service.upload_firmware(
hw_type=hw_type, hw_type=hw_type,
channel=channel, channel=channel,
version=version, version=version,
file_bytes=file_bytes, file_bytes=file_bytes,
update_type=update_type, update_type=update_type,
min_fw_version=min_fw_version, min_fw_version=min_fw_version,
changelog=changelog, notes=notes,
release_note=release_note,
bespoke_uid=bespoke_uid,
) )
await log_action(db, _user.sub, _user.name or _user.email, "CREATE", "firmware",
fw.id, f"{hw_type} v{version} ({channel})")
return fw
@router.get("", response_model=FirmwareListResponse) @router.get("", response_model=FirmwareListResponse)
@@ -60,28 +44,11 @@ def list_firmware(
@router.get("/{hw_type}/{channel}/latest", response_model=FirmwareMetadataResponse) @router.get("/{hw_type}/{channel}/latest", response_model=FirmwareMetadataResponse)
def get_latest_firmware( def get_latest_firmware(hw_type: str, channel: str):
hw_type: str,
channel: str,
hw_version: Optional[str] = Query(None, description="Hardware revision from NVS, e.g. '1.0'"),
current_version: Optional[str] = Query(None, description="Currently running firmware semver, e.g. '1.2.3'"),
):
"""Returns metadata for the latest firmware for a given hw_type + channel. """Returns metadata for the latest firmware for a given hw_type + channel.
No auth required — devices call this endpoint to check for updates. No auth required — devices call this endpoint to check for updates.
""" """
return service.get_latest(hw_type, channel, hw_version=hw_version, current_version=current_version) return service.get_latest(hw_type, channel)
@router.get("/{hw_type}/{channel}/latest/changelog", response_class=PlainTextResponse)
def get_latest_changelog(hw_type: str, channel: str):
"""Returns the full changelog for the latest firmware. Plain text."""
return service.get_latest_changelog(hw_type, channel)
@router.get("/{hw_type}/{channel}/{version}/info/changelog", response_class=PlainTextResponse)
def get_version_changelog(hw_type: str, channel: str, version: str):
"""Returns the full changelog for a specific firmware version. Plain text."""
return service.get_version_changelog(hw_type, channel, version)
@router.get("/{hw_type}/{channel}/{version}/info", response_model=FirmwareMetadataResponse) @router.get("/{hw_type}/{channel}/{version}/info", response_model=FirmwareMetadataResponse)
@@ -103,94 +70,9 @@ def download_firmware(hw_type: str, channel: str, version: str):
) )
@router.put("/{firmware_id}", response_model=FirmwareVersion)
async def edit_firmware(
firmware_id: str,
channel: Optional[str] = Form(None),
version: Optional[str] = Form(None),
update_type: Optional[UpdateType] = Form(None),
min_fw_version: Optional[str] = Form(None),
changelog: Optional[str] = Form(None),
release_note: Optional[str] = Form(None),
bespoke_uid: Optional[str] = Form(None),
file: Optional[UploadFile] = File(None),
_user: TokenPayload = Depends(require_permission("manufacturing", "add")),
db: AsyncSession = Depends(get_pg_session),
):
file_bytes = await file.read() if file and file.filename else None
fw = service.edit_firmware(
doc_id=firmware_id,
channel=channel,
version=version,
update_type=update_type,
min_fw_version=min_fw_version,
changelog=changelog,
release_note=release_note,
bespoke_uid=bespoke_uid,
file_bytes=file_bytes,
)
await log_action(db, _user.sub, _user.name or _user.email, "UPDATE", "firmware",
firmware_id, f"{fw.hw_type} v{fw.version} ({fw.channel})" if fw else firmware_id)
return fw
@router.delete("/{firmware_id}", status_code=204) @router.delete("/{firmware_id}", status_code=204)
async def delete_firmware( def delete_firmware(
firmware_id: str, firmware_id: str,
_user: TokenPayload = Depends(require_permission("manufacturing", "delete")), _user: TokenPayload = Depends(require_permission("manufacturing", "delete")),
db: AsyncSession = Depends(get_pg_session),
): ):
fw = service.get_firmware(firmware_id) if hasattr(service, "get_firmware") else None
service.delete_firmware(firmware_id) service.delete_firmware(firmware_id)
label = f"{fw.hw_type} v{fw.version} ({fw.channel})" if fw else firmware_id
await log_action(db, _user.sub, _user.name or _user.email, "DELETE", "firmware",
firmware_id, label)
# ─────────────────────────────────────────────────────────────────────────────
# OTA event telemetry — called by devices (no auth, best-effort)
# ─────────────────────────────────────────────────────────────────────────────
class OtaDownloadEvent(BaseModel):
device_uid: str
hw_type: str
hw_version: str
from_version: str
to_version: str
channel: str
class OtaFlashEvent(BaseModel):
device_uid: str
hw_type: str
hw_version: str
from_version: str
to_version: str
channel: str
sha256: str
@ota_router.post("/events/download", status_code=204)
def ota_event_download(event: OtaDownloadEvent):
"""Device reports that firmware was fully written to flash (pre-commit).
No auth required — best-effort telemetry from the device.
"""
logger.info(
"OTA download event: device=%s hw=%s/%s %s → %s (channel=%s)",
event.device_uid, event.hw_type, event.hw_version,
event.from_version, event.to_version, event.channel,
)
service.record_ota_event("download", event.model_dump())
@ota_router.post("/events/flash", status_code=204)
def ota_event_flash(event: OtaFlashEvent):
"""Device reports that firmware partition was committed and device is rebooting.
No auth required — best-effort telemetry from the device.
"""
logger.info(
"OTA flash event: device=%s hw=%s/%s %s → %s (channel=%s sha256=%.16s...)",
event.device_uid, event.hw_type, event.hw_version,
event.from_version, event.to_version, event.channel, event.sha256,
)
service.record_ota_event("flash", event.model_dump())

227
backend/firmware/service.py
View File

@@ -1,9 +1,7 @@
import hashlib import hashlib
import logging
import uuid import uuid
from datetime import datetime, timezone from datetime import datetime, timezone
from pathlib import Path from pathlib import Path
from typing import Any
from fastapi import HTTPException from fastapi import HTTPException
@@ -12,11 +10,9 @@ from shared.firebase import get_db
from shared.exceptions import NotFoundError from shared.exceptions import NotFoundError
from firmware.models import FirmwareVersion, FirmwareMetadataResponse, UpdateType from firmware.models import FirmwareVersion, FirmwareMetadataResponse, UpdateType
logger = logging.getLogger(__name__)
COLLECTION = "firmware_versions" COLLECTION = "firmware_versions"
VALID_HW_TYPES = {"vesper", "vesper_plus", "vesper_pro", "chronos", "chronos_pro", "agnus", "agnus_mini", "bespoke"} VALID_HW_TYPES = {"vesper", "vesper_plus", "vesper_pro", "chronos", "chronos_pro", "agnus", "agnus_mini"}
VALID_CHANNELS = {"stable", "beta", "alpha", "testing"} VALID_CHANNELS = {"stable", "beta", "alpha", "testing"}
@@ -43,31 +39,24 @@ def _doc_to_firmware_version(doc) -> FirmwareVersion:
update_type=data.get("update_type", UpdateType.mandatory), update_type=data.get("update_type", UpdateType.mandatory),
min_fw_version=data.get("min_fw_version"), min_fw_version=data.get("min_fw_version"),
uploaded_at=uploaded_str, uploaded_at=uploaded_str,
changelog=data.get("changelog"), notes=data.get("notes"),
release_note=data.get("release_note"),
is_latest=data.get("is_latest", False), is_latest=data.get("is_latest", False),
bespoke_uid=data.get("bespoke_uid"),
) )
def _fw_to_metadata_response(fw: FirmwareVersion) -> FirmwareMetadataResponse: def _fw_to_metadata_response(fw: FirmwareVersion) -> FirmwareMetadataResponse:
download_url = f"/api/firmware/{fw.hw_type}/{fw.channel}/{fw.version}/firmware.bin" download_url = f"/api/firmware/{fw.hw_type}/{fw.channel}/{fw.version}/firmware.bin"
is_emergency = fw.update_type == UpdateType.emergency
is_mandatory = fw.update_type in (UpdateType.mandatory, UpdateType.emergency)
return FirmwareMetadataResponse( return FirmwareMetadataResponse(
hw_type=fw.hw_type, hw_type=fw.hw_type,
channel=fw.channel, # firmware validates this matches requested channel channel=fw.channel,
version=fw.version, version=fw.version,
size=fw.size_bytes, # firmware reads "size" size_bytes=fw.size_bytes,
size_bytes=fw.size_bytes, # kept for admin-panel consumers
sha256=fw.sha256, sha256=fw.sha256,
update_type=fw.update_type, # urgency enum — for admin panel display update_type=fw.update_type,
mandatory=is_mandatory, # firmware reads this to decide auto-apply
emergency=is_emergency, # firmware reads this to decide immediate apply
min_fw_version=fw.min_fw_version, min_fw_version=fw.min_fw_version,
download_url=download_url, download_url=download_url,
uploaded_at=fw.uploaded_at, uploaded_at=fw.uploaded_at,
release_note=fw.release_note, notes=fw.notes,
) )
@@ -78,59 +67,33 @@ def upload_firmware(
file_bytes: bytes, file_bytes: bytes,
update_type: UpdateType = UpdateType.mandatory, update_type: UpdateType = UpdateType.mandatory,
min_fw_version: str | None = None, min_fw_version: str | None = None,
changelog: str | None = None, notes: str | None = None,
release_note: str | None = None,
bespoke_uid: str | None = None,
) -> FirmwareVersion: ) -> FirmwareVersion:
if hw_type not in VALID_HW_TYPES: if hw_type not in VALID_HW_TYPES:
raise HTTPException(status_code=400, detail=f"Invalid hw_type. Must be one of: {', '.join(sorted(VALID_HW_TYPES))}") raise HTTPException(status_code=400, detail=f"Invalid hw_type. Must be one of: {', '.join(sorted(VALID_HW_TYPES))}")
if channel not in VALID_CHANNELS: if channel not in VALID_CHANNELS:
raise HTTPException(status_code=400, detail=f"Invalid channel. Must be one of: {', '.join(sorted(VALID_CHANNELS))}") raise HTTPException(status_code=400, detail=f"Invalid channel. Must be one of: {', '.join(sorted(VALID_CHANNELS))}")
if hw_type == "bespoke" and not bespoke_uid:
raise HTTPException(status_code=400, detail="bespoke_uid is required when hw_type is 'bespoke'")
db = get_db()
sha256 = hashlib.sha256(file_bytes).hexdigest()
now = datetime.now(timezone.utc)
# For bespoke firmware: if a firmware with the same bespoke_uid already exists,
# overwrite it (delete old doc + file, reuse same storage path keyed by uid).
if hw_type == "bespoke" and bespoke_uid:
existing_docs = list(
db.collection(COLLECTION)
.where("hw_type", "==", "bespoke")
.where("bespoke_uid", "==", bespoke_uid)
.stream()
)
for old_doc in existing_docs:
old_data = old_doc.to_dict() or {}
old_path = _storage_path("bespoke", old_data.get("channel", channel), old_data.get("version", version))
if old_path.exists():
old_path.unlink()
try:
old_path.parent.rmdir()
except OSError:
pass
old_doc.reference.delete()
dest = _storage_path(hw_type, channel, version) dest = _storage_path(hw_type, channel, version)
dest.parent.mkdir(parents=True, exist_ok=True) dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_bytes(file_bytes) dest.write_bytes(file_bytes)
sha256 = hashlib.sha256(file_bytes).hexdigest()
now = datetime.now(timezone.utc)
doc_id = str(uuid.uuid4()) doc_id = str(uuid.uuid4())
db = get_db()
# Mark previous latest for this hw_type+channel as no longer latest # Mark previous latest for this hw_type+channel as no longer latest
# (skip for bespoke — each bespoke_uid is its own independent firmware) prev_docs = (
if hw_type != "bespoke": db.collection(COLLECTION)
prev_docs = ( .where("hw_type", "==", hw_type)
db.collection(COLLECTION) .where("channel", "==", channel)
.where("hw_type", "==", hw_type) .where("is_latest", "==", True)
.where("channel", "==", channel) .stream()
.where("is_latest", "==", True) )
.stream() for prev in prev_docs:
) prev.reference.update({"is_latest": False})
for prev in prev_docs:
prev.reference.update({"is_latest": False})
doc_ref = db.collection(COLLECTION).document(doc_id) doc_ref = db.collection(COLLECTION).document(doc_id)
doc_ref.set({ doc_ref.set({
@@ -143,10 +106,8 @@ def upload_firmware(
"update_type": update_type.value, "update_type": update_type.value,
"min_fw_version": min_fw_version, "min_fw_version": min_fw_version,
"uploaded_at": now, "uploaded_at": now,
"changelog": changelog, "notes": notes,
"release_note": release_note,
"is_latest": True, "is_latest": True,
"bespoke_uid": bespoke_uid,
}) })
return _doc_to_firmware_version(doc_ref.get()) return _doc_to_firmware_version(doc_ref.get())
@@ -169,11 +130,9 @@ def list_firmware(
return items return items
def get_latest(hw_type: str, channel: str, hw_version: str | None = None, current_version: str | None = None) -> FirmwareMetadataResponse: def get_latest(hw_type: str, channel: str) -> FirmwareMetadataResponse:
if hw_type not in VALID_HW_TYPES: if hw_type not in VALID_HW_TYPES:
raise HTTPException(status_code=400, detail=f"Invalid hw_type '{hw_type}'") raise HTTPException(status_code=400, detail=f"Invalid hw_type '{hw_type}'")
if hw_type == "bespoke":
raise HTTPException(status_code=400, detail="Bespoke firmware is not served via auto-update. Use the direct download URL.")
if channel not in VALID_CHANNELS: if channel not in VALID_CHANNELS:
raise HTTPException(status_code=400, detail=f"Invalid channel '{channel}'") raise HTTPException(status_code=400, detail=f"Invalid channel '{channel}'")
@@ -214,52 +173,6 @@ def get_version_info(hw_type: str, channel: str, version: str) -> FirmwareMetada
return _fw_to_metadata_response(_doc_to_firmware_version(docs[0])) return _fw_to_metadata_response(_doc_to_firmware_version(docs[0]))
def get_latest_changelog(hw_type: str, channel: str) -> str:
if hw_type not in VALID_HW_TYPES:
raise HTTPException(status_code=400, detail=f"Invalid hw_type '{hw_type}'")
if channel not in VALID_CHANNELS:
raise HTTPException(status_code=400, detail=f"Invalid channel '{channel}'")
db = get_db()
docs = list(
db.collection(COLLECTION)
.where("hw_type", "==", hw_type)
.where("channel", "==", channel)
.where("is_latest", "==", True)
.limit(1)
.stream()
)
if not docs:
raise NotFoundError("Firmware")
fw = _doc_to_firmware_version(docs[0])
if not fw.changelog:
raise NotFoundError("Changelog")
return fw.changelog
def get_version_changelog(hw_type: str, channel: str, version: str) -> str:
if hw_type not in VALID_HW_TYPES:
raise HTTPException(status_code=400, detail=f"Invalid hw_type '{hw_type}'")
if channel not in VALID_CHANNELS:
raise HTTPException(status_code=400, detail=f"Invalid channel '{channel}'")
db = get_db()
docs = list(
db.collection(COLLECTION)
.where("hw_type", "==", hw_type)
.where("channel", "==", channel)
.where("version", "==", version)
.limit(1)
.stream()
)
if not docs:
raise NotFoundError("Firmware version")
fw = _doc_to_firmware_version(docs[0])
if not fw.changelog:
raise NotFoundError("Changelog")
return fw.changelog
def get_firmware_path(hw_type: str, channel: str, version: str) -> Path: def get_firmware_path(hw_type: str, channel: str, version: str) -> Path:
path = _storage_path(hw_type, channel, version) path = _storage_path(hw_type, channel, version)
if not path.exists(): if not path.exists():
@@ -267,98 +180,6 @@ def get_firmware_path(hw_type: str, channel: str, version: str) -> Path:
return path return path
def record_ota_event(event_type: str, payload: dict[str, Any]) -> None:
"""Persist an OTA telemetry event (download or flash) to Firestore.
Best-effort — caller should not raise on failure.
"""
try:
db = get_db()
db.collection("ota_events").add({
"event_type": event_type,
"received_at": datetime.now(timezone.utc),
**payload,
})
except Exception as exc:
logger.warning("Failed to persist OTA event (%s): %s", event_type, exc)
def edit_firmware(
doc_id: str,
channel: str | None = None,
version: str | None = None,
update_type: UpdateType | None = None,
min_fw_version: str | None = None,
changelog: str | None = None,
release_note: str | None = None,
bespoke_uid: str | None = None,
file_bytes: bytes | None = None,
) -> FirmwareVersion:
db = get_db()
doc_ref = db.collection(COLLECTION).document(doc_id)
doc = doc_ref.get()
if not doc.exists:
raise NotFoundError("Firmware")
data = doc.to_dict() or {}
hw_type = data["hw_type"]
old_channel = data.get("channel", "")
old_version = data.get("version", "")
effective_channel = channel if channel is not None else old_channel
effective_version = version if version is not None else old_version
if channel is not None and channel not in VALID_CHANNELS:
raise HTTPException(status_code=400, detail=f"Invalid channel. Must be one of: {', '.join(sorted(VALID_CHANNELS))}")
updates: dict = {}
if channel is not None:
updates["channel"] = channel
if version is not None:
updates["version"] = version
if update_type is not None:
updates["update_type"] = update_type.value
if min_fw_version is not None:
updates["min_fw_version"] = min_fw_version if min_fw_version else None
if changelog is not None:
updates["changelog"] = changelog if changelog else None
if release_note is not None:
updates["release_note"] = release_note if release_note else None
if bespoke_uid is not None:
updates["bespoke_uid"] = bespoke_uid if bespoke_uid else None
if file_bytes is not None:
# Move binary if path changed
old_path = _storage_path(hw_type, old_channel, old_version)
new_path = _storage_path(hw_type, effective_channel, effective_version)
if old_path != new_path and old_path.exists():
old_path.unlink()
try:
old_path.parent.rmdir()
except OSError:
pass
new_path.parent.mkdir(parents=True, exist_ok=True)
new_path.write_bytes(file_bytes)
updates["sha256"] = hashlib.sha256(file_bytes).hexdigest()
updates["size_bytes"] = len(file_bytes)
elif (channel is not None and channel != old_channel) or (version is not None and version != old_version):
# Path changed but no new file — move existing binary
old_path = _storage_path(hw_type, old_channel, old_version)
new_path = _storage_path(hw_type, effective_channel, effective_version)
if old_path.exists() and old_path != new_path:
new_path.parent.mkdir(parents=True, exist_ok=True)
old_path.rename(new_path)
try:
old_path.parent.rmdir()
except OSError:
pass
if updates:
doc_ref.update(updates)
return _doc_to_firmware_version(doc_ref.get())
def delete_firmware(doc_id: str) -> None: def delete_firmware(doc_id: str) -> None:
db = get_db() db = get_db()
doc_ref = db.collection(COLLECTION).document(doc_id) doc_ref = db.collection(COLLECTION).document(doc_id)
@@ -390,9 +211,9 @@ def delete_firmware(doc_id: str) -> None:
db.collection(COLLECTION) db.collection(COLLECTION)
.where("hw_type", "==", hw_type) .where("hw_type", "==", hw_type)
.where("channel", "==", channel) .where("channel", "==", channel)
.order_by("uploaded_at", direction="DESCENDING")
.limit(1)
.stream() .stream()
) )
if remaining: if remaining:
# Sort in Python to avoid needing a composite Firestore index
remaining.sort(key=lambda d: d.to_dict().get("uploaded_at", ""), reverse=True)
remaining[0].reference.update({"is_latest": True}) remaining[0].reference.update({"is_latest": True})

38
backend/main.py
View File

@@ -15,24 +15,19 @@ from staff.router import router as staff_router
from helpdesk.router import router as helpdesk_router from helpdesk.router import router as helpdesk_router
from builder.router import router as builder_router from builder.router import router as builder_router
from manufacturing.router import router as manufacturing_router from manufacturing.router import router as manufacturing_router
from firmware.router import router as firmware_router, ota_router from firmware.router import router as firmware_router
from admin.router import router as admin_router from admin.router import router as admin_router
from crm.router import router as crm_products_router from crm.router import router as crm_products_router
from crm.customers_router import router as crm_customers_router from crm.customers_router import router as crm_customers_router
from crm.orders_router import router as crm_orders_router, global_router as crm_orders_global_router from crm.orders_router import router as crm_orders_router
from crm.comms_router import router as crm_comms_router from crm.comms_router import router as crm_comms_router
from crm.media_router import router as crm_media_router from crm.media_router import router as crm_media_router
from crm.nextcloud_router import router as crm_nextcloud_router from crm.nextcloud_router import router as crm_nextcloud_router
from crm.quotations_router import router as crm_quotations_router from crm.quotations_router import router as crm_quotations_router
from public.router import router as public_router
from notes.router import router as notes_router
from tickets.router import router as tickets_router
from audit.router import router as audit_router
from search.router import router as search_router
from crm.nextcloud import close_client as close_nextcloud_client, keepalive_ping as nextcloud_keepalive from crm.nextcloud import close_client as close_nextcloud_client, keepalive_ping as nextcloud_keepalive
from crm.mail_accounts import get_mail_accounts from crm.mail_accounts import get_mail_accounts
from mqtt.client import mqtt_manager from mqtt.client import mqtt_manager
import database as db from mqtt import database as mqtt_db
from melodies import service as melody_service from melodies import service as melody_service
app = FastAPI( app = FastAPI(
@@ -63,21 +58,14 @@ app.include_router(staff_router)
app.include_router(builder_router) app.include_router(builder_router)
app.include_router(manufacturing_router) app.include_router(manufacturing_router)
app.include_router(firmware_router) app.include_router(firmware_router)
app.include_router(ota_router)
app.include_router(admin_router) app.include_router(admin_router)
app.include_router(crm_products_router) app.include_router(crm_products_router)
app.include_router(crm_customers_router) app.include_router(crm_customers_router)
app.include_router(crm_orders_router) app.include_router(crm_orders_router)
app.include_router(crm_orders_global_router)
app.include_router(crm_comms_router) app.include_router(crm_comms_router)
app.include_router(crm_media_router) app.include_router(crm_media_router)
app.include_router(crm_nextcloud_router) app.include_router(crm_nextcloud_router)
app.include_router(crm_quotations_router) app.include_router(crm_quotations_router)
app.include_router(public_router)
app.include_router(notes_router)
app.include_router(tickets_router)
app.include_router(audit_router)
app.include_router(search_router)
async def nextcloud_keepalive_loop(): async def nextcloud_keepalive_loop():
@@ -97,27 +85,14 @@ async def email_sync_loop():
print(f"[EMAIL SYNC] Error: {e}") print(f"[EMAIL SYNC] Error: {e}")
async def crm_poll_loop():
while True:
await asyncio.sleep(24 * 60 * 60) # once per day
try:
from crm.service import poll_crm_customer_statuses
poll_crm_customer_statuses()
except Exception as e:
print(f"[CRM POLL] Error: {e}")
@app.on_event("startup") @app.on_event("startup")
async def startup(): async def startup():
init_firebase() init_firebase()
from database.core import init_db as sqlite_init_db await mqtt_db.init_db()
await sqlite_init_db()
await melody_service.migrate_from_firestore() await melody_service.migrate_from_firestore()
mqtt_manager.start(asyncio.get_event_loop()) mqtt_manager.start(asyncio.get_event_loop())
asyncio.create_task(db.partition_manager_loop()) asyncio.create_task(mqtt_db.purge_loop())
asyncio.create_task(db.purge_loop())
asyncio.create_task(nextcloud_keepalive_loop()) asyncio.create_task(nextcloud_keepalive_loop())
asyncio.create_task(crm_poll_loop())
sync_accounts = [a for a in get_mail_accounts() if a.get("sync_inbound") and a.get("imap_host")] sync_accounts = [a for a in get_mail_accounts() if a.get("sync_inbound") and a.get("imap_host")]
if sync_accounts: if sync_accounts:
print(f"[EMAIL SYNC] IMAP configured for {len(sync_accounts)} account(s) - starting sync loop") print(f"[EMAIL SYNC] IMAP configured for {len(sync_accounts)} account(s) - starting sync loop")
@@ -129,8 +104,7 @@ async def startup():
@app.on_event("shutdown") @app.on_event("shutdown")
async def shutdown(): async def shutdown():
mqtt_manager.stop() mqtt_manager.stop()
from database.core import close_db as sqlite_close_db await mqtt_db.close_db()
await sqlite_close_db()
await close_nextcloud_client() await close_nextcloud_client()

2
backend/manufacturing/audit.py
View File

@@ -1,6 +1,6 @@
import json import json
import logging import logging
from database import get_db from mqtt.database import get_db
logger = logging.getLogger("manufacturing.audit") logger = logging.getLogger("manufacturing.audit")

46
backend/manufacturing/models.py
View File

@@ -15,7 +15,7 @@ class BoardType(str, Enum):
BOARD_TYPE_LABELS = { BOARD_TYPE_LABELS = {
"vesper": "Vesper", "vesper": "Vesper",
"vesper_plus": "Vesper Plus", "vesper_plus": "Vesper+",
"vesper_pro": "Vesper Pro", "vesper_pro": "Vesper Pro",
"chronos": "Chronos", "chronos": "Chronos",
"chronos_pro": "Chronos Pro", "chronos_pro": "Chronos Pro",
@@ -23,28 +23,6 @@ BOARD_TYPE_LABELS = {
"agnus": "Agnus", "agnus": "Agnus",
} }
# Family codes (BS + 4 chars = segment 1 of serial number)
BOARD_FAMILY_CODES = {
"vesper": "VSPR",
"vesper_plus": "VSPR",
"vesper_pro": "VSPR",
"agnus": "AGNS",
"agnus_mini": "AGNS",
"chronos": "CRNS",
"chronos_pro": "CRNS",
}
# Variant codes (3 chars = first part of segment 3 of serial number)
BOARD_VARIANT_CODES = {
"vesper": "STD",
"vesper_plus": "PLS",
"vesper_pro": "PRO",
"agnus": "STD",
"agnus_mini": "MIN",
"chronos": "STD",
"chronos_pro": "PRO",
}
class MfgStatus(str, Enum): class MfgStatus(str, Enum):
manufactured = "manufactured" manufactured = "manufactured"
@@ -55,13 +33,6 @@ class MfgStatus(str, Enum):
decommissioned = "decommissioned" decommissioned = "decommissioned"
class LifecycleEntry(BaseModel):
status_id: str
date: str # ISO 8601 UTC string
note: Optional[str] = None
set_by: Optional[str] = None
class BatchCreate(BaseModel): class BatchCreate(BaseModel):
board_type: BoardType board_type: BoardType
board_version: str = Field( board_version: str = Field(
@@ -91,9 +62,6 @@ class DeviceInventoryItem(BaseModel):
owner: Optional[str] = None owner: Optional[str] = None
assigned_to: Optional[str] = None assigned_to: Optional[str] = None
device_name: Optional[str] = None device_name: Optional[str] = None
lifecycle_history: Optional[List["LifecycleEntry"]] = None
customer_id: Optional[str] = None
user_list: Optional[List[str]] = None
class DeviceInventoryListResponse(BaseModel): class DeviceInventoryListResponse(BaseModel):
@@ -104,19 +72,11 @@ class DeviceInventoryListResponse(BaseModel):
class DeviceStatusUpdate(BaseModel): class DeviceStatusUpdate(BaseModel):
status: MfgStatus status: MfgStatus
note: Optional[str] = None note: Optional[str] = None
force_claimed: bool = False
class DeviceAssign(BaseModel): class DeviceAssign(BaseModel):
customer_id: str customer_email: str
customer_name: Optional[str] = None
class CustomerSearchResult(BaseModel):
id: str
name: str = ""
email: str = ""
organization: str = ""
phone: str = ""
class RecentActivityItem(BaseModel): class RecentActivityItem(BaseModel):

22
backend/manufacturing/orm.py
View File

@@ -1,22 +0,0 @@
from datetime import datetime, timezone
from sqlalchemy import BigInteger, Column, DateTime, Index, String, Text
from database.postgres import Base
def _now():
return datetime.now(timezone.utc)
class MfgAuditLog(Base):
__tablename__ = "mfg_audit_log"
__table_args__ = (
Index("idx_mfg_audit_time", "timestamp"),
Index("idx_mfg_audit_action", "action"),
)
id = Column(BigInteger, primary_key=True, autoincrement=True)
timestamp = Column(DateTime(timezone=True), nullable=False, default=_now)
admin_user = Column(String(256), nullable=False)
action = Column(String(128), nullable=False)
serial_number = Column(String(128))
detail = Column(Text)

536
backend/manufacturing/router.py
View File

@@ -1,9 +1,7 @@
from fastapi import APIRouter, Depends, Query, HTTPException, UploadFile, File from fastapi import APIRouter, Depends, Query, HTTPException
from fastapi.responses import Response from fastapi.responses import Response
from fastapi.responses import RedirectResponse from fastapi.responses import RedirectResponse
from typing import Optional from typing import Optional
from pydantic import BaseModel
from sqlalchemy.ext.asyncio import AsyncSession
from auth.models import TokenPayload from auth.models import TokenPayload
from auth.dependencies import require_permission from auth.dependencies import require_permission
@@ -14,26 +12,8 @@ from manufacturing.models import (
ManufacturingStats, ManufacturingStats,
) )
from manufacturing import service from manufacturing import service
from shared.audit import log_action from manufacturing import audit
from shared.exceptions import NotFoundError from shared.exceptions import NotFoundError
from shared.firebase import get_db as get_firestore
from database.postgres import get_pg_session
class LifecycleEntryPatch(BaseModel):
index: int
date: Optional[str] = None
note: Optional[str] = None
class LifecycleEntryCreate(BaseModel):
status_id: str
date: Optional[str] = None
note: Optional[str] = None
VALID_FLASH_ASSETS = {"bootloader.bin", "partitions.bin"}
VALID_HW_TYPES_MFG = {"vesper", "vesper_plus", "vesper_pro", "agnus", "agnus_mini", "chronos", "chronos_pro"}
# Bespoke UIDs are dynamic — we allow any non-empty slug that doesn't clash with
# a standard hw_type name. The flash-asset upload endpoint checks this below.
router = APIRouter(prefix="/api/manufacturing", tags=["manufacturing"]) router = APIRouter(prefix="/api/manufacturing", tags=["manufacturing"])
@@ -45,21 +25,26 @@ def get_stats(
return service.get_stats() return service.get_stats()
@router.get("/audit-log")
async def get_audit_log(
limit: int = Query(20, ge=1, le=100),
_user: TokenPayload = Depends(require_permission("manufacturing", "view")),
):
entries = await audit.get_recent(limit=limit)
return {"entries": entries}
@router.post("/batch", response_model=BatchResponse, status_code=201) @router.post("/batch", response_model=BatchResponse, status_code=201)
async def create_batch( async def create_batch(
body: BatchCreate, body: BatchCreate,
user: TokenPayload = Depends(require_permission("manufacturing", "add")), user: TokenPayload = Depends(require_permission("manufacturing", "add")),
db: AsyncSession = Depends(get_pg_session),
): ):
result = service.create_batch(body) result = service.create_batch(body)
await log_action( await audit.log_action(
db, user.sub, user.email, admin_user=user.email,
action="CREATE", action="batch_created",
entity_type="device_batch", detail={
entity_id=result.batch_id, "batch_id": result.batch_id,
entity_label=f"Batch {result.batch_id} ({result.board_type}, qty {len(result.serial_numbers)})",
meta={
"board_type": result.board_type, "board_type": result.board_type,
"board_version": result.board_version, "board_version": result.board_version,
"quantity": len(result.serial_numbers), "quantity": len(result.serial_numbers),
@@ -95,207 +80,18 @@ def get_device(
return service.get_device_by_sn(sn) return service.get_device_by_sn(sn)
@router.get("/customers/search")
def search_customers(
q: str = Query(""),
_user: TokenPayload = Depends(require_permission("manufacturing", "view")),
):
"""Search CRM customers by name, email, phone, organization, or tags."""
results = service.search_customers(q)
return {"results": results}
@router.get("/customers/{customer_id}")
def get_customer(
customer_id: str,
_user: TokenPayload = Depends(require_permission("manufacturing", "view")),
):
"""Get a single CRM customer by ID."""
db = get_firestore()
doc = db.collection("crm_customers").document(customer_id).get()
if not doc.exists:
raise HTTPException(status_code=404, detail="Customer not found")
data = doc.to_dict() or {}
loc = data.get("location") or {}
city = loc.get("city") if isinstance(loc, dict) else None
return {
"id": doc.id,
"name": data.get("name") or "",
"surname": data.get("surname") or "",
"email": data.get("email") or "",
"organization": data.get("organization") or "",
"phone": data.get("phone") or "",
"city": city or "",
}
@router.patch("/devices/{sn}/status", response_model=DeviceInventoryItem) @router.patch("/devices/{sn}/status", response_model=DeviceInventoryItem)
async def update_status( async def update_status(
sn: str, sn: str,
body: DeviceStatusUpdate, body: DeviceStatusUpdate,
user: TokenPayload = Depends(require_permission("manufacturing", "edit")), user: TokenPayload = Depends(require_permission("manufacturing", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
# Guard: claimed requires at least one user in user_list result = service.update_device_status(sn, body)
# (allow if explicitly force_claimed=true, which the mfg UI sets after adding a user manually) await audit.log_action(
if body.status.value == "claimed": admin_user=user.email,
db = get_firestore() action="status_updated",
docs = list(db.collection("devices").where("serial_number", "==", sn).limit(1).stream()) serial_number=sn,
if docs: detail={"status": body.status.value, "note": body.note},
data = docs[0].to_dict() or {}
user_list = data.get("user_list", []) or []
if not user_list and not getattr(body, "force_claimed", False):
raise HTTPException(
status_code=400,
detail="Cannot set status to 'claimed': device has no users in user_list. "
"Assign a user first, then set to Claimed.",
)
# Guard: sold requires a customer assigned
if body.status.value == "sold":
db = get_firestore()
docs = list(db.collection("devices").where("serial_number", "==", sn).limit(1).stream())
if docs:
data = docs[0].to_dict() or {}
if not data.get("customer_id"):
raise HTTPException(
status_code=400,
detail="Cannot set status to 'sold' without an assigned customer. "
"Use the 'Assign to Customer' action first.",
)
result = service.update_device_status(sn, body, set_by=user.email)
await log_action(
db, user.sub, user.email,
action="STATUS_CHANGE",
entity_type="device",
entity_id=sn,
entity_label=sn,
meta={"status": body.status.value, "note": body.note},
)
return result
@router.patch("/devices/{sn}/lifecycle", response_model=DeviceInventoryItem)
async def patch_lifecycle_entry(
sn: str,
body: LifecycleEntryPatch,
user: TokenPayload = Depends(require_permission("manufacturing", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
"""Edit the date and/or note of a lifecycle history entry by index."""
fs = get_firestore()
docs = list(fs.collection("devices").where("serial_number", "==", sn).limit(1).stream())
if not docs:
raise HTTPException(status_code=404, detail="Device not found")
doc_ref = docs[0].reference
data = docs[0].to_dict() or {}
history = data.get("lifecycle_history") or []
if body.index < 0 or body.index >= len(history):
raise HTTPException(status_code=400, detail="Invalid lifecycle entry index")
if body.date is not None:
history[body.index]["date"] = body.date
if body.note is not None:
history[body.index]["note"] = body.note
doc_ref.update({"lifecycle_history": history})
from manufacturing.service import _doc_to_inventory_item
result = _doc_to_inventory_item(doc_ref.get())
await log_action(
db, user.sub, user.email,
action="UPDATE",
entity_type="device",
entity_id=sn,
entity_label=sn,
meta={"lifecycle_index": body.index, "date": body.date, "note": body.note},
)
return result
@router.post("/devices/{sn}/lifecycle", response_model=DeviceInventoryItem, status_code=200)
async def create_lifecycle_entry(
sn: str,
body: LifecycleEntryCreate,
user: TokenPayload = Depends(require_permission("manufacturing", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
"""Upsert a lifecycle history entry for the given status_id.
If an entry for this status already exists it is overwritten in-place;
otherwise a new entry is appended. This prevents duplicate entries when
a status is visited more than once (max one entry per status).
"""
from datetime import datetime, timezone
fs = get_firestore()
docs = list(fs.collection("devices").where("serial_number", "==", sn).limit(1).stream())
if not docs:
raise HTTPException(status_code=404, detail="Device not found")
doc_ref = docs[0].reference
data = docs[0].to_dict() or {}
history = list(data.get("lifecycle_history") or [])
new_entry = {
"status_id": body.status_id,
"date": body.date or datetime.now(timezone.utc).isoformat(),
"note": body.note,
"set_by": user.email,
}
existing_idx = next(
(i for i, e in enumerate(history) if e.get("status_id") == body.status_id),
None,
)
is_update = existing_idx is not None
if is_update:
history[existing_idx] = new_entry
else:
history.append(new_entry)
doc_ref.update({"lifecycle_history": history})
from manufacturing.service import _doc_to_inventory_item
result = _doc_to_inventory_item(doc_ref.get())
await log_action(
db, user.sub, user.email,
action="UPDATE" if is_update else "CREATE",
entity_type="device",
entity_id=sn,
entity_label=sn,
meta={"lifecycle_status": body.status_id, "date": new_entry["date"], "note": body.note},
)
return result
@router.delete("/devices/{sn}/lifecycle/{index}", response_model=DeviceInventoryItem)
async def delete_lifecycle_entry(
sn: str,
index: int,
user: TokenPayload = Depends(require_permission("manufacturing", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
"""Delete a lifecycle history entry by index. Cannot delete the entry for the current status."""
fs = get_firestore()
docs = list(fs.collection("devices").where("serial_number", "==", sn).limit(1).stream())
if not docs:
raise HTTPException(status_code=404, detail="Device not found")
doc_ref = docs[0].reference
data = docs[0].to_dict() or {}
history = data.get("lifecycle_history") or []
if index < 0 or index >= len(history):
raise HTTPException(status_code=400, detail="Invalid lifecycle entry index")
current_status = data.get("mfg_status", "")
deleted_entry = history[index]
if deleted_entry.get("status_id") == current_status:
raise HTTPException(status_code=400, detail="Cannot delete the entry for the current status. Change the status first.")
history.pop(index)
doc_ref.update({"lifecycle_history": history})
from manufacturing.service import _doc_to_inventory_item
result = _doc_to_inventory_item(doc_ref.get())
await log_action(
db, user.sub, user.email,
action="DELETE",
entity_type="device",
entity_id=sn,
entity_label=sn,
meta={"lifecycle_status": deleted_entry.get("status_id"), "index": index},
) )
return result return result
@@ -303,20 +99,13 @@ async def delete_lifecycle_entry(
@router.get("/devices/{sn}/nvs.bin") @router.get("/devices/{sn}/nvs.bin")
async def download_nvs( async def download_nvs(
sn: str, sn: str,
hw_type_override: Optional[str] = Query(None, description="Override hw_type written to NVS (for bespoke firmware)"),
hw_revision_override: Optional[str] = Query(None, description="Override hw_revision written to NVS (for bespoke firmware)"),
nvs_schema: Optional[str] = Query(None, description="NVS schema to use: 'legacy' or 'new' (default)"),
user: TokenPayload = Depends(require_permission("manufacturing", "view")), user: TokenPayload = Depends(require_permission("manufacturing", "view")),
db: AsyncSession = Depends(get_pg_session),
): ):
binary = service.get_nvs_binary(sn, hw_type_override=hw_type_override, hw_revision_override=hw_revision_override, legacy=(nvs_schema == "legacy")) binary = service.get_nvs_binary(sn)
await log_action( await audit.log_action(
db, user.sub, user.email, admin_user=user.email,
action="COMMAND", action="device_flashed",
entity_type="device", serial_number=sn,
entity_id=sn,
entity_label=sn,
meta={"command": "nvs_flash", "hw_type_override": hw_type_override, "nvs_schema": nvs_schema or "new"},
) )
return Response( return Response(
content=binary, content=binary,
@@ -330,19 +119,13 @@ async def assign_device(
sn: str, sn: str,
body: DeviceAssign, body: DeviceAssign,
user: TokenPayload = Depends(require_permission("manufacturing", "edit")), user: TokenPayload = Depends(require_permission("manufacturing", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
try: result = service.assign_device(sn, body)
result = service.assign_device(sn, body) await audit.log_action(
except NotFoundError as e: admin_user=user.email,
raise HTTPException(status_code=404, detail=str(e)) action="device_assigned",
await log_action( serial_number=sn,
db, user.sub, user.email, detail={"customer_email": body.customer_email, "customer_name": body.customer_name},
action="UPDATE",
entity_type="device",
entity_id=sn,
entity_label=sn,
meta={"customer_id": body.customer_id},
) )
return result return result
@@ -352,7 +135,6 @@ async def delete_device(
sn: str, sn: str,
force: bool = Query(False, description="Required to delete sold/claimed devices"), force: bool = Query(False, description="Required to delete sold/claimed devices"),
user: TokenPayload = Depends(require_permission("manufacturing", "delete")), user: TokenPayload = Depends(require_permission("manufacturing", "delete")),
db: AsyncSession = Depends(get_pg_session),
): ):
"""Delete a device. Sold/claimed devices require force=true.""" """Delete a device. Sold/claimed devices require force=true."""
try: try:
@@ -361,121 +143,24 @@ async def delete_device(
raise HTTPException(status_code=404, detail="Device not found") raise HTTPException(status_code=404, detail="Device not found")
except PermissionError as e: except PermissionError as e:
raise HTTPException(status_code=403, detail=str(e)) raise HTTPException(status_code=403, detail=str(e))
await log_action( await audit.log_action(
db, user.sub, user.email, admin_user=user.email,
action="DELETE", action="device_deleted",
entity_type="device",
entity_id=sn,
entity_label=sn,
meta={"force": force},
)
@router.post("/devices/{sn}/email/manufactured", status_code=204)
async def send_manufactured_email(
sn: str,
user: TokenPayload = Depends(require_permission("manufacturing", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
"""Send the 'device manufactured' notification to the assigned customer's email."""
db = get_firestore()
docs = list(db.collection("devices").where("serial_number", "==", sn).limit(1).stream())
if not docs:
raise HTTPException(status_code=404, detail="Device not found")
data = docs[0].to_dict() or {}
customer_id = data.get("customer_id")
if not customer_id:
raise HTTPException(status_code=400, detail="No customer assigned to this device")
customer_doc = db.collection("crm_customers").document(customer_id).get()
if not customer_doc.exists:
raise HTTPException(status_code=404, detail="Assigned customer not found")
cdata = customer_doc.to_dict() or {}
email = cdata.get("email")
if not email:
raise HTTPException(status_code=400, detail="Customer has no email address")
name_parts = [cdata.get("name") or "", cdata.get("surname") or ""]
customer_name = " ".join(p for p in name_parts if p).strip() or None
hw_family = data.get("hw_family") or data.get("hw_type") or ""
from utils.emails.device_mfged_mail import send_device_manufactured_email
send_device_manufactured_email(
customer_email=email,
serial_number=sn, serial_number=sn,
device_name=hw_family.replace("_", " ").title(), detail={"force": force},
customer_name=customer_name,
)
await log_action(
db, user.sub, user.email,
action="COMMAND",
entity_type="device",
entity_id=sn,
entity_label=sn,
meta={"command": "email_manufactured", "recipient": email},
)
@router.post("/devices/{sn}/email/assigned", status_code=204)
async def send_assigned_email(
sn: str,
user: TokenPayload = Depends(require_permission("manufacturing", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
"""Send the 'device assigned / app instructions' email to the assigned user(s)."""
db = get_firestore()
docs = list(db.collection("devices").where("serial_number", "==", sn).limit(1).stream())
if not docs:
raise HTTPException(status_code=404, detail="Device not found")
data = docs[0].to_dict() or {}
user_list = data.get("user_list") or []
if not user_list:
raise HTTPException(status_code=400, detail="No users assigned to this device")
hw_family = data.get("hw_family") or data.get("hw_type") or ""
device_name = hw_family.replace("_", " ").title()
from utils.emails.device_assigned_mail import send_device_assigned_email
errors = []
for uid in user_list:
try:
user_doc = db.collection("users").document(uid).get()
if not user_doc.exists:
continue
udata = user_doc.to_dict() or {}
email = udata.get("email")
if not email:
continue
display_name = udata.get("display_name") or udata.get("name") or None
send_device_assigned_email(
user_email=email,
serial_number=sn,
device_name=device_name,
user_name=display_name,
)
except Exception as exc:
errors.append(str(exc))
if errors:
raise HTTPException(status_code=500, detail=f"Some emails failed: {'; '.join(errors)}")
await log_action(
db, user.sub, user.email,
action="COMMAND",
entity_type="device",
entity_id=sn,
entity_label=sn,
meta={"command": "email_assigned", "user_count": len(user_list)},
) )
@router.delete("/devices", status_code=200) @router.delete("/devices", status_code=200)
async def delete_unprovisioned( async def delete_unprovisioned(
user: TokenPayload = Depends(require_permission("manufacturing", "delete")), user: TokenPayload = Depends(require_permission("manufacturing", "delete")),
db: AsyncSession = Depends(get_pg_session),
): ):
"""Delete all devices with status 'manufactured' (never provisioned).""" """Delete all devices with status 'manufactured' (never provisioned)."""
deleted = service.delete_unprovisioned_devices() deleted = service.delete_unprovisioned_devices()
await log_action( await audit.log_action(
db, user.sub, user.email, admin_user=user.email,
action="DELETE", action="bulk_delete_unprovisioned",
entity_type="device_batch", detail={"count": len(deleted), "serial_numbers": deleted},
entity_id="bulk_unprovisioned",
entity_label=f"Bulk delete unprovisioned ({len(deleted)} devices)",
meta={"count": len(deleted), "serial_numbers": deleted},
) )
return {"deleted": deleted, "count": len(deleted)} return {"deleted": deleted, "count": len(deleted)}
@@ -490,144 +175,3 @@ def redirect_firmware(
""" """
url = service.get_firmware_url(sn) url = service.get_firmware_url(sn)
return RedirectResponse(url=url, status_code=302) return RedirectResponse(url=url, status_code=302)
# ─────────────────────────────────────────────────────────────────────────────
# Flash assets — bootloader.bin and partitions.bin per hw_type
# These are the binaries that must be flashed at fixed addresses during full
# provisioning (0x1000 bootloader, 0x8000 partition table).
# They are NOT flashed during OTA updates — only during initial provisioning.
# Upload once per hw_type after each PlatformIO build that changes the layout.
# ─────────────────────────────────────────────────────────────────────────────
@router.get("/flash-assets")
def list_flash_assets(
_user: TokenPayload = Depends(require_permission("manufacturing", "view")),
):
"""Return asset status for all known board types (and any discovered bespoke UIDs).
Checks the filesystem directly — no database involved.
Each entry contains: hw_type, bootloader (exists, size, uploaded_at), partitions (same), note.
"""
return {"assets": service.list_flash_assets()}
@router.delete("/flash-assets/{hw_type}/{asset}", status_code=204)
async def delete_flash_asset(
hw_type: str,
asset: str,
user: TokenPayload = Depends(require_permission("manufacturing", "delete")),
db: AsyncSession = Depends(get_pg_session),
):
"""Delete a single flash asset file (bootloader.bin or partitions.bin)."""
if asset not in VALID_FLASH_ASSETS:
raise HTTPException(status_code=400, detail=f"Invalid asset. Must be one of: {', '.join(sorted(VALID_FLASH_ASSETS))}")
try:
service.delete_flash_asset(hw_type, asset)
except NotFoundError as e:
raise HTTPException(status_code=404, detail=str(e))
await log_action(
db, user.sub, user.email,
action="DELETE",
entity_type="firmware",
entity_id=f"{hw_type}/{asset}",
entity_label=f"{hw_type} / {asset}",
meta={"hw_type": hw_type, "asset": asset},
)
class FlashAssetNoteBody(BaseModel):
note: str
@router.put("/flash-assets/{hw_type}/note", status_code=204)
async def set_flash_asset_note(
hw_type: str,
body: FlashAssetNoteBody,
user: TokenPayload = Depends(require_permission("manufacturing", "edit")),
db: AsyncSession = Depends(get_pg_session),
):
"""Save (or overwrite) the note for a hw_type's flash asset set.
The note is stored as note.txt next to the binary files.
Pass an empty string to clear the note.
"""
service.set_flash_asset_note(hw_type, body.note)
await log_action(
db, user.sub, user.email,
action="UPDATE",
entity_type="firmware",
entity_id=hw_type,
entity_label=hw_type,
meta={"note": body.note},
)
@router.post("/flash-assets/{hw_type}/{asset}", status_code=204)
async def upload_flash_asset(
hw_type: str,
asset: str,
file: UploadFile = File(...),
user: TokenPayload = Depends(require_permission("manufacturing", "add")),
db: AsyncSession = Depends(get_pg_session),
):
"""Upload a bootloader.bin or partitions.bin for a given hw_type.
These are build artifacts from PlatformIO (.pio/build/{env}/bootloader.bin
and .pio/build/{env}/partitions.bin). Upload them once per hw_type after
each PlatformIO build that changes the partition layout.
"""
if not hw_type or len(hw_type) > 128:
raise HTTPException(status_code=400, detail="Invalid hw_type/bespoke UID.")
if asset not in VALID_FLASH_ASSETS:
raise HTTPException(status_code=400, detail=f"Invalid asset. Must be one of: {', '.join(sorted(VALID_FLASH_ASSETS))}")
data = await file.read()
service.save_flash_asset(hw_type, asset, data)
await log_action(
db, user.sub, user.email,
action="CREATE",
entity_type="firmware",
entity_id=f"{hw_type}/{asset}",
entity_label=f"{hw_type} / {asset}",
meta={"hw_type": hw_type, "asset": asset, "size_bytes": len(data)},
)
@router.get("/devices/{sn}/bootloader.bin")
def download_bootloader(
sn: str,
hw_type_override: Optional[str] = Query(None, description="Override hw_type for flash asset lookup (for bespoke firmware)"),
_user: TokenPayload = Depends(require_permission("manufacturing", "view")),
):
"""Return the bootloader.bin for this device's hw_type (flashed at 0x1000)."""
item = service.get_device_by_sn(sn)
hw_type = hw_type_override or item.hw_type
try:
data = service.get_flash_asset(hw_type, "bootloader.bin")
except NotFoundError as e:
raise HTTPException(status_code=404, detail=str(e))
return Response(
content=data,
media_type="application/octet-stream",
headers={"Content-Disposition": f'attachment; filename="bootloader_{hw_type}.bin"'},
)
@router.get("/devices/{sn}/partitions.bin")
def download_partitions(
sn: str,
hw_type_override: Optional[str] = Query(None, description="Override hw_type for flash asset lookup (for bespoke firmware)"),
_user: TokenPayload = Depends(require_permission("manufacturing", "view")),
):
"""Return the partitions.bin for this device's hw_type (flashed at 0x8000)."""
item = service.get_device_by_sn(sn)
hw_type = hw_type_override or item.hw_type
try:
data = service.get_flash_asset(hw_type, "partitions.bin")
except NotFoundError as e:
raise HTTPException(status_code=404, detail=str(e))
return Response(
content=data,
media_type="application/octet-stream",
headers={"Content-Disposition": f'attachment; filename="partitions_{hw_type}.bin"'},
)

260
backend/manufacturing/service.py
View File

@@ -2,11 +2,9 @@ import logging
import random import random
import string import string
from datetime import datetime, timezone from datetime import datetime, timezone
from pathlib import Path
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
from config import settings
from shared.firebase import get_db from shared.firebase import get_db
from shared.exceptions import NotFoundError from shared.exceptions import NotFoundError
from utils.serial_number import generate_serial from utils.serial_number import generate_serial
@@ -33,18 +31,6 @@ def _get_existing_sns(db) -> set:
return existing return existing
def _resolve_user_list(raw_list: list) -> list[str]:
"""Convert user_list entries (DocumentReferences or path strings) to plain user ID strings."""
from google.cloud.firestore_v1 import DocumentReference
result = []
for entry in raw_list:
if isinstance(entry, DocumentReference):
result.append(entry.id)
elif isinstance(entry, str):
result.append(entry.split("/")[-1])
return result
def _doc_to_inventory_item(doc) -> DeviceInventoryItem: def _doc_to_inventory_item(doc) -> DeviceInventoryItem:
data = doc.to_dict() or {} data = doc.to_dict() or {}
created_raw = data.get("created_at") created_raw = data.get("created_at")
@@ -64,9 +50,6 @@ def _doc_to_inventory_item(doc) -> DeviceInventoryItem:
owner=data.get("owner"), owner=data.get("owner"),
assigned_to=data.get("assigned_to"), assigned_to=data.get("assigned_to"),
device_name=data.get("device_name") or None, device_name=data.get("device_name") or None,
lifecycle_history=data.get("lifecycle_history") or [],
customer_id=data.get("customer_id"),
user_list=_resolve_user_list(data.get("user_list") or []),
) )
@@ -95,19 +78,11 @@ def create_batch(data: BatchCreate) -> BatchResponse:
"created_at": now, "created_at": now,
"owner": None, "owner": None,
"assigned_to": None, "assigned_to": None,
"user_list": [], "users_list": [],
# Legacy fields left empty so existing device views don't break # Legacy fields left empty so existing device views don't break
"device_name": "", "device_name": "",
"device_location": "", "device_location": "",
"is_Online": False, "is_Online": False,
"lifecycle_history": [
{
"status_id": "manufactured",
"date": now.isoformat(),
"note": None,
"set_by": None,
}
],
}) })
serial_numbers.append(sn) serial_numbers.append(sn)
@@ -158,38 +133,14 @@ def get_device_by_sn(sn: str) -> DeviceInventoryItem:
return _doc_to_inventory_item(docs[0]) return _doc_to_inventory_item(docs[0])
def update_device_status(sn: str, data: DeviceStatusUpdate, set_by: str | None = None) -> DeviceInventoryItem: def update_device_status(sn: str, data: DeviceStatusUpdate) -> DeviceInventoryItem:
db = get_db() db = get_db()
docs = list(db.collection(COLLECTION).where("serial_number", "==", sn).limit(1).stream()) docs = list(db.collection(COLLECTION).where("serial_number", "==", sn).limit(1).stream())
if not docs: if not docs:
raise NotFoundError("Device") raise NotFoundError("Device")
doc_ref = docs[0].reference doc_ref = docs[0].reference
doc_data = docs[0].to_dict() or {} update = {"mfg_status": data.status.value}
now = datetime.now(timezone.utc).isoformat()
history = list(doc_data.get("lifecycle_history") or [])
# Upsert lifecycle entry — overwrite existing entry for this status if present
new_entry = {
"status_id": data.status.value,
"date": now,
"note": data.note if data.note else None,
"set_by": set_by,
}
existing_idx = next(
(i for i, e in enumerate(history) if e.get("status_id") == data.status.value),
None,
)
if existing_idx is not None:
history[existing_idx] = new_entry
else:
history.append(new_entry)
update = {
"mfg_status": data.status.value,
"lifecycle_history": history,
}
if data.note: if data.note:
update["mfg_status_note"] = data.note update["mfg_status_note"] = data.note
doc_ref.update(update) doc_ref.update(update)
@@ -197,115 +148,47 @@ def update_device_status(sn: str, data: DeviceStatusUpdate, set_by: str | None =
return _doc_to_inventory_item(doc_ref.get()) return _doc_to_inventory_item(doc_ref.get())
def get_nvs_binary(sn: str, hw_type_override: str | None = None, hw_revision_override: str | None = None, legacy: bool = False) -> bytes: def get_nvs_binary(sn: str) -> bytes:
item = get_device_by_sn(sn) item = get_device_by_sn(sn)
return generate_nvs_binary( return generate_nvs_binary(
serial_number=item.serial_number, serial_number=item.serial_number,
hw_family=hw_type_override if hw_type_override else item.hw_type, hw_type=item.hw_type,
hw_revision=hw_revision_override if hw_revision_override else item.hw_version, hw_version=item.hw_version,
legacy=legacy,
) )
def assign_device(sn: str, data: DeviceAssign) -> DeviceInventoryItem: def assign_device(sn: str, data: DeviceAssign) -> DeviceInventoryItem:
"""Assign a device to a customer by customer_id. from utils.email import send_device_assignment_invite
- Stores customer_id on the device doc.
- Adds the device to the customer's owned_items list.
- Sets mfg_status to 'sold' unless device is already 'claimed'.
"""
db = get_db() db = get_db()
CRM_COLLECTION = "crm_customers"
# Get device doc
docs = list(db.collection(COLLECTION).where("serial_number", "==", sn).limit(1).stream()) docs = list(db.collection(COLLECTION).where("serial_number", "==", sn).limit(1).stream())
if not docs: if not docs:
raise NotFoundError("Device") raise NotFoundError("Device")
doc_data = docs[0].to_dict() or {} doc_data = docs[0].to_dict() or {}
doc_ref = docs[0].reference doc_ref = docs[0].reference
current_status = doc_data.get("mfg_status", "manufactured")
# Get customer doc
customer_ref = db.collection(CRM_COLLECTION).document(data.customer_id)
customer_doc = customer_ref.get()
if not customer_doc.exists:
raise NotFoundError("Customer")
customer_data = customer_doc.to_dict() or {}
# Determine new status: don't downgrade claimed → sold
new_status = current_status if current_status == "claimed" else "sold"
now = datetime.now(timezone.utc).isoformat()
history = doc_data.get("lifecycle_history") or []
history.append({
"status_id": new_status,
"date": now,
"note": "Assigned to customer",
"set_by": None,
})
doc_ref.update({ doc_ref.update({
"customer_id": data.customer_id, "owner": data.customer_email,
"mfg_status": new_status, "assigned_to": data.customer_email,
"lifecycle_history": history, "mfg_status": "sold",
}) })
# Add to customer's owned_items (avoid duplicates) hw_type = doc_data.get("hw_type", "")
owned_items = customer_data.get("owned_items", []) or [] device_name = BOARD_TYPE_LABELS.get(hw_type, hw_type or "Device")
device_doc_id = docs[0].id
already_assigned = any( try:
item.get("type") == "console_device" send_device_assignment_invite(
and item.get("console_device", {}).get("device_id") == device_doc_id customer_email=data.customer_email,
for item in owned_items serial_number=sn,
) device_name=device_name,
if not already_assigned: customer_name=data.customer_name,
device_name = doc_data.get("device_name") or BOARD_TYPE_LABELS.get(doc_data.get("hw_type", ""), sn) )
owned_items.append({ except Exception as exc:
"type": "console_device", logger.error("Assignment succeeded but email failed for %s → %s: %s", sn, data.customer_email, exc)
"console_device": {
"device_id": device_doc_id,
"serial_number": sn,
"label": device_name,
},
})
customer_ref.update({"owned_items": owned_items})
return _doc_to_inventory_item(doc_ref.get()) return _doc_to_inventory_item(doc_ref.get())
def search_customers(q: str) -> list:
"""Search crm_customers by name, email, phone, organization, or tags."""
db = get_db()
CRM_COLLECTION = "crm_customers"
docs = db.collection(CRM_COLLECTION).stream()
results = []
q_lower = q.lower().strip()
for doc in docs:
data = doc.to_dict() or {}
loc = data.get("location") or {}
loc = loc if isinstance(loc, dict) else {}
city = loc.get("city") or ""
searchable = " ".join(filter(None, [
data.get("name"), data.get("surname"),
data.get("email"), data.get("phone"), data.get("organization"),
loc.get("address"), loc.get("city"), loc.get("postal_code"),
loc.get("region"), loc.get("country"),
" ".join(data.get("tags") or []),
])).lower()
if not q_lower or q_lower in searchable:
results.append({
"id": doc.id,
"name": data.get("name") or "",
"surname": data.get("surname") or "",
"email": data.get("email") or "",
"organization": data.get("organization") or "",
"phone": data.get("phone") or "",
"city": city or "",
})
return results
def get_stats() -> ManufacturingStats: def get_stats() -> ManufacturingStats:
db = get_db() db = get_db()
docs = list(db.collection(COLLECTION).stream()) docs = list(db.collection(COLLECTION).stream())
@@ -387,105 +270,6 @@ def delete_unprovisioned_devices() -> list[str]:
return deleted return deleted
KNOWN_HW_TYPES = ["vesper", "vesper_plus", "vesper_pro", "agnus", "agnus_mini", "chronos", "chronos_pro"]
FLASH_ASSET_FILES = ["bootloader.bin", "partitions.bin"]
def _flash_asset_path(hw_type: str, asset: str) -> Path:
"""Return path to a flash asset (bootloader.bin or partitions.bin) for a given hw_type."""
return Path(settings.flash_assets_storage_path) / hw_type / asset
def _flash_asset_info(hw_type: str) -> dict:
"""Build the asset info dict for a single hw_type by inspecting the filesystem."""
base = Path(settings.flash_assets_storage_path) / hw_type
note_path = base / "note.txt"
note = note_path.read_text(encoding="utf-8").strip() if note_path.exists() else ""
files = {}
for fname in FLASH_ASSET_FILES:
p = base / fname
if p.exists():
stat = p.stat()
files[fname] = {
"exists": True,
"size_bytes": stat.st_size,
"uploaded_at": datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc).isoformat(),
}
else:
files[fname] = {"exists": False, "size_bytes": None, "uploaded_at": None}
return {
"hw_type": hw_type,
"bootloader": files["bootloader.bin"],
"partitions": files["partitions.bin"],
"note": note,
}
def list_flash_assets() -> list:
"""Return asset status for all known board types plus any discovered bespoke directories."""
base = Path(settings.flash_assets_storage_path)
results = []
# Always include all known hw types, even if no files uploaded yet
seen = set(KNOWN_HW_TYPES)
for hw_type in KNOWN_HW_TYPES:
results.append(_flash_asset_info(hw_type))
# Discover bespoke directories (anything in storage/flash_assets/ not in known list)
if base.exists():
for entry in sorted(base.iterdir()):
if entry.is_dir() and entry.name not in seen:
seen.add(entry.name)
info = _flash_asset_info(entry.name)
info["is_bespoke"] = True
results.append(info)
# Mark known types
for r in results:
r.setdefault("is_bespoke", False)
return results
def save_flash_asset(hw_type: str, asset: str, data: bytes) -> Path:
"""Persist a flash asset binary. asset must be 'bootloader.bin' or 'partitions.bin'."""
if asset not in ("bootloader.bin", "partitions.bin"):
raise ValueError(f"Unknown flash asset: {asset}")
path = _flash_asset_path(hw_type, asset)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(data)
return path
def delete_flash_asset(hw_type: str, asset: str) -> None:
"""Delete a flash asset file. Raises NotFoundError if not present."""
path = _flash_asset_path(hw_type, asset)
if not path.exists():
raise NotFoundError(f"Flash asset '{asset}' for '{hw_type}' not found")
path.unlink()
def set_flash_asset_note(hw_type: str, note: str) -> None:
"""Write (or clear) the note for a hw_type's flash asset directory."""
base = Path(settings.flash_assets_storage_path) / hw_type
base.mkdir(parents=True, exist_ok=True)
note_path = base / "note.txt"
if note.strip():
note_path.write_text(note.strip(), encoding="utf-8")
elif note_path.exists():
note_path.unlink()
def get_flash_asset(hw_type: str, asset: str) -> bytes:
"""Load a flash asset binary. Raises NotFoundError if not uploaded yet."""
path = _flash_asset_path(hw_type, asset)
if not path.exists():
raise NotFoundError(f"Flash asset '{asset}' for hw_type '{hw_type}' — upload it first via POST /api/manufacturing/flash-assets/{{hw_type}}/{{asset}}")
return path.read_bytes()
def get_firmware_url(sn: str) -> str: def get_firmware_url(sn: str) -> str:
"""Return the FastAPI download URL for the latest stable firmware for this device's hw_type.""" """Return the FastAPI download URL for the latest stable firmware for this device's hw_type."""
from firmware.service import get_latest from firmware.service import get_latest

2
backend/melodies/database.py
View File

@@ -1,6 +1,6 @@
import json import json
import logging import logging
from database import get_db from mqtt.database import get_db
logger = logging.getLogger("melodies.database") logger = logging.getLogger("melodies.database")

1
backend/melodies/models.py
View File

@@ -30,7 +30,6 @@ class MelodyInfo(BaseModel):
isTrueRing: bool = False isTrueRing: bool = False
previewURL: str = "" previewURL: str = ""
archetype_csv: Optional[str] = None archetype_csv: Optional[str] = None
outdated_archetype: bool = False
class MelodyAttributes(BaseModel): class MelodyAttributes(BaseModel):

39
backend/melodies/orm.py
View File

@@ -1,39 +0,0 @@
from datetime import datetime, timezone
from sqlalchemy import Boolean, Column, DateTime, Index, String, Text
from sqlalchemy.dialects.postgresql import JSONB
from database.postgres import Base
def _now():
return datetime.now(timezone.utc)
class MelodyDraft(Base):
__tablename__ = "melody_drafts"
__table_args__ = (
Index("idx_melody_drafts_status", "status"),
)
id = Column(String(128), primary_key=True)
status = Column(String(32), nullable=False, default="draft")
# 'data' stores the full melody definition as JSON (was TEXT/JSON in SQLite)
data = Column(JSONB, nullable=False)
created_at = Column(DateTime(timezone=True), nullable=False, default=_now)
updated_at = Column(DateTime(timezone=True), nullable=False, default=_now, onupdate=_now)
class BuiltMelody(Base):
__tablename__ = "built_melodies"
id = Column(String(128), primary_key=True)
name = Column(String(500), nullable=False)
pid = Column(String(128), nullable=False)
# 'steps' is a JSON array of step definitions
steps = Column(JSONB, nullable=False)
binary_path = Column(String(1000))
progmem_code = Column(Text)
# JSON array of melody IDs this built melody is assigned to
assigned_melody_ids = Column(JSONB, nullable=False, default=list)
is_builtin = Column(Boolean, nullable=False, default=False)
created_at = Column(DateTime(timezone=True), nullable=False, default=_now)
updated_at = Column(DateTime(timezone=True), nullable=False, default=_now, onupdate=_now)

61
backend/melodies/router.py
View File

@@ -1,14 +1,11 @@
from fastapi import APIRouter, Depends, UploadFile, File, Query, HTTPException, Response from fastapi import APIRouter, Depends, UploadFile, File, Query, HTTPException, Response
from typing import Optional from typing import Optional
from sqlalchemy.ext.asyncio import AsyncSession
from auth.models import TokenPayload from auth.models import TokenPayload
from auth.dependencies import require_permission from auth.dependencies import require_permission
from melodies.models import ( from melodies.models import (
MelodyCreate, MelodyUpdate, MelodyInDB, MelodyListResponse, MelodyInfo, MelodyCreate, MelodyUpdate, MelodyInDB, MelodyListResponse, MelodyInfo,
) )
from melodies import service from melodies import service
from database.postgres import get_pg_session
from shared.audit import log_action
router = APIRouter(prefix="/api/melodies", tags=["melodies"]) router = APIRouter(prefix="/api/melodies", tags=["melodies"])
@@ -45,12 +42,8 @@ async def create_melody(
body: MelodyCreate, body: MelodyCreate,
publish: bool = Query(False), publish: bool = Query(False),
_user: TokenPayload = Depends(require_permission("melodies", "add")), _user: TokenPayload = Depends(require_permission("melodies", "add")),
db: AsyncSession = Depends(get_pg_session),
): ):
melody = await service.create_melody(body, publish=publish, actor_name=_user.name) return await service.create_melody(body, publish=publish, actor_name=_user.name)
await log_action(db, _user.sub, _user.name or _user.email, "CREATE", "melody",
melody.id, melody.information.name if melody.information else melody.id)
return melody
@router.put("/{melody_id}", response_model=MelodyInDB) @router.put("/{melody_id}", response_model=MelodyInDB)
@@ -58,61 +51,32 @@ async def update_melody(
melody_id: str, melody_id: str,
body: MelodyUpdate, body: MelodyUpdate,
_user: TokenPayload = Depends(require_permission("melodies", "edit")), _user: TokenPayload = Depends(require_permission("melodies", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
old = await service.get_melody(melody_id) return await service.update_melody(melody_id, body, actor_name=_user.name)
melody = await service.update_melody(melody_id, body, actor_name=_user.name)
_SKIP = {"updated_at", "id", "metadata", "information", "noteAssignments"}
changes = {
k: {"old": getattr(old, k, None), "new": getattr(melody, k, None)}
for k in body.model_fields_set
if k not in _SKIP and getattr(old, k, None) != getattr(melody, k, None)
}
# Surface the name change from inside the information sub-object
old_name = old.information.name if old.information else None
new_name = melody.information.name if melody.information else None
if old_name != new_name:
changes["name"] = {"old": old_name, "new": new_name}
await log_action(db, _user.sub, _user.name or _user.email, "UPDATE", "melody",
melody_id, new_name or melody_id, changes=changes or None)
return melody
@router.delete("/{melody_id}", status_code=204) @router.delete("/{melody_id}", status_code=204)
async def delete_melody( async def delete_melody(
melody_id: str, melody_id: str,
_user: TokenPayload = Depends(require_permission("melodies", "delete")), _user: TokenPayload = Depends(require_permission("melodies", "delete")),
db: AsyncSession = Depends(get_pg_session),
): ):
melody = await service.get_melody(melody_id)
label = melody.information.name if melody.information else melody_id
await service.delete_melody(melody_id) await service.delete_melody(melody_id)
await log_action(db, _user.sub, _user.name or _user.email, "DELETE", "melody",
melody_id, label)
@router.post("/{melody_id}/publish", response_model=MelodyInDB) @router.post("/{melody_id}/publish", response_model=MelodyInDB)
async def publish_melody( async def publish_melody(
melody_id: str, melody_id: str,
_user: TokenPayload = Depends(require_permission("melodies", "edit")), _user: TokenPayload = Depends(require_permission("melodies", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
melody = await service.publish_melody(melody_id) return await service.publish_melody(melody_id)
await log_action(db, _user.sub, _user.name or _user.email, "PUBLISH", "melody",
melody_id, melody.information.name if melody.information else melody_id)
return melody
@router.post("/{melody_id}/unpublish", response_model=MelodyInDB) @router.post("/{melody_id}/unpublish", response_model=MelodyInDB)
async def unpublish_melody( async def unpublish_melody(
melody_id: str, melody_id: str,
_user: TokenPayload = Depends(require_permission("melodies", "edit")), _user: TokenPayload = Depends(require_permission("melodies", "edit")),
db: AsyncSession = Depends(get_pg_session),
): ):
melody = await service.unpublish_melody(melody_id) return await service.unpublish_melody(melody_id)
await log_action(db, _user.sub, _user.name or _user.email, "UNPUBLISH", "melody",
melody_id, melody.information.name if melody.information else melody_id)
return melody
@router.post("/{melody_id}/upload/{file_type}") @router.post("/{melody_id}/upload/{file_type}")
@@ -182,23 +146,6 @@ async def get_files(
return service.get_storage_files(melody_id, melody.uid) return service.get_storage_files(melody_id, melody.uid)
@router.patch("/{melody_id}/set-outdated", response_model=MelodyInDB)
async def set_outdated(
melody_id: str,
outdated: bool = Query(...),
_user: TokenPayload = Depends(require_permission("melodies", "edit")),
):
"""Manually set or clear the outdated_archetype flag on a melody."""
melody = await service.get_melody(melody_id)
info = melody.information.model_dump()
info["outdated_archetype"] = outdated
return await service.update_melody(
melody_id,
MelodyUpdate(information=MelodyInfo(**info)),
actor_name=_user.name,
)
@router.get("/{melody_id}/download/binary") @router.get("/{melody_id}/download/binary")
async def download_binary_file( async def download_binary_file(
melody_id: str, melody_id: str,

178
backend/migrate_customer_flags.py
View File

@@ -1,178 +0,0 @@
"""
One-time migration script: convert legacy negotiating/has_problem flags to new structure.
Run AFTER deploying the new backend code:
cd backend && python migrate_customer_flags.py
What it does:
1. For each customer with negotiating=True:
- Creates an order subcollection document with status="negotiating"
- Sets relationship_status="active" (only if currently "lead" or "prospect")
2. For each customer with has_problem=True:
- Appends one entry to technical_issues with active=True
3. Removes negotiating and has_problem fields from every customer document
4. Initialises relationship_status="lead" on any customer missing it
5. Recomputes crm_summary for each affected customer
"""
import sys
import os
import uuid
from datetime import datetime
# Make sure we can import backend modules
sys.path.insert(0, os.path.dirname(__file__))
from shared.firebase import init_firebase, get_db
init_firebase()
def migrate():
db = get_db()
customers_ref = db.collection("crm_customers")
docs = list(customers_ref.stream())
print(f"Found {len(docs)} customer documents.")
migrated_neg = 0
migrated_prob = 0
now = datetime.utcnow().isoformat()
for doc in docs:
data = doc.to_dict() or {}
customer_id = doc.id
updates = {}
changed = False
# ── 1. Initialise new fields if missing ──────────────────────────────
if "relationship_status" not in data:
updates["relationship_status"] = "lead"
changed = True
if "technical_issues" not in data:
updates["technical_issues"] = []
changed = True
if "install_support" not in data:
updates["install_support"] = []
changed = True
if "transaction_history" not in data:
updates["transaction_history"] = []
changed = True
# ── 2. Migrate negotiating flag ───────────────────────────────────────
if data.get("negotiating"):
order_id = str(uuid.uuid4())
order_data = {
"customer_id": customer_id,
"order_number": f"ORD-{datetime.utcnow().year}-001-migrated",
"title": "Migrated from legacy negotiating flag",
"created_by": "system",
"status": "negotiating",
"status_updated_date": now,
"status_updated_by": "system",
"items": [],
"subtotal": 0,
"discount": None,
"total_price": 0,
"currency": "EUR",
"shipping": None,
"payment_status": {
"required_amount": 0,
"received_amount": 0,
"balance_due": 0,
"advance_required": False,
"advance_amount": None,
"payment_complete": False,
},
"invoice_path": None,
"notes": "Migrated from legacy negotiating flag",
"timeline": [{
"date": now,
"type": "note",
"note": "Migrated from legacy negotiating flag",
"updated_by": "system",
}],
"created_at": now,
"updated_at": now,
}
customers_ref.document(customer_id).collection("orders").document(order_id).set(order_data)
current_rel = updates.get("relationship_status") or data.get("relationship_status", "lead")
if current_rel in ("lead", "prospect"):
updates["relationship_status"] = "active"
migrated_neg += 1
print(f" [{customer_id}] Created negotiating order, set relationship_status=active")
# ── 3. Migrate has_problem flag ───────────────────────────────────────
if data.get("has_problem"):
existing_issues = list(updates.get("technical_issues") or data.get("technical_issues") or [])
existing_issues.append({
"active": True,
"opened_date": data.get("updated_at") or now,
"resolved_date": None,
"note": "Migrated from legacy has_problem flag",
"opened_by": "system",
"resolved_by": None,
})
updates["technical_issues"] = existing_issues
migrated_prob += 1
changed = True
print(f" [{customer_id}] Appended technical issue from has_problem flag")
# ── 4. Remove legacy fields ───────────────────────────────────────────
from google.cloud.firestore_v1 import DELETE_FIELD
if "negotiating" in data:
updates["negotiating"] = DELETE_FIELD
changed = True
if "has_problem" in data:
updates["has_problem"] = DELETE_FIELD
changed = True
if changed or data.get("negotiating") or data.get("has_problem"):
updates["updated_at"] = now
customers_ref.document(customer_id).update(updates)
# ── 5. Recompute crm_summary ──────────────────────────────────────────
# Re-read updated doc to compute summary
updated_doc = customers_ref.document(customer_id).get()
updated_data = updated_doc.to_dict() or {}
issues = updated_data.get("technical_issues") or []
active_issues = [i for i in issues if i.get("active")]
support = updated_data.get("install_support") or []
active_support = [s for s in support if s.get("active")]
TERMINAL = {"declined", "complete"}
active_order_status = None
active_order_status_date = None
active_order_title = None
latest_date = ""
for odoc in customers_ref.document(customer_id).collection("orders").stream():
odata = odoc.to_dict() or {}
if odata.get("status") not in TERMINAL:
upd = odata.get("status_updated_date") or odata.get("created_at") or ""
if upd > latest_date:
latest_date = upd
active_order_status = odata.get("status")
active_order_status_date = upd
active_order_title = odata.get("title")
summary = {
"active_order_status": active_order_status,
"active_order_status_date": active_order_status_date,
"active_order_title": active_order_title,
"active_issues_count": len(active_issues),
"latest_issue_date": max((i.get("opened_date") or "") for i in active_issues) if active_issues else None,
"active_support_count": len(active_support),
"latest_support_date": max((s.get("opened_date") or "") for s in active_support) if active_support else None,
}
customers_ref.document(customer_id).update({"crm_summary": summary})
print(f"\nMigration complete.")
print(f" Negotiating orders created: {migrated_neg}")
print(f" Technical issues created: {migrated_prob}")
print(f" Total customers processed: {len(docs)}")
if __name__ == "__main__":
migrate()

0
backend/migration/__init__.py
View File

65
backend/migration/migrate_built_melodies.py
View File

@@ -1,65 +0,0 @@
"""
Phase 1 — Step 1.2: built_melodies (SQLite → Postgres)
Run on VPS:
docker compose exec backend python -m migration.migrate_built_melodies
"""
import asyncio
import sys
from sqlalchemy.dialects.postgresql import insert as pg_insert
from melodies.orm import BuiltMelody
from migration.utils import open_sqlite, AsyncPgSession, parse_dt, parse_json, log_run, pg_count
SCRIPT = "migrate_built_melodies"
async def run() -> None:
sqlite = await open_sqlite()
rows = await sqlite.execute_fetchall("SELECT * FROM built_melodies")
await sqlite.close()
source_count = len(rows)
print(f"Source (SQLite): {source_count} built_melodies rows")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = []
for r in rows:
records.append({
"id": r["id"],
"name": r["name"],
"pid": r["pid"],
"steps": parse_json(r["steps"], default=[]),
"binary_path": r["binary_path"],
"progmem_code": r["progmem_code"],
"assigned_melody_ids": parse_json(r["assigned_melody_ids"], default=[]),
"is_builtin": bool(r["is_builtin"]) if r["is_builtin"] is not None else False,
"created_at": parse_dt(r["created_at"]),
"updated_at": parse_dt(r["updated_at"]),
})
async with AsyncPgSession() as session:
async with session.begin():
stmt = pg_insert(BuiltMelody).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["id"])
await session.execute(stmt)
dest_count = await pg_count(session, "built_melodies")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

73
backend/migration/migrate_commands.py
View File

@@ -1,73 +0,0 @@
"""
Phase 1 — Step 1.10: commands (SQLite → Postgres)
commands is a raw-SQL table (no ORM model). BIGSERIAL PK — SQLite integer IDs
are NOT preserved; rows are inserted in sent_at order.
Run on VPS:
docker compose exec backend python -m migration.migrate_commands
"""
import asyncio
import sys
from sqlalchemy import text
from migration.utils import open_sqlite, AsyncPgSession, parse_dt, log_run, pg_count
SCRIPT = "migrate_commands"
async def run() -> None:
sqlite = await open_sqlite()
rows = await sqlite.execute_fetchall("SELECT * FROM commands ORDER BY sent_at")
await sqlite.close()
source_count = len(rows)
print(f"Source (SQLite): {source_count} commands rows")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = [
{
"device_serial": r["device_serial"],
"command_name": r["command_name"],
"command_payload": r["command_payload"],
"status": r["status"] or "pending",
"response_payload": r["response_payload"],
"sent_at": parse_dt(r["sent_at"]),
"responded_at": parse_dt(r["responded_at"]),
}
for r in rows
]
async with AsyncPgSession() as session:
async with session.begin():
await session.execute(
text("""
INSERT INTO commands
(device_serial, command_name, command_payload, status,
response_payload, sent_at, responded_at)
VALUES
(:device_serial, :command_name, :command_payload, :status,
:response_payload, :sent_at, :responded_at)
"""),
records,
)
dest_count = await pg_count(session, "commands")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

84
backend/migration/migrate_crm_comms_log.py
View File

@@ -1,84 +0,0 @@
"""
Phase 1 — Step 1.9: crm_comms_log (SQLite → Postgres)
FK to crm_customers(id) (nullable, ON DELETE SET NULL) — FK enforcement
suppressed until Phase 2 populates crm_customers.
Run on VPS:
docker compose exec backend python -m migration.migrate_crm_comms_log
"""
import asyncio
import sys
from sqlalchemy import text
from sqlalchemy.dialects.postgresql import insert as pg_insert
from crm.orm import CrmCommsLog
from migration.utils import open_sqlite, AsyncPgSession, parse_dt, parse_json, log_run, pg_count
SCRIPT = "migrate_crm_comms_log"
async def run() -> None:
sqlite = await open_sqlite()
rows = await sqlite.execute_fetchall("SELECT * FROM crm_comms_log ORDER BY occurred_at")
await sqlite.close()
source_count = len(rows)
print(f"Source (SQLite): {source_count} crm_comms_log rows")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = []
for r in rows:
# attachments stored as JSON text in SQLite
attachments = parse_json(r["attachments"], default=[])
# is_important / is_read stored as INTEGER (0/1) in SQLite
is_important = bool(r["is_important"]) if r["is_important"] is not None else False
is_read = bool(r["is_read"]) if r["is_read"] is not None else True
records.append({
"id": r["id"],
"customer_id": r["customer_id"],
"type": r["type"],
"mail_account": r["mail_account"],
"direction": r["direction"],
"subject": r["subject"],
"body": r["body"],
"body_html": r["body_html"],
"attachments": attachments,
"ext_message_id": r["ext_message_id"],
"from_addr": r["from_addr"],
"to_addrs": r["to_addrs"],
"logged_by": r["logged_by"],
"is_important": is_important,
"is_read": is_read,
"occurred_at": parse_dt(r["occurred_at"]),
"created_at": parse_dt(r["created_at"]),
})
async with AsyncPgSession() as session:
async with session.begin():
await session.execute(text("SET LOCAL session_replication_role = replica"))
stmt = pg_insert(CrmCommsLog).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["id"])
await session.execute(stmt)
dest_count = await pg_count(session, "crm_comms_log")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

172
backend/migration/migrate_crm_customers.py
View File

@@ -1,172 +0,0 @@
"""
Phase 2 — Step 2.4: crm_customers (Firestore → Postgres)
Reads the 'crm_customers' Firestore collection.
- Strips legacy fields: 'negotiating', 'has_problem'
- Converts Firestore DatetimeWithNanoseconds → UTC datetime
- Converts nested dicts/lists → JSONB-ready Python objects
After this runs, the FK constraints on crm_quotations, crm_comms_log,
crm_media, and crm_orders (all inserted in Phase 1 with FK enforcement
suppressed) become valid.
Run on VPS:
docker compose exec backend python -m migration.migrate_crm_customers
"""
import asyncio
import sys
from datetime import datetime, timezone
from sqlalchemy.dialects.postgresql import insert as pg_insert
from crm.orm import CrmCustomer
from shared.firebase import init_firebase, get_db as get_firestore
from migration.utils import AsyncPgSession, parse_dt, log_run, pg_count
SCRIPT = "migrate_crm_customers"
COLLECTION = "crm_customers"
_LEGACY_FIELDS = {"negotiating", "has_problem"}
_VALID_STATUSES = {
"lead", "active", "inactive", "archived",
"prospect", "churned", "vip",
}
def _now_utc() -> datetime:
return datetime.now(timezone.utc)
def _coerce_dt(val) -> datetime | None:
"""Handle both Firestore DatetimeWithNanoseconds and ISO strings."""
if val is None:
return None
if isinstance(val, datetime):
return val.replace(tzinfo=timezone.utc) if val.tzinfo is None else val
return parse_dt(str(val))
def _coerce_list(val, default=None) -> list:
if isinstance(val, list):
return val
return default if default is not None else []
async def run() -> None:
init_firebase()
fs = get_firestore()
if fs is None:
print("ERROR: Firebase not initialised.", file=sys.stderr)
sys.exit(1)
docs = list(fs.collection(COLLECTION).stream())
source_count = len(docs)
print(f"Source (Firestore): {source_count} crm_customers documents")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = []
skipped = 0
for doc in docs:
d = doc.to_dict()
# Strip legacy fields
for f in _LEGACY_FIELDS:
d.pop(f, None)
# folder_id is NOT NULL UNIQUE — skip docs missing it
folder_id = d.get("folder_id") or ""
if not folder_id:
print(f" WARNING: customer {doc.id} has no folder_id — skipping", file=sys.stderr)
skipped += 1
continue
# relationship_status — normalise unknown values to 'lead'
rel_status = d.get("relationship_status") or "lead"
if rel_status not in _VALID_STATUSES:
rel_status = "lead"
# contacts / notes — Firestore stores as list of maps
contacts = _coerce_list(d.get("contacts"))
# Serialise nested Pydantic-style objects to plain dicts
contacts = [c if isinstance(c, dict) else vars(c) for c in contacts]
notes = _coerce_list(d.get("notes"))
notes = [n if isinstance(n, dict) else vars(n) for n in notes]
# location — may be a map or None
location = d.get("location")
if location and not isinstance(location, dict):
location = vars(location)
tags = _coerce_list(d.get("tags"))
owned_items = _coerce_list(d.get("owned_items"))
owned_items = [o if isinstance(o, dict) else vars(o) for o in owned_items]
linked_user_ids = _coerce_list(d.get("linked_user_ids"))
technical_issues = _coerce_list(d.get("technical_issues"))
install_support = _coerce_list(d.get("install_support"))
transaction_history = _coerce_list(d.get("transaction_history"))
crm_summary = d.get("crm_summary")
if crm_summary and not isinstance(crm_summary, dict):
crm_summary = vars(crm_summary)
created_at = _coerce_dt(d.get("created_at")) or _now_utc()
updated_at = _coerce_dt(d.get("updated_at")) or _now_utc()
records.append({
"id": doc.id,
"firestore_id": doc.id,
"title": d.get("title"),
"name": d.get("name") or "",
"surname": d.get("surname"),
"organization": d.get("organization"),
"religion": d.get("religion"),
"language": d.get("language") or "el",
"folder_id": folder_id,
"relationship_status": rel_status,
"nextcloud_folder": d.get("nextcloud_folder"),
"contacts": contacts,
"notes": notes,
"location": location,
"tags": tags,
"owned_items": owned_items,
"linked_user_ids": linked_user_ids,
"technical_issues": technical_issues,
"install_support": install_support,
"transaction_history": transaction_history,
"crm_summary": crm_summary,
"created_at": created_at,
"updated_at": updated_at,
})
actual_source = source_count - skipped
print(f" {skipped} skipped (missing folder_id), {actual_source} to insert")
async with AsyncPgSession() as session:
async with session.begin():
stmt = pg_insert(CrmCustomer).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["id"])
await session.execute(stmt)
dest_count = await pg_count(session, "crm_customers")
if dest_count < actual_source:
msg = f"Count mismatch: expected>={actual_source} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count,
notes=f"{skipped} skipped (no folder_id)" if skipped else None)
if __name__ == "__main__":
asyncio.run(run())

75
backend/migration/migrate_crm_media.py
View File

@@ -1,75 +0,0 @@
"""
Phase 1 — Step 1.8: crm_media (SQLite → Postgres)
FK to crm_customers(id) (nullable) — FK enforcement suppressed until Phase 2
populates crm_customers.
Run on VPS:
docker compose exec backend python -m migration.migrate_crm_media
"""
import asyncio
import sys
from sqlalchemy import text
from sqlalchemy.dialects.postgresql import insert as pg_insert
from crm.orm import CrmMedia
from migration.utils import open_sqlite, AsyncPgSession, parse_dt, parse_json, log_run, pg_count
SCRIPT = "migrate_crm_media"
async def run() -> None:
sqlite = await open_sqlite()
rows = await sqlite.execute_fetchall("SELECT * FROM crm_media ORDER BY created_at")
await sqlite.close()
source_count = len(rows)
print(f"Source (SQLite): {source_count} crm_media rows")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = []
for r in rows:
# SQLite stores tags as JSON text; Postgres column is JSONB
tags_raw = r["tags"]
tags = parse_json(tags_raw, default=[])
records.append({
"id": r["id"],
"customer_id": r["customer_id"],
"order_id": r["order_id"],
"filename": r["filename"],
"nextcloud_path": r["nextcloud_path"],
"thumbnail_path": r["thumbnail_path"],
"mime_type": r["mime_type"],
"direction": r["direction"],
"tags": tags,
"uploaded_by": r["uploaded_by"],
"created_at": parse_dt(r["created_at"]),
})
async with AsyncPgSession() as session:
async with session.begin():
await session.execute(text("SET LOCAL session_replication_role = replica"))
stmt = pg_insert(CrmMedia).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["id"])
await session.execute(stmt)
dest_count = await pg_count(session, "crm_media")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

156
backend/migration/migrate_crm_orders.py
View File

@@ -1,156 +0,0 @@
"""
Phase 2 — Step 2.5: crm_orders (Firestore → Postgres)
Orders are stored as a subcollection under each customer:
crm_customers/{customer_id}/orders/{order_id}
Uses collection_group("orders") to fetch all orders in one pass,
then inserts into crm_orders. crm_customers MUST already be in Postgres
(step 2.4) so the FK constraint is satisfied.
Run on VPS:
docker compose exec backend python -m migration.migrate_crm_orders
"""
import asyncio
import sys
from datetime import datetime, timezone
from sqlalchemy.dialects.postgresql import insert as pg_insert
from crm.orm import CrmOrder
from shared.firebase import init_firebase, get_db as get_firestore
from migration.utils import AsyncPgSession, parse_dt, log_run, pg_count
SCRIPT = "migrate_crm_orders"
def _now_utc() -> datetime:
return datetime.now(timezone.utc)
def _coerce_dt(val) -> datetime | None:
if val is None:
return None
if isinstance(val, datetime):
return val.replace(tzinfo=timezone.utc) if val.tzinfo is None else val
return parse_dt(str(val))
def _coerce_list(val) -> list:
return val if isinstance(val, list) else []
def _coerce_dict(val) -> dict:
return val if isinstance(val, dict) else {}
async def run() -> None:
init_firebase()
fs = get_firestore()
if fs is None:
print("ERROR: Firebase not initialised.", file=sys.stderr)
sys.exit(1)
# collection_group fetches from ALL customers' 'orders' subcollections
docs = list(fs.collection_group("orders").stream())
source_count = len(docs)
print(f"Source (Firestore): {source_count} order documents (via collection_group)")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
# First, collect all customer IDs already in Postgres so we can skip
# orphaned orders whose customer didn't migrate (missing folder_id edge case)
async with AsyncPgSession() as session:
from sqlalchemy import text
result = await session.execute(text("SELECT id FROM crm_customers"))
valid_customer_ids = {row[0] for row in result.fetchall()}
records = []
skipped = 0
seen_order_numbers: set[str] = set()
for doc in docs:
d = doc.to_dict()
# Extract customer_id from the document path:
# crm_customers/{customer_id}/orders/{order_id}
path_parts = doc.reference.path.split("/")
# path: crm_customers / <cid> / orders / <oid>
try:
customer_id = path_parts[1]
except IndexError:
print(f" WARNING: cannot parse customer_id from path {doc.reference.path} — skipping")
skipped += 1
continue
if customer_id not in valid_customer_ids:
print(f" WARNING: order {doc.id} references unknown customer {customer_id} — skipping")
skipped += 1
continue
order_number = d.get("order_number") or f"ORD-LEGACY-{doc.id}"
# Deduplicate: if this order_number was already seen in this batch,
# make it unique by appending the doc ID suffix.
if order_number in seen_order_numbers:
order_number = f"{order_number}-{doc.id[:8]}"
print(f" INFO: duplicate order_number — renamed to {order_number}")
seen_order_numbers.add(order_number)
created_at = _coerce_dt(d.get("created_at")) or _now_utc()
updated_at = _coerce_dt(d.get("updated_at")) or _now_utc()
status_updated_date = _coerce_dt(d.get("status_updated_date"))
records.append({
"id": doc.id,
"customer_id": customer_id,
"order_number": order_number,
"title": d.get("title"),
"created_by": d.get("created_by"),
"status": d.get("status") or "negotiating",
"status_updated_date": status_updated_date,
"status_updated_by": d.get("status_updated_by"),
"items": _coerce_list(d.get("items")),
"subtotal": float(d.get("subtotal") or 0),
"discount": d.get("discount") if isinstance(d.get("discount"), dict) else None,
"total_price": float(d.get("total_price") or 0),
"currency": d.get("currency") or "EUR",
"shipping": d.get("shipping") if isinstance(d.get("shipping"), dict) else None,
"payment_status": _coerce_dict(d.get("payment_status")),
"invoice_path": d.get("invoice_path"),
"notes": d.get("notes") if isinstance(d.get("notes"), str) else None,
"timeline": _coerce_list(d.get("timeline")),
"created_at": created_at,
"updated_at": updated_at,
})
actual_source = source_count - skipped
print(f" {skipped} skipped (orphaned/bad path), {actual_source} to insert")
if not records:
print("Nothing valid to insert.")
await log_run(SCRIPT, source_count, 0, notes=f"{skipped} all skipped")
return
async with AsyncPgSession() as session:
async with session.begin():
stmt = pg_insert(CrmOrder).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["id"])
await session.execute(stmt)
dest_count = await pg_count(session, "crm_orders")
if dest_count < actual_source:
msg = f"Count mismatch: expected>={actual_source} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count,
notes=f"{skipped} skipped" if skipped else None)
if __name__ == "__main__":
asyncio.run(run())

102
backend/migration/migrate_crm_products.py
View File

@@ -1,102 +0,0 @@
"""
Phase 2 — Step 2.3: crm_products (Firestore → Postgres)
Reads the 'crm_products' Firestore collection. The Firestore schema is richer
than the Postgres target (has costs, stock, name_en, etc.) — we extract only
what the Postgres ORM model covers. The rest stays in Firestore until the
service is fully cut over.
Run on VPS:
docker compose exec backend python -m migration.migrate_crm_products
"""
import asyncio
import sys
from datetime import datetime, timezone
from sqlalchemy.dialects.postgresql import insert as pg_insert
from crm.orm import CrmProduct
from shared.firebase import init_firebase, get_db as get_firestore
from migration.utils import AsyncPgSession, parse_dt, log_run, pg_count
SCRIPT = "migrate_crm_products"
COLLECTION = "crm_products"
_LEGACY_STATUS_MAP = {
"active": True,
"discontinued": False,
"planned": True,
}
def _now_utc() -> datetime:
return datetime.now(timezone.utc)
async def run() -> None:
init_firebase()
fs = get_firestore()
if fs is None:
print("ERROR: Firebase not initialised.", file=sys.stderr)
sys.exit(1)
docs = list(fs.collection(COLLECTION).stream())
source_count = len(docs)
print(f"Source (Firestore): {source_count} crm_products documents")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = []
for doc in docs:
d = doc.to_dict()
# is_active: prefer 'active' bool field, fall back to 'status' string
if "active" in d:
is_active = bool(d["active"])
else:
is_active = _LEGACY_STATUS_MAP.get(d.get("status", "active"), True)
# unit_cost: Firestore uses 'price'
unit_cost = d.get("unit_cost") or d.get("price") or 0
created_at = parse_dt(d.get("created_at")) or _now_utc()
updated_at = parse_dt(d.get("updated_at")) or _now_utc()
records.append({
"id": doc.id,
"firestore_id": doc.id,
"name": d.get("name") or d.get("name_en") or "",
"sku": d.get("sku"),
"category": d.get("category"),
"description": d.get("description") or d.get("description_en"),
"unit_cost": unit_cost,
"currency": d.get("currency") or "EUR",
"unit_type": d.get("unit_type") or "pcs",
"is_active": is_active,
"created_at": created_at,
"updated_at": updated_at,
})
async with AsyncPgSession() as session:
async with session.begin():
stmt = pg_insert(CrmProduct).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["id"])
await session.execute(stmt)
dest_count = await pg_count(session, "crm_products")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

84
backend/migration/migrate_crm_quotation_items.py
View File

@@ -1,84 +0,0 @@
"""
Phase 1 — Step 1.7: crm_quotation_items (SQLite → Postgres)
FK to crm_quotations(id) — quotations must be migrated first (step 1.6).
FK enforcement suppressed via session_replication_role for the same reason
as in migrate_crm_quotations (parent crm_customers not yet in PG).
Run on VPS:
docker compose exec backend python -m migration.migrate_crm_quotation_items
"""
import asyncio
import sys
from decimal import Decimal
from sqlalchemy import text
from sqlalchemy.dialects.postgresql import insert as pg_insert
from crm.orm import CrmQuotationItem
from migration.utils import open_sqlite, AsyncPgSession, log_run, pg_count
SCRIPT = "migrate_crm_quotation_items"
def _dec(val, default="0") -> Decimal:
try:
return Decimal(str(val)) if val is not None else Decimal(default)
except Exception:
return Decimal(default)
async def run() -> None:
sqlite = await open_sqlite()
rows = await sqlite.execute_fetchall(
"SELECT * FROM crm_quotation_items ORDER BY quotation_id, sort_order"
)
await sqlite.close()
source_count = len(rows)
print(f"Source (SQLite): {source_count} crm_quotation_items rows")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = []
for r in rows:
records.append({
"id": r["id"],
"quotation_id": r["quotation_id"],
"product_id": r["product_id"],
"description": r["description"],
"description_en": r["description_en"],
"description_gr": r["description_gr"],
"unit_type": r["unit_type"] or "pcs",
"unit_cost": _dec(r["unit_cost"]),
"discount_percent": _dec(r["discount_percent"]),
"vat_percent": _dec(r["vat_percent"], "24"),
"quantity": _dec(r["quantity"], "1"),
"line_total": _dec(r["line_total"]),
"sort_order": int(r["sort_order"]) if r["sort_order"] is not None else 0,
})
async with AsyncPgSession() as session:
async with session.begin():
await session.execute(text("SET LOCAL session_replication_role = replica"))
stmt = pg_insert(CrmQuotationItem).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["id"])
await session.execute(stmt)
dest_count = await pg_count(session, "crm_quotation_items")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

118
backend/migration/migrate_crm_quotations.py
View File

@@ -1,118 +0,0 @@
"""
Phase 1 — Step 1.6: crm_quotations (SQLite → Postgres)
NOTE: crm_quotations has a FK to crm_customers(id).
The customer rows DO NOT exist in Postgres yet (they migrate in Phase 2).
To avoid FK violations, this script temporarily disables FK checks for the
session using SET CONSTRAINTS ALL DEFERRED — but since customer_id is a real
FK with ON DELETE CASCADE, we instead insert with the constraint deferred.
Safer approach used here: insert with `customer_id` as-is and rely on the
fact that crm_customers will be populated in Phase 2 before any service
reads join across the two tables. The FK is not deferred — instead we disable
the FK constraint enforcement for this transaction only via a session-level
SET session_replication_role = replica; which suppresses FK checks in Postgres.
We restore it immediately after the transaction.
Run on VPS:
docker compose exec backend python -m migration.migrate_crm_quotations
"""
import asyncio
import sys
from decimal import Decimal
from sqlalchemy import text
from sqlalchemy.dialects.postgresql import insert as pg_insert
from crm.orm import CrmQuotation
from migration.utils import open_sqlite, AsyncPgSession, parse_dt, parse_json, log_run, pg_count
SCRIPT = "migrate_crm_quotations"
def _dec(val, default="0") -> Decimal:
try:
return Decimal(str(val)) if val is not None else Decimal(default)
except Exception:
return Decimal(default)
async def run() -> None:
sqlite = await open_sqlite()
rows = await sqlite.execute_fetchall("SELECT * FROM crm_quotations ORDER BY created_at")
await sqlite.close()
source_count = len(rows)
print(f"Source (SQLite): {source_count} crm_quotations rows")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = []
for r in rows:
records.append({
"id": r["id"],
"quotation_number": r["quotation_number"],
"title": r["title"],
"subtitle": r["subtitle"],
"customer_id": r["customer_id"],
"language": r["language"] or "en",
"status": r["status"] or "draft",
"order_type": r["order_type"],
"shipping_method": r["shipping_method"],
"estimated_shipping_date": r["estimated_shipping_date"],
"global_discount_label": r["global_discount_label"],
"global_discount_percent": _dec(r["global_discount_percent"]),
"vat_percent": _dec(r["vat_percent"], "24"),
"global_vat_percent": _dec(r["global_vat_percent"], "24"),
"shipping_cost": _dec(r["shipping_cost"]),
"shipping_cost_discount": _dec(r["shipping_cost_discount"]),
"install_cost": _dec(r["install_cost"]),
"install_cost_discount": _dec(r["install_cost_discount"]),
"extras_label": r["extras_label"],
"extras_cost": _dec(r["extras_cost"]),
"comments": parse_json(r["comments"], default=[]),
"quick_notes": parse_json(r["quick_notes"], default={}),
"subtotal_before_discount": _dec(r["subtotal_before_discount"]),
"global_discount_amount": _dec(r["global_discount_amount"]),
"new_subtotal": _dec(r["new_subtotal"]),
"vat_amount": _dec(r["vat_amount"]),
"final_total": _dec(r["final_total"]),
"nextcloud_pdf_path": r["nextcloud_pdf_path"],
"nextcloud_pdf_url": r["nextcloud_pdf_url"],
"client_org": r["client_org"],
"client_name": r["client_name"],
"client_location": r["client_location"],
"client_phone": r["client_phone"],
"client_email": r["client_email"],
"is_legacy": bool(r["is_legacy"]) if r["is_legacy"] is not None else False,
"legacy_date": r["legacy_date"],
"legacy_pdf_path": r["legacy_pdf_path"],
"created_at": parse_dt(r["created_at"]),
"updated_at": parse_dt(r["updated_at"]),
})
async with AsyncPgSession() as session:
async with session.begin():
# Disable FK enforcement so we can insert before crm_customers arrives in Phase 2.
await session.execute(text("SET LOCAL session_replication_role = replica"))
stmt = pg_insert(CrmQuotation).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["id"])
await session.execute(stmt)
dest_count = await pg_count(session, "crm_quotations")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

54
backend/migration/migrate_crm_sync_state.py
View File

@@ -1,54 +0,0 @@
"""
Phase 1 — Step 1.5: crm_sync_state (SQLite → Postgres)
Simple key/value table — small, no FK deps.
Run on VPS:
docker compose exec backend python -m migration.migrate_crm_sync_state
"""
import asyncio
import sys
from sqlalchemy.dialects.postgresql import insert as pg_insert
from crm.orm import CrmSyncState
from migration.utils import open_sqlite, AsyncPgSession, log_run, pg_count
SCRIPT = "migrate_crm_sync_state"
async def run() -> None:
sqlite = await open_sqlite()
rows = await sqlite.execute_fetchall("SELECT * FROM crm_sync_state")
await sqlite.close()
source_count = len(rows)
print(f"Source (SQLite): {source_count} crm_sync_state rows")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = [{"key": r["key"], "value": r["value"]} for r in rows]
async with AsyncPgSession() as session:
async with session.begin():
stmt = pg_insert(CrmSyncState).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["key"])
await session.execute(stmt)
dest_count = await pg_count(session, "crm_sync_state")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

69
backend/migration/migrate_device_alerts.py
View File

@@ -1,69 +0,0 @@
"""
Phase 1 — Step 1.4: device_alerts (SQLite → Postgres)
device_alerts is a "current state" table — one row per (device_serial, subsystem).
The SQLite PK is (device_serial, subsystem); Postgres adds a BIGSERIAL surrogate PK
with a unique constraint on the pair.
Run on VPS:
docker compose exec backend python -m migration.migrate_device_alerts
"""
import asyncio
import sys
from sqlalchemy import text
from migration.utils import open_sqlite, AsyncPgSession, parse_dt, log_run, pg_count
SCRIPT = "migrate_device_alerts"
async def run() -> None:
sqlite = await open_sqlite()
rows = await sqlite.execute_fetchall("SELECT * FROM device_alerts")
await sqlite.close()
source_count = len(rows)
print(f"Source (SQLite): {source_count} device_alerts rows")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = [
{
"device_serial": r["device_serial"],
"subsystem": r["subsystem"],
"state": r["state"],
"message": r["message"],
"updated_at": parse_dt(r["updated_at"]),
}
for r in rows
]
async with AsyncPgSession() as session:
async with session.begin():
await session.execute(
text("""
INSERT INTO device_alerts (device_serial, subsystem, state, message, updated_at)
VALUES (:device_serial, :subsystem, :state, :message, :updated_at)
ON CONFLICT (device_serial, subsystem) DO NOTHING
"""),
records,
)
dest_count = await pg_count(session, "device_alerts")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

93
backend/migration/migrate_device_logs.py
View File

@@ -1,93 +0,0 @@
"""
Phase 1 — Step 1.12: device_logs (SQLite → Postgres)
Largest table — migrated in batches of 10,000 rows to avoid memory issues.
device_logs is a partitioned table; rows route automatically to the correct
monthly partition based on received_at.
Run on VPS:
docker compose exec backend python -m migration.migrate_device_logs
"""
import asyncio
import sys
from sqlalchemy import text
from migration.utils import open_sqlite, AsyncPgSession, parse_dt, log_run, pg_count
SCRIPT = "migrate_device_logs"
BATCH_SIZE = 10_000
async def run() -> None:
sqlite = await open_sqlite()
# Total count first
count_row = await sqlite.execute_fetchall("SELECT COUNT(*) FROM device_logs")
source_count = count_row[0][0]
print(f"Source (SQLite): {source_count} device_logs rows")
if source_count == 0:
await sqlite.close()
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
offset = 0
total_inserted = 0
while offset < source_count:
rows = await sqlite.execute_fetchall(
"SELECT * FROM device_logs ORDER BY received_at LIMIT ? OFFSET ?",
(BATCH_SIZE, offset),
)
if not rows:
break
records = [
{
"device_serial": r["device_serial"],
"level": r["level"],
"message": r["message"],
"device_timestamp": r["device_timestamp"],
"received_at": parse_dt(r["received_at"]),
}
for r in rows
]
async with AsyncPgSession() as session:
async with session.begin():
await session.execute(
text("""
INSERT INTO device_logs
(device_serial, level, message, device_timestamp, received_at)
VALUES
(:device_serial, :level, :message, :device_timestamp, :received_at)
"""),
records,
)
total_inserted += len(records)
offset += BATCH_SIZE
pct = min(100, int(total_inserted / source_count * 100))
print(f" {total_inserted}/{source_count} rows inserted ({pct}%)")
await sqlite.close()
# Final count verify
async with AsyncPgSession() as session:
dest_count = await pg_count(session, "device_logs")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

73
backend/migration/migrate_heartbeats.py
View File

@@ -1,73 +0,0 @@
"""
Phase 1 — Step 1.11: heartbeats (SQLite → Postgres)
Raw-SQL table (no ORM model). BIGSERIAL PK — SQLite IDs not preserved.
Run on VPS:
docker compose exec backend python -m migration.migrate_heartbeats
"""
import asyncio
import sys
from sqlalchemy import text
from migration.utils import open_sqlite, AsyncPgSession, parse_dt, log_run, pg_count
SCRIPT = "migrate_heartbeats"
async def run() -> None:
sqlite = await open_sqlite()
rows = await sqlite.execute_fetchall("SELECT * FROM heartbeats ORDER BY received_at")
await sqlite.close()
source_count = len(rows)
print(f"Source (SQLite): {source_count} heartbeats rows")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = [
{
"device_serial": r["device_serial"],
"device_id": r["device_id"],
"firmware_version": r["firmware_version"],
"ip_address": r["ip_address"],
"gateway": r["gateway"],
"uptime_ms": r["uptime_ms"],
"uptime_display": r["uptime_display"],
"received_at": parse_dt(r["received_at"]),
}
for r in rows
]
async with AsyncPgSession() as session:
async with session.begin():
await session.execute(
text("""
INSERT INTO heartbeats
(device_serial, device_id, firmware_version, ip_address,
gateway, uptime_ms, uptime_display, received_at)
VALUES
(:device_serial, :device_id, :firmware_version, :ip_address,
:gateway, :uptime_ms, :uptime_display, :received_at)
"""),
records,
)
dest_count = await pg_count(session, "heartbeats")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

66
backend/migration/migrate_melody_drafts.py
View File

@@ -1,66 +0,0 @@
"""
Phase 1 — Step 1.1: melody_drafts (SQLite → Postgres)
Run on VPS:
docker compose exec backend python -m migration.migrate_melody_drafts
"""
import asyncio
import json
import sys
from sqlalchemy import text
from sqlalchemy.dialects.postgresql import insert as pg_insert
from melodies.orm import MelodyDraft
from migration.utils import open_sqlite, AsyncPgSession, parse_dt, parse_json, log_run, pg_count
SCRIPT = "migrate_melody_drafts"
async def run() -> None:
sqlite = await open_sqlite()
rows = await sqlite.execute_fetchall("SELECT * FROM melody_drafts")
await sqlite.close()
source_count = len(rows)
print(f"Source (SQLite): {source_count} melody_drafts rows")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = []
for r in rows:
data_raw = r["data"]
# SQLite stores data as JSON text; Postgres column is JSONB
data = parse_json(data_raw, default={})
records.append({
"id": r["id"],
"status": r["status"] or "draft",
"data": data,
"created_at": parse_dt(r["created_at"]),
"updated_at": parse_dt(r["updated_at"]),
})
async with AsyncPgSession() as session:
async with session.begin():
stmt = pg_insert(MelodyDraft).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["id"])
await session.execute(stmt)
dest_count = await pg_count(session, "melody_drafts")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

67
backend/migration/migrate_mfg_audit_log.py
View File

@@ -1,67 +0,0 @@
"""
Phase 1 — Step 1.3: mfg_audit_log (SQLite → Postgres)
Run on VPS:
docker compose exec backend python -m migration.migrate_mfg_audit_log
"""
import asyncio
import sys
from sqlalchemy import text
from migration.utils import open_sqlite, AsyncPgSession, parse_dt, log_run, pg_count
SCRIPT = "migrate_mfg_audit_log"
async def run() -> None:
sqlite = await open_sqlite()
rows = await sqlite.execute_fetchall("SELECT * FROM mfg_audit_log ORDER BY id")
await sqlite.close()
source_count = len(rows)
print(f"Source (SQLite): {source_count} mfg_audit_log rows")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
# mfg_audit_log uses a BIGSERIAL PK — we don't preserve SQLite integer IDs
# because the Postgres sequence will assign new ones. We insert in the same
# timestamp order so the audit trail remains coherent.
records = [
{
"timestamp": parse_dt(r["timestamp"]),
"admin_user": r["admin_user"],
"action": r["action"],
"serial_number": r["serial_number"],
"detail": r["detail"],
}
for r in rows
]
async with AsyncPgSession() as session:
async with session.begin():
await session.execute(
text("""
INSERT INTO mfg_audit_log (timestamp, admin_user, action, serial_number, detail)
VALUES (:timestamp, :admin_user, :action, :serial_number, :detail)
"""),
records,
)
dest_count = await pg_count(session, "mfg_audit_log")
if dest_count < source_count:
msg = f"Count mismatch: source={source_count} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

56
backend/migration/migrate_public_features.py
View File

@@ -1,56 +0,0 @@
"""
Phase 2 — Step 2.2: public_features (Firestore → Postgres)
Reads the single 'admin_settings/public_features' doc from Firestore and
flattens each field into a key/value row in public_features.
Run on VPS:
docker compose exec backend python -m migration.migrate_public_features
"""
import asyncio
import sys
from sqlalchemy.dialects.postgresql import insert as pg_insert
from settings.orm import PublicFeature
from shared.firebase import init_firebase, get_db as get_firestore
from migration.utils import AsyncPgSession, log_run, pg_count
SCRIPT = "migrate_public_features"
COLLECTION = "admin_settings"
DOC_ID = "public_features"
async def run() -> None:
init_firebase()
fs = get_firestore()
if fs is None:
print("ERROR: Firebase not initialised — check service account path.", file=sys.stderr)
sys.exit(1)
doc = fs.collection(COLLECTION).document(DOC_ID).get()
if not doc.exists:
print("No public_features document found in Firestore — skipping.")
await log_run(SCRIPT, 0, 0, notes="source doc not found")
return
data = doc.to_dict()
source_count = len(data)
print(f"Source (Firestore): {source_count} fields in {COLLECTION}/{DOC_ID}")
records = [{"key": k, "value": v} for k, v in data.items()]
async with AsyncPgSession() as session:
async with session.begin():
stmt = pg_insert(PublicFeature).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["key"])
await session.execute(stmt)
dest_count = await pg_count(session, "public_features")
print(f"Postgres public_features: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

56
backend/migration/migrate_settings.py
View File

@@ -1,56 +0,0 @@
"""
Phase 2 — Step 2.1: console_settings (Firestore → Postgres)
Reads the single 'admin_settings/melody_settings' doc from Firestore and
flattens each field into a key/value row in console_settings.
Run on VPS:
docker compose exec backend python -m migration.migrate_settings
"""
import asyncio
import sys
from sqlalchemy.dialects.postgresql import insert as pg_insert
from settings.orm import ConsoleSetting
from shared.firebase import init_firebase, get_db as get_firestore
from migration.utils import AsyncPgSession, log_run, pg_count
SCRIPT = "migrate_settings"
COLLECTION = "admin_settings"
DOC_ID = "melody_settings"
async def run() -> None:
init_firebase()
fs = get_firestore()
if fs is None:
print("ERROR: Firebase not initialised — check service account path.", file=sys.stderr)
sys.exit(1)
doc = fs.collection(COLLECTION).document(DOC_ID).get()
if not doc.exists:
print("No melody_settings document found in Firestore — skipping.")
await log_run(SCRIPT, 0, 0, notes="source doc not found")
return
data = doc.to_dict()
source_count = len(data)
print(f"Source (Firestore): {source_count} fields in {COLLECTION}/{DOC_ID}")
records = [{"key": k, "value": v} for k, v in data.items()]
async with AsyncPgSession() as session:
async with session.begin():
stmt = pg_insert(ConsoleSetting).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["key"])
await session.execute(stmt)
dest_count = await pg_count(session, "console_settings")
print(f"Postgres console_settings: {dest_count} rows ✓")
await log_run(SCRIPT, source_count, dest_count)
if __name__ == "__main__":
asyncio.run(run())

143
backend/migration/migrate_staff.py
View File

@@ -1,143 +0,0 @@
"""
Phase 3 — Step 3.1: admin_users (Firestore → Postgres staff table)
Reads every document in the 'admin_users' Firestore collection and inserts
a matching row into the Postgres 'staff' table.
Key transformations:
- Legacy role names mapped to canonical roles (superadmin→sysadmin, etc.)
- permissions=None stored as JSONB null (sysadmin/admin have no permission map)
- ui_prefs column NOT migrated (not part of the Postgres schema — dropped)
- Firestore doc ID preserved as staff.id and staff.firestore_id
- created_at/updated_at default to now() if missing from Firestore doc
Run on VPS:
docker compose exec backend python -m migration.migrate_staff
"""
import asyncio
import sys
from datetime import datetime, timezone
from sqlalchemy.dialects.postgresql import insert as pg_insert
from staff.orm import Staff
from shared.firebase import init_firebase, get_db as get_firestore
from migration.utils import AsyncPgSession, parse_dt, log_run, pg_count
SCRIPT = "migrate_staff"
COLLECTION = "admin_users"
_ROLE_MAP = {
"superadmin": "sysadmin",
"melody_editor": "editor",
"device_manager": "editor",
"user_manager": "editor",
"viewer": "user",
# canonical roles pass through unchanged
"sysadmin": "sysadmin",
"admin": "admin",
"editor": "editor",
"user": "user",
"staff": "user",
}
def _now_utc() -> datetime:
return datetime.now(timezone.utc)
def _coerce_dt(val) -> datetime | None:
if val is None:
return None
if isinstance(val, datetime):
return val.replace(tzinfo=timezone.utc) if val.tzinfo is None else val
return parse_dt(str(val))
async def run() -> None:
init_firebase()
fs = get_firestore()
if fs is None:
print("ERROR: Firebase not initialised.", file=sys.stderr)
sys.exit(1)
docs = list(fs.collection(COLLECTION).stream())
source_count = len(docs)
print(f"Source (Firestore): {source_count} admin_users documents")
if source_count == 0:
print("Nothing to migrate.")
await log_run(SCRIPT, 0, 0, notes="source empty")
return
records = []
skipped = 0
for doc in docs:
d = doc.to_dict()
hashed_password = d.get("hashed_password") or ""
if not hashed_password:
print(f" WARNING: {doc.id} ({d.get('email')}) has no hashed_password — skipping",
file=sys.stderr)
skipped += 1
continue
email = d.get("email") or ""
if not email:
print(f" WARNING: {doc.id} has no email — skipping", file=sys.stderr)
skipped += 1
continue
raw_role = d.get("role") or "user"
role = _ROLE_MAP.get(raw_role, "user")
# sysadmin/admin have no permission map
permissions = d.get("permissions")
if role in ("sysadmin", "admin"):
permissions = None
now = _now_utc()
records.append({
"id": doc.id,
"firestore_id": doc.id,
"email": email,
"name": d.get("name") or "",
"role": role,
"permissions": permissions,
"hashed_password": hashed_password,
"is_active": bool(d.get("is_active", True)),
"created_at": _coerce_dt(d.get("created_at")) or now,
"updated_at": _coerce_dt(d.get("updated_at")) or now,
})
actual_source = source_count - skipped
print(f" {skipped} skipped (missing email or password), {actual_source} to insert")
if not records:
print("Nothing to insert after filtering.")
await log_run(SCRIPT, source_count, 0, success=False,
notes="all docs skipped — missing required fields")
sys.exit(1)
async with AsyncPgSession() as session:
async with session.begin():
stmt = pg_insert(Staff).values(records)
stmt = stmt.on_conflict_do_nothing(index_elements=["id"])
await session.execute(stmt)
dest_count = await pg_count(session, "staff")
if dest_count < actual_source:
msg = f"Count mismatch: expected>={actual_source} postgres={dest_count}"
print(f"ERROR: {msg}", file=sys.stderr)
await log_run(SCRIPT, source_count, dest_count, success=False, notes=msg)
sys.exit(1)
print(f"Postgres: {dest_count} rows ✓")
note = f"{skipped} skipped (missing fields)" if skipped else None
await log_run(SCRIPT, source_count, dest_count, notes=note)
if __name__ == "__main__":
asyncio.run(run())

116
backend/migration/utils.py
View File

@@ -1,116 +0,0 @@
"""
Shared helpers for all Phase 1 SQLite → Postgres migration scripts.
Usage in each script:
from migration.utils import open_sqlite, get_pg, log_run, parse_dt, parse_json
"""
import json
import sys
from datetime import datetime, timezone
from pathlib import Path
import aiosqlite
from sqlalchemy import text
from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker, AsyncSession
from config import settings
# ── SQLite ────────────────────────────────────────────────────────────────────
async def open_sqlite() -> aiosqlite.Connection:
"""Open the SQLite database (read-only; no writes during migration)."""
db_path = Path(settings.sqlite_db_path)
if not db_path.exists():
print(f"ERROR: SQLite database not found at {db_path.resolve()}", file=sys.stderr)
sys.exit(1)
conn = await aiosqlite.connect(str(db_path))
conn.row_factory = aiosqlite.Row
return conn
# ── Postgres ──────────────────────────────────────────────────────────────────
def _make_pg_session() -> async_sessionmaker:
engine = create_async_engine(settings.database_url, pool_size=5, echo=False)
return async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
AsyncPgSession = _make_pg_session()
# ── Type helpers ──────────────────────────────────────────────────────────────
def parse_dt(value: str | None) -> datetime | None:
"""Parse a SQLite TEXT timestamp → timezone-aware datetime (UTC)."""
if not value:
return None
for fmt in (
"%Y-%m-%dT%H:%M:%S.%f",
"%Y-%m-%dT%H:%M:%S",
"%Y-%m-%d %H:%M:%S.%f",
"%Y-%m-%d %H:%M:%S",
"%Y-%m-%d",
):
try:
dt = datetime.strptime(value, fmt)
return dt.replace(tzinfo=timezone.utc)
except ValueError:
continue
# ISO format with offset — let fromisoformat handle it
try:
dt = datetime.fromisoformat(value)
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt
except ValueError:
pass
print(f"WARNING: could not parse timestamp {value!r} — using now()", file=sys.stderr)
return datetime.now(timezone.utc)
def parse_json(value: str | None, default=None):
"""Parse a SQLite TEXT JSON column → Python object."""
if value is None:
return default
try:
return json.loads(value)
except (json.JSONDecodeError, TypeError):
return default
# ── Migration run log ─────────────────────────────────────────────────────────
async def log_run(
script_name: str,
source_rows: int,
dest_rows: int,
success: bool = True,
notes: str | None = None,
) -> None:
"""Insert a row into _migration_runs recording this script's execution."""
async with AsyncPgSession() as session:
await session.execute(
text("""
INSERT INTO _migration_runs
(script_name, ran_at, source_rows, dest_rows, success, notes)
VALUES
(:script_name, now(), :source_rows, :dest_rows, :success, :notes)
"""),
{
"script_name": script_name,
"source_rows": source_rows,
"dest_rows": dest_rows,
"success": "ok" if success else "error",
"notes": notes,
},
)
await session.commit()
# ── Count helper ──────────────────────────────────────────────────────────────
async def pg_count(session: AsyncSession, table: str) -> int:
row = await session.execute(text(f"SELECT COUNT(*) FROM {table}"))
return row.scalar()

14
backend/database/core.py → backend/mqtt/database.py
View File

@@ -2,11 +2,10 @@ import aiosqlite
import asyncio import asyncio
import json import json
import logging import logging
import os
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from config import settings from config import settings
logger = logging.getLogger("database") logger = logging.getLogger("mqtt.database")
_db: aiosqlite.Connection | None = None _db: aiosqlite.Connection | None = None
@@ -163,8 +162,6 @@ SCHEMA_STATEMENTS = [
quotation_id TEXT NOT NULL, quotation_id TEXT NOT NULL,
product_id TEXT, product_id TEXT,
description TEXT, description TEXT,
description_en TEXT,
description_gr TEXT,
unit_type TEXT NOT NULL DEFAULT 'pcs', unit_type TEXT NOT NULL DEFAULT 'pcs',
unit_cost REAL NOT NULL DEFAULT 0, unit_cost REAL NOT NULL DEFAULT 0,
discount_percent REAL NOT NULL DEFAULT 0, discount_percent REAL NOT NULL DEFAULT 0,
@@ -180,7 +177,6 @@ SCHEMA_STATEMENTS = [
async def init_db(): async def init_db():
global _db global _db
os.makedirs(os.path.dirname(os.path.abspath(settings.sqlite_db_path)), exist_ok=True)
_db = await aiosqlite.connect(settings.sqlite_db_path) _db = await aiosqlite.connect(settings.sqlite_db_path)
_db.row_factory = aiosqlite.Row _db.row_factory = aiosqlite.Row
for stmt in SCHEMA_STATEMENTS: for stmt in SCHEMA_STATEMENTS:
@@ -201,14 +197,6 @@ async def init_db():
"ALTER TABLE crm_quotations ADD COLUMN client_location TEXT", "ALTER TABLE crm_quotations ADD COLUMN client_location TEXT",
"ALTER TABLE crm_quotations ADD COLUMN client_phone TEXT", "ALTER TABLE crm_quotations ADD COLUMN client_phone TEXT",
"ALTER TABLE crm_quotations ADD COLUMN client_email TEXT", "ALTER TABLE crm_quotations ADD COLUMN client_email TEXT",
"ALTER TABLE crm_quotations ADD COLUMN is_legacy INTEGER NOT NULL DEFAULT 0",
"ALTER TABLE crm_quotations ADD COLUMN legacy_date TEXT",
"ALTER TABLE crm_quotations ADD COLUMN legacy_pdf_path TEXT",
"ALTER TABLE crm_media ADD COLUMN thumbnail_path TEXT",
"ALTER TABLE crm_quotation_items ADD COLUMN description_en TEXT",
"ALTER TABLE crm_quotation_items ADD COLUMN description_gr TEXT",
"ALTER TABLE built_melodies ADD COLUMN is_builtin INTEGER NOT NULL DEFAULT 0",
"ALTER TABLE crm_quotations ADD COLUMN global_vat_percent REAL NOT NULL DEFAULT 24",
] ]
for m in _migrations: for m in _migrations:
try: try:

2
backend/mqtt/logger.py
View File

@@ -1,5 +1,5 @@
import logging import logging
import database as db from mqtt import database as db
logger = logging.getLogger("mqtt.logger") logger = logging.getLogger("mqtt.logger")

34
backend/mqtt/router.py
View File

@@ -8,7 +8,7 @@ from mqtt.models import (
CommandListResponse, HeartbeatEntry, CommandListResponse, HeartbeatEntry,
) )
from mqtt.client import mqtt_manager from mqtt.client import mqtt_manager
import database as db from mqtt import database as db
from datetime import datetime, timezone from datetime import datetime, timezone
router = APIRouter(prefix="/api/mqtt", tags=["mqtt"]) router = APIRouter(prefix="/api/mqtt", tags=["mqtt"])
@@ -129,29 +129,27 @@ async def mqtt_websocket(websocket: WebSocket):
try: try:
from auth.utils import decode_access_token from auth.utils import decode_access_token
from sqlalchemy import select from shared.firebase import get_db
from database.postgres import AsyncSessionLocal
from staff.orm import Staff
payload = decode_access_token(token) payload = decode_access_token(token)
role = payload.get("role", "") role = payload.get("role", "")
# sysadmin and admin always have MQTT access # sysadmin and admin always have MQTT access
if role not in ("sysadmin", "admin"): if role not in ("sysadmin", "admin"):
# Check MQTT permission for editor/user
user_sub = payload.get("sub", "") user_sub = payload.get("sub", "")
async with AsyncSessionLocal() as session: db_inst = get_db()
result = await session.execute( if db_inst:
select(Staff).where(Staff.id == user_sub).limit(1) doc = db_inst.collection("admin_users").document(user_sub).get()
) if doc.exists:
staff = result.scalar_one_or_none() perms = doc.to_dict().get("permissions", {})
if not perms.get("mqtt", False):
if staff is None: await websocket.close(code=4003, reason="MQTT access denied")
await websocket.close(code=4003, reason="User not found") return
return else:
await websocket.close(code=4003, reason="User not found")
perms = staff.permissions or {} return
if not perms.get("mqtt", {}).get("access", False): else:
await websocket.close(code=4003, reason="MQTT access denied") await websocket.close(code=4003, reason="Service unavailable")
return return
except Exception: except Exception:
await websocket.close(code=4001, reason="Invalid token") await websocket.close(code=4001, reason="Invalid token")

Some files were not shown because too many files have changed in this diff Show More