bellsystems-cp/docs/melody-binary-serving.md

Melody Binary Serving (Plain HTTP)

Why this exists

ESP32 devices download .bsm melody files directly, using a URL supplied through the Android app. The download previously pointed at a Firebase Storage public URL (HTTPS only). ESP32's TLS client needs 40KB+ of RAM to open an HTTPS connection, which these devices can't reliably spare. To fix this, melody .bsm binaries are now stored and served by our own backend over plain HTTP, instead of Firebase Storage.

This does not affect the audio preview file (information.previewURL) — that's only ever fetched by the browser admin UI, which is already HTTPS, so it stays on Firebase Storage unchanged.

PID vs UID — why storage is keyed on pid, not uid

• pid ("Playback ID") identifies the underlying archetype binary — the raw note sequence (A, B, C...) baked into a .bsm file. Multiple melodies can legitimately share one pid: each melody remaps those notes to actual bells via its own noteAssignments, speed, and duration settings. One binary can back many different "remix" melodies. Duplicate PIDs across melodies are correct and expected, not a bug.
• uid is meant to be each melody's own unique database identity. It is not currently populated anywhere in MelodyForm.jsx — this is a pre-existing, separate bug, deferred as its own task. Because of this, local binary storage is keyed on pid, not uid.

What changed

• Storage: .bsm binaries are written to ./data/melody_binaries/{pid}.bsm on the host (mounted into the backend container at /app/storage/melody_binaries, same pattern as ./data/firmware and ./data/built_melodies). Because storage is keyed on pid, multiple melodies sharing a pid share one file — saving from any of them overwrites the same file, which is expected.
• Upload path: SelectArchetypeModal / BuildOnTheFlyModal still call POST /api/melodies/{melodyId}/upload/binary exactly as before. The backend (backend/melodies/service.py::save_binary_for_melody) now writes the bytes to local disk instead of uploading to Firebase Storage, and returns a URL like http://melodies.bellsystems.net/download/{pid}. The melody must have a pid set before uploading a binary (the endpoint returns 400 if not).
• Stored URL: the melody's url field (Firestore + SQLite) now holds that plain-HTTP URL instead of a Firebase public URL. No schema change — url was already a plain string field.
• Download route: GET /api/melodies/download/{pid} (backend/melodies/router.py) is unauthenticated (devices have no login token) and resolves pid directly to {pid}.bsm on disk, same pattern as the existing firmware download route (backend/firmware/router.py::download_firmware).
• Deletion is share-aware: deleting a melody or its binary (service.delete_file / service._delete_storage_files) only removes the local .bsm file if no other melody still references the same pid (service._pid_used_by_other_melody). This prevents one melody's deletion from breaking playback for other melodies sharing its archetype binary.

Migrating existing melodies

Melodies created before this change still have a Firebase Storage URL in their url field — deploying this code does not touch existing data. They keep working exactly as before (still HTTPS to Firebase) until migrated.

Two ways to move a melody to the new plain-HTTP URL:

1. Per-melody, via the UI: open the melody and re-run "Select Archetype" or "Build on the Fly" — this naturally re-uploads through the same endpoint, which now writes to local disk and updates url.

2. Bulk, via script: backend/scripts/migrate_melody_binaries_to_local.py downloads each melody's current Firebase binary and writes it to local disk under its pid, then updates url (SQLite + Firestore if published). Run with --dry-run first:

   docker exec -it bellsystems-backend python scripts/migrate_melody_binaries_to_local.py --dry-run
   docker exec -it bellsystems-backend python scripts/migrate_melody_binaries_to_local.py
   

   Known caveat: some existing melodies share a pid despite their Firebase URLs pointing at different source files (observed on real data — e.g. voice-count variant filenames like 1N_/2N_/3N_/4N_ all filed under one pid). The script downloads only the first melody's file for each pid and reuses it for every other melody sharing that pid — it does not attempt to detect which file is "correct". It prints a warning list of every pid where this happened; use each melody's playback button afterward to verify the right archetype plays, and manually re-assign the correct one via "Select Archetype" if it's wrong.

Infrastructure (outside this repo)

The plain-HTTP requirement is handled by keeping melodies.bellsystems.net on a separate hostname from console.bellsystems.net, so the console's NPM proxy host can stay HTTPS-only with no per-path exceptions.

Required, one-time setup outside this repository:

1. DNS: add an A/CNAME record for melodies.bellsystems.net pointing at the same host as console.bellsystems.net.
2. NPM (Nginx Proxy Manager) proxy host: create a new proxy host for melodies.bellsystems.net forwarding to the nginx container's exposed port (90 per docker-compose.yml). Do not force SSL / do not enable the "Force SSL" redirect on this proxy host — it must remain reachable over plain HTTP.

The in-repo nginx/nginx.conf already has a dedicated server_name melodies.bellsystems.net block that maps /download/{pid} to the backend's /api/melodies/download/{pid} route, so no further nginx changes are needed once the NPM proxy host exists.

Verification

1. Build/select an archetype for a melody with a pid set → confirm a file appears at ./data/melody_binaries/{pid}.bsm and the melody's url becomes http://melodies.bellsystems.net/download/{pid}.
2. curl http://localhost:8000/api/melodies/download/{pid} (direct to backend, bypassing nginx) → returns the .bsm bytes, no auth header needed.
3. curl http://localhost:90/api/melodies/download/{pid} (through nginx on the console server block) → same result.
4. Once NPM is configured, curl http://melodies.bellsystems.net/download/{pid} → same result, over plain HTTP.
5. Delete a melody whose pid is not shared with any other melody → confirm the local .bsm file is removed. Delete one of two melodies sharing a pid → confirm the file survives until the last one is deleted.
6. Upload a preview audio file → confirm it still lands in Firebase Storage and previewURL still populates (regression check).