Wallflower (Jam & Sample Manager)

Overview

Wallflower is a local-first jam and sample manager for musicians who improvise. It solves a specific creative problem: transforming multi-hour jam sessions into usable samples without workflow interruption. Record a 2-hour session, let the app auto-detect key, tempo, sections, and separate instruments, then extract an 8-bar synth loop and drag it into your DAW. All with local AI, crash-safe recording, and passive background processing.

Core purpose: Stay in creative flow. Record, tag, browse, extract — without switching out of the musical headspace to manage files.

Target users: Electronic musicians and improvisers who accumulate hours of jam recordings and need to find and extract the good parts later.

Key Features

Implemented (all phases complete, v0.2.0):

• Audio import pipeline with atomic copy → SHA-256 content hashing → symphonia metadata extraction → duplicate detection → SQLite insert
• Auto-import from ~/wallflower folder via filesystem watching
• USB recorder detection (Zoom F3) with import prompting
• Multi-channel recording with crash-safe WAV writer, silence detection, dropout recovery, and priority scheduling (recording preempts all background ML)
• Audio device selection with real-time level meters
• Metadata CRUD — tags, collaborators, instruments, location, notes with immediate persistence
• Photo attachment for documenting patch configurations (drag-and-drop)
• Chronological timeline browser with waveform visualization (wavesurfer.js)
• Peak generation for 120-minute audio files (server-side, multi-resolution, canvas-based)
• Bookmarks — mark and annotate positions within jams, context menus, popovers
• ML analysis pipeline — tempo (TempoCNN), key/scale, section boundaries, loop detection via essentia over gRPC with streaming progress
• Source separation — drums, bass, vocals, other stems via demucs-mlx with chunked processing for 60+ minute files
• Stem mixer — play separated stems with per-stem volume/solo/mute
• Stem export — export individual stems or mixes as audio files
• Sample browser — browse, filter, preview, and extract samples across the library
• Analysis profiles — full, standard, lightweight — with hardware-aware recommendations
• Model management — download, cache, and version ML models from settings
• Accessibility — high-contrast mode, skip links, roving tab index, keyboard navigation
• macOS distribution — code-signed and notarized DMG for Apple Silicon (macOS 13+)
• Auto-launch — optional launch at login via Tauri plugin
• Native macOS notifications for import completion and analysis results
• Headless CLI for debugging and automation (import, list, status, settings, devices)
• Recording deletion — remove recordings from both database and filesystem

Architecture

Tech Stack

Layer	Technology
Native shell	Tauri 2.10 (WebKit, macOS)
Backend	Rust + Axum 0.8 (HTTP API, port 23516)
Database	rusqlite 0.39 (bundled SQLite, WAL mode)
Audio I/O	cpal 0.15 (recording), symphonia 0.5 (decoding WAV/FLAC/MP3)
Audio write	hound 3.5 (crash-safe WAV writer)
Frontend	Next.js 16.2 + React 19 + TypeScript 6
Styling	Tailwind CSS 4.2 + shadcn/ui
Waveform	wavesurfer.js 7.12
State	zustand 5.0 (6 client stores) + @tanstack/react-query 5.99 (server)
ML analysis	Python 3.13 sidecar via gRPC (essentia 2.1b6, librosa 0.10)
Source separation	demucs-mlx (MLX-optimized, 2.6x faster than PyTorch MPS on Apple Silicon)
IPC	tonic/prost 0.14 (gRPC between Rust and Python)

Structure

wallflower/
├── crates/
│   ├── wallflower-core/     # Database, import, recording, analysis, export, peaks, bookmarks (11,620 LOC Rust, 29 modules)
│   ├── wallflower-app/      # Tauri shell, Axum API, 18 IPC command files, sidecar management, tray
│   └── wallflower-cli/      # Headless CLI (import, list, status, settings, devices)
├── src/
│   ├── app/                 # Next.js pages (home, jam detail, settings)
│   ├── lib/                 # types.ts, tauri.ts IPC wrapper, 6 zustand stores
│   └── components/          # 50+ components: waveform, timeline, stems, samples, recording, analysis, settings
├── sidecar/
│   └── src/wallflower_sidecar/  # gRPC server, 5 analyzer modules, model manager (Python 3.13)
├── proto/                   # wallflower_analysis.proto (gRPC service definition)
├── migrations/              # SQLite schema (jams, tags, collaborators, instruments, photos, bookmarks, analysis, stems)
├── .planning/               # GSD artifacts (7 phases, 34 plans, all complete)
├── Cargo.toml              # Rust workspace
└── package.json            # Frontend deps (v0.2.0)

Key Patterns

1. Atomic import pipeline — Copy → SHA-256 hash → metadata extract → duplicate check → DB insert. No partial state if any step fails.
2. Recording priority — All background processing (ML analysis, peak generation, separation) pauses during active recording via a priority scheduler. Creative flow is sacred.
3. Crash-safe WAV writer — Incremental writes with WAV header updates on every flush. Power loss during a 2-hour recording loses at most the last buffer, not the file.
4. Polyglot via gRPC — Rust for audio reliability and crash safety, Python for ML flexibility (essentia, demucs-mlx), TypeScript for rich UI. gRPC bridge with protobuf contracts keeps languages in separate processes with streaming progress.
5. Content-addressed storage — SHA-256 hash on import means the same file imported twice is a no-op, not a duplicate.
6. Chunked ML processing — demucs-mlx processes 60+ minute files in memory-bounded chunks with streaming progress updates via gRPC. No OOM on long jams.
7. Event-to-store bridge — Tauri events (recording state, analysis progress, import status) push into zustand stores for reactive UI without polling.
8. IPC-first, API-second — Primary UI communication via Tauri IPC commands (18 handlers); Axum HTTP API serves the webview's internal fetch calls and future headless/daemon use.

Development History

v0.2.0 complete — 273 commits over 9 days, April 18-27, 2026:

Phase	Status	Focus	Plans
1	Complete	Tauri shell, SQLite storage, Axum API foundation, import pipeline	4
2	Complete	Playback, metadata CRUD, timeline, notifications, design system	4
3	Complete	Multi-channel recording, crash-safe WAV writer, silence detection, device selection, priority scheduler	6
4	Complete	ML analysis pipeline — tempo, key, sections, loops via Python sidecar over gRPC	6
5	Complete	Source separation (demucs-mlx), stem mixer, stem export	5
6	Complete	Accessibility (high-contrast, keyboard nav), code signing, notarization, auto-launch	5
7	Complete	Sample browser (backend, store, UI, preview panel, filtering)	4

34 plans across 7 phases + 4 quick tasks. Average plan execution: ~9 minutes. All phases gated on the previous — recording before ML, ML before separation, separation before sample extraction.

Velocity Profile

The 9-day build breaks into two distinct modes:

• Days 1-2 (Phases 1-2): Foundation sprint. 39 commits establishing the Rust workspace, import pipeline, Tauri shell, and React design system. The critical decision here was getting Tauri IPC commands type-safe end-to-end before writing any business logic.
• Days 3-9 (Phases 3-7): Feature phases at ~30 commits/day. The Python sidecar (Phase 4) was the technical crux — gRPC proto definition, server implementation, Rust client, streaming progress, error handling across the language boundary. Once that seam was clean, Phase 5 (separation) and Phase 7 (sample browser) flowed naturally.

Architectural Decisions

Decision	Rationale
Rust + Tauri over Electron	Audio reliability, 96% smaller binary, native WebKit performance for long recordings
SQLite (bundled, WAL mode)	Single-file local database, crash-safe, no external dependency, sync-folder portable
symphonia + cpal (pure Rust)	Codec-agnostic audio without FFmpeg dependency; cross-platform potential
hound for WAV writing	Crash-safe incremental writes; symphonia is decode-only
Python sidecar via gRPC	ML ecosystem (essentia, demucs, librosa) is Python-native; gRPC keeps processes separate with typed streaming contracts
demucs-mlx over PyTorch MPS	2.6x faster on Apple Silicon; MLX is Apple's native ML framework
Content-addressed imports (SHA-256)	Duplicate detection without filename dependency; safe re-imports
Recording preempts all background work	Creative flow is non-negotiable; background ML can wait
Static Next.js export in Tauri webview	Rich React UI without a web server; all state via IPC or local HTTP
Tauri IPC over REST for UI	Lower latency, type-safe, no HTTP overhead for same-process communication
Manual SQL migrations	Dropped refinery due to rusqlite 0.39 incompatibility; explicit .sql files in migrations/
Chunked demucs processing	Memory-bounded separation for 60+ minute files; streaming progress
Canvas-based peaks	Pre-computed multi-resolution peaks indexed for fast scrubbing of 120-minute files

Strengths

• Type-safe across three languages — Rust structs → SQLite → Tauri IPC → TypeScript types with protobuf contracts governing the Rust↔Python boundary. No runtime serialization surprises anywhere in the stack.
• Atomic safety throughout — Import pipeline hashes before copying, WAV writer survives power loss, SQLite WAL prevents corruption during long recordings, priority scheduler prevents resource contention.
• The polyglot seam is clean — gRPC with streaming progress means Rust doesn't need to know about essentia internals and Python doesn't need to know about Tauri. The proto file is the contract. This was the project's biggest technical risk and it's now the best-designed boundary.
• Designed for heavy workloads — 120-minute file peak generation, chunked demucs processing, hardware-aware analysis profiles, memory-bounded stem separation. Not bolted on — designed from Phase 1 specs.
• Test coverage at critical boundaries — 21+ Rust test modules covering the import pipeline, recording engine, peak generation, export, and analysis queue. 9 Python test files covering all analyzers. Tests concentrate where failures are most costly.
• Complete from record to extract — The full creative workflow works: record a jam → auto-analyze tempo/key/sections → separate stems → browse samples → export. No critical feature is stubbed.

Weaknesses & Risks

• HTTP API mostly stubbed — 19 of 26 Axum endpoints return 501 because functionality moved to Tauri IPC. The headless/daemon mode story is incomplete — CLI works for basics, but automated workflows can't access recording or analysis via HTTP.
• Spatial similarity explorer descoped — The Phase 6 headline feature (browse by harmonic distance instead of chronology) was replaced with accessibility and distribution work. The signature differentiator is now a backlog item.
• Frontend testing gap — Only 1 test file (useRovingTabIndex) for 89 TypeScript files and 50+ components. Backend and sidecar are well-tested; the UI layer is not.
• Single platform — macOS only (Apple Silicon, macOS 13+). No Linux or Windows builds. Tauri supports cross-platform, but cpal device handling and demucs-mlx are macOS-specific.
• Python sidecar packaging — The sidecar requires Python 3.13 with pip-installed dependencies. No bundled Python runtime in the DMG — users need a working Python environment for ML features. This is a distribution friction point.
• No user validation — Built for the developer's own workflow. No other musicians have used it. The recording → analysis → extraction loop makes sense in theory; whether it matches how improvisers actually work is untested.

Connection to Other Projects

• Music domain cluster — Fourth project joining Evolver (instrument mastery), Etyde (practice/theory), and AbletonBuddy (production/songwriting). Wallflower covers recording and sample management — the stage between live performance and production. Together they span the full arc: learn (Etyde) → master an instrument (Evolver) → record jams (Wallflower) → produce tracks (AbletonBuddy).
• Second non-TypeScript primary language — After Earworm (Go), Wallflower is the second project choosing the right language for the domain: Rust for audio reliability and crash safety. Both validate the CLAUDE.md principle of suggesting alternatives when the problem domain is a poor fit for Node.js.
• First polyglot architecture — Three languages (Rust, Python, TypeScript) in three separate processes with typed IPC boundaries. No other project in the portfolio has this kind of intentional language separation — the closest is Earworm's Go-wrapping-Python subprocess boundary, but that's sequential, not concurrent.
• No Ollama/VPN/cloud dependency — Like Earworm, sits entirely outside the Mac Mini compute center. All ML runs locally in a Python sidecar via MLX/essentia, not through Roughneck. This is deliberate: audio processing is real-time-adjacent and can't tolerate VPN latency.
• Roughneck — Referenced in early research as a potential integration point for source separation queuing, but no dependency exists. Wallflower's own priority scheduler handles job orchestration internally.

What Makes This Project Different

Wallflower is the portfolio's first:

• Rust application — Not a CLI (Earworm is Go CLI), but a full native app with audio I/O, database, ML pipeline, and rich UI
• Tauri project — First use of Tauri over Electron/Next.js for a desktop application (96% smaller binary)
• Audio I/O project — First project that records and plays back audio natively (Evolver uses Web MIDI, AbletonBuddy uses Max4Live, Etyde uses alphaTab notation)
• Polyglot architecture by design — Rust + Python + TypeScript in separate processes with gRPC and Tauri IPC contracts
• ML-on-device project — First project running real ML models (essentia CNNs, demucs neural network) locally via MLX on Apple Silicon
• Code-signed macOS distribution — First project shipped as a notarized DMG, not a web app or dev-mode binary

The spatial similarity explorer remains the conceptual differentiator — no existing jam manager lets you browse recordings by "what sounds like this" instead of "when did I record this" — but it's now a future milestone, not a shipped feature. What shipped instead is the complete operational loop: record → analyze → separate → browse → extract. The spatial exploration will build on this foundation.

The Build → Ship Arc

Wallflower is the portfolio's cleanest example of the Build posture carried to completion. 273 commits, 34 plans, 7 phases, 9 days, one released binary. No scope creep into adjacent features. No premature polish phase. No pause for user research that doesn't exist yet.

What's notable is what didn't happen: the spatial similarity explorer (the feature I'm most excited about) got descoped when Phase 6 reality-checked the timeline. Accessibility and distribution took its place. That's the right call — a code-signed binary with accessibility support ships; a half-built similarity map doesn't — but it required actively choosing distribution fitness over the feature I wanted to build. The backlog knows where to pick up.

The next posture should be Extend (dogfooding) or Wait (until enough jam recordings exist to validate the similarity concept). Not more features.