demon hunter labs
Projects18
Every project, one jump awayChoose what you want to build.
Open the full Atlas

Foundation

Creation

Operations

Research

← Squidly manual

What it is / interactive architecture

Squidly: Real-time second-screen AI coaching system

Squidly is a Windows companion that can see the desktop, hear a two-speaker conversation, retrieve grounded context, and assemble useful guidance while the moment is still happening. This diagram follows captured signals through recognition, memory, retrieval, configurable AI guidance, and the local Advisor interface.

CAST / 05

Squidly field guide

Iona CoilThe Signal Oracle
Open complete manual
Real-time second-screen AI coaching systemREV 03
100%
SHAPE KEY
Real-time second-screen AI coaching systemSquidly is a Windows companion that can see the desktop, hear a two-speaker conversation, retrieve grounded context, and assemble useful guidance while the moment is still happening. This diagram follows captured signals through recognition, memory, retrieval, configurable AI guidance, and the local Advisor interface. Every station and route can be selected for more information.Focused-window or desktop BMP data enters the selected recognition route.R01 / captured pixelsHotkey, tray, and Advisor actions update the active session and timeline.R02 / action + artifactNormalized mic and rendered-audio chunks enter live or manual transcription.R03 / speaker streamsCompleted audio segments are retained as bounded local evidence.R04 / finalized WAVRecognized screenshot text and analysis state join the active session.R05 / ordered OCRTranscript updates are merged, finalized, labeled, and made available to the conversation lane.R06 / partial + final turnsRelevant local reference chunks are selected for the current question and mode.R07 / ranked snippetsCaptures, typed notes, pinned turns, latest turns, or the full transcript establish the requested reasoning scope.R08 / selected scopeThe assembled evidence and references become a bounded backend request.R09 / grounded promptGenerated guidance and backend outcome return to the session record.R10 / response + healthA bounded public status shape is serialized for authorized Advisor clients.R11 / status snapshotStatic assets load over HTTP and changing session state streams over WebSocket.R12 / HTTP + WebSocketCapture, listening, analysis, pin, clear, and refresh commands return to the native orchestrator.R13 / paired actionsPresentation state is partitioned into guidance, implementation, and conversation concerns.R14 / three lane ownershipSnapshots and ordered event records make the live session inspectable after the moment.R15 / persistRecognized text and image analysis can directly ground the next request.R16 / screen evidenceThe selected final turns feed latest-turn or full-conversation analysis.R17 / conversation scopeCoding modules or conversation bubbles render in their owned Advisor lane.R18 / visible helpComponent health and latency metrics join the status payload without leaking secrets.R19 / health contractLatency points and verification outcomes preserve what was actually exercised.R20 / timing + proofWin32 tray controls, configurable hotkeys, focused-window capture, full-screen capture, and typed context create deliberate visual work items.INN0101 / VISUAL INPUTDESKTOP +HOTKEYSIN 00InputOUT 02Microphone and rendered-call audio are captured as distinct streams through WASAPI or named AudioFlow and VB-CABLE endpoints.INN0202 / AUDIO INPUTWASAPI +AUDIOFLOWIndependent mic and render thre...IN 00InputOUT 02Windows.Media.Ocr supplies the fast local text route while the configured Codex image path can perform deeper screenshot interpretation.FXN0303 / RECOGNITIONOCR + IMAGEANALYSISIN 01ProcessOUT 02Manual and live speech routes select among Whisper, Nemotron, sherpa-onnx, faster-whisper, Riva, OpenAI Realtime, or a development-only mock.FXN0404 / RECOGNITIONLIVE STTMULTIPLEXERSeparate batch and live backend...IN 01ProcessOUT 02One session identity owns current guidance, transcript scope, timeline, captures, assistant responses, health, and the references selected for the live moment.DBN0505 / MEMORYSESSIONSTATESnapshot plus append-only recordsIN 04State storeOUT 04Folder-backed ingestion expands supported documents, extracts text or images, chunks content, and ranks relevant snippets without moving the source library into a remote service.DBN0606 / RETRIEVALREFERENCEINDEXIN 00State storeOUT 01Mode inference and prompt assembly join current evidence, selected transcript scope, long-term profile, session notes, and ranked references.FXN0707 / GROUNDINGCONTEXTASSEMBLYCoding, sales, mentoring, and m...IN 04ProcessOUT 01Configurable model clients generate grounded coding or conversation help while explicit health state records which real backend produced or failed to produce it.APIN0808 / MODEL BOUNDARYCODEX +LOCAL GEMMACodex CLI verified primary pathIN 01BoundaryOUT 02A compact Winsock HTTP and WebSocket server publishes Advisor assets, authorized status, actions, capture images, and live state updates.APIN0909 / NETWORK BOUNDARYTOKEN-GATEDLOCAL SERVERIN 03BoundaryOUT 01The responsive second-screen UI separates grounded guidance, screenshot and implementation work, and live conversation across dense and mobile layouts.OUTN1010 / PRESENTATIONTHREE-LANEADVISORLeft guidance railIN 03OutputOUT 01Tests, degraded-mode reasons, latency timestamps, and security checks make the current scaffold's working paths and limits inspectable.QAN1111 / EVIDENCEHEALTH +VERIFICATIONIN 00EvidenceOUT 02Prompts, markdown guidance, JSON and JSONL records, recaps, WAV audio, and BMP captures preserve the evidence and output of each session locally.OUTN1212 / OUTPUTLOCAL SESSIONARTIFACTSInspectable event streamsIN 03OutputOUT 00DHL-VISUAL-PROGRAM / SQUIDLY / REV 04VARIABLE NODES / DISTRIBUTED PORTS / INSPECTABLE CONTRACTS