Coldeye Architecture Whitepaper
Design choices, detector benchmarks, custom dataset fine-tuning, and real-time event reasoning on local hardware.
01.System Premise · Local Hardware
Coldeye runs entirely on a single computer without any cloud connection. Camera feeds never leave the room, which protects personal privacy and removes network delays.
Running several neural models at the same time is the main challenge. Detection, pose estimation, tracking, facial recognition, and OCR all share the same GPU and CPU. Every component is chosen for speed and steady frame rates rather than raw model size.
02.Pipeline Overview
The architecture is structured as a sequential pipeline from raw frames to queryable database records:
Capture Loop
OpenCV thread
Perception
YOLO11m + Pose
Tracking
ByteTrack IDs
Event Logic
Posture & pickup
Persistence
SQLite WAL + API
03.Stream Ingestion
The system reads camera frames using OpenCV. It works with USB webcams, mobile phone cameras, RTSP streams, and video files.
If video capture and model inference run together in a single loop, frames queue up in memory. Any slowdown in the model creates delay on screen. To prevent this, video capture runs in its own thread. It always hands over the newest frame and drops older ones during heavy compute spikes.
04.Detector Selection · Benchmarks
The detector is the core of the pipeline. It finds people and room objects to feed the tracking and interaction logic. Choosing the right architecture requires balancing accuracy against continuous runtime footprint.
Why a CNN rather than a Vision Transformer? Vision transformers like RT-DETR require heavy memory bandwidth to run global attention across the image. On a local computer running in the background, this causes loud fan noise and frame drops whenever other applications are open. A convolutional network focuses on local pixel areas, using much less memory while staying quiet and steady.
Why single-stage rather than two-stage? Two-stage models like Faster R-CNN first suggest regions and then crop and classify them one by one. This causes delays when multiple people enter the room. A single-stage detector scans the full image in one pass, keeping processing time completely predictable.
Empirical model selection: We first tested YOLO11s for its minimal compute footprint. In real camera tests, bounding boxes flickered and jittered between frames, which broke tracking continuity on held objects. Upgrading to YOLO11m stabilized object boundaries across consecutive frames while still running smoothly in real time.
| Model | Architecture | Observed Test Behavior | Verdict |
|---|---|---|---|
| YOLO11s | Compact CNN | Bounding box flickering and unstable boundaries across frames | Rejected |
| RT-DETR | Vision Transformer | Heavy compute footprint causing frame drops on continuous streams | Ruled out |
| YOLO11m | Balanced CNN | Stable object boundaries with smooth real-time playback | Selected |
05.Custom Dataset · Fine-Tuning
Pretrained models only recognize the 80 classes in the COCO dataset. Everyday room objects like a gaming controller or glasses case are not included. The base model misclassified a PS5 controller as a phone or remote control.
We built a custom dataset pipeline in ml/ to fix this gap:
Samples images across varying room lighting, rotations, and hand grips directly from the camera feed.
Structures bounding box annotations, partitions train and validation splits, and formats dataset files.
Fine-tunes base weights using transfer learning until classification and localization loss stabilize.
Before: Pretrained COCO misclassifying controller
After: Custom checkpoint detecting controller
Convergence curves across bounding box regression, classification loss, and mAP metrics
06.Multi-Target Tracking
We chose ByteTrack over DeepSORT. DeepSORT runs a second neural network on every detected box to extract appearance embeddings, which slows down the system when multiple objects appear.
ByteTrack associates detections using Kalman filter motion predictions and box overlap. It tracks targets through brief occlusions without extra neural computation.
During tests, fast entries produced blurry edges that briefly created ghost entities. We resolved this by requiring detections to persist across multiple consecutive frames before confirming a new entity.
Persistent tracking with bounding box IDs across continuous frames
07.Pose Estimation · Posture Detection
We use YOLO11n-pose to extract 17 body keypoints per person and classify sitting versus standing. The nano model was selected because pose estimation needs much less capacity than dense object detection.
In early tests with a webcam framing only the upper torso, the model hallucinated ankles and reported standing while the person was sitting. We fixed this by checking lower-body joint confidence across consecutive frames. When legs are hidden behind a desk, posture estimation pauses so it does not guess wrong.
Geometric joint angle analysis classifying sitting state
08.Hand-Object Spatial Interactions
Detecting when someone picks up an item must run without wearable sensors. Heavy 3D hand mesh models were tested but discarded because they overloaded the GPU alongside detection and pose.
Instead, we estimate the hand position by projecting outward from the wrist along the forearm line. When the hand remains in contact with an object, the pickup is confirmed. The item stays linked to the person until sustained separation registers a put-down.
Hand-object proximity detection confirming item pickup from desk
09.Face Privacy · Biometric Enrollment
Privacy is active by default. Faces are detected and pixelated directly on each frame before any image is shown on screen or saved to disk. A toggle in the dashboard allows blurring to be turned on or off at any moment.
Biometric identification remains strictly opt-in. When a user chooses to enroll, DeepFace generates face embeddings stored locally in SQLite.
Biometric face verification is compute-heavy. Running it directly in the main capture loop freezes the video stream. We isolate face matching into a separate background process so video playback stays smooth.
Automatic frame-level face pixelation with opt-in demographic enrollment
10.On-Demand Text Extraction (OCR)
Reading text on held objects requires careful resource management. Running recognition on every video frame quickly overloads the machine.
Why on-demand? Coldeye triggers OCR only when an item is held up near the lens. This avoids extra workload during regular monitoring.
Why EasyOCR over Tesseract? Traditional Tesseract converts images to black and white first, which often fails under uneven room lighting or tilted packaging. EasyOCR finds and reads text directly with neural models, handling real-world objects reliably.
11.Concurrency · Non-Blocking Storage
The backend separates vision processing, data storage, and the web interface.
Events are written to SQLite in Write-Ahead Logging (WAL) mode. Standard SQLite locks the file during writes, which crashes web queries when the vision engine writes a rapid stream of vision events. WAL mode lets FastAPI read data freely while the vision loop appends records.
FastAPI serves live video and events to the Next.js dashboard for real-time monitoring and timeline playback.
Real-time activity log streamed to the web dashboard