Nurby
GitHub
Features

Sleep, because your cameras won't.

Local AI that knows who's home, what your driveway just saw, and whether that sound was a glass break or a dropped mug. Nothing gets uploaded to the cloud, and there's no subscription. Self-host it if you have the GPU. If you don't, the built-in cloud providers work out of the box.

View on GitHub
01· Feature

Identity that survives a turned head

Most home AI cameras tie identity to a single clean face shot. If the camera never sees a face, the person is a stranger forever. Nurby runs a parallel body identity pipeline alongside face recognition.

  • OSNet body embedding
    512-dim vector trained on Market1501 + MSMT17. Invariant to camera angle, distance, lighting.
  • HSV color histogram
    32 hue bins, 8 saturation bins. Dark and under-saturated pixels masked so clothing colour drives the signal.
  • InsightFace ArcFace 512-dim face embedding
    When a face is visible, ArcFace stamps identity. Acts as the durable anchor.
  • Per-camera IoU + Kalman tracker
    Consecutive frames of the same person stay linked. Stops one-off cluster spam.
  • Tentative vs confirmed identities
    Body-only matches live in a separate pgvector index. Promoted to confirmed the moment a face hit lands on the same frame.
  • Retroactive face confirmation
    A fusion sweeper walks the last hour every five minutes. Name someone on the front-door cam at 6 pm, every prior body-only sighting from 5 to 6 pm gets their name.
  • Colour tie-break under collision
    Two delivery drivers in identical uniforms? Re-rank the top-K body clusters within a cosine band using HSV chi-square against the query crop.
  • Tracklet-centroid clustering
    First frame produces a tentative ID for live UI. Mean-of-five-frames decides the cluster once enough evidence exists, correcting motion-blur stragglers.
  • Identity decay
    Tentative clusters with no activity for 14 days are pruned. Confirmed clusters never auto-prune.
Why it matters
  • Recognition does not require a perfectly framed face.
  • One naming click propagates across every camera and every prior sighting in the same hour.
  • Clothing-twin collisions resolve without manual intervention.
  • The system does not lose track of someone who walked from your front door to your kitchen because the kitchen camera only caught the back of their head.
02· Feature

Smart Track. PTZ that actually follows people

Closed-loop visual servoing on ONVIF ContinuousMove. The closer the target gets to centre, the slower the camera moves.

  • Per-camera follow rules
    Eligible YOLO targets, a hard ignore list, priority order, min-confidence floor. The resident dog never triggers a follow.
  • Identity gate
    ptz_smart_track_require_face takes a list of Person UUIDs. Track only fires for matching body or face identities. Works even when the kid's face is turned away because identity is body-anchored.
  • Lost-target home preset
    After N seconds without an eligible target, return to a configured ONVIF preset instead of drifting forward.
  • Mechanical wear budget
    Default cap of 30 ContinuousMove commands per minute, rolling window. Micro-corrections collapse into a single move.
  • Deadzone, max-speed, gain
    Tunable centre tolerance, velocity cap, and proportional gain on bbox error. Snappy or smooth, your call.
  • Auto-zoom with hysteresis
    Slow zoom-in below 5% bbox area, fast zoom-out above 35%. Off by default to avoid losing fast targets.
  • No-go pan/tilt boxes
    Define angle regions the tracker will never enter. Privacy boundaries enforced inside the tracker.
Why it matters
  • Tracking math respects ONVIF's actual capabilities rather than fighting them.
  • Identity-aware tracking does not require constant face visibility.
  • Motor wear is bounded so the hardware lasts.
  • Privacy boundaries are enforced inside the tracker, not bolted on later.
03· Feature

Smart Privacy Zones with AI detection

Nurby auto-detects privacy-sensitive objects and applies a blur before any downstream pipeline sees the frame.

  • Auto-detected targets
    Bed, bathroom door, toilet, bathtub, mirror, picture frame, monitor, laptop, TV, neighbour windows. Add a detector that recognises new privacy targets and it just works.
  • Pre-storage masking
    Blur applied before thumbnail save, VLM encode, embedding generation, and live preview. Unredacted frame never leaves perception.
  • PTZ pose tagging
    Each auto-zone is tagged with the pan/tilt/zoom at detection time. Only applies when the camera returns to that pose (±5° pan/tilt, ±0.1 zoom). Forgiving on creation, strict on enforcement.
  • Freshness gate
    Auto-zones expire after 60 idle seconds. Stops a single mistaken detection from blurring a region forever.
  • Per-person privacy blur
    Set privacy_blur on a Person row. Their face crops and observation thumbnails render blurred. Useful for housemates who opted out of summaries.
  • Tunable blur strength
    Per-camera Gaussian kernel size. From redacted-but-recognizable for debugging to completely opaque.
Why it matters
  • Privacy zones move with the camera instead of revealing what was supposed to be hidden.
  • Sensitive content is masked before AI ever sees it, not just before display.
  • The zone list updates itself when furniture moves or new privacy targets appear.
04· Feature

Detection model catalog with open-vocabulary support

Seventeen models across YOLOv8, YOLO11, YOLO11-seg, YOLOv8 OIV7, and YOLO-World v1 + v2. Each tagged with speed, accuracy, and GPU hints.

  • Per-camera multi-model fusion
    Run several models on the same frame. Merge by any (union with NMS dedup), consensus (require K agree), or best (highest confidence wins per location).
  • YOLO-World per-camera prompts
    Plain English. Front door detects "person, package, delivery driver, mail truck, stroller." Driveway detects "car, truck, license plate." Feeds model.set_classes; the default LVIS vocabulary is replaced with your targets.
  • Dynamic label vocabulary
    The label picker is sourced from the active model, not a hardcoded list. The UI even shows "Labels sourced from yolov8n.pt" so you know what's actually available.
  • Security-relevant default filter
    Without per-model overrides, restrict to a curated subset (people, vehicles, animals, common indoor objects). The full COCO set is one toggle away.
Why it matters
  • Detection is not a single model decision baked in at build time.
  • Open-vocabulary prompts let users add classes the model was never trained on, by name.
  • Multi-model consensus catches the failure modes of any single model.
  • Per-camera tuning means a hallway and a driveway can run different stacks.
05· Feature

Local-first AI with multi-provider VLM

Ollama-first deployment. The provider catalog and a per-call precedence chain mean expensive models run only where they matter.

  • One-click Ollama deploy
    A VRAM-aware model picker shows compatible Ollama VLMs with a recommended badge based on your GPU. Running a 200B-parameter VLM locally requires serious hardware like an NVIDIA Spark module or a workstation GPU with 48 GB+ VRAM. No GPU? Cloud providers work out of the box.
  • Provider precedence chain
    Per-call → per-camera → system default. Run cheap models normally; pay for the heavy one only when triggers fire.
  • Provider catalog
    OpenAI, Anthropic, Gemini, Ollama, any OpenAI-compatible endpoint. Mix and match per camera.
  • Cascade refiner
    After the primary VLM finishes, a refiner re-processes the output when configured triggers match. Heavy-model cost only on interesting frames.
  • Reinterpret with model picker
    Any journey, incident or conversation can be re-interpreted by any provider with full context. Single click for a second opinion.
  • Per-camera prompt override
    A baby cam describes movement and quiet. A driveway describes vehicles and arrivals.
  • Token budget per camera
    Max output tokens plus optional input cap on long-context calls.
Why it matters
  • Local inference is a first-class option for users with the right hardware. Think NVIDIA Spark, not a five-year-old laptop.
  • No API keys required if you have the GPU. Cloud providers are one toggle away if you don't.
  • The VLM stack is heterogeneous on purpose. Expensive only where it matters.
06· Feature

Rule builder with plain-language preview

Triggers are cards with icons, not a dropdown of column names. The composed rule renders as a single sentence in English.

  • Trigger-card grid
    Object detected, face matched, audio event, schedule, tripwire crossed, loiter, person leaves frame, person enters zone, and more.
  • Condition chips
    Cameras, time-of-day windows, confidence floors, cooldown periods. Chips AND together; rules OR together.
  • Action types
    Webhook, generic API call, in-app notification, SMTP email, dashboard broadcast. Mix actions per rule.
  • Plain-language preview
    "When a person is detected on Front Door after 10 pm, send me an email and record a thirty-second clip." If the preview reads wrong, the rule is wrong.
  • Cooldown badges
    Visible on the rule card so nothing silently de-dupes events you expected to fire.
  • VLM verify stage
    Drop a verify step into any rule chain. The VLM confirms the detection before the action fires. A swaying branch flagged as "person" gets rejected, the chain aborts, and the event records why it was suppressed. Reuses the frame cache, so re-verifying is free.
Why it matters
  • Rules read like English instead of like a SQL query.
  • The verify stage turns a raw detection guess into a VLM-confirmed fact before any notification fires.
  • Webhook + API call actions mean Nurby is a platform component, not a closed island.
  • Cooldowns are explicit, not buried in advanced settings.
07· Feature

Spatial events drawn directly on the live feed

Tripwires and loiter zones are drawn on top of the live camera feed, not in a separate diagram with abstract coordinates.

  • Inline canvas editor
    Points snap to the feed in real time so the geometry matches the operator's mental model.
  • Tripwire crossings
    Direction-aware polylines. "Crossed from inside to outside" is distinguishable from "outside to inside."
  • Loiter detection
    A polygon + a duration threshold. "Someone stood in front of the door for more than 45 seconds."
  • Per-zone tracking
    The tracker carries per-zone entry timestamps for each track, not just a flat in-zone boolean.
Why it matters
  • Geometry is set in the operator's mental model (the feed) rather than a separate editor.
  • Direction-aware crossings remove the false positives of "someone walked past a line."
08· Feature

Smart recording with retention policies

Four recording modes (off, always, on-motion, on-object, clip), with a pre/post buffer so the event sits in the middle.

  • Pre/post buffer
    Configurable seconds before and after the trigger. Rolling buffer finalises a clip the moment the trigger fires.
  • Trigger-specific recording
    Per-camera YOLO labels trigger on-object recording. Driveway records only on vehicles. Front door records only on people.
  • Retention policy
    Time-based (days) or size-based (gigabytes). Enforced automatically by a sweeper. Per camera.
Why it matters
  • Recording on objects, not motion, kills the "leaves blowing on the porch" tape.
  • Pre/post buffer means the event opens with context, not the trigger frame itself.
  • Storage caps are enforced for you.
09· Feature

Cross-camera journeys and incidents

A persistent identity-anchored story that spans cameras. One Journey for a front-door → hallway → kitchen walk, not three observations.

  • Idle window
    Configurable seconds without a matching detection before a journey closes. Used to be hardcoded; now a setting.
  • Incidents
    Server-side grouped observations with a stable ID, idle window, and a rolling summary. Configurable per camera.
  • Single-name display invariant
    Person display names are case-insensitively unique at the database level. Two people cannot share a display name because journey fusion would silently merge them. Enforced with a friendly 409.
Why it matters
  • The user gets one story per visit instead of fifteen disconnected motion alerts.
  • The cross-camera link survives camera handoffs.
  • The naming constraint stops invisible bugs before they happen.
10· Feature

Audio understanding

PANNs CNN14 turns the microphone into a 527-class semantic signal: baby cry, dog bark, glass break, smoke alarm, doorbell, knock, footsteps, speech, music, alarm clock.

  • Per-camera audio toggles
    Capture, transcription, raw storage, transcript-only redaction, summary-only redaction. Audio retention days and transcript retention days are independent.
  • STT budget per hour
    Per-provider per-camera minutes-per-hour cap so a hot mic never blows through your bill.
  • Conversation grouping
    Adjacent utterances within conversation_gap_seconds collapse into a Conversation. Long ones get their own VLM summary.
  • Conversation reinterpret
    Like every other AI artifact, a Conversation can be re-interpreted by any provider with full context.
Why it matters
  • Audio is not a binary "motion detected" flag; it's a 527-class semantic signal.
  • STT cost is bounded per camera, not a single global meter.
  • Audio and video events live in the same rule engine and the same timeline.
11· Feature

Natural-language search with pgvector

"Dog in the kitchen last night." Three results with thumbnails. The query embeds against observation description embeddings (Vector(384)) in pgvector.

  • Cross-modal scope
    Observations, journeys, incidents, conversations, transcripts and daily digests all carry embeddings. One search hits all of them.
  • Ask Nurby
    A natural-language answer panel sits next to the result list. Same query, but the VLM synthesises a response.
Why it matters
  • The user never has to learn a query syntax.
  • The same vector index powers search across every artifact type.
  • Search and Ask are two views on the same retrieval, not two unrelated features.
12· Feature

Household-wide daily digest and multi-day recaps

One configurable local hour fires build_daily_digest. Visitors, packages, vehicles, audio events, active cameras. Structured facts plus an LLM narrative. Need a week or a month? summarize_window chunks the range, builds deterministic mini-summaries per chunk at zero LLM cost, then runs one budget-gated LLM pass for the final synthesis.

  • Structured facts + LLM narrative
    The facts dictionary is persisted alongside the LLM summary. UI renders bullets from facts even when the LLM returns empty.
  • Week and month recaps
    summarize_window chunks any time range into deterministic mini-summaries (zero LLM cost), then one budget-gated LLM reduce produces the final narrative. Degrades gracefully with a partial summary if the token cap is hit.
  • Per-camera timezones
    Each camera carries an IANA timezone. A system timezone anchors the digest hour. Timestamps render in the camera's local time.
  • Provider-pickable
    Digest provider can differ from per-camera VLM provider.
Why it matters
  • One card in the morning instead of scrubbing last night's tape.
  • Week and month recaps stay within your token budget because the map step costs nothing. The LLM is spent only on the final synthesis.
  • The digest does not break when the LLM is offline; structured facts always render.
  • Timezones work even when the host is in UTC.
13· Feature

Timeline with filter sidebar

A collapsible filter panel keeps event cards in context. Filter by camera, label, person, time window, provider. The filter state lives in the URL so links are shareable.

Why it matters
  • Sharing a filtered timeline is a copy-paste, not a screenshot.
  • Returning to the same view across devices reuses the same URL.
14· Feature

People

Faces are a starting point, not the only signal. Unknown clusters surface for naming; body-only clusters can be linked or named in one click.

  • Auto-discovered face clusters
    Unknown faces cluster automatically. The UI surfaces them under "Who are these people?" with thumbnail grids.
  • Auto-labelled until named
    Each pending cluster gets an "Unknown 645" style label so it can be referenced before naming.
  • Appearance description
    A VLM pass produces a short demographic + clothing description per cluster ("woman in red jacket, dark jeans"). Useful in summaries that need to refer to an unnamed person.
  • Body-only suggestions
    Two paths: link an existing Person (the common case once you've named someone's face on another cam), or name a fresh Person from the body match alone.
  • Per-person recap prompt
    Each Person carries an optional recap prompt and a cached recap status. "Show me what Aisha did this week" runs the configured provider against her activity feed.
  • Star + consent flags
    Starred people surface to the top of the dashboard summary. Identities without consent are excluded from cross-camera matching against the known-person index.
Why it matters
  • Faces are a starting point, not the only signal.
  • Body-only clusters can be confirmed in one click via an existing Person, propagating identity backwards across cameras.
  • Each Person can have their own custom "what's their day been like" prompt.
15· Feature

Multi-user with role-based camera access

Invite keys, per-camera grants and supported SSO providers. The right family member sees the right cameras and nothing else.

  • Invite keys
    Single-use key with a role and a set of camera grants. Family members get only the cameras they should see.
  • Per-camera grants
    View-only on the baby cam, full control on the driveway. Granularity per camera, per action.
  • SSO / OAuth / passwordless welcome
    Link existing accounts via supported identity providers. Accounts are never created on behalf of the user.
Why it matters
  • Household access is not all-or-nothing.
  • Identity providers are linked, not impersonated.
16· Feature

Privacy controls baked in

Local inference, smart blur, pose-aware zones, and self-hosting. Privacy is where Nurby starts, not a setting you have to go find and switch on.

  • On-device inference via Ollama
    Frames never leave the box if you don't want them to.
  • NudeNet auto-blur
    Configurable minimum score, default 0.5. Default on.
  • Per-person privacy blur at render
    Per-Person flag flips face crops and observation thumbnails to blurred.
  • Smart privacy zones with PTZ pose awareness
    See section 03.
  • Self-hosted Docker Compose
    No data has to leave the network.
Why it matters
  • Self-host is supported, not penalised.
  • Privacy primitives chain together: blur, zones, per-person flag, on-device inference.
17· Feature

Developer platform

Rules are JSON. Inspectable, version-controllable, exportable. The reinterpret endpoint accepts any provider so models can be swapped from outside the UI.

  • API call action
    Fire arbitrary HTTP requests with a templated body. Twilio SMS, Home Assistant scenes, Slack webhooks, anything that speaks HTTP.
  • Webhook receiver
    Rules can also be triggered by inbound webhooks for two-way integrations.
  • Plain JSON rule payload
    Inspect, version-control, export, diff. Rules are not opaque blobs.
  • Reinterpret API
    Any artifact's reinterpret endpoint accepts a provider_id, so model swaps work from outside the UI too.
  • Per-camera ONVIF profile token
    Most cameras use Profile_1. Cameras that don't can override.
  • MCP server
    A read-only Model Context Protocol server re-exports all 10 read tools. Paste one config block into Claude Desktop and ask "who was at the door today?" Answered from your own cameras, with camera ACL respected and per-user budget counted. Behind a compose profile, opt-in.
Why it matters
  • Nurby is an SDK and a glue layer, not just an app.
  • The MCP server makes your cameras queryable from any AI app that supports the protocol. One config block, no custom integration.
  • Automation chains escape the UI without losing fidelity.
18· Feature

Integrations

RTSP, ONVIF (PTZ ContinuousMove, Stop, GetPresets, GotoPreset), MediaMTX for WebRTC publish, Ollama for local inference, OpenAI, Anthropic, Gemini, Twilio (via API call action), SMTP, generic webhooks. Phone-as-mic and network mic camera types for audio-only sources.

Why it matters
  • Talks to the gear and the services you already have.
  • Audio-only sources are first-class, not afterthoughts.
19· Feature

Theme and UX details

Dark-first with no-flash theme switching. Hover PTZ overlay. Live observation cards floating off tiles. Camera personas as one-click presets.

  • Dark-first, no-flash
    Theme is committed in the document before paint so there's no white flash on cold load.
  • Hover PTZ overlay
    Directional pad + zoom controls appear on camera tiles without a navigation step.
  • Live observation cards
    Float off the tiles when events fire, so the eye lands on the new thing first.
  • Notification bell with unread badge
    Plain old workhorse, working.
  • Camera personas
    One-click presets (Front Door, Baby Cam, Pet Cam, Wildlife, Driveway) bundle detection models, retention, summary mode and audio settings. Edit anything afterward; personas are starting points, not lock-ins.
Why it matters
  • Defaults are opinionated.
  • Power features stay discoverable without overwhelming first-time visitors.
20· Feature

Operational hygiene

Sticky failure flags, hardware autodetection, cluster decay, fusion sweeper, migrations on startup. The platform behaves itself.

  • Sticky load-failure flags
    If torch, torchreid, or InsightFace is missing, the feature disables itself once and stops retrying. No log spam.
  • MPS / CUDA / CPU auto-detection
    ReID and detection stacks pick the right backend without configuration. YOLO object detection runs on modest hardware. VLM inference requires a dedicated GPU with enough VRAM for the model you choose.
  • Cluster decay sweeper
    Tentative body clusters with no activity for 14 days are marked ignored.
  • Face-overlap fusion sweeper
    Body clusters get promoted to confirmed when a named face appears alongside them.
  • Migration on startup
    Alembic upgrade runs in the lifespan handler. Schema is always current.
Why it matters
  • The system fixes itself when it can.
  • Maintenance windows are smaller because hygiene is automated.

Self-host the whole stack. $0, forever.

Free and open. Local VLM inference needs a serious GPU. Think NVIDIA Spark or a workstation-class card with enough VRAM for a 200B-parameter model. Without one, the cloud providers work out of the box.

View on GitHub
Built with ♥ in Bangladesh.[email protected]