> Initializing Session...
Vcentre
Competitor Scan • FastAPI • Groq Fallbacks
Description
Vcentre scrapes competitor Instagram accounts nightly and scores Reels and Photos as separate cohorts, so one viral Reel can't skew the baseline for an account's regular photos. A 12-hour cooldown and a circuit breaker guard every scrape, and only posts clearing a 3x-median threshold, an absolute floor, and a minimum engagement rate route through a 10-provider LLM fallback chain — Groq and Cerebras handle the bulk of the analysis for free, Gemini only synthesizes the final brief. Briefs write straight into the same database the posting bots read from, and a closed-loop feedback job scores each post's real performance back against the pattern that produced it.
Timeline
10 Days
Role
Solo developer
Status
Production
Year
2026
Deployment
Daily scan @ 2AM UTC
Core Stack
FastAPI / SQLite
Why I Built It
Understanding the core problems, proposed solutions, and realized product achievements.
Scraping every competitor post nightly buries the handful of genuinely viral posts inside routine volume — a big account's normal Tuesday photo outscores a small account's true breakout in raw likes, and one 500k-view Reel sitting in the same list as an account's usual 2k-like photos would drag any shared median wildly upward, making every other post at that account look artificially unremarkable.
GitHub Actions cron fires main.py at 2AM UTC — config.load() validates niches.yaml/agents.yaml and key_ping.py pings all 10 providers before anything else runs.
db.set_pipeline_start() anchors the Gemini daily budget window, then OurPostsSync pulls the posting bots' own recent post history so the contrast engine isn't running blind.
scraper.scrape_batch() fires one parallel Apify call across every competitor + owned-account handle — but only for handles clear of the 12-hour cooldown and under the 2-hit/24h circuit breaker.
The Apify dataset resolves; posts are matched back to handles by ownerUsername, keyword-filtered per niche, and upserted into competitor_posts.
outlier.py splits each handle's posts into Reels vs. Photos cohorts, computes the rolling median per cohort, and flags a post only if it clears 3x median AND an absolute floor AND a 1% engagement rate.
Flagged outliers route through router.run() for vision audit, caption audit, hook scoring, and strategy contrast — Groq first, then Cerebras/NVIDIA/GitHub Models, with Gemini held back for the final step.
Gemini synthesizes the structured JSON brief only if the daily call budget has room left; the brief writes into creative_briefs, the same table the posting bots read from directly.
HealthChecker scans for zero-post niches, zero-outlier runs, and 14-day zero-brief streaks, then prune_old_posts() trims anything older than 60 days.
Batch every handle into one parallel Apify call
scraper.pyscrape_batch() sends all competitor + owned-account profile URLs into a single Apify actor run instead of one call per handle, matching results back by ownerUsername — cutting API calls from N down to 1 per pipeline run.
Gate every scrape behind a cooldown and a circuit breaker
scraper.py_is_cooldown_active() skips any handle successfully scraped in the last 12 hours; _is_circuit_breaker_active() kills a handle after 2 attempts in a rolling 24h window — both checked before a single Apify credit is spent.
Isolate cohorts before a median is ever computed
outlier.pydetect() splits posts into is_reel==1 vs is_reel==0 before statistics.median() runs, so a Reel's view count never enters a Photo cohort's baseline or vice versa.
Require three signals to agree before calling something viral
outlier.pyis_outlier only trips when a post clears the median multiplier, an absolute floor (3,000 views / 500 likes), and a minimum 1% composite engagement rate — a raw view spike alone isn't enough.
Route every AI task through a 10-provider fallback chain
router.pyAgentRouter.run() tries the task's preferred provider first, then walks a fixed fallback order across 10 providers, placing any rate-limited agent on a 5-minute cooldown instead of retrying it immediately.
Feed real post performance back into pattern scoring
db.pyprocess_closed_loop_feedback() compares each newly-synced post's real likes against a 30-day baseline and updates the viral_patterns table's success weight, so patterns that under-deliver lose priority automatically.
Nightly runs scan every configured niche, isolate true format-relative outliers instead of raw volume peaks, and queue structured creative briefs directly into the posting bots' database — with the whole pipeline, from scraping to brief synthesis, running on free-tier AI credits and a $0/month infrastructure bill.
- Monthly infra cost
- $0
- AI fallback depth
- 10 providers
- Outlier confirmation gates
- 3 (median + floor + ER)
- Scrape credit guards
- 2 (cooldown + circuit breaker)
Architecture Pipelines
Visual system flow diagrams representing data pipelines and structured checks.
core/scraper.py
Batches every competitor + owned handle into one parallel Apify actor call, gated by a 12-hour per-handle cooldown and a 2-hit/24h circuit breaker so credits are never spent on an account just checked.
core/outlier.py
Splits each handle's posts into Reels vs. Photos cohorts, computes a rolling median per cohort, and flags a post viral only when it clears the median multiplier, an absolute floor, and a minimum engagement rate together.
core/router.py
Routes every AI task — vision audits, caption audits, hook scoring, final synthesis — through a per-task preferred provider first, then a shared 10-provider fallback order with 5-minute rate-limit cooldowns.
core/brief.py
Runs vision, caption, hook, and strategy-contrast audits per outlier, then synthesizes everything into one structured JSON creative brief via Gemini, reserved for the final call to stay inside its free-tier budget.
core/pattern_engine.py
Clusters outliers into hook, caption, and visual-theme patterns, deduplicating via Supabase vector similarity search with an LLM semantic-matching fallback at a strict 0.85 confidence threshold.
core/health.py
Runs five layered checks after every run — zero posts, low scrape yield, zero outliers, budget-aware zero briefs, and a 14-day zero-brief streak — before writing a GitHub Actions summary and optional Discord alert.
Features & Capabilities
Key features and built-in components implemented inside the codebase.
- ✓Format-isolated median outlier detection (Reels vs. Photos scored separately)
- ✓Composite viral confirmation (median multiplier + absolute floor + engagement-rate gate)
- ✓12-hour cooldown + 2-hit/24h circuit breaker to protect scrape credits
- ✓Single parallel Apify batch call across every competitor + owned account
- ✓10-provider AI fallback chain with per-task routing and rate-limit cooldowns
- ✓Semantic pattern deduplication via vector similarity + LLM tiebreaker fallback
- ✓Closed-loop feedback scoring real post performance back into pattern weights
- ✓Layered health diagnostics distinguishing quiet days from silent failures
Solving Complex Bottlenecks
A deep dive into security rule architectures, race conditions, and verification solutions.
A single 500k-view Reel sitting in the same post list as an account's usual 2k-like photos would drag any shared median or average wildly upward, making every other post at that account look artificially unremarkable.
Reels and photos measure success on totally different scales — views vs. likes — and mixing them into one pooled median means the format with naturally bigger numbers can swamp the signal from the other.
Split posts by is_reel flag before any statistics call
outlier.pyUse views as the Reel metric, likes as the Photo metric — never mix them
Require a minimum cohort size of 3 valid posts before computing a median at all
"Median resistance to outliers only holds if it's computed over a genuinely comparable population — pooling two different measurement scales into one median just moves the distortion instead of removing it."
Development Journey
A chronological day-by-day log detailing core milestones and features added.
Skeleton, Config & Schema
config.py's lazy singleton loader, the niches.yaml/agents.yaml schema, and the initial SQLite schema (migrations.py) stood up before any scraping code existed.
Apify Scraper + Rate Limit Gates
scraper.py built out against Apify's instagram-scraper actor, with the 12-hour cooldown and 2-hit circuit breaker wired in from the start to protect the credit budget.
Format-Isolated Outlier Engine
outlier.py's cohort-splitting median logic and composite engagement-rate gate replaced an earlier single-pooled-average draft that kept flagging big accounts' routine posts as outliers.
10-Provider Router + Brief Generator
router.py's fallback order and brief.py's vision/caption/hook/contrast pipeline shipped together, synthesizing into one structured JSON brief via Gemini.
Pattern Engine, Closed-Loop Feedback & Health Checks
pattern_engine.py's semantic pattern matching, db.py's closed-loop performance scoring, and health.py's layered diagnostics finished before enabling the daily 2AM UTC cron.
Operational Metrics
No public-facing site to audit — these are the safeguards that keep an unattended pipeline honest.
minimum gap between successful scrapes of the same handle in the same niche, checked before any Apify credit is spent
a handle stops being scraped mid-day once it's already been attempted twice, protecting the credit budget from a misbehaving niche config
median multiplier, absolute floor, and minimum engagement rate must all pass together before a post is flagged viral
Groq → Cerebras → NVIDIA → OpenRouter → GitHub Models → Cohere → SambaNova → Mistral → Gemini → HuggingFace, with rate-limited agents cooled down for 5 minutes instead of retried
Trade-off Explanations
Concise rationales for technology selections and future roadmaps.
Reliable Scraping Without Maintaining Cookies
Apify (apify-client)Offloads the actual scraping to a managed actor instead of hand-rolling Instagram session/cookie management, trading a small per-run credit cost for scrapes that don't break every time Instagram tweaks its anti-bot detection.
Free, Fast Inference for the Bulk of the Analysis
Groq + CerebrasVision audits, caption audits, and hook scoring all run on Groq/Cerebras's free tiers first — Gemini is only called once per brief, right at the final synthesis step, to stay inside its daily call budget.
Local-First With an Optional Sync Layer
SQLite + SupabaseSQLite is the source of truth for a single run; Supabase sync (profiles, patterns, competitor posts) layers cross-run and cross-niche persistence on top without requiring a database server to stay up.
A Thin Control Layer, Not the Pipeline Itself
FastAPI dashboarddashboard_server.py exposes read/trigger endpoints over the same SQLite data the cron job writes — the dashboard observes and can kick off runs, but the pipeline logic itself lives entirely in schedulers/daily.py.
"Evaluating scaling adjustments and cache controls shows true engineering maturity."
- ✓Move the semantic pattern-matching fallback off a full LLM call and onto a cheaper local embedding comparison first, only escalating to the LLM matcher when cosine similarity is genuinely ambiguous.
- ✓Make the circuit breaker's 2-hit/24h window niche-configurable instead of hardcoded, since some niches have far tighter Apify budgets than others.
- ✓Add TikTok as a second scrape source so outlier detection isn't blind to a competitor's cross-platform virality.
- ✓Surface closed-loop feedback scores directly on the dashboard so a declining pattern's success rate is visible before it's automatically retired.
Visual Gallery
Scrollable preview tracks showcasing panel responsive structures.
Finished exploring this session?
End the workspace session to release resources and return back to the primary portfolio index.