SI
Sentinel Integrations
← Back to Research Index

AI Social Scan (ai-social-scsn) Implementation Plan

> For Hermes: Use subagent-driven-development skill to implement this plan task-by-task.

Goal: Build a highly focused, sovereign social scanning, opportunity identification, and long-term trend tracking pipeline running locally on our Orchestrator Node (Intel NUC) and Local Inference Node (M4 Mac Mini) cluster, utilizing Ollama local sub-agents.

Architecture:

Tech Stack:

Database Schema Design (`src/db/schema.sql`)

-- Core social postings tracked over time
CREATE TABLE IF NOT EXISTS postings (
    id TEXT PRIMARY KEY,               -- Hash of URL/ID to prevent duplicates
    platform TEXT NOT NULL,            -- 'x', 'hn', 'reddit', 'rss'
    external_id TEXT,                  -- Platform-specific identifier
    url TEXT NOT NULL UNIQUE,          -- Direct link to post
    author TEXT NOT NULL,              -- Author username/handle
    title TEXT,                        -- Thread/Post title if applicable
    content TEXT NOT NULL,             -- Raw text content of the post
    published_at DATETIME,             -- Original publish time
    first_seen_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    last_seen_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    engagement_metrics TEXT            -- JSON block of likes, reposts, comments
);

-- AI Opportunity & Trend analysis
CREATE TABLE IF NOT EXISTS postings_analysis (
    posting_id TEXT PRIMARY KEY,
    summary TEXT NOT NULL,
    key_themes TEXT NOT NULL,          -- JSON list of strings
    opportunity_hypothesis TEXT,       -- Potential B2B/consulting angle
    relevance_score INTEGER NOT NULL,  -- 0 to 10 score on Sentinel target match
    sentiment TEXT,                    -- 'positive', 'neutral', 'negative'
    analyzed_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY(posting_id) REFERENCES postings(id) ON DELETE CASCADE
);

-- Ranking and Lifecycle state tracking
CREATE TABLE IF NOT EXISTS postings_lifecycle (
    posting_id TEXT PRIMARY KEY,
    current_state TEXT NOT NULL,       -- 'DISCOVERED', 'ANALYZING', 'HIGH_VALUE', 'DECAYING', 'ARCHIVED'
    base_score REAL NOT NULL,          -- Initial ranking score
    current_score REAL NOT NULL,       -- Base score multiplied by decay factor
    decay_factor REAL DEFAULT 1.0,     -- Dynamic decay based on age and category
    last_updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY(posting_id) REFERENCES postings(id) ON DELETE CASCADE
);

-- Actions taken by Sentinel/Otto
CREATE TABLE IF NOT EXISTS audit_trail (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    posting_id TEXT NOT NULL,
    action_type TEXT NOT NULL,         -- 'PROMOTED_TO_TELEGRAM', 'DISMISSED', 'ARCHIVED'
    performed_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    notes TEXT,
    FOREIGN KEY(posting_id) REFERENCES postings(id) ON DELETE CASCADE
);

Implementation Tasks

Task 1: Initialize Database and Schema

Objective: Create the SQLite database structure on Orchestrator Node.

- Create: src/ai_social_scsn/db/connection.py

- Create: src/ai_social_scsn/db/schema.sql (use schema above)

- Create: tests/test_db.py

Verify the database file is initialized and tables exist.

Run: pytest tests/test_db.py -v (fails, files do not exist)

Write Python code to setup `./workspace/ and run schema statements on initialization.

Run: pytest tests/test_db.py -v (passes)

git commit -m "feat(scsn): init SQLite database and schema"

Task 2: Crawling & Scraper Modules (HN, Reddit, Stack Overflow, Lobste.rs, and Meta Tech Feeds)

Objective: Build high-signal, zero-credentials scrapers utilizing public feeds and APIs. This bypasses rate-limiting/auth hurdles while capturing elite developer and tech chatter.

- Create: src/ai_social_scsn/scrapers/hn_search.py (Hacker News Algolia API)

- Create: src/ai_social_scsn/scrapers/reddit_rss.py (Reddit RSS feeds for target subreddits like r/artificial, r/localllama)

- Create: src/ai_social_scsn/scrapers/stackoverflow.py (StackExchange API for tagged questions like large-language-models, langchain)

- Create: src/ai_social_scsn/scrapers/lobsters.py (Lobste.rs JSON/RSS feeds for high-signal system engineering/AI discussions)

- Create: src/ai_social_scsn/scrapers/meta_tech.py (Public RSS feeds for Meta Developer Blog / Engineering updates)

- Create: tests/test_scrapers.py

Step 1: Write failing test

Verify scrapers can fetch and parse JSON/XML structures for all five source feeds.

def test_all_scrapers_parse_valid_structures():
    # Test assertions for HN, Reddit, StackOverflow, Lobsters, and Meta
    pass

Step 2: Run test to verify failure

Run: pytest tests/test_scrapers.py -v

Expected: FAIL (modules not found)

Step 3: Implement feed scraping

Step 4: Run test to verify pass

Run: pytest tests/test_scrapers.py -v

Expected: PASS

Step 5: Commit

git add src/ai_social_scsn/scrapers/ tests/test_scrapers.py
git commit -m "feat(scsn): implement expanded public scraping pipeline (HN, Reddit, SO, Lobsters, Meta)"

Task 3: Crawling Module (X Placeholder Driver)

Objective: Create a modular scraper for X (Twitter) using a stubbed driver designed for easy swapping with official API/scraping drivers once ready.

- Create: src/ai_social_scsn/scrapers/x_scraper.py

- Create: tests/test_x_scraper.py

Verify standard interface returns a mock list of high-value postings when no credentials exist, but validates the interface contract.

Write structured fallback code that logs "X Account Mode: Mature Phase Pending. Emulating signal tracking..." and returns a designated set of highly relevant simulated X posts for testing pipeline flow.

git commit -m "feat(scsn): add modular X scraper stub"

Task 4: AI Analysis & Opportunity Scoring Pipeline

Objective: Send new posts to Local Inference Node via Ollama client, extracting structured analyses and relevance scores using local reasoning models.

- Create: src/ai_social_scsn/analysis/pipeline.py

- Create: src/ai_social_scsn/analysis/prompts.py

- Create: tests/test_analysis.py

Verify JSON parser extracts opportunity hypothesis and Sentinel target match score from raw model output.

Write logic to call Ollama on Local Inference Node (qwen2.5-coder:7b or deepseek-r1:7b) with precise system instructions to output a clean JSON block matching our target schema fields.

git commit -m "feat(scsn): implement Ollama-driven opportunity extractor"

Task 5: Dynamic Ranking and Lifecycle Manager

Objective: Implement aging and decay logic to track the life cycle of high-value postings over months, updating scores daily.

- Create: src/ai_social_scsn/lifecycle/decay.py

- Create: tests/test_lifecycle.py

Verify a post's score decreases correctly based on age, moving from HIGH_VALUE to DECAYING and finally ARCHIVED.

Use exponential decay formula: current_score = base_score exp(-decay_constant days_old).

If a post is marked as HIGH_VALUE (e.g., Opportunity Score >= 8/10), give it a much slower decay rate (half-life of 14 days) than generic news (half-life of 3 days). Once current_score falls below threshold (e.g. 2.0), transition state to ARCHIVED.

git commit -m "feat(scsn): implement dynamic lifecycle and exponential decay engine"

Task 6: Telegram Notification Dispatcher & Daily Cron

Objective: Tie the crawling, analysis, lifecycle updating, and alerting together, sending promoted HIGH_VALUE opportunities straight to Telegram.

- Create: src/ai_social_scsn/alerts/telegram.py

- Create: src/ai_social_scsn/cli.py

- Create: cron/scsn_daily.sh

Verify promoted postings are parsed into highly polished, executive-grade alert layouts.

- Deliver payload: MEDIA or native telegram message showing Title, Platform, Author, Opportunity Hypothesis, and action suggestions.

- Setup a daily cron job that triggers crawling, scoring, decay processing, and alerts.

git commit -m "feat(scsn): add Telegram alert dispatcher and daily CLI entrypoint"