Research Notes: Databricks Big Book of Generative AI (SFT/DPO Trajectory Mapping)

This document records the key architectural patterns from Databricks' The Big Book of Generative AI and maps them directly to Sentinel Integrations' SIA (Self-Improving AI) Weights Optimization pipeline (sia/weights/).

📚 Core Lifecycle Stages

Production-quality Generative AI applications evolve across five distinct technical stages, each shifting the balance between cost, latency, and domain specialization:

1. Foundation Models: Selecting pre-trained base weights (e.g., Gemma-4, Qwen-3-Coder, Gemini, Claude).

2. Prompt Engineering: Formatting instructions, system directives, and inline few-shot examples.

3. Retrieval-Augmented Generation (RAG): Dynamically injecting external databases or knowledge vaults (e.g., via MCP) into context.

4. Fine-Tuning (SFT & DPO): Domain-specific training to permanently alter model behavior, tone, style, or structured API compliance.

5. Pretraining from Scratch: Capital-intensive, raw weight training on billions of tokens (typically reserved for sovereign foundation weights).

🔄 Mapping to SIA Weights Optimization (SIA-W)

To achieve a truly Self-Improving AI framework without runaway cloud API costs, we leverage Stage 4 (Fine-Tuning) on our zero-marginal-cost local hardware (Local Inference Node Mac Mini with M4 Pro GPU).

The goal of the sia/weights/ package is to parse autonomous trajectory execution logs into structured datasets to fine-tune our local models (Gemma/Qwen), converting raw run history into permanent reasoning capacity.

       [Raw Trajectory Run] (agent_execution/*.json)
               │
               ▼
   [Reward / Success Evaluation] (SIA Orchestrator)
               │
         ┌─────┴────────────────────────────────┐
         ▼                                      ▼
    [SUCCESS (Reward > Threshold)]       [FAILED / ROLLED-BACK]
         │                                      │
         ▼                                      ▼
  SFT Positive Target                    DPO Dispreferred Pair
         │                                      │
         ▼                                      ▼
  (Fine-tunes model on                 (Fine-tunes model to 
   successful execution paths)          avoid failed/risky paths)

1. Supervised Fine-Tuning (SFT) Trajectory Builder

• Databricks Pattern: Supervised fine-tuning teaches a general-purpose model a specific instruction-response style or API execution pattern by feeding it high-quality target demonstrations.
• SIA-W Translation:

* Input: Raw JSON trace of an execution branch that successfully resolved a task (e.g., resolved a failing pytest suite, safely modified a schema without regressions).

* Formatting: Convert the sequential run trace into a standard ChatML prompt-response sequence. The model's actions that led to the positive reward are saved as the "target response" for the system prompt input.

* Outcome: When trained on these targets, local models (Qwen-3-Coder) learn the specific tool-call sequence of Sentinel Integrations, reducing the need for mid-turn corrections.

2. Direct Preference Optimization (DPO) Trajectory Builder

• Databricks Pattern: DPO aligns models to human or systematic preferences by presenting pairs of completions for the same prompt—one flagged as preferred (chosen) and one as dispreferred (rejected).
• SIA-W Translation:

* Input: A scenario where the agent made an initial error (rejected/dispreferred trajectory) but successfully resolved it in a subsequent turn or rollback (chosen/preferred trajectory).

* Chosen Trajectory: The corrected, clean edit sequence that resolved the task successfully.

* Rejected Trajectory: The initial failed attempt (e.g., introduced a shell-injection vulnerability or nested event loop exception).

* Outcome: DPO training systematically penalizes the weights associated with the failed pattern, training local models to instinctively avoid common "footguns" and security vulnerabilities.

📈 Next-Step Deliverables for `sia/weights/`

1. Scaffold `./workspace/

* Implement a parser that scans `./workspace/ logs.

* Filter trajectories by reward metric (success/fail).

* Format outputs into standard Hugging Face SFT datasets ({"messages": [...]}) and DPO datasets ({"prompt": "...", "chosen": "...", "rejected": "..."}).

2. Local Fine-Tuning Infrastructure (Local Inference Node):

* Configure a lightweight training loop (using Hugging Face TRL or standard llama.cpp / MLX fine-tuning scripts on Apple Silicon) to consume the generated SFT/DPO datasets.