SI
Sentinel Integrations
← Back to Research Index

RUNBOOK: LLaMA FACTORY INSTALLATION AND FINE-TUNING ON MIRA (macOS MPS)

Document ID: SI-RB-2026-LF1

Target Host: Local Inference Node (Apple M4 Pro Mac mini, 24GB Unified Memory)

Operating System: macOS (Apple Silicon)

Security Context: macOS socketfilterfw active (Firewall enabled)

Ecosystem Standard: Integrates with local dataset outputs from parse_session_trajectories.py.

1. OBJECTIVE AND SYSTEM DESIGN

The goal is to establish a high-integrity, completely local fine-tuning pipeline on Local Inference Node to train open-weights models (such as Gemma-2-9B-It or Qwen-2.5-7B-Instruct) on custom trajectory logs generated by Sentinel's agent runs.

LLaMA Factory provides a unified, zero-code CLI and Gradio-based Web UI that manages SFT (Supervised Fine-Tuning), DPO (Direct Preference Optimization), and LoRA (Low-Rank Adaptation) workflows. Because Local Inference Node leverages Apple Silicon, we must configure the installation to utilize PyTorch MPS (Metal Performance Shaders) for GPU-accelerated training.

2. PRE-INSTALLATION SYSTEM CHECKS

Execute these check commands on Local Inference Node before proceeding to verify the baseline development environment:

1. Verify Xcode Command Line Tools:

`bash

xcode-select -p || xcode-select --install

`

2. Verify Python 3 Environment:

Ensure Python 3.10 or 3.11 is active. (Python 3.12 is supported but 3.10/3.11 provides broader compiled wheel compatibility for machine learning libraries).

`bash

python3 --version

`

3. Verify Git LFS (for Hugging Face model downloads):

`bash

brew install git-lfs

git lfs install

`

3. STEP-BY-STEP BARE-IRON INSTALLATION

Navigate to your training workspace on Local Inference Node (e.g., ~/sia_train/) and perform the installation:

Step 3.1: Clone LLaMA Factory Repository

cd ~/sia_train
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory

Step 3.2: Configure the Virtual Environment

Activate your existing training venv (~/sia_train_env/) or create a clean, isolated environment:

# If creating fresh:
python3 -m venv ~/sia_train_env
source ~/sia_train_env/activate

Step 3.3: Install Apple Silicon PyTorch (MPS)

PyTorch requires native Metal acceleration. Install the stable macOS nightly or release build:

pip install --upgrade pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

Note: PyTorch automatically includes MPS acceleration in the standard macOS CPU/Unified Memory wheel package. Verify MPS availability in Python:

python3 -c "import torch; print('MPS Available:', torch.backends.mps.is_available())"
# Expected output: MPS Available: True

Step 3.4: Install LLaMA Factory Dependencies

Install the package in editable mode with metrics and optional components:

pip install -e .[metrics,bitsandbytes]

Step 3.5: MacOS Compatibility Pitfall - bitsandbytes & Quantization

4. DATASET INTEGRATION

To integrate your custom compiled SFT datasets (e.g., local_session_dataset.json compiled by parse_session_trajectories.py on Orchestrator Node):

Step 4.1: Copy Dataset to Local Inference Node

Transfer the JSON file from Orchestrator Node to Local Inference Node over your Tailscale mesh:

scp ./workspace/ mira:~/sia_train/LLaMA-Factory/data/local_session_dataset.json

Step 4.2: Register Dataset in dataset_info.json

LLaMA Factory indexes all available datasets via data/dataset_info.json. You must append your custom file configuration:

1. Open data/dataset_info.json in your editor.

2. Add this entry to the dictionary:

  "local_session_sft": {
    "file_name": "local_session_dataset.json",
    "formatting": "sharegpt",
    "columns": {
      "messages": "conversations"
    }
  }

5. LAUNCHING THE PIPELINE

Step 5.1: Launching the Gradio Web UI

To spin up the interactive training dashboard:

llamafactory-cli webui

Step 5.2: CLI Training Command (Alternative to Web UI)

If executing the training loop natively in the terminal to bypass browser memory overhead:

1. Create a configuration file named gemma_mps_lora_sft.yaml under examples/lora_single_gpu/:

### model
model_name_or_path: google/gemma-2-9b-it

### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: all

### dataset
dataset: local_session_sft
template: gemma
cutoff_len: 2048
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 4

### output
output_dir: saves/gemma-2-9b/lora/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 4
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
fp16: false
bf16: false
device: mps

2. Run the training CLI command:

llamafactory-cli train examples/lora_single_gpu/gemma_mps_lora_sft.yaml

6. MODEL EXPORT & OLLAMA PACKAGING

Once SFT is complete and the adapter weights are saved under saves/gemma-2-9b/lora/sft:

Step 6.1: Merge and Export Model

To merge the trained LoRA adapters back into the base FP16 model:

llamafactory-cli export examples/lora_single_gpu/gemma_mps_lora_sft.yaml \
    --export_dir models/gemma-2-9b-sft-merged \
    --export_size 2 \
    --export_device cpu \
    --export_legacy_format false

Step 6.2: Quantize to GGUF

Using llama.cpp tools on Local Inference Node, quantize the merged FP16 model to Q4_K_M or Q8_0 GGUF:

python3 ~/llama.cpp/convert_hf_to_gguf.py models/gemma-2-9b-sft-merged/ \
    --outtype q8_0 \
    --outfile models/gemma-2-9b-sft-q8.gguf

Step 6.3: Import into Ollama

Create a custom Modelfile under ~/sia_train/:

FROM ./LLaMA-Factory/models/gemma-2-9b-sft-q8.gguf
TEMPLATE "{{ if .System }}<|im_start|>system\n{{ .System }}<|im_end|>\n{{ end }}{{ if .Prompt }}<|im_start|>user\n{{ .Prompt }}<|im_end|>\n{{ end }}<|im_start|>assistant\n{{ .Response }}<|im_end|>\n"
PARAMETER stop "<|im_end|>"

Create and run the model natively in Ollama on Local Inference Node:

ollama create gemma-2-9b-sft -f ./Modelfile
ollama run gemma-2-9b-sft