Skip to content
BranchesTags

Repository files navigation

Loop Engineering Blog Refinement System

Production-quality AI application that continuously improves technical blog posts until they meet a quality threshold.

Stack: Python 3.12 · Microsoft Agent Framework · Azure OpenAI · FastAPI · Pydantic · OpenTelemetry · Docker

This is Loop Engineering, not a one-shot prompt chain:

Draft → Evaluate → Diagnose → Improve → Evaluate Again → Stop OR Continue → Human Approval (if needed)

Table of Contents

  1. System Architecture
  2. Component Architecture
  3. Loop Engineering Flow
  4. State Architecture
  5. Data & Storage Architecture
  6. Agent Architecture
  7. API Architecture
  8. Observability Architecture
  9. Deployment Architecture
  10. Project Structure
  11. Setup
  12. Configuration
  13. Future Improvements

System Architecture

Continuous improvement through observation, verification & feedback:

Loop Engineering Architecture

Flow: INPUT → PLAN → EXECUTE → OBSERVE → VERIFY → UPDATE PLAN → REPEAT until success.

Stage Purpose
1. Input Bound the goal: topic, constraints, target score, max iterations
2. Plan Break down the task, choose strategy and tools
3. Execute Agent writes or modifies content with tools + context
4. Observe Collect outputs, metrics, tool results, side effects
5. Evidence Gate Accept / Retry / Escalate based on what the result proved
6. Verify Deterministic checks + LLM evaluation → score, issues, recommendations
7. Update Plan Incorporate learnings; the next attempt must change the plan
8. Persistence & Observability Versions, memory, traces, storage — deployable local or Azure

Loop stops when: score ≥ target · max iterations reached · no improvement > 2 points for two consecutive iterations.

Application request flow

High-level view of how requests flow through the system:

                         ┌─────────────────────────┐
                         │        Clients          │
                         │  curl · Postman · UI    │
                         └────────────┬────────────┘
                                      │ HTTP/JSON
                                      ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         FastAPI Application                             │
│  POST /blog · GET /status/{id} · GET /history/{id} · GET /blog/{id}     │
│  POST /blog/{id}/rollback · POST /blog/{id}/approve · GET /health       │
└────────────────────────────────┬────────────────────────────────────────┘
                                 │
                                 ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                    BlogRefinementWorkflow (Loop Engine)                 │
│                                                                         │
│   ┌──────────────────┐  ┌────────────────────┐  ┌────────────────────┐  │
│   │ TerminationPolicy│  │ Version Manager    │  │ Human Approval Gate│  │
│   │ score / max /    │  │ v0..vN + rollback  │  │ plateau / budget   │  │
│   │ plateau rules    │  │                    │  │                    │  │
│   └──────────────────┘  └────────────────────┘  └────────────────────┘  │
└───────┬──────────────────────────┬──────────────────────────┬───────────┘
        │                          │                          │
        ▼                          ▼                          ▼
┌───────────────┐        ┌─────────────────┐        ┌─────────────────┐
│ WriterAgent   │        │ EvaluatorAgent  │        │ RefinerAgent    │
│ (optional     │        │ structured      │        │ improved blog + │
│  drafting)    │        │ quality JSON    │        │ change summary  │
└───────┬───────┘        └────────┬────────┘        └────────┬────────┘
        │                         │                          │
        └────────────┬────────────┴──────────────┬───────────┘
                     ▼                           ▼
        ┌────────────────────────┐   ┌─────────────────────────┐
        │  LLM Service Port      │   │  Deterministic Tools    │
        │  Agent Framework       │   │  grammar · readability  │
        │  Azure OpenAI  OR      │   │  keywords · headings    │
        │  Heuristic (local/CI)  │   │  markdown · duplicates  │
        └────────────┬───────────┘   └────────────┬────────────┘
                     │                            │
        ┌────────────┴───────────┐   ┌────────────┴────────────┐
        ▼                        ▼   ▼                         ▼
┌───────────────┐      ┌──────────────┐      ┌──────────────────────────┐
│ Persistence   │      │ Blob Storage │      │ Telemetry                │
│ SQLite  ⇄     │      │ Local FS  ⇄  │      │ OpenTelemetry spans      │
│ Cosmos DB     │      │ Azure Blob   │      │ App Insights compatible  │
└───────────────┘      └──────────────┘      └──────────────────────────┘

Design principles

Principle How it is applied
Loop Engineering Shared state + evaluate/refine cycle + explicit stop conditions
Clean Architecture API → Workflow → Agents/Tools → Adapters
SOLID One responsibility per agent/tool/repository; ports for LLM/storage/DB
Azure-ready Local adapters swap to Cosmos + Blob with env config only
No infinite loops Target score, max iterations, plateau detection, hard safety cap

Component Architecture

flowchart TB
    subgraph API["API Layer"]
        R[routes.py]
        M[main.py]
    end

    subgraph Domain["Domain / Orchestration"]
        W[workflow.py]
        TP[LoopTerminationPolicy]
        S[WorkflowState]
    end

    subgraph Agents["Agents"]
        WA[WriterAgent]
        EA[EvaluatorAgent]
        RA[RefinerAgent]
    end

    subgraph Tools["Deterministic Tools"]
        G[grammar]
        RD[readability]
        K[keyword_checker]
        SC[scoring]
        ST[storage]
    end

    subgraph Services["Infrastructure Services"]
        LLM[llm.py]
        P[persistence.py]
        T[telemetry.py]
    end

    subgraph Models["Contracts"]
        EM[evaluation.py]
        RM[refinement.py]
        SM[state.py]
    end

    M --> R
    R --> W
    W --> TP
    W --> S
    W --> EA
    W --> RA
    W --> WA
    EA --> LLM
    RA --> LLM
    WA --> LLM
    W --> SC
    SC --> G
    SC --> RD
    SC --> K
    W --> P
    W --> ST
    W --> T
    EA --> EM
    RA --> RM
    S --> SM
Loading

Layer responsibilities

Layer Path Responsibility
API app/api/ HTTP transport, request/response validation
Entrypoint app/main.py App factory, lifespan, DI wiring
Workflow app/workflow.py Loop orchestration, termination, versioning
Agents app/agents/ Single-purpose Agent Framework agents
Tools app/tools/ Non-LLM quality checks + blob storage port
Services app/services/ LLM, persistence, telemetry adapters
Models app/models/ Pydantic contracts for state and agent I/O
Prompts app/prompts/ Markdown prompts loaded at runtime
Config app/config.py Environment-driven settings

Loop Engineering Flow

Control loop

flowchart TD
    A[Load Blog → Version 0] --> B[Run Deterministic Checks]
    B --> C[LLM Evaluation → Structured JSON]
    C --> D[Blend Score = 70% LLM + 30% Deterministic]
    D --> E{Score ≥ Target?}
    E -->|Yes| F[Auto-Approve]
    E -->|No| G{Max iterations reached?}
    G -->|Yes| H[Await Human Approval]
    G -->|No| I{Plateau: Δ ≤ 2 for 2 consecutive iterations?}
    I -->|Yes| H
    I -->|No| J[Diagnose Issues + Recommendations]
    J --> K[Refiner Agent Improves Blog]
    K --> L[Persist Next Version]
    L --> B
Loading

Termination rules

The loop must stop when any condition is true:

Condition Default Outcome
Score ≥ target 85 approved
Max iterations reached 5 evaluations max_iterations_reached + human approval
Plateau improvement ≤ 2 points for 2 consecutive iterations plateau_reached + human approval
Hard safety cap max_iterations * 2 Forced stop (never infinite)

Sequence (one iteration)

sequenceDiagram
    participant API as FastAPI
    participant WF as Workflow
    participant Tools as Deterministic Tools
    participant Eval as EvaluatorAgent
    participant LLM as Agent Framework / Azure OpenAI
    participant Ref as RefinerAgent
    participant DB as Persistence
    participant Blob as Storage

    API->>WF: start(content)
    WF->>Blob: save version 0
    WF->>DB: save state

    loop Until stop condition
        WF->>Tools: compute_deterministic_metrics(content)
        Tools-->>WF: DeterministicMetrics
        WF->>Eval: evaluate(content, metrics)
        Eval->>LLM: structured EvaluationResult
        LLM-->>Eval: scores + issues + recommendations
        Eval-->>WF: EvaluationResult
        WF->>DB: record score + evaluation

        alt Target / Max / Plateau
            WF-->>API: terminal status
        else Continue
            WF->>Ref: refine(content, evaluation, history)
            Ref->>LLM: structured RefinementResult
            LLM-->>Ref: improved_content + changes
            Ref-->>WF: RefinementResult
            WF->>Blob: save next version
            WF->>DB: update state
        end
    end
Loading

State Architecture

One shared WorkflowState object drives the loop:

stateDiagram-v2
    [*] --> pending
    pending --> running: POST /blog
    running --> approved: score ≥ target
    running --> plateau_reached: quality no longer improves
    running --> max_iterations_reached: iteration budget exhausted
    running --> failed: unhandled exception
    plateau_reached --> approved: POST /approve
    max_iterations_reached --> approved: POST /approve
    approved --> rolled_back: POST /rollback
    rolled_back --> awaiting_human_approval
    awaiting_human_approval --> approved: POST /approve
Loading

State fields

WorkflowState
├── document_id
├── original_content
├── current_content
├── iteration / maximum_iterations / target_score
├── score_history[]
├── evaluation_history[]
├── refinement_history[]
├── versions[]          # v0 original, v1..vN revisions, rollbacks
├── status / approved / termination_reason
├── human_review_required
└── timestamps (created / updated / started / completed)

Version history

Version Meaning
v0 Original draft
v1..vN Each refinement
rollback Restored snapshot of a prior version (new version entry)

Data & Storage Architecture

Adapters hide local vs Azure backends behind interfaces:

                    ┌────────────────────────┐
                    │   WorkflowRepository   │  (port)
                    └───────────┬────────────┘
                     ┌──────────┴──────────┐
                     ▼                     ▼
            ┌────────────────┐    ┌─────────────────┐
            │ SQLite (local) │    │ Cosmos DB Azure │
            └────────────────┘    └─────────────────┘

                    ┌────────────────────────┐
                    │    DocumentStorage     │  (port)
                    └───────────┬────────────┘
                     ┌──────────┴──────────┐
                     ▼                     ▼
            ┌────────────────┐    ┌─────────────────┐
            │ Local filesystem│    │ Azure Blob      │
            └────────────────┘    └─────────────────┘
Concern Local Azure
Workflow state SQLite (aiosqlite) Cosmos DB
Document versions data/storage/{id}/vN.md Blob container
Switch PERSISTENCE_BACKEND=sqlite PERSISTENCE_BACKEND=cosmos
Switch STORAGE_BACKEND=local STORAGE_BACKEND=azure_blob

Agent Architecture

Agents use Microsoft Agent Framework (Agent + Azure-routed OpenAIChatClient) with Pydantic structured outputs.

prompts/*.md  ──load──►  Agent(instructions, response_format=Model)
                                │
                                ▼
                         agent.run(user_input)
                                │
                                ▼
                         AgentResponse.value  →  validated Pydantic model
Agent Input Output Prompt
Writer Topic / requirements WriterResult prompts/writer.md
Evaluator Blog + deterministic metrics EvaluationResult prompts/evaluator.md
Refiner Blog + evaluation + score history RefinementResult prompts/refiner.md

Evaluator JSON contract

{
  "overall_score": 76,
  "clarity": 80,
  "accuracy": 70,
  "structure": 74,
  "readability": 82,
  "originality": 71,
  "technical_depth": 77,
  "issues": ["..."],
  "recommendations": ["..."]
}

Deterministic tools (no LLMs)

Tool Purpose
Grammar score Typos, repeated words, punctuation heuristics
Readability Flesch-based score, word count, reading time
Keyword checker Required-term coverage / stuffing penalty
Heading structure H1/H2 hierarchy and skip detection
Markdown validation Fences, empty headings, malformed links
Duplicate paragraphs Near-duplicate block detection

Blended publication score: 70% LLM + 30% deterministic.

For local/CI without Azure credentials, LLM_MODE=heuristic simulates evaluate/refine dynamics so the full loop remains testable.

API Architecture

Client
  │
  ├─ POST   /blog                      Start loop (sync or async_mode)
  ├─ GET    /status/{document_id}      Workflow status + score history
  ├─ GET    /history/{document_id}     Versions + evaluation/refinement history
  ├─ GET    /blog/{document_id}        Latest content
  ├─ POST   /blog/{document_id}/rollback
  ├─ POST   /blog/{document_id}/approve
  └─ GET    /health

OpenAPI docs: http://localhost:8000/docs

Observability Architecture

Workflow / Agents / LLM / Tools
            │
            ▼
     OpenTelemetry TracerProvider
            │
     ┌──────┼──────────────────┐
     ▼      ▼                  ▼
 Console   OTLP endpoint    Azure Monitor
 (local)   (collector)      Application Insights

Traced / recorded signals:

  • LLM calls (agent name, model, latency, tokens)
  • Iteration count and score deltas
  • Prompt/agent execution
  • Failures and exceptions
  • Termination reason

Deployment Architecture

Local                          Azure
─────                          ─────
Docker Compose                 Container Apps / App Service
  └─ FastAPI API                 └─ FastAPI container
SQLite                         Cosmos DB
Local filesystem               Azure Blob Storage
Heuristic or Azure OpenAI      Azure OpenAI
Console / OTLP                 Application Insights

Minimal Azure cutover:

  1. LLM_MODE=azure + Azure OpenAI settings
  2. PERSISTENCE_BACKEND=cosmos + Cosmos credentials
  3. STORAGE_BACKEND=azure_blob + storage connection string
  4. APPLICATIONINSIGHTS_CONNECTION_STRING=...

Project Structure

loop-engineering/
├── app/
│   ├── main.py                 # FastAPI entrypoint
│   ├── config.py               # Settings from .env
│   ├── workflow.py             # Loop engine + termination policy
│   ├── state.py                # State re-exports
│   ├── prompts/
│   │   ├── writer.md
│   │   ├── evaluator.md
│   │   └── refiner.md
│   ├── agents/
│   │   ├── writer.py
│   │   ├── evaluator.py
│   │   └── refiner.py
│   ├── tools/
│   │   ├── grammar.py
│   │   ├── readability.py
│   │   ├── keyword_checker.py
│   │   ├── scoring.py
│   │   └── storage.py
│   ├── models/
│   │   ├── evaluation.py
│   │   ├── refinement.py
│   │   └── state.py
│   ├── services/
│   │   ├── llm.py
│   │   ├── telemetry.py
│   │   └── persistence.py
│   ├── api/
│   │   └── routes.py
│   └── utils/
│       ├── logger.py
│       └── prompts.py
├── docs/
│   └── loop-engineering-architecture.png
├── tests/
├── sample_blogs/
├── scripts/
├── postman/
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── pyproject.toml
└── README.md

Setup

Prerequisites

  • Python 3.12+
  • uv (recommended) or pip
  • Docker (optional)
  • Azure OpenAI (optional; heuristic mode works offline)

Install & run

cd loop-engineering
cp .env.example .env
uv python install 3.12
uv venv
uv pip install -r requirements.txt
uv run uvicorn app.main:app --reload --port 8000

Tests

LLM_MODE=heuristic ENABLE_TELEMETRY=false uv run pytest -q

Docker

cp .env.example .env
docker compose up --build

Seed + examples

uv run python scripts/seed_data.py
./scripts/example_requests.sh
# Import: postman/Loop_Engineering.postman_collection.json

Example request

curl -X POST http://localhost:8000/blog \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Rough Azure Agents",
    "keywords": ["azure", "agent", "observability"],
    "target_score": 85,
    "maximum_iterations": 4,
    "content": "# Draft\n\nteh enviroment needs work..."
  }'

Configuration

Variable Description Default
AZURE_OPENAI_ENDPOINT Azure OpenAI endpoint empty
AZURE_OPENAI_KEY API key (AZURE_OPENAI_API_KEY also accepted) empty
AZURE_OPENAI_MODEL Deployment name gpt-4o
TARGET_SCORE Approval threshold 85
MAX_ITERATIONS Evaluation budget 5
LLM_MODE heuristic or azure heuristic
PERSISTENCE_BACKEND sqlite or cosmos sqlite
STORAGE_BACKEND local or azure_blob local
APPLICATIONINSIGHTS_CONNECTION_STRING Azure Monitor empty

Azure OpenAI mode:

LLM_MODE=azure
AZURE_OPENAI_ENDPOINT=https://YOUR_RESOURCE.openai.azure.com/
AZURE_OPENAI_KEY=...
AZURE_OPENAI_MODEL=gpt-4o

Future Improvements

  • Magentic / group-chat multi-agent critique via Agent Framework orchestrations
  • Content safety and groundedness evaluators
  • Streaming iteration events (SSE / WebSockets)
  • Managed Identity for Azure OpenAI, Cosmos, and Blob
  • Policy-as-code termination rules
  • CI publish to Azure Container Registry

License

MIT

About

LoopForge is an AI-powered engineering framework for designing, testing, and optimizing reliable feedback loops, autonomous agents, and continuous improvement workflows

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages