Skip to content
QuadKit Python Framework

AI Integration

Quadkit ships a modular AI stack built on the same contract-first foundation as the rest of the framework. You program against protocols (LLMClientProtocol, RAGPipelineProtocol, …), so providers and models are swappable through configuration alone.

The AI layer is composed of focused, independently installable packages:

PackagePurpose
an AI extensionOrchestration layer — discovers and wires the AI subsystems below
quadkit-ai-llmMulti-provider LLM client (OpenAI, Anthropic, Gemini, Ollama, Groq, Mistral, …)
a RAG extensionRetrieval-augmented generation pipeline
a vector-store extensionVector store backends (pgvector, Qdrant, Pinecone, in-memory)
quadkit-ai-agentsAgents with tools and strategies (ReAct, plan-and-execute)
an AI memory extensionEpisodic, semantic, and working memory
an AI extensionConversation sessions — branching, checkpointing, multi-agent
an AI skills extensionSkill/tool registry and executor
quadkit-ai-mcpModel Context Protocol server and client
an AI workers extensionBackground AI work — batch embedding, document ingestion
an AI extensionTracing, metrics, and health checks for AI calls
an AI evaluation extensionFeedback collection and processing
an AI extensionLLM output evaluation (Q&A, string/embedding distance, trajectory, criteria) and reproducible experiment tracking

1. Configuring the LLM Client

Section titled “1. Configuring the LLM Client”

quadkit-ai-llm exposes a single LLMClientProtocol and selects the concrete provider from configuration. Wire it through the published LLM module:

from quadkit import Application
from quadkit.ai.llm import LLMModule, ClientConfig




def create_app() -> Application:
    app = Application(name="my-ai-app")
    app.add_module(
        LLMModule.configure(
            ClientConfig(provider="anthropic", model="claude-sonnet-4-6")
        )
    )
    return app

Equivalent YAML — providers are an ordered list under the ai_llm section (the first is highest priority):

application.yaml
ai_llm:
  enabled: true
  strategy: sequential          # sequential | parallel_race | cost_optimized | latency_optimized
  providers:
    - name: primary
      model: claude-sonnet-4-6
      api_key: "${ANTHROPIC_API_KEY}"
  defaults:
    temperature: 0.2

2. Calling the LLM

Section titled “2. Calling the LLM”

Inject LLMClientProtocol and call complete(). It returns a Result — there are no exceptions for expected failures (rate limits, provider errors):

from quadkit.contracts.ai.llm import LLMClientProtocol
from quadkit.result import Result




class ChatService:
    def __init__(self, llm: LLMClientProtocol) -> None:
        self._llm = llm


    async def reply(self, prompt: str) -> str:
        result = await self._llm.complete(
            messages=[{"role": "user", "content": prompt}],
        )
        if result.is_err():
            return f"LLM error: {result.unwrap_err()}"
        return result.unwrap().content

complete() accepts a plain message list and supports model, temperature, max_tokens, tools, and stop_sequences overrides. For token-by-token output, use stream_chat(...), which returns an async stream of chunks.

3. Thinking Suppression

Section titled “3. Thinking Suppression”

Some models (Qwen3, Gemma, and other reasoning models served via LM Studio / vLLM / SGLang) emit chain-of-thought tokens by default, adding 20–30s of latency. Quadkit can suppress this at the provider level via ThinkingConfig:

from quadkit.contracts.ai.thinking import ThinkingConfig
from quadkit.ai.llm import ClientConfig


ClientConfig(
    provider="lmstudio",
    model="qwen3",
    thinking=ThinkingConfig(suppress=True),   # inject `enable_thinking: false`
)

Or per provider in the routing config / via env var:

Terminal window
QK_AI_LLM__PROVIDERS__PRIMARY__SUPPRESS_THINKING=true

ThinkingConfig also exposes budget_tokens (Anthropic, Gemini 2.5), effort (OpenAI o-series), and level (Gemini 3) for models where you want reasoning but with a bound.

4. Retrieval-Augmented Generation (RAG)

Section titled “4. Retrieval-Augmented Generation (RAG)”

A RAG extension can coordinate chunking, embedding, vector retrieval, and synthesis behind RAGPipelineProtocol. A module-level RAG configuration API is not part of the published packages, so the example below focuses on the stable contract boundary.

Then query through the injected pipeline:

from quadkit.contracts.ai.rag import RAGPipelineProtocol, RAGContext




class DocsService:
    def __init__(self, rag: RAGPipelineProtocol) -> None:
        self._rag = rag


    async def ask(self, question: str) -> str:
        result = await self._rag.execute(RAGContext(query=question))
        answer = result.unwrap()
        return answer.answer  # plus citations / sources when enabled

The vector backend (pgvector, Qdrant, Pinecone, or in-memory for tests) is provided by a vector-store extension and selected via the vector config section — your RAG code never changes when you switch stores.

5. Agents, Memory & Sessions

Section titled “5. Agents, Memory & Sessions”

For multi-step reasoning, quadkit-ai-agents provides agents that call tools and follow strategies such as ReAct and plan-and-execute. Pair them with:

  • an AI skills extension — a registry of callable tools the agent can invoke.
  • an AI memory extension — episodic / semantic / working memory across turns.
  • an AI extension — durable conversations with branching and checkpointing.

These compose through the container like any other Quadkit services. See the per-package guides under the ecosystem for the exact tool-registration and executor APIs.

6. Observability

Section titled “6. Observability”

an AI extension adds tracing, metrics, and health checks around AI calls — giving you visibility into latency, token usage, and retrieval steps without changing your service code:

application.yaml
ai_observability:
  enabled: true
  metrics_enabled: true
  tracing_enabled: true
  health_checks_enabled: true

Next Steps

Section titled “Next Steps”