AI Engineer Stack — A Visual Reference

The Master Key

00The 8 Layers (the whole game)

When you watch a demo or read a new architecture post, you're never seeing something new — you're seeing a tool plugged into one of these 8 slots. Top = closest to the human. Bottom = closest to silicon. Memorize the question each layer answers.

1 · Control / UX“How does the human talk to it?” Chat UI, Telegram, web frontend.

2 · Orchestration / Agent“Who decides what to do?” Routing, memory, tool-calling, loops.

3 · API / Serving“What's the wire contract?” The HTTP endpoint code talks to.

4 · Augmentation / Tools“Where's the extra knowledge?” RAG, vector DB, function calls.

5 · Inference Engine“What runs the weights?” The runtime that executes the model.

6 · ML Framework“What defines the math?” Tensors, autograd, the model graph.

7 · Model Weights“What is the actual model?” The trained parameter file.

8 · Compute / Accel“What hardware crunches it?” GPU, CUDA, edge silicon.

The Reference Ladder

01The Universal Stack

A request falls down the ladder (human → silicon), the answer climbs back up. Every example later on this page is just this same ladder with different tools in the slots.

LAYER 1Control / UX PlaneWhere the human gives intent & reads the answer

Synthwave web UITelegram / DiscordReact / Open WebUICLI

LAYER 2Orchestration / AgentThe brain that routes, remembers, calls tools, loops

HermesLangChain / LangGraphRouter decision logicLlamaIndex

LAYER 3API / ServingThe HTTP contract between orchestrator and model

FastAPIOpenAI-compatible /v1Anthropic APIgRPC / Triton

LAYER 4Augmentation / ToolsExternal knowledge & actions injected before/around generation

Pinecone (vector DB)RAG retrieverEmbeddings modelWeb / SQL tools

LAYER 5Inference Engine / RuntimeThe program that loads weights & produces tokens

OllamaOsaurus (MLX)vLLMllama.cppTensorRT-LLM

LAYER 6ML FrameworkDefines the math: tensors, layers, autograd, the graph

PyTorchtorchvisionTensorFlowscikit-learn*MLX

LAYER 7Model WeightsThe actual trained parameters — the “knowledge”

Llama 3.xQwen3NemotronYOLOv5GPT-4o / Claude*

LAYER 8Compute / AccelerationThe silicon + driver layer that does the matrix math

NVIDIA GPU + CUDAJetson Orin NanoApple Silicon / MetalCisco UCS

* scikit-learn and hosted models like GPT-4o / Claude are shown for placement — scikit-learn is a framework for classical ML; hosted models hide layers 5–8 behind someone else's API.

Build vs Run

02Two Paths Through the Stack

The same 8 layers serve two very different jobs. The clearest way to reason about any system is to separate build-time (making the weights) from run-time (serving the weights). This is where Pandas, NumPy & scikit-learn actually live.

🛠️ Build-Time (Training / Data)

Most of the job: turn messy data into a model file.

Raw dataCSV, logs, images, scraped text

▼

Pandasclean, filter, merge, handle missing rows

▼

NumPyarrays & matrix math under the hood

▼

scikit-learn · or · PyTorchclassical ML (boosting, forests) vs deep learning

▼

Trained weightsthe artifact you'll later serve

⚡ Run-Time (Inference / Serving)

A live user request falling down the ladder and back up.

User in the UIasks a question

▼

Orchestrator routespicks model, pulls memory + RAG context

▼

API callstructured request to an endpoint

▼

Inference engineloads weights, generates tokens on the GPU

▲ answer climbs back up

Rendered answergrounded, streamed to the user

From the Video

03The 7 Tools — and Their Slot

Each tool from the “7 tools every AI engineer should learn” list, tagged with its layer and — crucially — how it differs from its neighbors in the same slot. That contrast is the part worth internalizing.

Foundation · Language

Python

The baseline everything is built on

Readable, fast to prototype — the lingua franca of ML.
PyTorch, TF, LangChain, HuggingFace are all Python-native.

vs others: Rust/C++ are faster but you only drop to them for the hot inner loop (e.g. inside llama.cpp). Python is the glue.

Environment

Conda

Isolated, conflict-free environments

Project A needs TF 2.10, Project B needs 2.15 — keep them apart.
Manages non-Python deps too (CUDA toolkit, MKL).

vs venv / pip / uv: venv isolates only Python; Conda also handles binary/CUDA deps. vs Docker: Docker isolates the whole OS — heavier, reproducible for deploy.

Data · Build-time

Pandas + NumPy

Clean the data before any model sees it

Pandas: high-level tables — load, filter, merge, fill gaps.
NumPy: low-level N-D arrays & matrix math underneath.

vs Polars / Spark: Polars = faster single-node Pandas; Spark = distributed for data too big for one box.

Classical ML Framework

scikit-learn

Most “AI” in production is still this

Structured-data tasks: fraud, churn, demand forecasting.
Gradient Boosting, Random Forests — fast, CPU, explainable.

vs PyTorch/TF: no neural nets / GPU / autograd. Reach for it when the data is tabular, not text or pixels.

Orchestration / Agent

LangChain

Wires an LLM to the outside world

Chains, agents, memory, tool-calling, RAG plumbing.
Search docs, hold conversation state, call APIs mid-response.

vs LlamaIndex: LangChain = general agent framework; LlamaIndex = RAG/indexing specialist. vs Hermes: Hermes is a self-hosted harness/router around the same ideas.

Augmentation · Vector DB

Pinecone

Search by meaning, not by keyword

Stores embeddings (numeric vectors of meaning) at scale.
“dog” → finds “puppies / canines”. The R in RAG.

vs Qdrant/Weaviate/Chroma: open-source, self-host. vs pgvector: just a Postgres extension — fine until you need huge scale.

API / Serving + Weights

Model APIs (OpenAI / Anthropic)

Rent the model instead of hosting it

Manage system prompts, token limits, rate limits, cost.
Multi-model routing: swap GPT-4o ↔ Claude per sub-task.

vs local (Ollama/Osaurus): APIs hide layers 5–8 — no GPU to own, but you pay per token & your data leaves the building.

Bonus · Control

The Frontend

Not in the “7” but always present

Telegram bot, web chat, synthwave dashboard, CLI.
Pure presentation — never put routing logic here.

Design tell: if model/tool decisions live in the UI, that's a red flag. Decisions belong in Layer 2.

The Core Skill

04Same Slot, Different Tool

“Why X over Y?” is the question that comes up on every design review. Here are the swap-in options for each slot and the one-line reason you'd pick each.

Layer 5 — Inference Engines

“What actually runs the weights?” Same model file, very different runtime trade-offs.

Tool	Sweet spot	Format / hardware	Pick it when…
Ollama	Easiest local single-node	GGUF · CPU/GPU · wraps llama.cpp	Home lab, dev, “just run a model.”
Osaurus	Apple Silicon native	MLX · Mac unified memory	Running on a MacBook, want Metal/MLX speed.
llama.cpp	Bare-metal portability	GGUF · runs anywhere, even CPU	You need it on weird/edge hardware.
vLLM	High-throughput serving	HF weights · NVIDIA GPU · PagedAttention	Many concurrent users, production batching.
TensorRT-LLM / Triton	Max enterprise perf	Compiled · NVIDIA · NVAIE	Datacenter, lowest latency, you own the GPUs.

Layer 2 — Orchestrators

“Who holds the logic — routing, memory, tools, loops?”

Tool	Shape	Best for
LangChain	Python library of chains/agents	General app glue, fast prototyping.
LangGraph	Graph/state-machine agents	Complex multi-step, cyclic agent control.
LlamaIndex	RAG & indexing specialist	Doc-heavy retrieval pipelines.
Hermes	Self-hosted agent harness + router	Owning the whole loop on your own box, quota-aware model routing.
Raw SDK	Just call the API yourself	Simple, predictable, no framework lock-in.

Layer 4 — Vector Databases

“Where do embeddings live for RAG?”

Tool	Model	Pick it when…
Pinecone	Managed SaaS	You want zero ops, scale handled for you.
Qdrant / Weaviate / Milvus	Open-source, self-host	Data must stay in-house, want control.
Chroma	Lightweight, local	Prototyping, small RAG, runs in-process.
pgvector	Postgres extension	You already run Postgres & scale is modest.

Layer 6 — Frameworks

“What defines the math?” Match the framework to the data type.

Tool	Domain	Differentiator
PyTorch	Deep learning (text, vision)	Dynamic graphs, autograd, GPU — the research default.
torchvision	Computer vision add-on	Datasets, transforms, pretrained CV models (feeds YOLO etc.).
TensorFlow / Keras	Deep learning	Strong production/mobile (TF Lite, TF Serving) heritage.
scikit-learn	Classical ML, tabular	No GPU/NN — boosting, forests, fast & explainable.
MLX	Apple Silicon DL	Built for Mac unified memory (powers Osaurus).

Layer 8 — Compute & Acceleration

“What silicon, and what's the software bridge to it?”

Tool	What it is	Role
CUDA	NVIDIA's GPU programming layer	The bridge frameworks use to reach the GPU. Not a model — it's the road.
Datacenter GPU (H100 / L40S)	Server accelerator	Big training + high-throughput serving.
Jetson Orin Nano	Edge AI module (has CUDA)	On-device vision/robotics, low power, no cloud.
Apple Silicon / Metal	Mac unified-memory chip	Local LLMs via MLX/Osaurus, no discrete GPU.
Cisco UCS + GPUs	Enterprise rack server	The physical floor of an AI Factory.

Map It Onto Real Systems

05Example Stacks (same colors, every time)

Now the payoff. Each system below is the same ladder — read the colored tag on each row and you instantly know which layer you're looking at. Notice how some stacks skip layers (a hosted API hides 5–8; a vision pipeline has no orchestrator).

🏠 Hermes ▸ Ollama ▸ Llama 3.x

Home-lab LLM agent

The classic self-hosted setup: your own agent driving a local model, no cloud, no per-token bill.

ControlTelegram on phonechat in, answers out

OrchestrationHermesrouting, memory, tool-calling

APIOllama /api (OpenAI-compatible)local HTTP endpoint

Inference EngineOllamallama.cpp under the hood, GGUF

WeightsLlama 3.xquantized GGUF file

ComputeHome GPU / CPU + CUDAon the Unraid box

🍎 Hermes ▸ Osaurus ▸ Qwen3-8B-MLX

Apple-Silicon local

Same orchestrator, different engine slot — swapped to run natively on a MacBook's unified memory.

ControlTelegram / web UIsame front door

OrchestrationHermesidentical logic, new backend

APIOsaurus OpenAI-compatible serverover the LAN / NFS

Inference EngineOsaurusbuilt on MLX

FrameworkMLXApple's array framework

WeightsQwen3-8B-MLX (Q4)quantized for unified memory

ComputeApple Silicon / Metalno discrete GPU

📷 Jetson Orin Nano ▸ YOLOv5

Edge computer vision

No LLM, no orchestrator — a pure vision pipeline. Shows the bottom half of the ladder running solo.

ControlCamera feed / displayframes in, boxes out

AugmentationPre/post-processingresize, NMS, draw boxes

FrameworkPyTorch + torchvisiondefines & runs the net

WeightsYOLOv5object-detection model

ComputeJetson Orin Nano + CUDAedge GPU, low power, offline

🏢 Cisco Secure AI Factory

Enterprise / NVAIE

The same ladder, hardened for the datacenter: orchestration on Kubernetes, NVIDIA's enterprise software suite, your own racks.

ControlInternal app / portalemployee-facing

OrchestrationOpenShift (K8s) + NeMoscheduling, pipelines, guardrails

API / ServingTriton Inference Serverstandardized model endpoints

Inference EngineTensorRT-LLM (NVAIE)compiled, optimized

FrameworkPyTorch / NeMotrain + fine-tune

WeightsEnterprise / fine-tuned modelgoverned, private

ComputeCisco UCS + NVIDIA GPUs + CUDAon-prem racks

🧠 The “YouTube Hermes” Multi-Brain Reference Stack

Map any flashy demo onto this

When a creator shows off a synthwave-UI agent with “sensory” and “reasoning” brains, this is what you're actually looking at. The trick: one orchestrator, a router, and several model slots behind it — each model is just Layer 7 weights on its own engine.

L1 · ControlSynthwave web UIthe pretty front-end

L3 · API/ServingFastAPI backendthe wire contract

L2 · OrchestrationRouter decision logicpicks the right “brain” per task

L4 · AugmentationRAG + memory + toolscontext fed in before generation

L7 · Weights (brain A)Sensory brainNemotron Nano Omni — vision/audio/multimodal

L7 · Weights (brain B)Reasoning brain“Jack” / Qwen3-class ~27B — planning & logic

L5 · EnginevLLM / Ollama per braineach model gets a runtime

L8 · ComputeRemote GPU(s) + CUDAwhere the brains actually run

📌 The mapping move: “sensory brain / reasoning brain” are just Layer-7 weights selected by the Layer-2 router. Multimodal, multi-model demos are not a new layer — they're multiple Layer-7 slots hanging off one orchestrator. Nemotron Nano Omni / Qwen sizes are example/demo names — verify exact versions before quoting them.

The Money Diagram

06How RAG Actually Flows

The single most-misunderstood piece: “How does a vector DB feed an LLM?” Trace it left-to-right — that's the whole mechanism.

Layer 1User query“What's our refund policy?”

→

Layer 4Embed the querytext → vector

→

Vector DBPinecone searchnearest-meaning chunks

→

Layer 2Orchestrator builds promptquery + retrieved context

→

Layer 3→7LLM generatesgrounded, cited answer

The one-line version: “The vector DB (Pinecone, L4) retrieves semantically-relevant context, the orchestrator (LangChain/Hermes, L2) injects it into the prompt, and the LLM (via the API, L3) generates an answer grounded in that context instead of hallucinating.”

The Takeaway

07One Habit

Everything above collapses into a single reflex — and it is what separates thinking in architecture from reciting buzzwords.

💡 When you meet any new tool, name its slot first. Then name what it swaps out. “Pinecone is a Layer-4 vector DB; I'd swap it for Qdrant if the data had to stay in-house.” You're not listing tools — you're placing them on a ladder you already know. New tools stop being new; they're just a fresh occupant of a slot you understand.