LLM & AI Calculators 2025 — VRAM Requirements, Token Cost, GPU Comparison, RAG & Fine-Tuning | ZeonTools

01 How to Calculate LLM VRAM Requirements

Calculating the VRAM required to run a Large Language Model (LLM) is a multi-component problem that trips up even experienced ML engineers. The naive approach — "multiply parameters by 2 bytes for FP16" — only accounts for model weights and ignores KV cache, activations, framework overhead, and batch scaling. Understanding every component is critical for production deployments and hardware purchasing decisions.

The complete VRAM formula: Total VRAM = Model Weights + KV Cache + Activation Memory + Framework Overhead. Model weights: parameters × bytes_per_precision (FP32=4, FP16/BF16=2, INT8=1, INT4=0.5 bytes). Framework overhead: PyTorch/vLLM typically adds 0.5–1 GB. Activation memory during inference is small (a few hundred MB for batch=1), but scales with batch size during training.

16 GB

Llama 3 8B in FP16

4.5 GB

Llama 3 8B in INT4

140 GB

Llama 3 70B in FP16

38 GB

Llama 3 70B in INT4

GPU VRAM Fit Reference

Consumer GPUs

RTX 4090 — 24 GB: Llama 3 8B in FP16, 70B in INT4 (2× GPU)
RTX 4080 — 16 GB: 7B/8B FP16, 13B INT4
RTX 3090 — 24 GB: Same as 4090 but slower bandwidth
RTX 4070 Ti — 12 GB: 7B INT4 comfortably, 7B INT8 tight

Data Centre GPUs

H100 SXM5 — 80 GB: 70B FP16, 405B INT4
A100 SXM4 — 80 GB: 70B FP16, fine-tuning 7B full
A100 PCIe — 40 GB: 30B FP16, 70B INT4 (2× GPU)
T4 — 16 GB: 7B FP16 inference only

KV Cache Tip: For long-context inference (32K+ tokens), KV cache can exceed model weights. Llama 3 8B with standard MHA at 32K context, batch=8 requires ~20 GB for KV cache alone — larger than the model weights. Use GQA models (Llama 3, Mistral) for long-context work to reduce KV cache by 4–8×.

02 Model Quantization: FP32, BF16, INT8, INT4 Compared

Quantization is the single most impactful lever for making large models fit on available hardware. It reduces numerical precision of model weights, dramatically cutting memory requirements with varying degrees of quality impact. Modern techniques like GPTQ, AWQ, and GGUF have made INT4 quantization nearly lossless for most real-world tasks.

4 bytes

FP32 per parameter

2 bytes

FP16/BF16 per parameter

1 byte

INT8 per parameter

0.5 bytes

INT4 per parameter

Precision → Quality Impact

FP32: Baseline (training precision), no loss
BF16: ≈0% quality loss, standard inference
INT8 (LLM.int8): <1% perplexity increase
GPTQ-4bit: ~3–5% perplexity increase
AWQ-4bit: ~2–4% perplexity increase (better)
GGUF Q4_K_M: ~3% perplexity, most popular local format
GPTQ-3bit: ~8–15% perplexity, significant loss

Quantization Formats

GPTQ: GPU-optimised, works with ExLlamaV2/AutoGPTQ
AWQ: Activation-aware, often better than GPTQ at same bitwidth
GGUF: CPU+GPU hybrid (llama.cpp), cross-platform
SpQR: Mixed-precision, outlier-aware (research)
bitsandbytes: Load-time INT8/INT4, easy Hugging Face integration
EXL2: Variable bit-width per layer, best quality/size ratio

Best Practice: For production serving, prefer AWQ-4bit over GPTQ-4bit — it delivers better perplexity at the same memory footprint by calibrating around activation outliers. For local llama.cpp use, Q4_K_M (GGUF) is the community gold standard: best balance of size, speed, and quality.

03 Context Window & KV Cache Memory Explained

The context window defines how much text an LLM can "see" at once — including your input prompt, conversation history, retrieved RAG documents, and the model's own output. Modern models range from 4K tokens (older GPT-3.5) to 1M tokens (Gemini 1.5 Pro). Longer context enables more powerful applications but has significant VRAM and latency implications.

The KV cache formula: KV_GB = 2 × layers × kv_heads × head_dim × seq_len × batch × bytes ÷ 1e9. For Llama 3 8B (32 layers, 8 KV heads with GQA, 128 head_dim) at FP16 with 8K context, batch=1: ≈0.53 GB. At 128K context: ≈8.5 GB — larger than the model itself in INT4. Grouped Query Attention (GQA) reduces KV heads from 32 to 8 in Llama 3, shrinking KV cache by 4× compared to standard MHA.

128K

Llama 3.1 context window

Gemini 1.5 Pro context

4×

GQA KV cache reduction

O(n)

KV cache growth rate

PagedAttention (vLLM): vLLM's PagedAttention manages KV cache like OS virtual memory — allocating memory in "pages" dynamically rather than pre-allocating for max sequence length. This allows up to 24× more efficient GPU memory utilisation and enables continuous batching, dramatically increasing throughput on production serving infrastructure.

04 GPU Comparison for LLM Inference: H100 vs A100 vs RTX 4090

Hardware choice is one of the most consequential decisions in LLM deployment. Three key metrics determine suitability: memory capacity (fits the model), memory bandwidth (determines decode speed), and compute (determines prefill speed). These are not always correlated — the RTX 4090 has impressive bandwidth but limited VRAM; the A100-40G has 40 GB but lower bandwidth than H100.

3.35 TB/s

H100 SXM5 bandwidth

2.0 TB/s

A100 SXM4 bandwidth

1.008 TB/s

RTX 4090 bandwidth

0.3 TB/s

T4 bandwidth

H100 SXM5 80GB — $30K+

989 TFLOPS BF16 (with sparsity: 1979 TFLOPS)
3.35 TB/s HBM3 bandwidth
NVLink 4.0: 900 GB/s inter-GPU
Best for: 70B+ production serving, training
~210 tok/s Llama3-70B INT4

A100 SXM4 80GB — $15K

312 TFLOPS BF16
2.0 TB/s HBM2e bandwidth
ECC memory, NVLink 3.0
Best for: Enterprise inference + fine-tuning
~125 tok/s Llama3-70B INT4

RTX 4090 24GB — $1,600

82.6 TFLOPS FP32
1.008 TB/s GDDR6X bandwidth
No ECC, consumer driver limits
Best for: Local inference, prototyping
~100 tok/s Llama3-8B FP16

Apple M2 Ultra 192GB — $5K

Unified memory: CPU + GPU share 192 GB
800 GB/s memory bandwidth
Best for: 70B FP16 local, power efficiency
~30–40 tok/s Llama3-70B FP16
No CUDA — Metal/MLX framework

05 Understanding Tokens: Counting, Cost & Optimization

Tokens are the fundamental unit of text in LLMs — not words or characters, but subword pieces from the model's vocabulary (typically 32K–128K token vocabulary using Byte Pair Encoding or SentencePiece). Understanding tokenisation is critical for cost estimation, prompt engineering, and context window planning.

Rule of thumb: 1 token ≈ 4 characters ≈ 0.75 words in English. However, this varies significantly by content type: code is denser (more tokens per line), non-Latin scripts like Chinese use 1–2 characters per token, and structured data (JSON, XML) is very token-inefficient. A 1,000-word essay is roughly 1,333 tokens; a 10-page PDF might be 3,000–5,000 tokens.

4 chars

≈ 1 token (English)

0.75 words

≈ 1 token (English)

$2.50

GPT-4o per 1M input tokens

$0.075

Gemini Flash per 1M tokens

Cost Optimisation Strategies: (1) Prompt caching — Anthropic and Google charge 90% less for cached prompt tokens. (2) Batching — OpenAI's Batch API offers 50% discount. (3) Model selection — Gemini Flash vs GPT-4o is a 33× price difference for many tasks. (4) Output compression — instruct the model to be concise; output tokens are 3–5× more expensive than input. (5) Structured outputs — JSON mode reduces back-and-forth re-prompting.

06 Vector Embeddings & RAG Architecture

Embedding models transform text into dense numerical vectors that capture semantic meaning. Similar texts produce vectors close together in high-dimensional space, enabling semantic search, recommendation, and retrieval-augmented generation. The quality of embeddings directly determines the quality of RAG — garbage embeddings mean irrelevant retrieved context, which can make LLM responses worse than no RAG at all.

1,536

text-embedding-3-small dims

$0.02

OpenAI embed-3-small / 1M tok

6 KB

Float32 1536-dim vector size

~1ms

Typical ANN query latency

Embedding Model Comparison

text-embedding-3-small: 1536-dim, $0.02/1M — best value
text-embedding-3-large: 3072-dim, $0.13/1M — highest quality
Cohere embed-v3: 1024-dim, $0.10/1M — multilingual
BGE-M3 (local): 1024-dim, free — state-of-art open
nomic-embed-text: 768-dim, free — 8K context window

Vector DB Options

Pinecone Serverless: $0.08/GB/month, managed, easy
Weaviate Cloud: $25/month starter, GraphQL API
Qdrant: Self-host free, cloud from $25/month
pgvector: PostgreSQL extension, free, SQL interface
Chroma: Local dev, free, SQLite-backed
Milvus: Enterprise scale, Kubernetes-native

07 Fine-Tuning LLMs: LoRA, QLoRA & Full Fine-Tune

Fine-tuning adapts a pre-trained LLM to your specific domain, style, or task format. The choice of technique has massive implications for hardware requirements, training time, and cost. Full fine-tuning delivers maximum quality but requires enormous GPU resources. Parameter-efficient methods like LoRA and QLoRA achieve near-equivalent results at a fraction of the cost.

~80–120 GB

Full fine-tune 7B VRAM

~20–28 GB

LoRA 7B VRAM

~8–12 GB

QLoRA 7B VRAM

0.1–1%

LoRA trainable parameters

Full Fine-Tuning

All parameters updated
VRAM: model × 16–20× (FP32 grads + Adam)
7B model: needs 4–8× A100 80GB
Best quality ceiling
Risk of catastrophic forgetting
Cost: $500–$5,000 for 7B model

LoRA (Low-Rank Adaptation)

Trains low-rank matrices only (r=4–64)
VRAM: model (FP16) + LoRA adapters + optimizer
7B model: 1× A100-40G or 2× RTX 4090
Within 1% of full fine-tune quality
Adapters can be merged or swapped
Cost: $50–$200 for 7B model

QLoRA (4-bit Quantized LoRA)

Base model in 4-bit NF4 quantization
LoRA adapters in BF16
7B model: fits on single RTX 4090 (24GB)
70B model: fits on single A100-80G
Within 1–2% of full fine-tune quality
Cost: $10–$50 for 7B model on cloud GPU

Recommended Stack: For most organisations, QLoRA with Unsloth (2× faster than standard QLoRA) on a single A100-40G is the sweet spot. Use Weights & Biases for experiment tracking, Axolotl or LLaMA-Factory for training orchestration, and merge LoRA adapters before serving with vLLM for maximum throughput.

08 Inference Optimization: Batching, Streaming & KV Caching

Inference optimisation is where production LLM engineering lives. Raw model performance (tokens/second at batch=1) is rarely the relevant metric for real applications — you need to maximise throughput across concurrent users while keeping per-request latency acceptable. The key techniques are continuous batching, prompt caching, and speculative decoding.

~100 tok/s

RTX 4090, Llama3-8B FP16

~210 tok/s

H100, Llama3-70B INT4

24×

vLLM throughput vs naive serving

2–3×

Speculative decoding speedup

Framework Comparison: vLLM (PagedAttention, continuous batching, OpenAI-compatible API) is the production standard. TGI (HuggingFace Text Generation Inference) offers Flash Attention 2 and tensor parallelism. llama.cpp is the standard for CPU+consumer GPU inference supporting GGUF format. Ollama wraps llama.cpp with a Docker-like model management UX. TensorRT-LLM (NVIDIA) provides maximum throughput on NVIDIA hardware with kernel fusion and INT8/FP8 quantization.

09 LLM API Providers: Cost & Feature Comparison 2025–2026

The LLM API market has become fiercely competitive, with pricing dropping 90%+ since 2023. Choosing the right provider and model for your use case can mean a 100× difference in cost with similar quality. The key dimensions: context window size, input/output pricing ratio, rate limits, latency, multimodal capabilities, and structured output support.

$2.50

GPT-4o / 1M input tokens

$0.075

Gemini Flash / 1M tokens

$3.00

Claude 3.5 Sonnet / 1M in

$0.20

Mistral Small / 1M tokens

OpenAI Models (2025)

GPT-4o: $2.50/$10.00 per 1M in/out, 128K ctx
GPT-4o-mini: $0.15/$0.60 per 1M, 128K ctx
o1: $15/$60 per 1M, reasoning model
o1-mini: $3/$12 per 1M, fast reasoning
GPT-4o Batch: 50% discount, 24hr window
Function calling, JSON mode, vision

Anthropic Claude (2025)

Claude 3.5 Sonnet: $3/$15 per 1M, 200K ctx
Claude 3.5 Haiku: $0.80/$4 per 1M, 200K ctx
Claude 3 Opus: $15/$75 per 1M, top quality
Prompt caching: 90% discount on cached tokens
Best for: long-doc analysis, coding, safety
Tool use, vision, computer use

Google Gemini (2025)

Gemini 1.5 Pro: $1.25/$5 per 1M (≤128K)
Gemini 1.5 Flash: $0.075/$0.30 per 1M
Gemini 2.0 Flash: Next-gen, competitive pricing
1M token context window (1.5 Pro)
Best for: long-context, multimodal, cost
Free tier: 15 RPM on Flash

10 RAG vs Fine-Tuning: When to Use Each

RAG and fine-tuning are complementary techniques, not competitors. They solve different problems: RAG addresses what the model knows (factual knowledge), while fine-tuning addresses how the model behaves (style, format, reasoning patterns). Choosing incorrectly is one of the most common and expensive mistakes in production AI development.

Use RAG When:

Knowledge changes frequently (docs, news, policies)
You need source citations/attribution
Knowledge base is very large (10K+ documents)
Multiple knowledge domains with different access controls
Fast time-to-production (<1 week to deploy)
Budget is limited — no GPU training cost

Use Fine-Tuning When:

Consistent output format is critical (structured JSON, code)
Domain-specific vocabulary/jargon (medical, legal, finance)
Reducing prompt length and API cost at scale
Teaching new reasoning capabilities
Proprietary communication style/tone
Latency reduction (shorter system prompts)

Advanced Pattern — RAG + Fine-Tuning Together: The highest-performing production systems often combine both. Fine-tune the base model on domain vocabulary and output format, then layer RAG on top for current knowledge retrieval. This gives you style consistency from fine-tuning with knowledge currency from RAG — at the cost of higher implementation complexity.

11 AI ROI & Business Case for LLM Implementation

Building a defensible AI business case requires quantifying both the benefits (labour savings, output quality improvements, speed gains) and the total cost of ownership (API costs, infrastructure, engineering time, prompt maintenance, quality assurance). Many organisations underestimate the "hidden costs" of AI: prompt engineering, evaluation pipelines, safety reviews, and ongoing maintenance.

150–400%

Typical Year 1 AI ROI

2–4 months

Common payback period

30–50%

Realistic efficiency realisation

$0.01–$0.10

AI cost per automated task

ROI Formula: Annual Savings = (tasks/day × 365 × minutes_per_task/60 × hourly_rate × team_size) × efficiency_factor. Subtract annual AI costs (API + infrastructure + maintenance). Divide implementation cost by monthly net savings for payback period. Apply 30–50% efficiency discount for real-world adoption friction, quality assurance overhead, and edge cases that still require human review.

12 Quick Reference: GPU Specs, Token Costs & Key Formulas

VRAM Formula: VRAM_GB = params_B × bytes_per_param + (2 × layers × kv_heads × head_dim × seq_len × batch × bytes / 1e9) + 0.5 (overhead)

Tokens/Second (decode): tok_s = memory_bandwidth_GBs / (model_size_GB / params_B) × batch_size

Fine-Tune FLOPs: FLOPs = 6 × params × training_tokens | Time_hr = FLOPs / (gpu_TFLOPS × 1e12 × util × 3600)

$176,100

SS wage base (irrelevant here, for reference) — H100 peak TFLOPS: 1979 w/sparsity

2,080 hrs/yr

Standard work year

989 TFLOPS

H100 BF16 dense compute

$2–3/hr

A100-80G cloud spot price

13 Frequently Asked Questions

How much VRAM does Llama 3 70B need?

Llama 3 70B has approximately 70 billion parameters. The VRAM requirement depends heavily on the quantization precision used. At full FP16/BF16 precision (2 bytes per parameter), the model weights alone require 140 GB of VRAM — beyond a single A100-80GB. With INT8 quantization (1 byte/param), it fits in ~70 GB. With INT4/GPTQ-4bit (0.5 bytes/param), you need roughly 35–38 GB, making it runnable on 2× RTX 4090 (24 GB each) with tensor parallelism. With AWQ-4bit or GPTQ-3bit, memory drops to ~28 GB. Add KV cache overhead for inference: at 4K sequence length with batch size 1, an additional 4–8 GB is required depending on the architecture. For Llama 3 70B with GQA (8 KV heads), KV cache is significantly smaller than standard MHA models.

What is the difference between FP16 and INT4 quantization?

FP16 (16-bit floating point) stores each model weight as a 2-byte float, preserving full numerical precision and achieving near-identical quality to FP32 training. INT4 quantization compresses each weight to 4 bits (0.5 bytes), reducing model size by 4× compared to FP16. A 7B parameter model that requires 14 GB in FP16 drops to roughly 3.5–4.5 GB in INT4 (including quantization overhead). The quality tradeoff: INT4 introduces quantization error, typically increasing perplexity by 3–8% for well-implemented methods like GPTQ or AWQ, which is imperceptible for most tasks. INT8 is a middle ground: 2× memory reduction vs FP16 with less than 1% quality loss. For production deployments, BF16 or FP16 is preferred for maximum accuracy; INT4 is ideal for local consumer GPU deployment where VRAM is the constraint.

How do LLM API token costs work?

LLM APIs charge separately for input tokens (your prompt, context, and retrieved documents) and output tokens (the model's generated response). Pricing is expressed per 1 million tokens. As of 2025-2026: GPT-4o costs $2.50/1M input and 10.00/1M output; GPT-4o-mini is $0.15/0.60; Claude 3.5 Sonnet is $3.00/15.00; Gemini 1.5 Flash is $0.075/0.30. Output tokens are typically 3–5× more expensive than input tokens because generation is computationally more intensive. One token equals approximately 4 characters or 0.75 words in English. A typical ChatGPT-style conversation (2,000 input + 500 output tokens) costs about $0.0055 with GPT-4o. For high-volume applications sending 1 million requests/day, provider choice and prompt optimization can mean the difference of tens of thousands of dollars monthly.

What is a KV cache and why does it matter for VRAM?

The KV (Key-Value) cache stores intermediate attention computations during inference so they do not need to be recomputed for each new token generated. When a model processes a 2,048-token prompt, it computes key and value matrices for every layer and every token. Without caching, generating each output token would require recomputing all previous tokens' attention — scaling as O(n²) in time. With KV caching, these computations are stored in VRAM and reused, making decode-phase token generation nearly constant time per step. The VRAM cost: KV_cache_GB ≈ 2 × num_layers × num_heads × head_dim × seq_len × batch_size × bytes_per_param ÷ 1e9. For Llama 3 8B (32 layers, 32 heads, 128 head_dim) at FP16 with 4K context and batch=1: ~2 GB. Grouped Query Attention (GQA) reduces KV cache by using fewer KV heads than query heads, shrinking this overhead by 4–8×.

H100 vs A100 vs RTX 4090 — which GPU for LLM inference?

H100 SXM5 (80 GB HBM3, $30,000+): The gold standard for production LLM inference. 3.35 TB/s memory bandwidth, 989 TFLOPS BF16, NVLink for multi-GPU scaling. Best for large models (70B+) and high-throughput serving. A100 SXM4 80GB (15,000–$20,000): 2 TB/s bandwidth, 312 TFLOPS BF16. Excellent for 7B–70B inference and fine-tuning. The standard in cloud ML (AWS p4d, Google A3). RTX 4090 (24 GB, ~1,600): 1.008 TB/s bandwidth, 82.6 TFLOPS. Best consumer option — fits Llama 3 8B in FP16, 70B models in INT4 with 2+ GPUs. No ECC memory, consumer driver limitations for multi-GPU. For cost-efficiency: RTX 4090 delivers the best tokens/$ for local inference. H100 wins on absolute throughput and reliability for production. A100 is the sweet spot for enterprise deployments with mixed workloads of inference and fine-tuning.

What is the context window and how does it affect VRAM?

The context window is the maximum number of tokens an LLM can process in a single forward pass — including both the input prompt and the generated output. Modern models range from 4K tokens (older GPT-3.5) to 1M tokens (Gemini 1.5 Pro). VRAM impact is significant and grows linearly (with GQA/MQA) or quadratically (with standard MHA) with context length. For a Llama 3 8B model with standard MHA, doubling context from 4K to 8K tokens roughly doubles KV cache memory. The KV cache formula: GB = 2 × layers × heads × head_dim × seq_len × batch × bytes ÷ 1e9. At 128K context (Llama 3.1 8B), the KV cache alone can exceed the model weights in size. Techniques like sliding window attention (Mistral), Grouped Query Attention, and page attention (vLLM) reduce this overhead substantially for long-context workloads.

What is RAG (Retrieval-Augmented Generation)?

RAG is an AI architecture pattern where a language model's response is augmented with relevant information retrieved from an external knowledge base at query time. Instead of relying solely on the LLM's training knowledge (which has a knowledge cutoff and may hallucinate facts), RAG retrieves the most relevant document chunks from a vector database, inserts them into the prompt context, and lets the LLM generate a grounded answer. The pipeline: (1) Ingestion — documents are split into chunks, converted to embedding vectors, and stored in a vector DB (Pinecone, Weaviate, Qdrant, pgvector). (2) Query — the user question is embedded, the top-k similar chunks are retrieved, and the LLM generates a response using those chunks as context. RAG dramatically reduces hallucination, keeps knowledge current without retraining, and is more cost-effective than fine-tuning for most knowledge-update tasks.

LoRA vs QLoRA vs full fine-tuning — which to choose?

Full fine-tuning updates all model parameters and requires VRAM equal to roughly 16–20× the model weights in FP32 (model + gradients + optimizer states). For a 7B model, that is 80–120 GB — requiring multiple A100s. LoRA (Low-Rank Adaptation) adds trainable low-rank matrices to attention layers while freezing the base model, reducing trainable parameters to 0.1–1% of total. VRAM drops to ~3–4× the model size in INT8/FP16. For a 7B model: ~20–28 GB, runnable on a single A100-40G or 2× RTX 4090. QLoRA combines LoRA with 4-bit quantization of the frozen base model, enabling fine-tuning on a single RTX 4090 (24 GB) for models up to 30B parameters. Quality: QLoRA produces results within 1–2% of full fine-tuning on most benchmarks while using 10× less GPU memory. For most use cases, QLoRA is the clear choice; full fine-tuning only when maximum accuracy is critical.

How many tokens can an LLM generate per second?

LLM inference throughput (tokens/second) is primarily memory-bandwidth limited during the decode phase. The formula: tokens/s ≈ memory_bandwidth_TBs / (model_size_bytes / num_parameters) × batch_size. For Llama 3 8B in FP16 (16 GB model) on an RTX 4090 (1.008 TB/s): ~63 tokens/s at batch=1. On A100 SXM4 (2 TB/s): ~125 tokens/s. On H100 SXM5 (3.35 TB/s): ~210 tokens/s. Larger batches increase throughput but add latency per request. INT4 quantization halves model size, roughly doubling token throughput. Prefill (prompt processing) is compute-bound and scales with context length. For a practical benchmark: GPT-4o on OpenAI's infrastructure delivers ~100–150 tokens/s for typical API requests. Local Llama 3 8B INT4 on RTX 4090 achieves 80–100 tokens/s with llama.cpp or vLLM.

What is GQA (Grouped Query Attention)?

Grouped Query Attention is an architectural optimization that reduces KV cache memory without sacrificing model quality. In standard Multi-Head Attention (MHA), every attention head has its own unique Key and Value projections. In GQA, multiple query heads share a single KV head — reducing KV cache size by the grouping factor. For example, Llama 3 8B uses 32 query heads but only 8 KV heads (ratio 4:1), reducing KV cache to 25% of what standard MHA would require. Llama 3 70B uses 64 query heads and 8 KV heads (ratio 8:1), saving 87.5% on KV cache memory. Multi-Query Attention (MQA) is the extreme version where all query heads share a single KV head. GQA delivers a balance: near-MQA efficiency with near-MHA quality. This is why long-context inference is far more practical with newer models like Mistral, Llama 3, and Gemma than with older MHA architectures.

How are embedding vectors stored and what do they cost?

Embedding vectors are dense numerical arrays representing semantic meaning of text. OpenAI's text-embedding-3-small produces 1,536-dimensional vectors; text-embedding-3-large produces 3,072 dimensions. Storage cost per vector: at float32 (4 bytes/dim), a 1,536-dim vector = 6,144 bytes ≈ 6 KB. One million document embeddings = ~6 GB in float32, ~3 GB in float16. API cost to generate embeddings: text-embedding-3-small costs $0.02/1M tokens — embedding 1 million 500-token documents costs 10. Vector databases (Pinecone, Weaviate, Qdrant, pgvector) store vectors with metadata and enable approximate nearest-neighbor (ANN) search in milliseconds. Cloud vector DB pricing: Pinecone Serverless charges ~$0.08/GB/month for storage plus query costs. For 10 million vectors at 1,536 dims in float32 (~60 GB): ~4.80/month storage. Self-hosted Qdrant or pgvector eliminates cloud storage costs entirely.

What is model quantization and does it hurt quality?

Model quantization reduces the numerical precision of model weights from the training precision (FP32 or BF16) to lower-bit formats, reducing memory footprint and often increasing inference speed. Common formats: FP32 (4 bytes) → FP16/BF16 (2 bytes) → INT8 (1 byte) → INT4/GPTQ/AWQ (0.5 bytes) → GPTQ-3bit (0.375 bytes). Quality impact measured by perplexity increase vs FP16 baseline: BF16 ≈ 0% loss; INT8 ≈ 0.5–1% perplexity increase; INT4 (GPTQ/AWQ) ≈ 3–6% perplexity increase; INT3 ≈ 8–20% increase. GPTQ and AWQ are post-training quantization methods that calibrate quantization weights using a small dataset, significantly reducing quality loss vs naive INT4. For conversational tasks and instruction following, INT4 quality is nearly indistinguishable from FP16. For precise numerical tasks, code generation, or math, FP16 or INT8 is strongly preferred.

How do I estimate AI ROI for my business?

AI ROI calculation: Annual_Labor_Savings = (tasks_automated/day × 365 × minutes_per_task/60 × hourly_rate × team_size). Annual_AI_Cost = (tasks/day × 365 × cost_per_task). Net_Annual_Savings = Labor_Savings − Annual_AI_Cost. ROI% = ((Net_Savings − Implementation_Cost) / Implementation_Cost) × 100. Payback_Months = Implementation_Cost / (Net_Monthly_Savings). Example: 100 tasks/day, 30 min saved each, $50/hr employees, 10-person team, 0.05 AI cost/task, $50,000 implementation. Annual labor savings: 100 × 365 × 0.5 × 50 × 10 = $9,125,000. Annual AI cost: 100 × 365 × 0.05 = $1,825. Net savings: 9,123,175. ROI: 18,146%. Payback: <1 month. Real-world ROI is typically 150–400% in year 1 after accounting for quality assurance overhead, prompt engineering time, integration development, and adoption friction (typically 30–50% efficiency realization rate).

What GPU should I buy for running LLMs locally?

Budget under $500: Use cloud APIs (no local GPU). Budget 500–$1,000: RTX 3060 12GB — runs Llama 3 8B in INT4 (~4.5 GB), comfortable for 7B models. Budget 1,000–$1,600: RTX 4080 16GB — runs 13B models in INT4, 7B in INT8, smooth performance with 1.008 TB/s bandwidth. Best value: RTX 4090 24GB (~1,600) — the sweet spot for local inference. Runs Llama 3 70B in INT4 (requires 38–40 GB; combine 2× RTX 4090 for $3,200). Runs any 7B/8B model in FP16 comfortably. ~100 tokens/s for 8B models. For multi-GPU setups: 2× RTX 4090 with NVLink (requires NVLink bridge) handles 70B INT4 models. Professional GPU route: Used A100 40GB (8,000–$12,000) gives ECC memory, NVLink, and enterprise reliability. Mac Studio / Mac Pro with M-series chips: Unified memory architecture makes 192GB M2 Ultra excellent for 70B FP16 inference at 5,000–$7,000.

What is the difference between prefill and decode in LLM inference?

LLM inference has two distinct computational phases. Prefill (prompt processing): The model processes all input tokens simultaneously in a single parallel forward pass — this is compute-bound. Prefill throughput scales with model FLOPS. Time to first token (TTFT) is determined by prefill speed. For a 2,048-token prompt on an A100 with Llama 3 8B, prefill takes ~50–100ms. Decode (autoregressive generation): The model generates output tokens one at a time — each new token attends to all previous tokens. This is memory-bandwidth-bound because the model weights must be read from GPU memory for each token generated. Decode speed (tokens/second) = memory_bandwidth / model_size. Improving decode: quantization reduces model size; larger batch sizes amortize memory reads across multiple concurrent requests (improving throughput but not per-request latency); speculative decoding uses a smaller draft model to propose multiple tokens, then verifies with the large model in one pass.