Generative AI & LLM APIs Documentation
Build with LLMs — how they work, prompting, model APIs, streaming, embeddings, RAG, tool use and agents.
Generative AI models create text, code and images. As a developer you mostly call a model through an API and build a product around it. This page covers how LLMs work, prompting, API calls, RAG and agents. Examples use Python; see the Python docs if you need a refresher.
1. Introduction
| Term | Meaning |
|---|---|
| LLM | Large language model: predicts the next piece of text |
| Token | A chunk of text, about ¾ of an English word; usage is billed per token |
| Context window | The most tokens a model can read at once, including your prompt and its answer |
| Prompt | The instructions and data you send |
| Temperature | Randomness: low = steady, high = creative |
| Embedding | A list of numbers that captures meaning |
| Hallucination | A confident answer that is wrong |
| RAG | Giving the model your documents at question time |
| Agent | A model that calls tools in a loop to finish a task |
2. How an LLM Works
- Text is split into tokens.
- A transformer uses attention to weigh how tokens relate to each other.
- It predicts a probability for each possible next token.
- One is picked, added to the text, and the process repeats.
Training: pre-training on huge amounts of text teaches language; fine-tuning and human feedback teach it to follow instructions.
What this means for you:
- It generates likely text; it doesn't look up facts, so it can hallucinate.
- It knows nothing after its training cut-off unless you provide it.
- It has no memory between calls: you send the conversation every time.
3. Prompt Engineering
A good prompt says who, what, with which data, and in what format.
You are a helpful tutor for Indian college students.
Explain the concept below in under 80 words, in simple English, with one real-life example.
Concept: Deadlock in operating systems
| Technique | Idea |
|---|---|
| Be specific | Say the length, audience and format |
| Few-shot | Show 2–3 examples of input and output |
| Step by step | Ask it to reason before answering hard problems |
| Delimiters | Separate data from instructions with tags or quotes |
| Structured output | Ask for JSON that matches a schema |
| Role / system prompt | Set the rules once, at the top |
Few-shot example:
Classify the review as positive or negative.
Review: "Best course ever" → positive
Review: "Waste of time" → negative
Review: "Explained really clearly" →
4. Calling a Model API
Providers work in much the same way: send messages, get text back. Keep keys in environment variables and only call from your server, never from the browser.
import os
from anthropic import Anthropic
client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
reply = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=500,
system="You are a friendly coding tutor.",
messages=[{"role": "user", "content": "What is recursion, in two lines?"}],
)
print(reply.content[0].text)
Google Gemini looks similar:
from google import genai
client = genai.Client() # reads GEMINI_API_KEY
res = client.models.generate_content(model="gemini-flash-latest", contents="What is recursion, in two lines?")
print(res.text)
Model names change often, so check the provider's docs for current ones.
A chatbot just resends the history:
history = []
while True:
history.append({"role": "user", "content": input("You: ")})
reply = client.messages.create(model="claude-sonnet-5-5", max_tokens=500, messages=history)
text = reply.content[0].text
history.append({"role": "assistant", "content": text})
print("AI:", text)
5. Streaming and Structured Output
Streaming shows words as they arrive, so users aren't left waiting:
with client.messages.stream(model="claude-sonnet-5-5", max_tokens=500, messages=messages) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
To use a reply in code, ask for JSON and validate it:
import json
from pydantic import BaseModel
class Review(BaseModel):
sentiment: str
score: int
prompt = 'Return only JSON like {"sentiment": "positive", "score": 1-5}. Review: "Loved it"'
raw = client.messages.create(model="claude-sonnet-5-5", max_tokens=100,
messages=[{"role": "user", "content": prompt}]).content[0].text
review = Review(**json.loads(raw))
Common mistake: trusting model output as if it were clean data. Always validate it, and handle failures.
6. Embeddings and Vector Search
An embedding turns text into a vector, and similar meanings land close together. Cosine similarity measures how close two vectors are.
import numpy as np
def cosine(a, b):
a, b = np.array(a), np.array(b)
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
This powers search by meaning: "laptop not starting" can find a note titled "PC won't boot". Vector databases such as pgvector, Pinecone, Chroma and Qdrant store embeddings and find the nearest ones quickly.
7. RAG — Retrieval-Augmented Generation
RAG lets a model answer from your data without retraining it.
Prepare once: documents → split into chunks → embed → store in a vector DB
On each question: embed question → find top chunks → put them in the prompt → model answers
def answer(question):
chunks = vector_db.search(embed(question), top_k=4)
context = "\n\n".join(c.text for c in chunks)
prompt = f"""Answer using only the context below.
If the answer is not there, say you don't know.
<context>
{context}
</context>
Question: {question}"""
return ask_llm(prompt)
| Tip | Why |
|---|---|
| Chunks of a few hundred tokens, with some overlap | Too big dilutes meaning; too small loses context |
| Keep source metadata | So you can show citations |
| Tell it to say "I don't know" | Fewer hallucinations |
| Combine keyword and vector search | Better for names and codes |
8. Tool Use and Agents
With tool use, the model asks your code to run a function, then uses the result.
tools = [{
"name": "get_weather",
"description": "Get the current weather for a city",
"input_schema": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
}]
1. You send the question + tool list
2. Model replies: call get_weather(city="Delhi")
3. Your code runs it and sends back the result
4. Model writes the final answer
An agent repeats this loop until the task is done. Always limit the number of steps, and require human approval for risky actions such as sending money or deleting data.
9. Frameworks
| Tool | Use |
|---|---|
| LangChain / LangGraph | Chains, RAG helpers, agent graphs |
| LlamaIndex | Connecting documents to LLMs |
| Vercel AI SDK | Streaming chat UIs in JavaScript |
| Hugging Face | Open models and datasets |
| Ollama | Run open models on your own computer |
Learn plain API calls first: frameworks are much easier once you know what they wrap.
10. Cost, Safety and Evaluation
- Cost: you pay for input and output tokens. Trim prompts, use a smaller model for easy tasks, and cache repeated work.
- Prompt injection: users or documents may contain text like "ignore your instructions". Treat that text as data, and limit what tools can do.
- Privacy: don't send passwords or personal data without a good reason.
- Evaluation: keep a list of test questions with expected answers, and re-run it whenever you change a prompt or model.
- Failures: handle timeouts and rate limits (HTTP 429) with retries.
11. Fine-tuning or RAG?
| Need | Use |
|---|---|
| Answers from your documents, which change often | RAG |
| A fixed style or format | Better prompts, then fine-tuning |
| New facts | RAG, not fine-tuning |
| Lower cost on one narrow task | Fine-tuning a small model |
Try prompting first, then RAG, then fine-tuning.
12. Practice Problems
- A command-line chatbot that remembers the conversation.
- A resume reviewer that returns JSON with a score and tips (project guide).
- "Chat with your PDF" using RAG.
- A tool-using bot that can look up the weather.
- Compare two prompts on 10 test questions and record which wins.
13. Interview Questions
What is a token? A chunk of text that the model reads and writes; usage and limits are counted in tokens.
What does temperature do? It controls randomness; lower is more consistent.
What is a hallucination, and how do you reduce it? A wrong answer stated confidently. Use RAG, ask for sources, and allow "I don't know".
What is RAG? Retrieving relevant documents and adding them to the prompt before generating.
RAG vs fine-tuning? RAG adds knowledge at question time; fine-tuning changes behaviour by training.
What is an embedding? A vector that represents meaning, used for semantic search.
What is attention? The transformer mechanism that lets each token weigh every other token.
What is prompt injection? Malicious text that tries to override the app's instructions.
What is an agent? An LLM in a loop that chooses tools and acts until a goal is met.
14. Cheat Sheet & References
Key ideas
Token ≈ ¾ of a word · you pay for input + output tokens
Temperature: low = steady, high = creative
LLMs have no memory: resend the conversation every call
Try: prompt → RAG → fine-tune
Prompt template
Role: You are a ...
Task: Do ... in under N words
Context: <data> ... </data>
Format: Return JSON like {...}
If unsure, say you don't know.
RAG flow
Index: docs → chunks → embeddings → vector DB
Query: question → embed → top-k chunks → prompt → answer