Getting everything ready for you…

Loading courses, problems and your progress. This takes just a moment.

0s

Generative AI & LLM APIs Documentation

Build with LLMs — how they work, prompting, model APIs, streaming, embeddings, RAG, tool use and agents.

Generative AI models create text, code and images. As a developer you mostly call a model through an API and build a product around it. This page covers how LLMs work, prompting, API calls, RAG and agents. Examples use Python; see the Python docs if you need a refresher.

1. Introduction

Term Meaning
LLM Large language model: predicts the next piece of text
Token A chunk of text, about ¾ of an English word; usage is billed per token
Context window The most tokens a model can read at once, including your prompt and its answer
Prompt The instructions and data you send
Temperature Randomness: low = steady, high = creative
Embedding A list of numbers that captures meaning
Hallucination A confident answer that is wrong
RAG Giving the model your documents at question time
Agent A model that calls tools in a loop to finish a task

2. How an LLM Works

  1. Text is split into tokens.
  2. A transformer uses attention to weigh how tokens relate to each other.
  3. It predicts a probability for each possible next token.
  4. One is picked, added to the text, and the process repeats.

Training: pre-training on huge amounts of text teaches language; fine-tuning and human feedback teach it to follow instructions.

What this means for you:

  • It generates likely text; it doesn't look up facts, so it can hallucinate.
  • It knows nothing after its training cut-off unless you provide it.
  • It has no memory between calls: you send the conversation every time.

3. Prompt Engineering

A good prompt says who, what, with which data, and in what format.

You are a helpful tutor for Indian college students.
Explain the concept below in under 80 words, in simple English, with one real-life example.

Concept: Deadlock in operating systems
Technique Idea
Be specific Say the length, audience and format
Few-shot Show 2–3 examples of input and output
Step by step Ask it to reason before answering hard problems
Delimiters Separate data from instructions with tags or quotes
Structured output Ask for JSON that matches a schema
Role / system prompt Set the rules once, at the top

Few-shot example:

Classify the review as positive or negative.

Review: "Best course ever" → positive
Review: "Waste of time" → negative
Review: "Explained really clearly" →

4. Calling a Model API

Providers work in much the same way: send messages, get text back. Keep keys in environment variables and only call from your server, never from the browser.

import os
from anthropic import Anthropic

client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

reply = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=500,
    system="You are a friendly coding tutor.",
    messages=[{"role": "user", "content": "What is recursion, in two lines?"}],
)
print(reply.content[0].text)

Google Gemini looks similar:

from google import genai

client = genai.Client()   # reads GEMINI_API_KEY
res = client.models.generate_content(model="gemini-flash-latest", contents="What is recursion, in two lines?")
print(res.text)

Model names change often, so check the provider's docs for current ones.

A chatbot just resends the history:

history = []
while True:
    history.append({"role": "user", "content": input("You: ")})
    reply = client.messages.create(model="claude-sonnet-5-5", max_tokens=500, messages=history)
    text = reply.content[0].text
    history.append({"role": "assistant", "content": text})
    print("AI:", text)

5. Streaming and Structured Output

Streaming shows words as they arrive, so users aren't left waiting:

with client.messages.stream(model="claude-sonnet-5-5", max_tokens=500, messages=messages) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

To use a reply in code, ask for JSON and validate it:

import json
from pydantic import BaseModel

class Review(BaseModel):
    sentiment: str
    score: int

prompt = 'Return only JSON like {"sentiment": "positive", "score": 1-5}. Review: "Loved it"'
raw = client.messages.create(model="claude-sonnet-5-5", max_tokens=100,
                             messages=[{"role": "user", "content": prompt}]).content[0].text
review = Review(**json.loads(raw))

Common mistake: trusting model output as if it were clean data. Always validate it, and handle failures.

7. RAG — Retrieval-Augmented Generation

RAG lets a model answer from your data without retraining it.

Prepare once:  documents → split into chunks → embed → store in a vector DB
On each question: embed question → find top chunks → put them in the prompt → model answers
def answer(question):
    chunks = vector_db.search(embed(question), top_k=4)
    context = "\n\n".join(c.text for c in chunks)
    prompt = f"""Answer using only the context below.
If the answer is not there, say you don't know.

<context>
{context}
</context>

Question: {question}"""
    return ask_llm(prompt)
Tip Why
Chunks of a few hundred tokens, with some overlap Too big dilutes meaning; too small loses context
Keep source metadata So you can show citations
Tell it to say "I don't know" Fewer hallucinations
Combine keyword and vector search Better for names and codes

8. Tool Use and Agents

With tool use, the model asks your code to run a function, then uses the result.

tools = [{
    "name": "get_weather",
    "description": "Get the current weather for a city",
    "input_schema": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
}]
1. You send the question + tool list
2. Model replies: call get_weather(city="Delhi")
3. Your code runs it and sends back the result
4. Model writes the final answer

An agent repeats this loop until the task is done. Always limit the number of steps, and require human approval for risky actions such as sending money or deleting data.

9. Frameworks

Tool Use
LangChain / LangGraph Chains, RAG helpers, agent graphs
LlamaIndex Connecting documents to LLMs
Vercel AI SDK Streaming chat UIs in JavaScript
Hugging Face Open models and datasets
Ollama Run open models on your own computer

Learn plain API calls first: frameworks are much easier once you know what they wrap.

10. Cost, Safety and Evaluation

  • Cost: you pay for input and output tokens. Trim prompts, use a smaller model for easy tasks, and cache repeated work.
  • Prompt injection: users or documents may contain text like "ignore your instructions". Treat that text as data, and limit what tools can do.
  • Privacy: don't send passwords or personal data without a good reason.
  • Evaluation: keep a list of test questions with expected answers, and re-run it whenever you change a prompt or model.
  • Failures: handle timeouts and rate limits (HTTP 429) with retries.

11. Fine-tuning or RAG?

Need Use
Answers from your documents, which change often RAG
A fixed style or format Better prompts, then fine-tuning
New facts RAG, not fine-tuning
Lower cost on one narrow task Fine-tuning a small model

Try prompting first, then RAG, then fine-tuning.

12. Practice Problems

  1. A command-line chatbot that remembers the conversation.
  2. A resume reviewer that returns JSON with a score and tips (project guide).
  3. "Chat with your PDF" using RAG.
  4. A tool-using bot that can look up the weather.
  5. Compare two prompts on 10 test questions and record which wins.

13. Interview Questions

What is a token? A chunk of text that the model reads and writes; usage and limits are counted in tokens.

What does temperature do? It controls randomness; lower is more consistent.

What is a hallucination, and how do you reduce it? A wrong answer stated confidently. Use RAG, ask for sources, and allow "I don't know".

What is RAG? Retrieving relevant documents and adding them to the prompt before generating.

RAG vs fine-tuning? RAG adds knowledge at question time; fine-tuning changes behaviour by training.

What is an embedding? A vector that represents meaning, used for semantic search.

What is attention? The transformer mechanism that lets each token weigh every other token.

What is prompt injection? Malicious text that tries to override the app's instructions.

What is an agent? An LLM in a loop that chooses tools and acts until a goal is met.

14. Cheat Sheet & References

Key ideas

Token ≈ ¾ of a word · you pay for input + output tokens
Temperature: low = steady, high = creative
LLMs have no memory: resend the conversation every call
Try: prompt → RAG → fine-tune

Prompt template

Role: You are a ...
Task: Do ... in under N words
Context: <data> ... </data>
Format: Return JSON like {...}
If unsure, say you don't know.

RAG flow

Index: docs → chunks → embeddings → vector DB
Query: question → embed → top-k chunks → prompt → answer

References