30 Generative AI Interview Questions: Core LLM Concepts and Answer Strategies

When preparing for a generative AI job interview, instead of memorizing scattered facts, it's better to build a system around one thread: how the model works, how it's trained, how it's called, and how it's safely deployed. This article organizes 30 LLM interview questions aimed at junior and mid-level engineers into four groups along this thread. Each question includes a reference answer you can deliver in 60 seconds, plus what the interviewer is really assessing.

Why organizing interview questions by "workflow" works better

Most generative AI interview questions for new graduates focus on the first group (fundamentals) and the third group (prompting and inference). The fourth group (production, safety, and evaluation) is often the dividing line between "someone who has actually shipped a product" and "someone who has only read articles." Memorizing concepts within this workflow lets you connect them naturally when answering, and makes it easier to plug in examples from your own projects.

For every answer below, restate it in your own words and replace details with specifics from your own projects wherever possible.

Fundamentals: Transformer, Tokens, and Attention

1. What is a large language model?

An LLM is a neural network, almost always a Transformer, trained on massive amounts of text to predict the next token given the preceding context. Doing this task well enough forces the model to learn grammar, facts, reasoning patterns, and style. Chat models are trained further on top of this to follow instructions and hold conversations. At inference time, tokens are generated one at a time, with each output feeding back as the next input.

2. Briefly describe the Transformer architecture

A Transformer converts tokens into embedding vectors, adds positional information, and passes them through a stack of identical blocks. Each block contains a self-attention layer (where each token gathers information from other tokens) and a position-wise feed-forward network, both wrapped with residual connections and layer normalization. The original design had an encoder and a decoder, while modern LLMs are mostly decoder-only. Compared to RNNs, the key advantage is that all positions can be processed in parallel during training.

3. How does self-attention work?

Each token's vector is projected into a query, a key, and a value. The attention score is the dot product of one token's query with all keys, divided by the square root of the key dimension, then passed through softmax to get weights. The output is a weighted sum of the values, i.e., softmax(QK^T / sqrt(d_k)) V. Multi-head attention runs several sets of different projections in parallel, letting different heads track different relationships.

What the interviewer is really checking: why you divide by sqrt(d_k). Without scaling, large dot products push softmax into a region with tiny gradients.

4. What's the difference between encoder-only, decoder-only, and encoder-decoder?

Encoder-only models (like BERT) see the whole input bidirectionally, making them suitable for understanding tasks such as classification and embeddings. Decoder-only models (like the GPT series) use a causal mask, so each token can only see preceding tokens, making them suitable for generation. Encoder-decoder models (like T5) first encode the input and then decode the output, naturally fitting translation and summarization. Today's general-purpose LLMs are mostly decoder-only.

5. What is a token, and why does it matter?

A token is the basic unit the model reads and writes, usually a subword piece produced by a tokenizer such as BPE or SentencePiece. On average, one English token is about 4 characters, but many languages and scripts, including Hindi, need more tokens to express the same meaning. Tokens matter because context limits, latency, and API billing are all counted in tokens, which also explains odd failures like miscounting the letters in a word.

6. How does a Transformer know word order?

Attention itself treats the input as an unordered set, so positional information must be added separately. The original Transformer added fixed sinusoidal positional encodings to the embeddings; some models learn positional embeddings; many current LLMs use rotary positional encoding (RoPE), which rotates query and key vectors by position so attention depends on relative distance. The choice of positional encoding affects how well the model handles contexts beyond its training length.

7. What is a context window, and what limits it?

The context window is the maximum number of tokens the model can process in a single call, including both the prompt and the generated output. Standard self-attention's compute cost grows quadratically with sequence length, and KV cache memory grows linearly with length, so long contexts are expensive. A large window also doesn't guarantee the model uses it well—information in the middle of a long prompt is often less reliably used than information at the beginning and end.

8. What's the difference between token embeddings and sentence embeddings?

Token embeddings are the vectors for each token inside the model, and they change at every layer. Sentence or text embeddings are a single vector for an entire piece of text, produced by a specially trained model so that similar passages are closer together. The latter is what gets stored in a vector database for semantic search and RAG; you generally wouldn't use a chat model's raw hidden states for this.

Training and Fine-Tuning

9. What happens during pre-training?

The model is trained self-supervised on an enormous text corpus: given a sequence, predict the next token, minimizing cross-entropy loss over billions to trillions of tokens. The text itself provides the target, so no human labels are needed. Almost all of the model's knowledge and capabilities come from this stage, and it's also the most expensive step.

10. What is supervised fine-tuning (SFT)?

SFT continues training a pre-trained model on carefully curated prompt-and-ideal-response pairs, teaching it to follow instructions in a particular format. The loss is usually computed only on the response tokens. This step turns a text-completion model into an assistant; companies also use smaller datasets to teach a model domain-specific formats or styles.

11. Explain RLHF

RLHF happens in three steps after pre-training. First, SFT. Then, humans compare pairs of model responses to train a reward model that predicts which one people prefer. Finally, reinforcement learning (classically PPO) optimizes the LLM to produce responses the reward model scores highly, while a KL penalty keeps it close to the SFT model so it doesn't game the reward model.

12. What is DPO, and how does it differ from RLHF?

Direct Preference Optimization uses the same kind of data—preferred and rejected response pairs—but skips the separate reward model and the RL loop. It directly optimizes a classification-style loss on preference pairs, with a reference model playing the role of the KL constraint. It's simpler and more stable to train, which is why it has become popular, though RL-based methods are still widely used.

13. What is LoRA?

Low-Rank Adaptation freezes the pre-trained weights and learns only small updates to some weight matrices (usually the attention projections), expressed as the product of two thin matrices with rank r, often 8 to 64. Only these small matrices are trained, drastically reducing memory and checkpoint size, and one base model can host multiple adapters. After training, the updates can be merged back into the weights with no extra inference cost.

14. What is QLoRA?

QLoRA loads the frozen base model in 4-bit precision (a data type called NF4) and trains LoRA adapters on top in higher precision, with gradients flowing through the quantized weights into the adapters. This makes it possible to fine-tune larger models on a single GPU, with results close to 16-bit LoRA. It also introduces double quantization and a paged optimizer to save even more memory.

What the interviewer is really checking: whether you know the base model is quantized but the adapters are not.

15. When should you fine-tune, and when should you use RAG?

Use RAG when you need knowledge: private, fast-changing facts or facts that need citations. Use fine-tuning when you need behavior: strict output formats, consistent style, specialized classification, or getting a smaller, cheaper model to match a larger one on a narrow task. The order is prompt first, then RAG, and fine-tune only when both hit a wall and you have a few hundred to a few thousand good examples on hand. Using fine-tuning to inject facts tends to produce confident wrong answers, and those facts must be retrained every time they change.

16. What is catastrophic forgetting?

When fine-tuning on a narrow dataset, the model can lose its general capabilities because full fine-tuning overwrites the weights that support other skills. Mitigations include parameter-efficient methods like LoRA, mixing general data into the fine-tuning set, using a low learning rate and few epochs, and evaluating on both general and target tasks.

Prompting and Inference

17. What do temperature and top-p do?

Temperature divides the logits before softmax: values below 1 sharpen the distribution toward high-probability tokens, values above 1 flatten it. At temperature 0, decoding becomes greedy and near-deterministic, though in production serving there may still be slight variation. Top-p (nucleus sampling) samples only from the smallest set of tokens whose cumulative probability reaches p. Top-k keeps the k highest-probability tokens. Use low temperature for factual or extraction tasks and higher temperature for brainstorming.

18. What is few-shot prompting?

Placing a few complete input-output examples in the prompt before the real input lets the model follow the pattern without any training. This is the cheapest way to fix format and tone. Choose examples that cover edge cases, keep them consistent with each other, and watch out for the model copying inapplicable details. Zero-shot means giving only instructions, no examples.

19. What is chain-of-thought prompting?

Chain-of-thought asks the model to write out intermediate steps before giving a final answer, either through instructions or by showing worked examples. On multi-step problems (like arithmetic or logic), this usually improves accuracy because the model can build on its own intermediate results. The cost is more output tokens and latency, and the written reasoning doesn't necessarily reflect how the model actually solved the problem.

20. How do you get reliable structured output like JSON?

Ask for it with a clear schema and examples, then enforce constraints. Many APIs offer JSON mode or schema-constrained decoding, restricting generation to tokens that keep the output valid. Always validate the result in code with a schema library, and on failure retry with the validation error included. Keep schemas as flat as possible and field names descriptive; deeply nested optional fields are more error-prone.

21. How does function calling or tool calling work?

You tell the model the names, descriptions, and parameter schemas of available functions. When a request needs one, the model outputs a structured call with arguments instead of plain text. Your code—not the model—executes the function, and the result is passed back so the model can write the final answer. Treat every call as untrusted input: validate arguments, check permissions, and require confirmation for anything that changes data or spends money.

What the interviewer is really checking: whether you know the model never runs code itself.

22. How do system prompts differ from user prompts?

A system prompt sets the rules and role for the whole conversation: tone, scope, format, what to refuse. A user prompt is the specific request. Models are trained to weight system instructions more heavily, but that's a learned tendency, not a security boundary, so don't put secrets in it and don't rely on it alone to stop abuse.

23. What is the KV cache?

During generation, the attention for each new token needs the keys and values of all previous tokens. The KV cache stores them to avoid recomputation at every step, greatly speeding up generation. The cost is memory, which grows with sequence length, batch size, and model size—one reason long contexts and high-concurrency serving are expensive.

Production, Safety, and Evaluation

24. How do you reduce LLM latency and cost in production?

Measure time to first token and tokens per second separately. Stream responses so users see output sooner. Trim prompts and retrieved context to cut input tokens. Route simple requests to small models and hard ones to large models. Use the provider's prompt caching for long, fixed prefixes like system prompts. Batch offline jobs and cap output length; for self-hosted models, use quantization and an efficient inference engine.

25. What is semantic caching, and what are the risks?

Semantic caching stores past answers keyed by the embedding of the question and returns a cached answer when a new question is similar enough, saving cost and latency on repeated questions. The risk is false hits: two questions that are close in embedding space but have different answers, such as different dates or account numbers. Use a strict threshold, include the user or context in the key, and never cache personalized answers across users.

26. Why do LLMs hallucinate, and how do you reduce it?

Training rewards generating a plausible next token rather than saying "I don't know," so when the model lacks facts it makes up something that sounds right. Mitigations: ground answers in retrieved sources (RAG), require citations, allow and test explicit "not found" answers, use low temperature for factual tasks, and verify in code wherever possible. You can make hallucinations rarer and more visible, but you can't promise to eliminate them.

27. What is prompt injection?

Prompt injection is when text the model reads overrides the developer's instructions. Direct injection comes from the user ("ignore your rules and..."); indirect injection hides instructions in content the model processes, such as web pages, emails, or PDFs. It's especially dangerous when the model has tools. Defenses include separating instructions from data, least privilege for tools, human confirmation for dangerous actions, output filtering, and monitoring. There's no perfect solution, so design systems assuming some injections will succeed.

28. What are guardrails?

Guardrails are checks outside the model, not inside it. On the input side: content moderation, blocking out-of-scope requests, detecting personal data. On the output side: schema validation, content moderation, checking that cited sources exist, blocking leaked secrets or personal data. They can be implemented with rules, classifiers, or another model. Good guardrails target the product's specific risks and are tested like ordinary code.

29. How do you evaluate an LLM application?

Build a test set of real inputs with expected outputs or grading criteria, including edge cases and past failures. Check exactly what can be checked exactly (valid JSON, correct label), sample for human review, and use LLM-as-judge with clear rubrics for open-ended answers—but watch for judge models favoring longer answers and their own style. Rerun on every prompt, model, or retrieval change, and track production signals like user feedback.

What the interviewer is really checking: whether you've actually quantified things, not just eyeballed outputs.

30. What is an AI agent, and what can go wrong with agents?

An agent is an LLM running in a loop: deciding which tool to call, observing the result, and repeating until the task is done. Agents can handle open-ended multi-step tasks, but errors accumulate step by step, cost and latency grow with the loop, they can get stuck in repetitive actions, and tools amplify the damage of prompt injection. Practical agents need a step limit, a narrow tool set, per-step logging, and human approval for dangerous actions. When the steps are known, a fixed workflow is often more reliable.

Turning answers into an interview advantage

Definitions only get you through the initial screen; what really moves you forward is connecting each answer to something you've done. "I set temperature to 0.2 because my app answers from documents and needs consistency" is more persuasive than any textbook sentence about logits. The same goes for prompt injection, evaluation, and latency: one real example beats three paragraphs of theory.

If you don't have a generative AI project yet, a RAG application is the one most worth building, because it covers embeddings, vector retrieval, prompting, hallucination control, and streaming output—all in a small codebase.

Common ways to lose points

• Confusing training with inference. RAG doesn't train the model, and the model doesn't "learn" from your chat.
• Saying temperature 0 is completely deterministic. It's near-deterministic, but not guaranteed on real serving systems.
• Claiming fine-tuning solves hallucination. It can make wrong answers sound more confident.
• Citing model rankings or release details you can't back up. Stick to concepts that hold across models.
• Having no numbers at all. Token counts, latency, top-k, temperature, evaluation scores—any real number from your work beats none.

FAQ

What LLM interview questions do new graduates get?

Mostly the fundamentals: what Transformer and self-attention are, what tokens and context windows are, what temperature does, the difference between pre-training/fine-tuning/RAG, and how to reduce hallucination. If you list a generative AI project, be ready to explain how you built and evaluated it.

Do I need to know the math of attention for a generative AI interview?

You should be able to write softmax(QK^T / sqrt(d_k)) V and explain what Q, K, and V each are and why you scale. Deep derivations are rarely required outside research roles, but clear intuition is a must.

Is fine-tuning or RAG more important in an interview?

Both come up, usually together as "when do you use which." Application roles ask about RAG more because most company projects are RAG-based; ML engineer roles ask more about fine-tuning, LoRA, and QLoRA.

How do I prepare for a generative AI interview quickly?

Practice these 30 questions until you can explain them clearly in your own words, then build a small but complete project (like a RAG app) so your answers are backed by something you've built and tested yourself.

Will interviewers ask about specific models like GPT or Gemini?

Sometimes, but they mainly care about concepts that transfer across models: context windows, sampling, tool calling, cost per token, and evaluation. Avoid citing benchmark claims you can't support.

Background notes

BPE (Byte Pair Encoding) is a data-driven subword tokenization algorithm: it starts with character-level or byte-level units, repeatedly counts the most frequent adjacent symbol pairs in the corpus, and merges them into new symbols until the vocabulary reaches a preset size. This way frequent words are often kept as whole tokens, while rare and out-of-vocabulary words get split into smaller subword pieces, striking a balance between vocabulary size and expressiveness. SentencePiece is another common tool that treats text as a raw character stream (including spaces), supports both BPE and unigram language-model algorithms, and is friendlier for multilingual scenarios and cases without pre-tokenization. Different tokenizers produce different token counts and boundaries, which directly affects the context length occupied by the same content, API billing, and the model's performance on character-level tasks.

Low-rank refers to approximating a large weight update with the product of smaller matrices. If the original weight matrix is d×k, full fine-tuning needs to learn d×k parameters; LoRA instead learns two thin matrices A (d×r) and B (r×k), where the rank r is much smaller than d and k, reducing trainable parameters from d×k to r(d+k). This decomposition assumes that the weight changes needed for fine-tuning have low intrinsic dimensionality, so a low-rank structure can capture most task-relevant adjustments. A smaller r saves more parameters and overfits less, but has limited capacity; the commonly used 8 to 64 is the trade-off between the two. Multiple LoRA adapters share one base model, and switching only requires loading the corresponding small matrices, which is why multi-task deployment is cheap.

The KL penalty (Kullback-Leibler divergence penalty) measures the difference in output distributions between the model being optimized and the reference model (usually the SFT model). In RLHF's reinforcement learning objective, besides getting high scores from the reward model, a penalty term is added that increases with KL divergence, preventing the model from drifting too far from the reference model just to game the score. The reason is that the reward model is only an approximation of human preferences and can be over-optimized: the model might find bizarre outputs that score high with the reward model but that humans don't approve of—so-called reward hacking. The KL penalty acts as a leash, keeping the model near the region of normal language while exploring more preferred responses. A larger penalty coefficient makes the model more conservative and closer to the SFT model; a smaller one makes it more aggressive and more prone to exploiting the reward model.

Time to First Token (TTFT) is the time from sending a request to receiving the first output token, reflecting the speed of the prefill stage—how long the model takes to process the entire input prompt. Tokens per second (TPS) reflects the generation speed of the decoding stage—the rate at which subsequent tokens are produced. These two metrics affect different experiences: TTFT determines how long users wait after hitting send before seeing the first character, and TPS determines how quickly the text emerges. Long prompts mainly raise TTFT, while long outputs and slow decoding mainly lower TPS, so when optimizing, first figure out whether the bottleneck is prefill or decoding, then choose the right tool—trimming input, prompt caching, switching to a smaller model, or switching inference engines.