Databricks Generative AI Engineer Lesson 6: Frameworks and Prompt Engineering in Practice

Framework Selection: No Silver Bullet, Only Scenario Matching

The debate over frameworks never stops when building generative AI applications. Some people favor LangChain's modularity, others insist on using pure Python to stay lightweight, and still others are beginning to explore DSPy's automated optimization. This lesson from Databricks makes it clear: the key to choosing a framework lies in the task itself, not in chasing trends.

LangChain provides building blocks—prompt templates, retrievers, output parsers—connected into chains through pipelines. LangGraph (from the same team) models agents as graphs with state, loops, and branches, making it suitable for complex workflows. The OpenAI Agents SDK is another popular way to write agent loops; especially when tasks are simple, pure Python combined with this SDK is sufficient because Databricks endpoints directly accept OpenAI request formats. LlamaIndex focuses on document indexing and retrieval, while DSPy automatically optimizes prompts through metrics rather than manual tuning.

Integration with Databricks: The AI Bridge Package

Regardless of which framework you choose, Databricks provides a unified connection layer through the AI Bridge integration package. After installing the Databricks package for the corresponding framework (such as the LangChain or OpenAI package), you gain platform-native components: chat models served on Databricks, AI Search indexes as retrievers, Unity Catalog functions as tools, and Genie agents as data sources. In LangChain, the chat model class points directly to the serving endpoint, and the retriever class points to the index. Teams can keep their familiar framework while swapping components without rewriting chains. MLflow's automatic logging calls cover all frameworks, ensuring that tracing and evaluation are unaffected by framework choice.

DSPy is a programmatic prompt optimization framework proposed by a team at Stanford University. Unlike manually writing prompts, DSPy abstracts the components of a prompt (instructions, examples, reasoning steps) into learnable modules and allows developers to define one or more evaluation metrics (such as accuracy or F1). DSPy's optimizers (such as BootstrapFewShot and MIPRO) then automatically search for the best combination of prompt templates and few-shot examples to maximize that metric. This effectively turns prompt engineering from a "handcraft" into an "optimizable parameter search" problem, making it especially suitable for scenarios with clear evaluation data and high manual tuning costs.

AI Bridge is a set of integration packages provided by Databricks, designed to seamlessly connect mainstream LLM application frameworks (LangChain, LlamaIndex, OpenAI SDK, etc.) with capabilities on the Databricks platform such as model serving, vector retrieval, and function calling. Its core value lies in standardized interfaces: developers do not need to change the framework's programming paradigm; they only need to replace components such as model clients and retrievers in the framework with the corresponding Databricks implementations provided by AI Bridge to gain platform-level security, monitoring, and governance. For example, LangChain's ChatDatabricks class points directly to a Databricks serving endpoint, while DatabricksVectorSearch wraps an AI Search index as a retriever.

Enriching Prompts: Mining Signals from the Question

A user's question often contains clues that go beyond its literal meaning. The lesson uses an example where a Copperline customer asks about pairing the CX450 headset: the assistant instead returns pages for other models. The solution is to break down the question's key fields (product code, order number), key terms, and intent (pairing, warranty, returns), and turn them into retrieval augmentation.

The product code becomes a metadata filter for search, the order number triggers an order lookup, and the intent selects appropriate instructions or examples. You can also rewrite vague questions into clearer search queries, or expand abbreviations spelled out in manuals. Each step adds context that the model would otherwise have to guess.

Quick Check: Using the Product Code in the Question

A customer writes: "My CX450 won't pair after the recent update." The assistant searches all manuals and often answers with other models. What is the best improvement? The answer is to extract the product code as a key field and filter the search so that only the correct manual is retrieved. Option B (using a larger model on the same mixed pages) and Option C (stuffing all manuals into the prompt) do not address the root problem, and Option D (pushing the work onto the customer) ignores the fact that the user has already specified the product.

Adjusting Responses: From Baseline to Target Output

Prompt engineering iteration requires methodology. First establish a baseline prompt, run it on real questions, and record the differences between the output and the target—too long, too vague, wrong tone, or missing sources. Then fix only one problem at a time, using explicit instructions such as "answer in three sentences," "cite the manual page number," or "keep a friendly, plain tone."

When instructions are not enough, add few-shot examples of ideal answers that simultaneously demonstrate length, tone, and format. Keep every version of the prompt so that you can compare new answers with the baseline on the same set of questions. The final check in the lesson: the answer is correct but long and technical, and the customer wants a short, friendly reply. Adding a "concise" instruction helps; what should you do next? The answer is to add few-shot examples of ideal answers, because examples convey length and tone that instructions alone cannot specify.

Few-shot Examples are several "input-desired output" examples provided directly in the prompt, allowing the model to understand task requirements through pattern matching. Compared with pure instructions (such as "please answer concisely"), examples can simultaneously convey multiple dimensions of information such as length, tone, format, and content structure, making them especially suitable for implicit requirements that are difficult to describe precisely with instructions. In prompt engineering iteration, when explicit instructions cannot fully correct output deviations, adding 2-3 high-quality examples is usually the most effective next step. Note that examples should cover typical scenarios and avoid being overly similar to the current question, which could lead to overfitting.

Key Takeaways and Next Lesson Preview

Framework selection should serve the task: LangChain builds chains, LangGraph and the OpenAI Agents SDK build agents, LlamaIndex focuses on retrieval, and DSPy optimizes prompts. Databricks AI Bridge packages connect them to serving models, AI Search, Unity Catalog functions, and Genie. Enrich prompts through key fields, terms, and intent, using filters, queries, and instructions. Start from a baseline, change one thing at a time, adjust responses with explicit instructions, and add few-shot examples when instructions are insufficient. The next lesson will discuss model selection.