Building a Real-Time Voice Cooking Assistant with Firebase AI Logic

Imagine this scenario: you're cooking in the kitchen, hands covered in flour, needing to check the next step in a recipe while suddenly remembering you just used the last clove of garlic and need to add it to your shopping list. How much easier would it be to have an assistant that can answer questions in real time and help manage your grocery list? In a technical demo video, Google showcased how to build exactly this kind of real-time cooking assistant app — called Friendly Meals — using Gemini Live API and Function Calling, powered by Firebase AI Logic.

What makes this approach particularly noteworthy is that the entire AI assistant — complete with voice, video, and action capabilities — can be built securely on the client side, with virtually no complex server-side architecture required.

How Friendly Meals Works

Friendly Meals is an Android app that generates recipes based on a list of ingredients and additional requirements the user provides (such as dietary restrictions). When a user opens a recipe, they see a "Live Cooking Assistant" button. Tapping it opens the assistant interface, where the app activates the phone's camera and microphone to provide real-time help throughout the cooking process.

The demo video shows a real conversation: the user holds up a block of cheese and asks, "Is this the right cheese for this dish?" The assistant recognizes it through the camera and responds: "That looks like cheddar or a similar hard cheese, but the recipe calls for brie — a soft cheese that melts differently. Would you like me to find a substitute or add brie to your list?" The user says to add it to the list, and the assistant completes the action automatically.

This combination of visual recognition, conversational understanding, and real-world action is precisely what distinguishes an "agentic" AI assistant from an ordinary chatbot.

Firebase AI Logic: Direct Client-to-Model Connectivity

The ability to build such a real-time assistant directly on the client side is made possible by Firebase AI Logic. Through its AI Logic SDK, developers can securely call Google's generative models directly from mobile or web clients, without needing to build and maintain their own middleware server.

For developers, this dramatically lowers the barrier to delivering voice- and video-driven agentic experiences. The core idea in the demo is pairing the Gemini Live model with function calling capabilities to enable this kind of real-time, hands-free interaction.

Step 1: Declare Permissions

Since this is a video-based assistant on Android, you'll need to declare permissions for microphone and camera access in the Android manifest file. The video highlights one important detail: declaring camera functionality as "not required." This ensures that users on devices without cameras can still download the app and use features that don't depend on camera input — a practical compatibility consideration that's easy to overlook.

Step 2: Initialize the Live Model

Friendly Meals uses a dedicated class to manage the lifecycle of streaming sessions. Initialization requires passing in the model name, a voice character profile, response modality, and the set of tools to use.

The model also needs system instructions to define the "cooking expert" persona and describe the recipe the user is currently preparing. The instructions in the video go something like this: "You are a helpful real-time cooking assistant. The user is preparing the following recipe… The user will stream live cooking video and ask questions in real time. Answer accurately based on the recipe context and video content. Be concise and helpful. If the user asks to add an ingredient or item to the shopping list, call the add ingredients to grocery list function."

Traditional mobile AI app architecture typically requires a self-hosted middleware server: the client sends requests to the server, the server holds the API key and forwards requests to the AI model, then returns responses to the client. This protects the API key from being exposed through reverse engineering, but it also introduces operational overhead, latency, and scaling pressure. The core breakthrough of Firebase AI Logic is that it pushes key management and access control down to the Firebase platform layer — via App Check and Firebase's authentication mechanisms — allowing the client SDK to communicate directly with generative models without exposing credentials. Developers are freed from maintaining a proxy server, while Firebase still enforces quota management and access policies on the server side.

Function Calling: Giving the Assistant Real Hands

A conversational interface is useful, but a truly capable assistant must be able to affect the app itself. The video makes a critical distinction: when the user says "I just used the last of the olive oil — add it to the shopping list," the model shouldn't just respond "Sure, noted" and do nothing. It should actually execute that action within the app.

The first step is declaring a tool for managing grocery list items, which requires providing the function name, a description, and the parameters the function expects. This tool is added to the configuration when the Live model instance is created.

Session Handling and Data Persistence

The next step is updating the view model to pass a session handler when starting a new Live session. This handler's type is a function response part, and it defines what should happen when the add ingredient to grocery list tool is called.

The handling logic works in a few steps: first, it checks whether the function being called matches the registered function; then it validates the data — determining whether it's a JSON object or a plain string and cleaning it accordingly, and confirming that the extracted ingredient name is not empty; finally, it retrieves the currently logged-in user from Firebase Auth and writes the ingredient to that user's shopping list in Firestore.

The complete flow is: user says "figs and onions" → model recognizes intent → maps to the add ingredient to grocery list tool → extracts string parameter → triggers function to write to the list. The entire process requires no hands-on screen interaction.

Function Calling is the standard paradigm for large language models to interact with external systems. It works like this: when calling the model, the developer includes a "tool declaration" describing the available functions — their names, purposes, and a JSON Schema of required parameters. When the model determines that a user's intent requires calling a function, it doesn't execute code directly. Instead, it outputs a structured "function call request" in its response, containing the function name and extracted parameter values. The client or server receives this request, and application code is responsible for actually executing the action, then returning the result to the model, which uses it to generate a final natural-language reply. This separation — where the model handles intent understanding and the app handles execution — ensures both security (the model can't directly access the database) and gives the AI assistant the genuine ability to affect application state.

Security Recommendations Before Going Live

The video closes with two key recommendations before shipping to production. The first is to protect the app with App Check, ensuring that only legitimate apps running on uncompromised devices can call Firebase backend services. Google emphasizes this so strongly that App Check is effectively mandatory for every Firebase AI Logic call.

The second is to consult the official production checklist, which covers additional considerations before releasing AI-powered features. The video also provides a link to the specific Pull Request implementing the assistant functionality, making it easy for developers to review the Jetpack Compose UI changes as well as the complete code for handling Live sessions and bidirectional audio/video streaming.

App Check is Firebase's application integrity verification service. It uses a Device Attestation mechanism to confirm that requests originate from genuine, uncompromised app instances. On Android, App Check integrates with the Google Play Integrity API; on iOS, it uses Apple DeviceCheck or App Attest. Only verified clients receive valid tokens, granting them access to protected Firebase backend resources. For scenarios where AI models are called directly from the client, this protection layer is especially critical: without App Check, anyone with access to the app's configuration could potentially forge requests, bypass quota limits, or abuse AI calls — leading to unexpected costs and security risks.

Conclusion

The Friendly Meals case illustrates a broader trend: with tools like Firebase AI Logic, developers no longer need extensive backend infrastructure to deliver voice- and video-driven agentic experiences in mobile and web apps. Gemini Live handles real-time conversation and visual understanding, while function calling bridges intent and real-world action — together forming an assistant that can listen, see, and act. For teams looking to integrate similar capabilities into their own apps, this is a highly practical implementation pattern worth studying.