Open-Source AI Desktop Pet Cat: Qwen 3.5 Omni + ESP32 Create a Full-Modal Intelligent Companion

A palm-sized AI kitten that can see, hear, speak, and even record everything you do throughout the day — this open-source project packs a full omni-modal large model into pocket-sized hardware, with surprisingly impressive results.

Project Overview: A Desktop Pet with a Soul

This open-source project creates a miniature AI desktop cat — small enough to fit in your pocket or hang around your neck — yet packed with features. On the hardware side, it's equipped with four servo motors (positioned at the neck and ears), an ultra-high-definition LCD display, and a camera, all powered by an ESP32-S3 microcontroller. On the software side, the project integrates the Qwen 3.5 Omni full omni-modal model, giving this little cat genuine "perception" and "communication" capabilities.

The most interesting design concept: you can even photograph your real cat and let it "cyber-possess" this hardware, giving you a digital cat that will never knock your stuff off the table.

Core Capabilities: The Interaction Revolution Brought by Omni-Modal Models

Emotionally Expressive Voice Interaction

Unlike traditional AI hardware, this cat's voice responses carry emotion. To understand why this matters, consider the limitations of conventional AI voice interaction systems: they use a pipeline architecture consisting of four independent modules chained together — ASR (Automatic Speech Recognition), NLU (Natural Language Understanding), NLG (Natural Language Generation), and TTS (Text-to-Speech). The problem with this architecture is that each module introduces its own latency, with cumulative delays typically reaching 1-3 seconds. More critically, paralinguistic information carried in speech — emotion, tone, speaking rate — is completely lost when ASR converts audio to plain text. When TTS regenerates speech, it can only "guess" the emotion based on text content, resulting in mechanical and stilted output.

Qwen 3.5 Omni uses an end-to-end architecture that directly maps speech input to speech output through a unified model. Emotional features are preserved as continuous vectors throughout the entire processing pipeline, bypassing the lengthy "speech recognition → text understanding → text-to-speech" chain to achieve direct speech-to-speech interaction. This means lower latency and more natural tone and emotional expression — the technical foundation that enables this AI cat to genuinely "act cute."

When you ask it to act cute, it actually responds in a coy, affectionate tone; ask it to greet you excitedly, and its intonation shifts accordingly. This nuanced emotional expression was previously difficult to achieve in AI hardware because emotional information gradually degrades through multiple conversion steps.

Visual Perception and Gesture Interaction

With a camera and connection to the omni-modal model, this AI desktop pet can "see" you. An omni model refers to a unified neural network capable of simultaneously processing multiple input and output modalities — text, speech, images, and video. Unlike traditional multimodal systems, omni-modal models use an end-to-end training paradigm where understanding and generation across all modalities share the same parameters and representation space, allowing cross-modal information to naturally fuse within a unified semantic space. Therefore, this cat's responses are based not only on your voice but also incorporate visual information through integrated reasoning, rather than two separate systems processing independently and stitching results together.

Even more fun: the project features gesture interaction — wave your hand, and the cat will follow your direction to interact with you, genuinely creating a "playing with a cat" feeling. You can even ask it "How do I look?" to test its visual capabilities. While the answer might make you laugh or cry, it can indeed see and understand your appearance.

Daily Recording and Intelligent Review

This is the most practically valuable feature of the entire project. It's essentially a "lifelogging" application — using a wearable device to continuously record personal visual and auditory experiences, then leveraging AI for structured organization and retrieval. This concept was first systematically proposed by Microsoft Research in the 2000s, but remained in the academic stage due to storage costs and limited AI comprehension capabilities at the time. Only with recent breakthroughs in large vision-language models has this type of application become accessible to ordinary users, with products like Rewind.ai and Limitless exploring similar directions.

Hang the cat around your neck when you go out, and it records your day from a first-person perspective. The large model analyzes every frame of audio and video in the background. Qwen 3.5 Omni supports longer video understanding context windows, enabling temporal correlation analysis across consecutive frames rather than isolated understanding of individual images, making cross-timespan reasoning possible. In the evening, you can click "one-click summary" and it will help you review — where you went, who you met, what interesting things you did. For forgetful people, this is essentially a portable memory assistant.

Desktop Monitoring and Health Reminders

Place the cat on your desk, and it will quietly observe your behavior: Have you been scrolling your phone too long? How long since you last drank water? Have you gotten up to stretch? How long have you been slacking off? All this information is stored in memory, and during your next conversation, it will proactively remind you to take care of your health.

This feature cleverly leverages the Omni model's audio-visual understanding capabilities, transforming a toy-level desktop pet into a genuinely useful productivity tool.

Technical Architecture Analysis: Why Qwen 3.5 Omni

The project creator admitted that they previously couldn't decide which model to use to "drive this shell."