How Can World Models Carry Memory and Intergenerational Legacy? A Deep Dive into Microsoft Project Fez

When World Models Meet Memory: An Overlooked Design Exploration

When most people discuss World Models, the conversation tends to revolve around environment simulation, physics prediction, or game generation. Interactive world models like Google's Genie demonstrate astonishing technical capabilities, but if you search through their demos, you'd be hard-pressed to find a single example that genuinely answers "what does this mean for ordinary people's everyday lives?"

Jiwon, a UX research intern at Microsoft Project Fez—a master's student in Design Engineering from Imperial College London—set out to fill that gap. Her core research question during her internship was: How can world models empower truly meaningful personal experiences that people actually care about? Her answer: start with memory.

This choice has solid reasoning behind it. Memory is a deeply personal, relational, and ethically sensitive domain. The team drew on the 5R framework of memory (Retrieve, Recall, Reminisce, Reflect, Reconstruct) and found that existing technology already supports practical functions like "retrieval" and "recall" quite well—but the more emotionally rich dimensions, "reminisce" and "reflect," are almost entirely unexplored.

World Models are a core concept in AI, referring to systems that learn internal representations of environments through large-scale data, enabling them to predict future states, simulate scenario evolution, and generate coherent interactive experiences. Google DeepMind's Genie series is a prime example: it can generate an interactive 2D game world from a single image, letting users operate in real time within the generated virtual environment. Similarly, models like OpenAI's Sora and Meta's V-JEPA have demonstrated early forms of world models in video prediction and physical reasoning. Currently, the main applications of world models are concentrated in game generation, robot training, and autonomous driving simulation, with evaluation criteria typically focused on visual fidelity, physical consistency, and interaction fluidity. What makes Jiwon's research unique is that she pulls world models out of the "environment simulation" frame and instead asks: when a system can generate arbitrary visual worlds, can it serve humanity's most intimate and emotional needs—the preservation, transmission, and sharing of memory? This question connects the technical capabilities of world models with humanistic care, opening up a previously overlooked design space.

Four Groups, One Common Answer: Family

To understand people's real needs around memory, Jiwon recruited four groups of participants with different relationships: cross-generational family members, long-distance couples, people separated by the loss of a loved one, and deeply digitized teenagers. The recruitment process was quite challenging—the team searched extensively on LinkedIn, Facebook groups, WhatsApp communities, and even cold emails, ultimately assembling a diverse sample spanning three generations (ages 16 to over 65).

The goal of the first round of focus groups was not to design future technology, but to understand people's deepest longings around memory. The researchers designed a "memory box" exercise: participants could select only one memory they wanted to preserve and explain why. This constraint forced people to reveal their most valued priorities and desires.

The results were surprisingly consistent: when asked what matters most, people repeatedly returned to family relationships. Despite the study deliberately recruiting diverse relationship groups, family emerged as the main thread running through all narratives. Two complementary desires surfaced—older participants wanted to pass down memories, stories, and values to future generations; and both younger and older participants expressed a desire to learn about previous generations. This desire was often driven by absence: stories never told, loved ones no longer alive, family history difficult to access.

When AI Says "This Looks Like a 1940s Soldier": An Ethical Experiment About Authenticity

The second round of focus groups faced a core challenge: how do you help people without a design or machine learning background imagine entirely new forms of memory interaction within 90 minutes?

Jiwon's solution was quite ingenious. She designed a set of "capability cards" that translated key world model capabilities into "superpowers" anyone could understand—deliberately avoiding phrases like "AI features," because that would immediately anchor people to their existing assumptions about AI. Participants also received "scenario cards" based on first-round themes, such as "create a memory gift for a friend."

Only after completing their creative concepts did the researchers reveal the AI capabilities behind these "superpowers" and ask participants: what would you keep, change, or reject?

The results revealed a thought-provoking tension. People's initial reactions to AI intervening in memory were generally negative, especially when AI threatened the authenticity or personal perspective of memories. Multiple participants expressed strong reservations about AI-generated interpretations, worrying that generated content could be mistaken for fact or override personally meaningful recollections.

But as discussion deepened, a contradiction emerged: many of the desires people expressed—connecting fragmented memories, understanding memories in broader context, exploring new perspectives—were precisely the areas where AI could provide meaningful support. The unmet needs people described aligned closely with AI capabilities, despite their initial resistance to AI involvement.

The key insight: once these desires were explored more carefully, the conversation shifted from "should AI be involved" to "how should AI be involved." Acceptance depended on how AI communicated its contributions: people were more willing to accept AI presenting content as suggestions, possibilities, or interpretations rather than authoritative statements of reality. They also wanted AI to be honest about uncertainty rather than pretending to generate perfect, factually accurate visual content.

The common thread across all three points was a strong desire: to preserve space for human judgment and agency.

The 5R Framework of Memory originates from the intersection of memory studies and cognitive psychology, and is widely used to understand how humans interact with the past. The five dimensions are: Retrieve—finding information from external storage, such as searching photos or scrolling through chat history; Recall—actively extracting past experiences from the brain; Reminisce—revisiting the past with emotional warmth, often accompanied by narrative and sharing; Reflect—constructing meaning from memories, understanding how they shape the self; Reconstruct—the brain reassembles information each time it remembers, making memory variable and reshaped by present context. Existing technology (photo albums, timelines, chat search) mainly supports the first two, while reminisce and reflect—the two most emotionally rich dimensions—lack dedicated technological support. This framework explains why Jiwon chose to start from memory: world models' generative capabilities may be uniquely suited to creating new interaction forms for reminiscence and reflection, but they also raise ethical questions about the boundaries of "reconstruction"—when AI participates in reconstruction, how is the authenticity of memory safeguarded?

Steve's Digital Legacy: A Design Vision About "Being Remembered"

Based on these findings, Jiwon constructed a character scenario named Steve. Steve is a retired professor and grandfather, inspired by a real participant who was already using AI to create digital storybooks for his grandchildren.

In the envisioned future, Steve uses a pair of AI glasses and a digital legacy toolkit to begin creating a gift that transcends time for his granddaughter Sophie. The entire process is presented through five short films:

Act One: Establishing Intent. Steve opens the toolkit and finds a guide and AI glasses inside. The system invites him to write a future letter to his granddaughter and select 4-5 cherished items. Through guided conversation, AI helps Steve clarify his intent and purpose. The design insight: rather than asking Steve to define the world from the start, let objects trigger thinking, and AI responds with suggestions for possible intents. This rich, reflective dialogue allows Steve's complex and nuanced emotional desires to gradually find expression.

Act Two: Meaning-Making in Family Conversation. After dinner, Steve sits with his partner and son-in-law, laying out cherished objects. The conversation turns to Steve's own childhood and ancestors. "I want Sophie to know her roots, not just me, but the generations before." AI plays the role of "context surfacer" here—retrieving relevant information from the web, triggering reflection—but the construction of meaning still belongs to the user.

A key design choice: AI did not generate an image of Steve's grandfather. In the second round of focus groups, only 2 out of 17 participants said they would be willing to view such speculative visual content. Most preferred not to see a generated version at all, partly because knowing the image isn't factually real diminishes its emotional significance. When no original image exists, this concern is even stronger.

Act Three: The Human-AI Iteration Loop. After seeing the first version of the world model, Steve asks for a "warmer atmosphere" and gradually adjusts it to look more like Sophie's room. When AI discovers Steve doesn't have a single photo of his grandfather, it organizes fragmented information into a "puzzle board"—rather than creating a literal "AI grandfather." This deliberate avoidance of literal reconstruction responds to ethical concerns about AI resurrecting the deceased.

Act Four: The World as Shared Space. Steve invites his partner to explore this world together. The generated photo album becomes a shared reflective space, where his partner naturally asks questions and draws out meaning. AI stays in the background, capturing what emerges rather than intervening in the conversation. The design insight: the experience deliberately preserves the place of Steve's most intimate relationships, acknowledging that those who know him and share his history can offer care and understanding that AI should not attempt to replace.

Act Five: Receiver-Centered Design. Steve's granddaughter Sophie sits down to imagine a new version together. "I love ice castles and magic," she says. "I've always wanted a talking friend to go on adventures with." This scene emphasizes that the receiver's wishes matter equally, while preserving the original world co-created by Steve and his partner—multiple versions can branch and coexist rather than being forcibly merged.

Digital Legacy is a rapidly emerging interdisciplinary topic involving law, ethics, product design, and AI technology. It refers to the digital assets individuals leave behind after death—including social media accounts, photos, chat records, audio and video, and increasingly in recent years, AI-generated content. Current mainstream platforms handle this differently: Facebook allows accounts to be converted to memorial status, Google offers an Inactive Account Manager to designate data heirs, but most of these solutions remain at the level of "data access rights" and rarely touch the deeper question of "how to help future generations understand the meaning behind this data." What gives Steve's scenario design value is that it redefines digital legacy from "storage and inheritance" to "meaning transmission and relationship maintenance." AI glasses, the digital legacy toolkit, and world models together form a support system, but the core of the design is not technological showmanship—it's helping Steve complete intent clarification, family dialogue, and meaning-making during his lifetime. This stands in sharp contrast to controversial current practices like AI resurrecting the dead or generating virtual avatars, proposing a more restrained and human-centered path.

What Matters More Than Technical Capability: Four Design Principles

Jiwon's research ultimately crystallized into a set of design principles whose applicability extends far beyond world models, pointing toward a broader human-centered AI relationship:

First, AI experiences should enrich rather than replace the qualities that make us human. Experiences should create space for genuine relationships, thoughtful conversation, reflection, and imagination.

Second, establish new models of human-AI collaboration. AI empowers imagination and collaboration, but places people, relationships, and core values at the center of the experience. The process should be as valuable as the outcome—when the final work carries visible traces of people's conversations, perspectives, and care, it gains additional meaning from human effort and relationships, and ceases to be "AI slop."

Third, embed AI in familiar activities and social practices. Leverage users' environments, relationships, and online and offline resources to make interactions feel comfortable and intuitive.

Fourth, preserve time for intent and reflection before accelerating production. In emotionally significant contexts, speed should come from AI's responsiveness and ability to eliminate unnecessary effort—not from rushing reflection, interpretation, and critical evaluation, which are precisely where meaning is born.

On the ethical level, the research proposes several clear positions: making deliberate ethical choices in emotionally significant contexts depends not on limiting AI capabilities, but on deciding when, how, and whether to use them. Focus on emotionally appropriate representations rather than literal reconstruction; selectively use generative AI at different stages of the experience; preserve agency and control by making AI's contributions and roles transparent; and respect multiple ownership in collaborative creation.

AI Slop is a term that gained popularity around 2024, referring to the flood of low-quality, homogeneous AI-generated content lacking human intent. It typically has several characteristics: mass-produced, stylistically identical, factually vague, emotionally hollow, and often oriented toward traffic rather than meaning. The proliferation of AI slop has drawn widespread criticism because it not only dilutes the information ecosystem but also creates generalized distrust of AI-generated content. In her design principles, Jiwon explicitly states that when the final work carries visible traces of people's conversations, perspectives, and care, it is no longer AI slop—a criterion that places "human participation" and "relational visibility" at the core. In other words, the key to distinguishing meaningful AI experiences from AI slop lies not in the technical quality of generated content, but in whether human intent, reflection, and emotional investment are preserved in the creative process. This view has direct implications for current AI product design: rather than pursuing faster generation speeds, design interaction rhythms that can carry human relationships and reflective processes.

A Future Vision That's Less "Sci-Fi"

Jiwon closed her talk with a thought-provoking point: what she presented may be a less common, future-oriented AI vision. No human replicas, virtual cities, or sci-fi experiences. But technical ambition doesn't automatically make an experience meaningful or desirable.

Current AI is already good at creating practical value by helping people act faster and produce more. But whether this "super productivity" is truly desirable is another question. For Jiwon, desirable utility creates space for people to learn, reflect, and maintain everyday connections—the very things that make us human.

The project is planned for publication in the Late Breaking Work track at the CHI conference. The team also produced a proposal on world models as a new extension of social media, a detailed report on the second round of focus groups, and a reusable toolkit of capability cards and scenario cards.

While everyone chases bigger, faster, more realistic models, this work reminds us: the most valuable future may not be about what amazing content AI can generate, but about how its capabilities can help people better reflect, connect, and live more meaningfully.