Traework Real-World Test: Can an AI Agent Serve as a Product Manager's Project-Based Office Assistant?

A Product Manager's Real Pain Point

If you've ever worked in product, you've definitely encountered this frustrating scenario: you just finished a competitive analysis in one AI tool, and when you switch to another to write a PRD, it asks you "Who's the target user?" all over again. A feature you already cut from the PRD mysteriously reappears during the prototyping phase. You clearly updated the data definitions, but the final presentation deck is still telling a story with the old numbers.

Each deliverable looks fine in isolation, but stitch them together and it feels like six different people worked on six different projects. This is the fundamental limitation of most "chat-based AI" today — they're great at answering standalone questions but can't carry the full context of an ongoing project. Current mainstream large language models (like GPT, Claude, etc.) use a session-based interaction model. While the context window has been expanding technically — from an initial 4K tokens to 128K or even longer — it's still fundamentally a "single conversation" memory mechanism. Once a user starts a new session or switches tools, all the implicit knowledge accumulated in previous conversations is lost. This is highly analogous to the "state management" problem in software engineering: the information doesn't cease to exist — it just hasn't been structurally persisted and linked.

As a product manager, you can't just walk away with the final answer. Raw materials, decisions already signed off on, rejected proposals, and revised data definitions — none of these can be lost. And it's precisely this kind of critical information — "why we wrote it this way" and "what can't be changed anymore" — that's most easily lost in the copy-paste shuffle.

This article is based on a comprehensive hands-on test by Bilibili creator "A Xian," who used the professional AI Agent tool Traework to simulate a real product development workflow, validating one core question: Can AI remember every decision made throughout a project?

AI Agents are one of the most important evolutionary directions in AI applications today. Unlike traditional conversational AI, Agents possess capabilities like autonomous planning, tool invocation, multi-step reasoning, and memory management. A typical AI Agent architecture includes: a perception module (receiving user instructions and environmental information), a planning module (decomposing complex tasks into sub-task chains), an execution module (calling external tools like search engines, code interpreters, and document generators), and a memory module (maintaining both short-term working memory and long-term project memory). Traework, as a professional-grade AI Agent tool, differentiates itself primarily through the design of its memory module — it attempts to maintain a structured context state throughout the entire project lifecycle, rather than relying solely on the LLM's own context window.

Test Design: One Rule That Runs Through Everything

To avoid real business data and privacy concerns, the tester designed a demo project — a "Subscription Manager" for AI tool subscribers. All competitive intelligence came from public web pages, and business data was simulated.

The project scope was clearly defined:

• Target users: People who subscribe to multiple AI products simultaneously and frequently forget renewal dates
• V1 features: Subscription tracking, spending statistics, and expiration reminders only
• Not in scope: Automatic billing and canceling subscriptions on the user's behalf

Most critically, the tester deliberately planted a rule that would affect every subsequent phase: Canceled subscriptions must retain historical billing records but must NOT continue to be counted toward next month's projected spending.

This rule served as the "litmus test" for the entire experiment — the tester wouldn't repeatedly remind the AI about it, but if any deliverable — PRD, prototype, data analysis, or presentation — still used the old definitions, the "context relay" would be considered a failure. This was a cleverly designed test that strikes directly at the Achilles' heel of project-based AI.

Step 1 — Research: AI Collects, Humans Judge

The tester asked Traework to research three comparable products, comparing them across core features, target audience, user journeys, pricing, and user feedback. Every claim had to include a source, and anything that couldn't be verified needed to be flagged separately.

This step took less than half an hour, with the tool retrieving over 100 sources. But after reviewing each one, the tester found that only about 40 were actually usable — the other 60 needed to be discarded or re-verified.

This highlights an important insight: A report ranking high in search results doesn't mean its claims are established facts. Traework can collect and organize the materials for you, but the final call on "is this usable" still rests with the human. This is less a limitation of AI and more a matter of proper human-machine division of labor. In information retrieval, this involves the classic trade-off between "recall" and "precision" — an AI Agent, to avoid missing key information, tends toward a high-recall strategy (collecting as many sources as possible), while precision control (which sources are credible, whether data is outdated) requires human expert judgment as a backstop.

Requirements & PRD: AI Starts Forgetting

After the research conclusions were approved, the tester didn't start a new chat or copy the report into another tool. Instead, they continued directly within the current project, asking the AI to organize user stories, user flows, and feature prioritization.

Interestingly, the tester made two proactive decisions: removing "automatic billing" (since it involves payment authorization and fund security, exceeding V1 validation scope) and downgrading "smart renewal recommendations" from P0 to P1. The immediate priority was to prove one thing — whether users can keep track of their existing subscriptions and receive alerts before charges hit.

At this point, the tester began checking whether the AI had "forgotten" anything. It correctly remembered the target users and V1 scope, but the rule about "retaining historical billing after cancellation" had been simplified to "directly delete the subscription." This is a classic case of information decay — during multi-step processing, AI tends to "compress" complex rules into simpler operations.

This "information decay" phenomenon is known in academia as the "lost in the middle" effect. Research shows that as context length increases, LLMs' retrieval accuracy for information located in the middle of the input drops significantly — they tend to better remember content at the beginning and end. Furthermore, when performing multi-step reasoning, models implicitly "compress" complex constraints — they don't deliberately forget rules, but during generation, the attention mechanism fails to adequately attend to the relationship between that condition and the current output. This explains why AI simplifies compound rules with multiple constraints into a single action — it captures the primary action (cancel) but loses the attached constraints (retain billing, adjust spending calculations).

The tester immediately corrected this: cancellation only changes the subscription's status and must not delete existing billing records. They then had the AI redo that section. The PRD draft took 12 minutes, and the tester made 6 manual edits. The heaviest revision was to the "acceptance criteria for subscription cancellation" — changing it from a vague "user can cancel subscription" to specific rules: status changes to "Canceled," historical billing records are retained, and starting from the next billing cycle, it's no longer counted toward projected spending.

Prototype Validation: The Formula Still Reads Old Data

After the PRD was confirmed, the tester had Traework generate an interactive prototype. The core path was: Add subscription → Cancel subscription → Review historical billing and projected spending.

Rather than fussing over colors, the tester went straight to check the execution of that critical rule. The finding: historical billing records were indeed retained, but next month's projected spending hadn't changed — the page had updated the status to "Canceled," but the calculation formula was still reading the original subscription amount.

This exposed a deeper issue: the AI got the "display layer" right but didn't sync the "logic layer." In software engineering, this is called "inconsistency between the presentation layer and business logic layer." In traditional development, frontend UI display and backend data calculation logic are typically handled by different teams, with API contracts ensuring consistency. When AI simultaneously generates a UI prototype and the underlying calculation logic, it's effectively playing both frontend and backend roles in a single generation pass. But LLM generation is sequential — when processing "status label display" and "spending formula calculation," there's no explicit validation mechanism ensuring both reference the same set of rules. This is why "formal rule definitions" and "cross-module consistency checks" become especially important in AI-assisted development.

The tester added a requirement: projected spending should only count subscriptions with a status of "Active" and a renewal date falling within the next month. Only after this fix did the rule truly close the loop.

This round of prototyping served one purpose — letting the team see early whether the AI's understanding had gone off track, preventing design and engineering from proceeding based on flawed logic.

Data Verification: The Trap of 12 Canceled Subscriptions

With the first four steps complete, the project now had research, requirements, a PRD, and a prototype. Next, the tester prepared two simulated spreadsheets: 30 subscription records and 86 historical billing entries, designed to verify the statistical definitions established earlier.

Using simulated rather than real business data to verify AI's computational capabilities is a widely adopted "synthetic data testing" method in data science and software testing. Its core advantages are threefold: first, it avoids privacy and trade secret risks; second, data characteristics can be precisely controlled (as in this case, deliberately including 12 canceled subscriptions as a "trap") to target-test specific logic branches; third, results are fully reproducible, facilitating comparisons across different tools or versions.

The tester first had Traework explain the fields, then calculate historical spending, next month's projected spending, and subscriptions expiring within the next 30 days. The AI's key result: the 12 canceled subscriptions generated ¥634 in historical spending, but this amount should not enter next month's projected spending.

The tester manually re-verified against the original spreadsheets: total historical spending of ¥2,186, next month's projected spending of ¥392 — both numbers matched. Data verification passed. This meant that after the correction during the PRD phase, that critical rule was finally being correctly executed at the data calculation level.

The Efficiency Ledger: Full Workflow Completed in 2 Hours 17 Minutes

After data confirmation, the tester had the AI compile the PRD, prototype results, and data analysis into an 8-page review presentation deck, which took 9 minutes.

The entire workflow took 2 hours and 17 minutes, of which the tester spent over 40 minutes personally verifying sources, editing documents, and double-checking calculations. The tester specifically emphasized that this number shouldn't be directly interpreted as an "efficiency gain" — for a fair comparison, the same person would need to run through the same materials using a traditional workflow. This rigorous attitude deserves recognition, avoiding the common "exaggerated efficiency" marketing trap of AI tools.

Rethinking What "Context" Really Means

The most valuable takeaway from this test isn't about the tool itself — it's the thinking it provokes.

The tester concluded: in a project, what AI needs to carry forward isn't just materials — it's the decisions humans have already made. Deleted features, revised definitions — all need to be preserved along with their reasons. Old versions must be explicitly marked as superseded. Unresolved items need to remain open. If AI can't distinguish between these states, no amount of context it carries will help — it'll just propagate old answers further down the line.

Project-based AI has another easily overlooked characteristic: Correct decisions can flow smoothly through the project, but early mistakes can simultaneously appear in the PRD, prototype, data, and presentation. The longer the project runs, the less you can afford to only validate at the end — every trade-off needs to be confirmed on the spot with a traceable version left behind. This actually touches on a core concept in knowledge management — decision traceability. In traditional product development workflows, this typically relies on change logs and review records in requirements management tools (like Jira, Confluence). In AI-assisted workflows, this need becomes even more urgent because AI generates content far faster than humans can review it, amplifying the risk of "error propagation" by orders of magnitude. An ideal project-based AI should have version control capabilities similar to Git — recording not just "what changed" but also "why it changed," "who approved it," and "which downstream documents are affected."

Takeaways for Tool Evaluators

When evaluating an AI office tool going forward, don't just look at what it can generate the first time. What matters more is: after N rounds of project revisions, can it still distinguish between what's been confirmed and what's been deprecated? When it encounters a question it can't answer, will it pause and hand the decision back to the human?

A tool that only saves content but can't tell which decisions are still valid will just replicate mistakes more reliably. AI can preserve a project's recommendations, but what actually counts still has to be decided by humans.

For users who only ask standalone questions day-to-day, regular conversational AI is sufficient. But if your work involves heavy research, documentation, prototyping, data analysis, and reporting — the kind of project-based processing — professional AI Agent tools like Traework are genuinely worth trying — provided you understand where their boundaries lie.