Unverified50% confidenceFactExact time
When a UI element is selected in the desktop client, a screenshot is captured and sent to the model along with the corresponding React component, props, and type information.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/6/2026
First Seen
Valid until: 1/4/2027
Sources
Related Entities
Related Claims
Unverified视觉模型接收每张屏幕截图并输出结构化信息，包括OCR识别文本、页面布局、UI控件的像素坐标框及元素状态75% similarUnverifiedComputer Use通过将屏幕截图作为视觉输入，模型识别UI元素后输出鼠标点击或键盘操作指令，与依赖DOM结构的Selenium、Playwright有本质区别74% similarVerifiedComputer Use understands interface layout and content semantics by directly viewing screenshots through a multimodal large model, which is fundamentally different from traditional RPA's reliance on predefined rules and fixed UI element locating74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/981593API
curl https://kongchang.com/api/v1/knowledge/claims/981593MCP
get_claim(id=981593)