S2TDR

S2TDR is a structured multimodal reconstruction framework that converts dialogue turns containing memes into contextually appropriate textual utterances.

S2TDR decomposes meme interpretation into three interpretable sub-steps: visual cue extraction, emotion-intent reasoning, and context-aware dialogue reconstruction.

S2TDR uses OCR text only as an auxiliary cue, used when available, rather than as a core dependency.

The reconstructed dialogue can be directly fed into existing text dialogue models without any architectural modification to the downstream model.

The paper evaluates S2TDR on three typical dialogue understanding tasks: sentiment analysis, intent recognition, and dialogue summarization.

Compared with directly concatenating memes as visual input to the model, S2TDR's structured reconstruction yields general improvements across multiple downstream models.

The authors conducted component ablation experiments to verify the contributions of the visual extraction, intent reasoning, and reconstruction modules.

The authors specifically tested meme scenarios without OCR text, confirming that the framework does not depend on OCR.

S2TDR's method implicitly assumes that converting memes to text does not lose key information.

For memes that highly depend on visual style (such as the humor of a specific meme template), textual reconstruction may still involve information loss.

S2TDR's explicit reconstruction brings two benefits: interpretability (intermediate steps are auditable) and compatibility (existing text models require no changes).