Smatch++
An enhanced version of the Smatch metric used to evaluate the quality of graph-based semantic parsing results, such as AMR parsing. It measures parsing accuracy by computing the F1 score of the optimal alignment between nodes and edges of a predicted graph and a reference graph.
Timeline (last 90 days)
Under strict length constraints, APD improves DRS task performance over local masking by 5.1 and 2.4 points on the Smatch++ metric
In data-sufficient scenarios, the Smatch++ gap between a simple post-hoc filtering strategy and APD is within 0.2 points
Penalizing automaton-rejected tokens during training can improve development-set Smatch++ by 3.6 to 11.0 points after 10% of training is completed, but this advantage gradually diminishes as training progresses
Smatch++ is the primary metric for evaluating graph-based semantic parsing results, measuring parsing quality by computing the optimal-alignment F1 score between nodes and edges of the predicted graph and the reference graph
All Facts (4)
Under strict length constraints, APD improves DRS task performance over local masking by 5.1 and 2.4 points on the Smatch++ metric
50%UnverifiedIn data-sufficient scenarios, the Smatch++ gap between a simple post-hoc filtering strategy and APD is within 0.2 points
50%UnverifiedPenalizing automaton-rejected tokens during training can improve development-set Smatch++ by 3.6 to 11.0 points after 10% of training is completed, but this advantage gradually diminishes as training progresses
50%UnverifiedSmatch++ is the primary metric for evaluating graph-based semantic parsing results, measuring parsing quality by computing the optimal-alignment F1 score between nodes and edges of the predicted graph and the reference graph
50%