LGMT
An open-source tool for detecting logical reasoning consistency in LLMs based on metamorphic testing. It defines metamorphic relations using first-order logic and verifies their logical correctness with the Z3 theorem prover, identifying self-contradictions on equivalent problems without manual annotation.
Timeline (last 90 days)
LGMT is an open-source tool that detects reasoning inconsistencies in large language models through metamorphic relations based on first-order logic.
LGMT has 20 built-in metamorphic relations built on first-order logic, such as swapping the order of premises, substituting equivalent propositions, and adjusting quantifier structure.
LGMT supports user-defined metamorphic relations and verifies their logical correctness through the Z3 theorem prover, ensuring that the generated equivalent cases are formally reliable.
LGMT comes with a benchmark containing 3,091 metamorphic groups.
Tested on six widely used LLMs, the inconsistency rate ranges from 20.84% to 29.44%.
LGMT is released as open source on GitHub, offering two ways to use it: a command-line interface (CLI) and Web Studio.
LGMT provides an offline mock mode that can run without an API key.
The LGMT project address is https://github.com/MaxwelsDonc/lgmt-tool, and the demo video address is https://youtu.be/gou4opptu1U.
All Facts (8)
LGMT is an open-source tool that detects reasoning inconsistencies in large language models through metamorphic relations based on first-order logic.
50%UnverifiedLGMT has 20 built-in metamorphic relations built on first-order logic, such as swapping the order of premises, substituting equivalent propositions, and adjusting quantifier structure.
50%UnverifiedLGMT supports user-defined metamorphic relations and verifies their logical correctness through the Z3 theorem prover, ensuring that the generated equivalent cases are formally reliable.
50%UnverifiedLGMT comes with a benchmark containing 3,091 metamorphic groups.
50%UnverifiedTested on six widely used LLMs, the inconsistency rate ranges from 20.84% to 29.44%.
50%UnverifiedLGMT is released as open source on GitHub, offering two ways to use it: a command-line interface (CLI) and Web Studio.
50%UnverifiedLGMT provides an offline mock mode that can run without an API key.
50%UnverifiedThe LGMT project address is https://github.com/MaxwelsDonc/lgmt-tool, and the demo video address is https://youtu.be/gou4opptu1U.
50%