Ontologies are crucial for organizing domain knowledge, but creating them is a complex task that demands significant expertise. Ontology learning, which aims to automate this process, has advanced considerably in the past decade, especially with the rise of Large Language Models (LLMs). A recent study, conducted by researchers Roos M. Bakker, Daan L. Di Scala, Maaike H. T. de Boer, and Stephan A. Raaijmakers, delves into the challenge of identifying axioms—fundamental components that define logical relations within ontologies.
The study introduces a new benchmark called OntoAxiom, which systematically tests LLMs on their ability to identify axioms. The benchmark comprises nine medium-sized ontologies, totaling 17,118 triples and 2,771 axioms. The researchers focused on five types of axioms: subclass, disjoint, subproperty, domain, and range. To evaluate LLM performance, they compared twelve different models using three-shot settings and two prompting strategies: a Direct approach, where all axioms are queried at once, and an Axiom-by-Axiom (AbA) approach, where each prompt queries for one axiom only.
The findings reveal that the AbA prompting strategy leads to higher F1 scores than the Direct approach. However, performance varies across different types of axioms, indicating that some axioms are more challenging to identify. The domain of the ontology also influences performance. For instance, the FOAF ontology achieved a score of 0.642 for the subclass axiom, while the music ontology only reached 0.218. Larger LLMs generally outperformed smaller ones, but smaller models may still be viable in resource-constrained settings.
Overall, while the performance of LLMs is not yet sufficient to fully automate axiom identification, they can provide valuable candidate axioms. This support can significantly aid ontology engineers in the development and refinement of ontologies. As LLMs continue to evolve, their role in ontology learning is likely to become even more prominent, potentially revolutionizing how we structure and access domain knowledge.
