Use of AI in Ontology Development

 

The report examines the role and potential of artificial intelligence methods, particularly large language models (LLMs), in the process of developing and maintaining ontologies. It initially emphasizes that the success of these methods significantly depends on the decades-long efforts of human experts in creating and managing ontologies, as these ontologies are included in the training data of LLMs.

The challenges in evaluating the performance of AI in this field include limitations arising from the training data cutoff date of the models and the subjective nature of ontology construction. Accurate evaluation requires using terms not present in the models' training data and leveraging human expertise to determine the correctness and quality of ontologies generated or suggested by AI.

The "Future Directions" section proposes strategies to improve the use of AI, such as customizing the Retrieval Augmented Generation (RAG) method to prioritize higher-quality terms and utilizing hybrid vector store and graph database backends for more precise information retrieval.

The report explores various methods for integrating AI into ontology editing environments. These methods include creating plugins for existing tools like Protégé, integration into tabular editing environments, designing new user interfaces focused on text-based interactions (such as CurateGPT), and direct integration into LLM chat interfaces (like GPTs in ChatGPT). In all these methods, receiving user feedback and enabling the acceptance, rejection, or modification of AI suggestions for continuous system improvement are of paramount importance.

To enhance the quality and reliability of AI-generated results, the use of automated validation methods is also discussed. These methods include employing OWL reasoners for filtering and inferring relationships and using RAG to find evidence in scientific literature.

Beyond generating new terms, the report emphasizes the need to develop AI capabilities to support additional workflows such as the maintenance, correction, and refactoring of existing ontologies. Utilizing information from issue tracking systems to automate requested changes is also presented as a promising approach.

In conclusion, the report finds that AI has significant potential to facilitate the process of building and maintaining ontologies, but the goal is not to replace human experts but rather to augment their expertise with tools that reduce repetitive tasks and increase efficiency. Evaluations have shown that AI-generated relationships have high precision but may not be comprehensive, and the quality of AI-generated term definitions is acceptable but not as good as human-generated definitions. Therefore, close collaboration between AI and human experts is essential for achieving accurate and high-quality ontologies.

Source: Dynamic Retrieval Augmented Generation of Ontologies using Artificial Intelligence (DRAGON-AI)