Document of bibliographic reference 405999

BibliographicReference record

Type

Bibliographic resource

Type of document

Book chapters

Type of document

Conference paper

BibLvlCode

AM

Title

Benchmarking large language models for image classification of marine mammals

Abstract

As Artificial Intelligence (AI) has developed rapidly over the past few decades, the new generation of AI, Large Language Models (LLMs) trained on massive datasets, has achieved ground-breaking performance in many applications. Further progress has been made in multimodal LLMs, with many datasets created to evaluate LLMs with vision abilities. However, none of those datasets focuses solely on marine mammals, which are indispensable for ecological equilibrium. In this work, we build a benchmark dataset with 1,423 images of 65 kinds of marine mammals, where each animal is uniquely classified into different levels of class, ranging from species-level to medium-level to group-level. Moreover, we evaluate several approaches for classifying these marine mammals: (1) machine learning (ML) algorithms using embeddings provided by neural networks, (2) influential pre-trained neural networks, (3) zero-shot models: CLIP and LLMs, and (4) a novel LLM-based multi-agent system (MAS). The results demonstrate the strengths of traditional models and LLMs in different aspects, and the MAS can further improve the classification performance. The dataset is available on GitHub: https://github.com/yeyimilk/LLM-Vision-Marine-Animals.git.

Bibliographic citation

Qi, Y.; Cai, S.; Zhao, Z.; Li, J.; Lin, Y.; Wang, Z. (2024). Benchmarking large language models for image classification of marine mammals, in: Che, H. et al. 2024 IEEE International Conference on Knowledge Graph (ICKG), Abu Dhabi, United Arab Emirates, 11-12 December 2024. pp. 258-265. https://dx.doi.org/10.1109/ickg63256.2024.00040

Access rights

open access

Is accessible for free

true