{"refrec":{"BRefID":405999,"RR":"<b>Qi, Y.; Cai, S.; Zhao, Z.; Li, J.; Lin, Y.; Wang, Z.</b> (2024). Benchmarking large language models for image classification of marine mammals, <b><i>in</i></b>: Che, H. <i>et al.</i> <i>2024 IEEE International Conference on Knowledge Graph (ICKG), Abu Dhabi, United Arab Emirates, 11-12 December 2024.</i> pp. 258-265. <a href=\"https://dx.doi.org/10.1109/ickg63256.2024.00040\" target=\"_blank\">https://dx.doi.org/10.1109/ickg63256.2024.00040</a>","BEntID":403794,"PublicFlag":1,"CheckedFlag":0,"wosflag":0,"vabbflag":0,"RefStringPartII":", <b><i>in</i></b>: Che, H. <i>et al.</i> <i>2024 IEEE International Conference on Knowledge Graph (ICKG), Abu Dhabi, United Arab Emirates, 11-12 December 2024.</i> pp. 258-265. <a href=\"https://dx.doi.org/10.1109/ickg63256.2024.00040\" target=\"_blank\">https://dx.doi.org/10.1109/ickg63256.2024.00040</a>","DocTypID":17,"DocType":"Book chapters","MarineFlag":0,"FreshFlag":0,"BrackishFlag":0,"TerrestrialFlag":0,"Authorstring":"Qi, Y.; Cai, S.; Zhao, Z.; Li, J.; Lin, Y.; Wang, Z.","OrigTitleTranslFlag":0,"Authorstringtrunc":"Qi, Y. <i>et al.</i>","Englishabstract":"As Artificial Intelligence (AI) has developed rapidly over the past few decades, the new generation of AI, Large Language Models (LLMs) trained on massive datasets, has achieved ground-breaking performance in many applications. Further progress has been made in multimodal LLMs, with many datasets created to evaluate LLMs with vision abilities. However, none of those datasets focuses solely on marine mammals, which are indispensable for ecological equilibrium. In this work, we build a benchmark dataset with 1,423 images of 65 kinds of marine mammals, where each animal is uniquely classified into different levels of class, ranging from species-level to medium-level to group-level. Moreover, we evaluate several approaches for classifying these marine mammals: (1) machine learning (ML) algorithms using embeddings provided by neural networks, (2) influential pre-trained neural networks, (3) zero-shot models: CLIP and LLMs, and (4) a novel LLM-based multi-agent system (MAS). The results demonstrate the strengths of traditional models and LLMs in different aspects, and the MAS can further improve the classification performance. The dataset is available on GitHub: https://github.com/yeyimilk/LLM-Vision-Marine-Animals.git.","AbstractOtherLang":null,"BibLvlCode":"AM","StandardTitle":"Benchmarking large language models for image classification of marine mammals","OrigTitleLangCode":"en","OrigTitleLangCodeExtended":"eng","OrigTitleLangID":15,"DateLastModified":{"date":"2025-03-17 01:37:34.728976","timezone_type":1,"timezone":"+01:00"},"UserAccessRight":null,"UserAccID":null,"AuthorKeywords":null,"OtherDescriptors":null,"Notes":null,"AnaPub":2024,"MonPub":null,"DateUpdate":"2025-03-10","DateCreate":"2025-03-10","SecASFANote":null,"ConfID":null,"PeerRev":0,"VlizCoreFlag":1,"WoScode":null,"VABBcode":null,"OpenAcc":1,"DOI":"10.1109/ickg63256.2024.00040"},"refs":null,"anarec":{"AnaID":405999,"PubliDate":2024,"Pagination":"258-265","XtraPublOfAnaID":null,"ISBN":"979-8-3315-0882-1","Volume":null,"Issue":null,"BRefMon":405997,"BRefMonRR":"<b>Che, H. <i>et al.</i></b> (2024). 2024 IEEE International Conference on Knowledge Graph (ICKG), Abu Dhabi, United Arab Emirates, 11-12 December 2024. IEEE: Piscataway. ISBN 979-8-3315-0882-1. 512 pp.","BRefXtra":null,"BRefXtraRR":null,"SerBRefID":null,"SerRR":null,"StandardTitleSer":null,"ISSN":null,"AbbrevSer":null,"StandardTitleMon":"2024 IEEE International Conference on Knowledge Graph (ICKG), Abu Dhabi, United Arab Emirates, 11-12 December 2024","StartPage":258,"Pages":8,"ToPubliDate":null,"BRefBibLvlCode":"M","SerNotes":null,"AutString":"Che, H. <i>et al.</i>"},"monrec":null,"serrec":null,"relations":null,"relationsRev":null,"addrec":null,"othpubs":null,"ownerships":null,"authors":[{"AutName":"Qi","Firstname":"Yijiashun","Initials":"Y.","Affiliation":"University of Michigan","Discriminator":null,"CorporateFlag":0,"BEntID":403794,"AutID":582458,"OrderNr":1,"DegrID":null,"EditorFlag":0,"CorrespFlag":0,"IllustratorFlag":0,"ReviserFlag":0,"TranslatorFlag":0,"InsAcronym":null,"InsFSN":null,"ORCID":null,"PersID":null,"InsID":null},{"AutName":"Cai","Firstname":"Shuzhang","Initials":"S.","Affiliation":"University of Texas at Dallas","Discriminator":null,"CorporateFlag":0,"BEntID":403794,"AutID":582459,"OrderNr":2,"DegrID":null,"EditorFlag":0,"CorrespFlag":0,"IllustratorFlag":0,"ReviserFlag":0,"TranslatorFlag":0,"InsAcronym":null,"InsFSN":null,"ORCID":null,"PersID":null,"InsID":null},{"AutName":"Zhao","Firstname":"Zunduo","Initials":"Z.","Affiliation":"New York University","Discriminator":null,"CorporateFlag":0,"BEntID":403794,"AutID":582460,"OrderNr":3,"DegrID":null,"EditorFlag":0,"CorrespFlag":0,"IllustratorFlag":0,"ReviserFlag":0,"TranslatorFlag":0,"InsAcronym":null,"InsFSN":null,"ORCID":null,"PersID":null,"InsID":null},{"AutName":"Li","Firstname":"Jiaming","Initials":"J.","Affiliation":"Stony Brook University","Discriminator":null,"CorporateFlag":0,"BEntID":403794,"AutID":582461,"OrderNr":4,"DegrID":null,"EditorFlag":0,"CorrespFlag":0,"IllustratorFlag":0,"ReviserFlag":0,"TranslatorFlag":0,"InsAcronym":null,"InsFSN":null,"ORCID":null,"PersID":null,"InsID":null},{"AutName":"Lin","Firstname":"Yanbin","Initials":"Y.","Affiliation":"Florida Atlantic University","Discriminator":null,"CorporateFlag":0,"BEntID":403794,"AutID":582462,"OrderNr":5,"DegrID":null,"EditorFlag":0,"CorrespFlag":0,"IllustratorFlag":0,"ReviserFlag":0,"TranslatorFlag":0,"InsAcronym":null,"InsFSN":null,"ORCID":null,"PersID":null,"InsID":null},{"AutName":"Wang","Firstname":"Zhiqiang","Initials":"Z.","Affiliation":null,"Discriminator":null,"CorporateFlag":0,"BEntID":403794,"AutID":491550,"OrderNr":6,"DegrID":null,"EditorFlag":0,"CorrespFlag":0,"IllustratorFlag":0,"ReviserFlag":0,"TranslatorFlag":0,"InsAcronym":null,"InsFSN":null,"ORCID":null,"PersID":null,"InsID":null}],"mapdetails":null,"datasets":null,"monographs":null,"monparts":null,"serparts":null,"BEntOpen":null,"BEntPrivate":null,"availability":[{"BInstID":411912,"LibID":36,"BRefID":405999,"EmbargoDate":null,"FullEmbargoDate":null,"PhysMedID":16,"hasOCRd":1,"ShelfLocCode":"411912","RFID":null,"PaidValue":null,"Medium":"Server","Description":"VLIZ Open Access","Acronym":"VLIZ","Library":"Vlaams Instituut voor de Zee","DutchTerm":"Open access","URL":null,"ClassifID":53,"Classification":"Open access","ReqLink":null,"ClassifTypID":1,"URLLocation":"https://www.vliz.be/imisdocs/publications/","SubDir":null,"InternalReq":0,"LoggedInReq":0,"Disclaimer":null,"DutchDisclaimer":null,"FileFormat":".pdf","FileDescr":"pdf","InsPub":1,"InsID":36,"FileFormID":6,"LendableFlag":null,"PublicFlag":1,"orderLib":"A","Notes":"Reprint","AccConID":null,"AccessConstraint":null,"LicURL":null}],"litstyles":[{"LitStyID":3,"Style":"Conference paper"}],"thespers":null,"arch2discl":null,"SERpubls":null,"MONpubls":[{"PublName":"IEEE","Place":"Piscataway"}],"pictures":[],"thestermsPath":null,"thestermsASFA":null,"taxtermsASFA":null,"geotermsASFA":null,"collections":[{"Collection":"VLIZ Acknowledged Publications","ShortName":"VLIZ ackn"}],"conf":null,"proj":null,"Physdatasets":null,"spcols":{"941":{"SpName":"LifeWatch Species Information Backbone","SpColID":941,"ParSpColID":39,"TopParID":39,"ShortName":"LifeWatch Species Information Backbone","URLLocation":null,"LibID":36,"OpenRepoFlag":null,"SpTypID":null,"TopParIDNotWebsite":39,"SpColPath":"VLIZ ackn/LifeWatch Species Information Backbone"},"39":{"SpName":"VLIZ Acknowledged Publications","SpColID":39,"ParSpColID":null,"TopParID":null,"ShortName":"VLIZ ackn","URLLocation":null,"LibID":36,"OpenRepoFlag":null,"SpTypID":null,"TopParIDNotWebsite":null,"SpColPath":"VLIZ ackn"},"507":{"SpName":"World Register of Marine Species","SpColID":507,"ParSpColID":null,"TopParID":null,"ShortName":"WoRMS website","URLLocation":null,"LibID":null,"OpenRepoFlag":null,"SpTypID":null,"TopParIDNotWebsite":null,"SpColPath":"WoRMS website"},"915":{"SpName":"World Register of Marine Species (WoRMS) acknowledged","SpColID":915,"ParSpColID":941,"TopParID":39,"ShortName":"WoRMS ackn","URLLocation":null,"LibID":36,"OpenRepoFlag":null,"SpTypID":null,"TopParIDNotWebsite":39,"SpColPath":"VLIZ ackn/LifeWatch Species Information Backbone/WoRMS ackn"},"947":{"SpName":"WoRMS ackn - direct reference","SpColID":947,"ParSpColID":915,"TopParID":39,"ShortName":"WoRMS ackn - direct","URLLocation":null,"LibID":36,"OpenRepoFlag":null,"SpTypID":null,"TopParIDNotWebsite":39,"SpColPath":"VLIZ ackn/LifeWatch Species Information Backbone/WoRMS ackn/WoRMS ackn - direct"}},"doi":null,"publs":null,"serparttypes":null,"monauthors":[{"AutName":"Che","Initials":"H.","CorporateFlag":0,"BEntID":403792,"AutID":582450,"OrderNr":1,"DegrID":null,"EditorFlag":1,"CorrespFlag":0,"IllustratorFlag":0,"ReviserFlag":0,"TranslatorFlag":0,"AutStrTrunc":"Che, H. <i>et al.</i>"},{"AutName":"Fensel","Initials":"A.","CorporateFlag":0,"BEntID":403792,"AutID":582451,"OrderNr":2,"DegrID":null,"EditorFlag":1,"CorrespFlag":0,"IllustratorFlag":0,"ReviserFlag":0,"TranslatorFlag":0,"AutStrTrunc":"Che, H. <i>et al.</i>"},{"AutName":"Zhu","Initials":"H.(H)","CorporateFlag":0,"BEntID":403792,"AutID":582453,"OrderNr":3,"DegrID":null,"EditorFlag":1,"CorrespFlag":0,"IllustratorFlag":0,"ReviserFlag":0,"TranslatorFlag":0,"AutStrTrunc":"Che, H. <i>et al.</i>"},{"AutName":"Wattenhofer","Initials":"R.","CorporateFlag":0,"BEntID":403792,"AutID":582454,"OrderNr":4,"DegrID":null,"EditorFlag":1,"CorrespFlag":0,"IllustratorFlag":0,"ReviserFlag":0,"TranslatorFlag":0,"AutStrTrunc":"Che, H. <i>et al.</i>"},{"AutName":"Wu","Initials":"X.","CorporateFlag":0,"BEntID":403792,"AutID":582455,"OrderNr":5,"DegrID":null,"EditorFlag":1,"CorrespFlag":0,"IllustratorFlag":0,"ReviserFlag":0,"TranslatorFlag":0,"AutStrTrunc":"Che, H. <i>et al.</i>"}],"MParts":null,"SParts":null,"hLibs":null,"langs":[{"BEntID":403794,"AbstractFlag":0,"LangID":15,"LangCode":"en","Lang":"English","DutchTerm":"Engels","LangCodeExtended":"eng"},{"BEntID":403794,"AbstractFlag":1,"LangID":15,"LangCode":"en","Lang":"English","DutchTerm":"Engels","LangCodeExtended":"eng"}],"urls":[{"URL":"https://dx.doi.org/10.1109/ickg63256.2024.00040","externalID":"10.1109/ickg63256.2024.00040","URLTypeCode":"DOI","URLID":140986,"URLTypID":13,"URLType":"DOI","URLPrefix":"http://dx.doi.org/"}],"thesterms":null,"taxterms":null,"geoterms":null,"othterms":null,"asfacodes":null,"asfa2codes":null,"thestermsFRIS":null,"taxtermsFRIS":null,"geotermsFRIS":null,"othtermsFRIS":null,"resmessage":"","complete":1,"sessions":{"newSesName":"Chisala, Chilekwa, C.","newSesDate":{"date":"2025-03-10 07:48:24.167000","timezone_type":3,"timezone":"Europe/Brussels"},"updSesName":"Chisala, Chilekwa, C.","updSesDate":{"date":"2025-03-10 07:48:24.167000","timezone_type":3,"timezone":"Europe/Brussels"}}}