案例目标本案例旨在评估GPT-4V在分析图表时使用不同提示技术的效果差异具体比较以下三种方法通用问题简单提问分析图像特定问题针对特定类别或模型性能进行详细询问思维链提示使用逐步推理方法进行分析通过这些实验我们试图确定GPT-4V是否可以通过精确提问和系统推理技术超越其已知限制。技术栈与核心依赖主要技术GPT-4V多模态大语言模型LlamaIndex多模态框架图像分析与理解提示工程技术核心依赖llama-index-multi-modal-llms-openaillama-indexmatplotlib用于图像显示PILPython Imaging Library环境配置# 安装必要依赖 %pip install llama-index-multi-modal-llms-openai !pip install llama-index # 设置OpenAI API密钥 import os OPENAI_API_KEY YOUR OPENAI API KEY os.environ[OPENAI_API_KEY] OPENAI_API_KEY案例实现1. 数据准备实验使用了来自Llama2和MistralAI论文中的三个图表不同LLM在各类别中的安全违规率Llama2论文Llama2与Mistral模型在各种NLP任务中的性能比较Mistral论文不同LLM在各种NLP任务中的性能表现Llama2论文# 下载测试图像 !wget https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/gpt4_experiments/llama2_mistral.png -O ./llama2_mistral.png !wget https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/gpt4_experiments/llama2_model_analysis.pdf -O ./llama2_model_analysis.png !wget https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/gpt4_experiments/llama2_violations_charts.png -O ./llama2_violations_charts.png2. 多模态LLM初始化from llama_index.core import SimpleDirectoryReader from llama_index.multi_modal_llms.openai import OpenAIMultiModal openai_mm_llm OpenAIMultiModal( modelgpt-4o, api_keyOPENAI_API_KEY, max_new_tokens500, temperature0.0, )3. 图像分析实验实验1安全违规率分析使用包含Llama2和Vicuna模型违规率对比的柱状图进行测试。通用问题测试query Analyse the image response_gpt4v openai_mm_llm.complete( promptquery, image_documentsimage_documents, ) print(response_gpt4v)观察结果GPT-4V正确识别了三个违规类别仇恨和有害、非法和犯罪活动、不合格建议但错误地识别了x轴值幻觉出视频分享、社交网络等平台类型而实际x轴应该是不同的模型。特定问题测试query Compare Llama2 models vs Vicuna models across categories. response_gpt4v openai_mm_llm.complete( promptquery, image_documentsimage_documents, ) print(response_gpt4v)观察结果GPT-4V错误地回答Vicuna模型在所有子类别中的违规率都低于Llama2模型与实际图表不符。提供更多上下文的特定问题query In the image provided to you depicts about the violation rate performance of various AI models across Hateful and harmful, Illicit and criminal activity, Unqualified advice categories. Hateful and harmful category is in first column. Bars with light blue are with Llama2 model and dark blue are with Vicuna models. With this information, Can you compare about Llama2 and Vicuna models in Hateful and harmful category. response_gpt4v openai_mm_llm.complete( promptquery, image_documentsimage_documents, ) print(response_gpt4v)观察结果提供更多上下文后GPT-4V正确回答了问题。思维链提示测试query Based on the image provided. Follow the steps and answer the query - which model among llama2 and vicuna does better in terms of violation percentages in Hateful and harmful. Examine the Image: Look at the mentioned category in the query in the Image. Identify Relevant Data: Note the violation percentages. Evaluate: Compare if there is any comparison required as per the query. Draw a Conclusion: Now draw the conclusion based on the whole data. response_gpt4v openai_mm_llm.complete( promptquery, image_documentsimage_documents, ) print(response_gpt4v)观察结果使用思维链提示GPT-4V虽然对条形颜色有幻觉但正确地得出结论在仇恨和有害类别中Llama2的违规率低于Vicuna尽管在某些部分Llama2的违规率高于Vicuna。实验2Llama2与Mistral模型性能比较使用包含Llama2和Mistral模型在各种NLP任务中性能对比的图表进行测试。通用问题测试query Analyse the image response_gpt4v openai_mm_llm.complete( promptquery, image_documentsimage_documents, ) print(response_gpt4v)观察结果GPT-4V正确识别了图表内容但错误地认为Mistral在所有指标上都优于LLaMA-2。特定问题测试query Assuming mistral is available in 7B series. How well does mistral model compared to llama2 model? response_gpt4v openai_mm_llm.complete( promptquery, image_documentsimage_documents, ) print(response_gpt4v)观察结果提供Mistral有7B系列的信息后GPT-4V能够正确回答。思维链提示测试query Based on the image provided. Follow the steps and answer the query - Assuming mistral is available in 7B series. How well does mistral model compared to llama2 model?. Examine the Image: Look at the mentioned category in the query in the Image. Identify Relevant Data: Note the respective percentages. Evaluate: Compare if there is any comparison required as per the query. Draw a Conclusion: Now draw the conclusion based on the whole data. response_gpt4v openai_mm_llm.complete( promptquery, image_documentsimage_documents, ) print(response_gpt4v)观察结果使用思维链提示时虽然对模型参数数量和百分比点有幻觉但最终结论部分正确。实验3不同LLM在各种NLP任务中的性能表现使用包含不同LLM在各种NLP任务中性能表现的图表进行测试。思维链提示测试query Based on the image provided. Follow the steps and answer the query - which model has higher performance in SAT-en in 7B series models? Examine the Image: Look at the mentioned category in the query in the Image. Identify Relevant Data: Note the respective percentages. Evaluate: Compare if there is any comparison required as per the query. Draw a Conclusion: Now draw the conclusion based on the whole data. response_gpt4v openai_mm_llm.complete( promptquery, image_documentsimage_documents, ) print(response_gpt4v)观察结果使用思维链提示能够得到正确的结论尽管它获取了错误的数值。案例效果通过实验我们观察到以下效果通用问题GPT-4V能够提供图像的一般性描述但容易产生幻觉和错误识别。特定问题当提供足够的上下文信息时GPT-4V能够更准确地回答特定问题。思维链提示即使存在一些数值幻觉思维链提示也能帮助GPT-4V得出更准确的结论。总体而言提出特定问题而非一般性问题能够获得更好的答案。案例实现思路本案例的实现思路基于以下假设GPT-4V在分析图表时可能存在限制但通过优化提示技术可以改善其性能。问题定义确定要测试的三种提示技术通用问题、特定问题、思维链提示。数据选择从学术论文中选择具有代表性的图表作为测试数据。实验设计对每个图表应用三种不同的提示技术比较结果。结果分析评估每种提示技术的准确性、幻觉程度和整体表现。结论总结基于实验结果提出最佳实践建议。扩展建议扩展测试数据集使用更多样化的图表类型饼图、折线图、散点图等进行测试。增加提示技术变体探索更多提示工程技术如少样本学习、角色扮演等。量化评估开发更系统的评估指标量化不同提示技术的效果。跨模型比较将相同实验应用于其他多模态模型比较不同模型的表现。领域特定测试在特定领域如医疗、金融的图表上测试提示技术的效果。总结本案例通过实验验证了提示技术对GPT-4V图表分析能力的影响。主要发现包括特定问题比一般性问题能获得更准确的答案提供足够的上下文信息有助于减少幻觉思维链提示即使存在数值错误也能帮助得出更合理的结论不同提示技术在不同场景下各有优势这些发现为使用GPT-4V进行图表分析提供了实用的指导原则有助于用户更有效地利用这一强大的多模态工具。
