MLLM Interpretability Research

Interpretability of Multimodal Large Language Models

This research focuses on understanding and improving the interpretability of multimodal large language models, particularly LLaVA, through attention mechanism analysis and adaptive pruning.

Key Contributions

• Attention Analysis: Identified strong correlations between visual inputs and token-level outputs in LLaVA
• Mechanistic Understanding: Enhanced interpretability by uncovering underlying attention mechanisms
• Adaptive Pruning: Proposed technique to selectively prune hierarchical attention layers while preserving high-attention heads
• Efficiency Gains: Improved both accuracy and efficiency through targeted pruning strategies

Technical Innovation

Developed novel analysis frameworks for understanding how visual information flows through the model's attention layers, enabling more targeted optimization strategies for multimodal architectures.

Period: May 2024 - September 2024
Institution: Rutgers University
Advisor: Prof. Hao Wang