BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning

Changdae Oh, Hyeji Hwang, Hee Young Lee, Yong Taek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, Kyungwoo Song

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

4 Scopus citations


With the surge of large-scale pre-trained models (PTMs), fine-tuning these models to numerous downstream tasks becomes a crucial problem. Consequently, parameter efficient transfer learning (PETL) of large models has grasped huge attention. While recent PETL methods showcase impressive performance, they rely on optimistic assumptions: 1) the entire parameter set of a PTM is available, and 2) a sufficiently large memory capacity for the fine-tuning is equipped. However, in most real-world applications, PTMs are served as a black-box API or proprietary software without explicit parameter accessibility. Besides, it is hard to meet a large memory requirement for modern PTMs. In this work, we propose black-box visual prompting (Black-VIP), which efficiently adapts the PTMs without knowledge about model architectures and parameters. Black-VIP has two components; 1) Coordinator and 2) simultaneous perturbation stochastic approximation with gradient correction (SPSA-GC). The Coordinator designs inputdependent image-shaped visual prompts, which improves few-shot adaptation and robustness on distribution/location shift. SPSA-GC efficiently estimates the gradient of a target model to update Coordinator. Extensive experiments on 16 datasets demonstrate that BlackVIP enables robust adaptation to diverse domains without accessing PTMs' parameters, with minimal memory requirements. Code:

Original languageEnglish
Title of host publicationProceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
PublisherIEEE Computer Society
Number of pages12
ISBN (Electronic)9798350301298
StatePublished - 2023
Event2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 - Vancouver, Canada
Duration: 18 Jun 202322 Jun 2023

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
ISSN (Print)1063-6919


Conference2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023


  • Transfer
  • continual
  • low-shot
  • meta
  • or long-tail learning


Dive into the research topics of 'BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning'. Together they form a unique fingerprint.

Cite this