‘Self-contained’ here means a captioning model
‘Self-contained’ here means a captioning model you can bundle and run offline, no hosted API. A few that work well embedded:
– BLIP / BLIP-2 (Salesforce) — the standard open captioning models. Salesforce/blip-image-captioning-large via Hugging Face transformers gives solid one-line descriptions; runs on CPU (slowly) or a modest GPU.
– Florence-2 (Microsoft) — newer, small (0.23B / 0.77B), Apache-2.0, does captioning plus grounding/detection from one model. Best quality-to-size ratio today.
– ViT-GPT2 (nlpconnect/vit-gpt2-image-captioning) — lighter and older; fine for short captions with minimal footprint.
– moondream2 — a small vision-language model built to run locally, easy to prompt for a caption.
Minimal BLIP example:
from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image
proc = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")
image = Image.open("photo.jpg").convert("RGB")
inputs = proc(image, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=40)
print(proc.decode(out[0], skip_special_tokens=True))
For shipping inside other software:
– Footprint / licence — check each model’s licence for redistribution. Florence-2 and ViT-GPT2 are permissive; some larger VLMs aren’t.
– Runtime — export to ONNX (or use optimum) to avoid a full PyTorch dependency in the host app. That’s usually what ‘self-contained’ ends up meaning in production.
– Caption vs alt text — raw captions (‘a dog sitting on grass’) aren’t always good alt text, which is context-dependent (same image needs different alt on a vet’s site vs a pet-food shop). If accuracy matters, add a prompt/template layer on top rather than using the raw caption.
Pick on your size and licence budget: Florence-2 for best quality-per-MB, ViT-GPT2 if you need it tiny.