AI Augmented DevRel Part 1

Apache NiFi, HuggingFace, BLIP, Image Captioning, Vision, AI, ML, DL

Salesforce/blip-image-captioning-large · Hugging Face

So I want to automate some DevRel activities like preparing images for publications after events and meetups. So I hooked it up to Slack, so an image can be uploaded to Slack, NiFi will retrieve the Slack message, parse out the image link, download that with OAuth, clean it up, prepare it and then run YOLO v8 and BLIP on it. This will find people and some items in the image and then create a caption for it. The processed image and the text will be posted to a Slack channel. AI Generated images are nice augmentations but as you can see they are not the best as they have weird wrong words and sometimes are a bit generic. I will still be using real-world images of my cats.

I found that BLIP with HuggingFace Transformers is really easy to run as you can see below. I am currently just running this as a execute command from NiFi, but I will make this into a processor.

python3.10 -W ignore /Users/tspann/Downloads/servers/nifi-2.0.0-M2/testblip.py $1

import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
import sys
import io

processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")
img = sys.argv[1]
raw_image = Image.open(img).convert('RGB')
inputs = processor(raw_image, return_tensors="pt")
out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))

You can see this in part of this application as for processing Meetups I want people to ask questions, I am trying Mixtral to answer them.

Mixtral: Generative Sparse Mixture of Experts in DataFlows

See also this article on Meetup interactivity.

Building an LLM Bot for Meetups and Conference Interactivity

I output my materials to #reports so everyone can get the information real-time. Then I can build Medium articles around them. I am wondering if I can have those mostly generated from my materials and maybe a model trained on my articles.

Source Code

meetups/buildinganllmformeetups at main · tspannhw/meetups

After I wrote this I tried to make a NiFi Python processor from the original code. It worked! That article is coming out soon.

I am also working on a processor to add Zilliz Milvus for RAG.

RESOURCES

Mixtral: Generative Sparse Mixture of Experts in DataFlows

Streaming Street Cams to YOLO v8 With Python and NiFi to MinIO/S3

Google Gemma for Real-Time Lightweight Open LLM Inference

LLM Pipelines with Pinecone and HuggingFace with Python and Apache NiFi