How to upload 50 OpenCV frames into cloud storage within 1 second

How to read RTSP/Video frames and upload them into cloud storage in an asynchronous way

Bala Venkatesh
Nov 12, 2020 · 4 min read

USE CASE

The solution is, we can upload frames in an asynchronous way using Celery.

from celery import Celery
from celery.result import AsyncResult
from celery.result import allow_join_result
from celery.decorators import periodic_task
app = Celery(‘tasks’, backend=’redis://guest@127.0.0.1:6379', broker=’redis://guest@127.0.0.1:6379')
@app.task(bind=True, max_retries=30)
def upload_frames_gcs(self, file_name):
try:
url = upload_file_to_gcs(file_name)
return url
except Exception as e:
raise self.retry(exc=e)

Chains in Celery

Groups in Celery

jobs = group(upload_frames_gcs.s(file_name, ts) for ts, file_name in file_name_dic.items())result = jobs.apply_async()
def taskid_status(task_id_array):
for task in task_id_array:
if task.successful():
task_id_array.remove(task)
with allow_join_result():
frames_array = []
for results in task.join():
frame_dic = {}
frame_dic['frame_url'] = results[0]
frames_array.append(frame_dic)
return task_id_array, frames_array

I have tested the performance with multiple dissimilar test cases.

If you want further assistance or help. please feel free to contact me balavenkatesh.com 📬 I am happy to help you.🖥️

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data…

Sign up for Analytics Vidhya News Bytes

By Analytics Vidhya

Latest news from Analytics Vidhya on our Hackathons and some of our best articles! Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Bala Venkatesh

Written by

I have a passion for understanding technology at a fundamental level and Sharing ideas and code. * Aspire to Inspire before I expire* https://balavenkatesh.com

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Bala Venkatesh

Written by

I have a passion for understanding technology at a fundamental level and Sharing ideas and code. * Aspire to Inspire before I expire* https://balavenkatesh.com

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data Science professionals. We are building the next-gen data science ecosystem https://www.analyticsvidhya.com

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store