Identifying Patterns in a Sea of Video Data

Every day, billions of hours of video are uploaded to the internet or stored in local directories. The ability to analyze and understand this vast content is more crucial than ever. NeuTube, a recent hackathon project here at Neudesic, emerged as an innovative response to this challenge. It combines the latest technology in artificial intelligence and vector search to revolutionize the way we interact with videos.

Querying videos: Finding the needle in the haystack

Video content reigns supreme. Yet, the task of pinpointing specific moments within long videos or extensive libraries is daunting, akin to finding a needle in a haystack.

During a recent Neudesic hackathon, a team embarked on a mission initially focused on detecting traffic patterns. As their exploration deepened, they envisioned a broader application — a tool capable of identifying specific frames in a multitude of videos through natural language queries. This tool, aimed at streamlining the analysis and summarization of video content, stands to benefit engineers, product managers, and designers, potentially revolutionizing the development process in various industries reliant on video analysis.

Understanding the magnitude of this challenge is crucial. In a world inundated with video content, an efficient means to navigate and extract valuable data from these digital libraries is indispensable. From traffic pattern analysis to enhancing security protocols, the potential applications are vast, marking a significant stride toward more accessible and effective video analysis, offering actionable insights for professionals across numerous disciplines.

Introducing NeuTube: Simplifying Video Analysis with AI Image Recognition

NeuTube is an innovative solution that merges the capabilities of OpenAI’s GPT-4 Vision with Azure Vector Search databases, focusing on understanding, summarizing, and accurately locating video content through ChatGPT-like interactions. This unique integration allows NeuTube not only to visualize but also to analyze and respond to queries in natural language, efficiently and precisely connecting user questions with the most relevant frames. With this capability, NeuTube opens new possibilities for transforming a variety of fields, from air traffic management to sports broadcasting, security, and journalism, among others.

Imagine a security team quickly searching for suspicious behavior in hours of footage. With NeuTube, they can simply ask: ‘At what times do two people appear together near the entrance?’ and immediately get the exact moments in the video where this occurs.

A clear example of NeuTube’s versatility is seen in the development of our Web App, exclusively focused on processing cat videos. This specialization has given us the opportunity to explore and demonstrate NeuTube’s capability to handle queries in a specific and charming theme. Through this application, we have been able to test the tool with a wide range of requests, from identifying specific moments of play and feline curiosities to detecting more complex patterns in cat behavior.

A memorable moment in development was using “cute cats licking paws” as a test query. This light-hearted example showcased the tool’s broad query handling capabilities, from simple to complex requests.

Building NeuTube: A Journey of Evolution and Adaptation

NeuTube’s inception was simple: to analyze traffic at intersections. Yet, as we delved into OpenAI’s GPT-4 Vision, the project’s scope expanded. The team’s agility allowed a shift towards a versatile video analysis tool, integrating various technologies for comprehensive video content querying through natural language with the following outcomes:

• Efficiency in analysis: NeuTube’s ability to swiftly locate specific frames using natural language queries promises to save significant time, enhancing productivity for professionals across various sectors.
• Cross-industry application: From traffic management to security and surveillance, NeuTube’s broad applicability could lead to safer and more efficient operations across numerous industries.
• User-centric design: While design thinking wasn’t the project’s focus, creating an efficient user interface was essential. We wanted to develop a straightforward web interface for video uploading and querying, ensuring the tool’s primary function — accurate video content analysis and responsive natural language queries — remained the highlight.

Navigating the hurdles: The development of NeuTube

Creating NeuTube was a journey filled with technical challenges and steep learning curves, particularly due to the pioneering integration of OpenAI’s GPT-4 Vision service with Azure’s Vector Search databases. These nascent technologies, while powerful, demanded innovative solutions to overcome early-day limitations.

Overcoming token limitations with video

A significant challenge in using GPT-4 Vision is its token limits, impacting video analysis. The solution to this hurdle lies in segmenting videos into smaller, more manageable parts. This approach ensures comprehensive coverage of the video content while maintaining the analysis’s efficiency and integrity.

Key Strategies

• Segmentation: Divide the video into short scenes or time intervals.
• Context Management: Ensure coherence within each segment and between adjacent segments.
• Summarization and Prioritization: Condense key information and focus on relevant parts to optimize analysis.
• Batch Processing and Post-Processing: Analyze segments in groups and then integrate the results for a complete analysis.

This methodology effectively overcomes the token limitations of GPT-4 Vision, ensuring a comprehensive and coherent analysis of extensive videos.

Tackling storage challenges with video

Another key challenge was storage. Fortunately, our friends at Microsoft built Azure Blob Storage for efficient video and frame management. We used this and the Azure AI Search for quick retrieval of frame summaries. By batching uploads for the GPT-4 Vision service, we ensured the tool’s effectiveness within the set limitations.

Time-boxed hackathon

The hackathon’s time constraints pushed us to make decisive technical choices swiftly. We opted for Python for its proficiency in AI applications and Angular for the UI for its modular structure, enhancing the development speed and product robustness. The decision to initially bypass You Only Look Once (YOLO), a tool for real-time object detection, allowed us to concentrate on refining NeuTube’s core capabilities, particularly in storing and analyzing frame summaries.

Solution teardown — a deep dive into NeuTube

Going deeper on the technical layer of NeuTube, we unravel the layers that empower this tool to stand out in the realm of video analysis.

Choice of Technologies

The selection of technologies for NeuTube was strategic, prioritizing efficiency, scalability, and ease of development:

• Angular for UI: Chosen for its component-based architecture and lightweight nature, Angular facilitates rapid development of rich user interfaces.
• Python for APIs: Python’s widespread acceptance in AI development circles, coupled with its simplicity and power, made it the ideal choice for scripting and backend development.
• Azure Blob Storage and AI Search: These Azure services provide reliable storage solutions and advanced search capabilities, crucial for managing large volumes of data and ensuring fast retrieval of video analysis results.

Modular Design for Optimal Functionality

NeuTube’s interface is crafted for seamless integration and interaction among its components, ensuring optimal functionality:

• User Interface (UI): Split into three pivotal sections: video uploading, query management, and video display with descriptions, making the user experience intuitive and efficient.
• Document Ingestion API: Bridges the UI with the backend, handling video uploads and initiating the segmentation process, crucial for detailed video analysis.

Core Processes: From Upload to Analysis

The journey of a video through NeuTube involves several key stages, each integral to the tool’s comprehensive analysis capabilities:

• Video Segmentation: Videos are segmented into frames upon upload, setting the stage for in-depth analysis.
• Frame Analysis: Each frame undergoes a thorough analysis by OpenAI’s GPT-4 Vision, extracting and summarizing detailed content information.
• Embedding Generation: Summaries are further processed through the ADA model to generate embeddings, vital for semantic searches.
• Storage: The system stores embeddings, summaries, and video/frame links in Azure Vector Database, ensuring quick and efficient data retrieval.

NLP and search functionality

NeuTube’s comprehension of and responses to natural language queries is powered by an NLP pipeline comprised of Backend for Frontend (BFF) Application and the ChatGPT API. The BFF acts as the intermediary, processing queries through Azure Vector Search database to fetch relevant summaries while the ChatGPT API enables NeuTube to analyze summaries and query terms, identifying and returning the most pertinent frames along with their timestamps.

A simple user experience

NeuTube’s UI is thoughtfully designed to ensure user-friendly navigation and interactive elements:

• Search Results Display: A list view of search results allows for easy identification and selection of relevant frames.
• Video Navigation: Selecting a frame from the search results automatically navigates to the specific moment in the video player, enhancing user interaction.

Realizing NeuTube’s Potential: Impact and Future Horizons

The journey towards refining NeuTube involves rigorous testing and gathering user feedback to ensure its effectiveness and utility in real-world scenarios. This iterative process will be crucial in shaping NeuTube into a tool that truly meets the needs of its users. That said, we do have an eye on the following enhancements.

• Granular Frame Analysis: Future versions may allow for deeper analysis specificity, catering to the diverse requirements of different video types.
• Expanded Video Selection: Users could select specific videos or collections for analysis, offering greater control over their queries.
• Live Feed Processing: Incorporating live video feed analysis could open up new avenues for real-time decision-making and insights.
• Audio Analysis Integration: Extending capabilities to include audio analysis would provide a more holistic understanding of video content.
• Domain-Specific Customization: Future iterations could offer customization options, allowing users to tailor analyses to specific domains or categories.

Feedback is a crucial element of any software product or service, and your contributions are highly valued. We welcome the developer community and tech enthusiasts to join us on this thrilling path. If you have suggestions, comments, or simply want to delve deeper into NeuTube, hit us up in the comments! Your involvement is vital and can make a significant impact in shaping the future of video analysis!