A Fast AI Content Authentication Using Parallel Distributed Computing Architecture

Sushmitha, Intern, Adori Labs, Santa Clara, California

In an age of rapidly evolving AI generated content and sophisticated deep fakes, creators face an unprecedented challenge: safeguarding ownership and authenticity. Addressing the AI risks and safety, the White House recently released an Executive Order* underscoring the need for robust media authentication to guarantee authenticity and integrity.

Adori Labs* has a patented audio watermark technology to combat these challenges.

Adori’s watermarking solution effectively authenticates ownership and empowers creators by imperceptibly encoding the creator’s unique ID into the audio that is capable of surviving any manipulations on the media.

As a software engineering intern, I had the distinct opportunity to develop an AI content authentication software using their watermark technology with guidance of the CEO of Adori Labs, Nathan Iyer, who architected the design.

This application aims to simplify and streamline the content authentication process for users while ensuring robust scalability and efficiency.

The Key Challenges

Building this application for production, exposed a number of workflow problems that need an elegant solution. Some of these challenges are listed below:

1. Scalability: The backend should be capable of handling large volumes of diverse audio and video file formats.
2. Speed or Performance: Watermarking is a computing intensive task. However, creators want this to happen quickly, without having to wait for a long time for publication of their content.
3. Simplified workflow: The application needed a simple user experience wherein users can upload multiple files, view a progress for each file, and a notification needs to be sent to the user after it is done with a download link.
4. Cost: The solution needs to be cost effective.

The Solution

A Parallel Approach: Splitting the Burden

The answer to these challenges lay in parallelization. We split large files into smaller parts, encoding each parallelly using Amazon’s serverless Lambda service. This ensured speed and minimised user wait times. The files are then stitched together and uploaded to a S3 bucket.

Diving into Architecture: A Serverless Symphony

To achieve this parallel processing efficiently, we adopted a “Distributed Serverless Event-driven” architecture using Amazon Web Services (AWS).

Figure 1: Serverless, Event-Driven, Distributed Architecture

Here’s the flow:

1. User uploads a file: It lands in an S3 bucket, triggering an event to Lambda function.
2. Lambda checks file duration: If long, it splits the file into parts and uploads them to another S3 bucket.
3. Each part triggers another Lambda function: AWS Lambda scales automatically, spinning up new instances to handle multiple uploads simultaneously.
4. Parallel encoding: Each Lambda function encodes its assigned part.
5. Merging the encoded parts: Once all parts are encoded, the final Lambda merges them back into a single file.
6. Email notification: Upon completion, an email with the download link is sent to the user.
7. Delete parts from S3: The split parts are no longer needed and can be deleted.

The advantages of this architecture include

✔ Scalability: Automatic scaling handles large uploads effortlessly.
✔ Speed: Parallelization ensures fast encoding, reducing user wait times.
✔ Cost-effectiveness: Serverless pay-per-use model keeps costs low.

The Application in Action:

Figure 2: Encoding creator’s watermark into the files

Figure 3: Decoding the watermark from the file

Other Potential Applications

The core approach of splitting large tasks, parallel processing with serverless functions, and event-driven triggering holds potential for various use cases beyond this specific project.

Other potential use cases may include -

1. Training AI models: Large datasets necessitate efficient processing. Splitting them into smaller chunks and utilizing parallel processing with Lambda functions can significantly reduce training times.
2. Data processing pipelines: Breaking down complex data processing tasks into smaller, independent steps triggered by events (e.g., data arrival) creates a scalable and manageable pipeline.
3. Image/video processing: Similar to audio watermarking, image/video resizing, manipulation, or feature extraction can benefit from parallel processing on smaller segments.
4. Log analysis: Processing large log files in real-time for fraud detection, anomaly detection, or security analysis can be optimized through this architecture.

My internship with Adori Labs was more than just a technical learning experience; it was a chance to contribute to a solution that empowers creators and tackles a critical challenge in the digital age. I am immensely grateful to Nathan Iyer and the talented team at Adori Labs for this opportunity, and I look forward to applying the knowledge and skills gained to future endeavors.

Cheers, thanks for reading! 😊

References

1. https://www.whitehouse.gov/briefing-room/statements-releases/2023/10/30/fact-sheet-president-biden-issues-executive-order-on-safe-secure-and-trustworthy-artificial-intelligence/
2. https://www.adorilabs.com/