Creating a live video streaming application in Flutter
So have you ever been on the streets when your country has won the World Cup? Well if not you can imagine the situation of driving through all that traffic. That is what happens when you try to create an HTTP server for streaming videos. So today I’ll be walking you through the entire process of creating a live video server to display the live video feed in a Flutter application.
Table of Contents
- Workings of video streaming
- What is WebSocket
- Creating a WebSocket
- Image construction in Flutter
- Using WebSockets in Flutter
- Results and Conclusion
- Project Repository
Workings of video streaming
Video is a series of images shown fast, these images are better known as frames. Videos have a unit called Frames Per Second (FPS) to determine how many frames pass through each second. Usually, the higher the fps the smoother the video feels.
Thus, to transmit a camera feed from one place to another we need to read the frames and send each frame from the server to the client. Now if we do this in an unoptimized manner the server will have a lot of traffic and the final video frames will have a lot of lag. So usually the frame is read as a byte array and then encoded in some kind of format like base64 encoding before sending it to the client.
Great! Now that you know the basics of how video streaming works, we can jump to the concepts of WebSockets and why we need them.
What is a WebSocket?
Basics of HTTP Protocol
Before jumping to WebSockets we must first understand the basics of the HTTP protocol. Not going into formal definitions, the HTTP protocol is request-response-based. Meaning that unless the client requests some data the server will not respond, or in other words, the server responds to requests sent by the client.
Introduction to WebSockets
Now WebSockets are built over this conventional HTTP protocol, thus they provide some additional benefits. Formally WebSockets are defined as
A computer communications protocol, that provides full-duplex communication channels over a single TCP connection
WebSockets work in a really interesting way, for initializing a connection between the client and the server both ends need to perform a handshake.
A handshake, in this case, is exactly what it sounds like, both the client and server check and change their respective settings to match each other and once all the configurations are done a handshake denotes the final connection has been established.
Now, after a handshake is done since the connection is a full-duplex connection the server can send data to the client without it specifically requesting it and vice versa.
Now, you can imagine how easy it is to overcome our world cup problem, because now instead of you going through all that traffic the groceries are directly delivered to you with you just calling the store. Thus, WebSockets are really useful for several applications where there is live data transfer involved.
A great example of WebSockets is any form of chatting application, like WhatsApp, you receive a message from your friend directly, there is no need for you to specifically open the chat and do something to get messages.
If you know any other examples do put them in the comments for everyone.
OK !! Enough with the theoretical parts, let us get into the implementation of all these concepts and do some real coding. So in this tutorial, I am going to use Python to create WebSockets, but there are a lot of tools available out there for your specific needs.
Creating a WebSocket
Now before the creation of the WebSocket, I am assuming that you know the basics of Python, if not I highly recommend you watch this video series before moving forward
Nice, now that you know the basics let’s set up the Python environment to start the WebSockets programming
Installing Dependencies
First, we will install the required Dependencies, we will be doing it in a virtual environment, if you don’t know what it is you can read all about it here. We are going to need 3 packages for this project
- websockets
- asyncio
- cv2 (OpenCV)
#!bin/bash
# Run this bash sript to create and install required dependencies
# Install pipenv for virtual environment
python -m pip install pipenv
# Create virtual environment and install required packages
pipenv install websockets asyncio opencv-python
# Activate virtual environment
pipenv shellNow as we have all the packages needed, you may have noticed a package called Asyncio, this package is used for asynchronous programming in Python, otherwise, everything in Python is synchronous by default.
If you don’t have any idea of what synchronous and asynchronous mean you should watch this video before moving forward
Creating the WebSocket Server
Now as explained in the video above we are going to use some synchronous and some asynchronous features in this application. We will start by reading the camera input using OpenCV (cv2 package) this is done using the following code. Here note that we are using the yield keyword and not the return which will give the output but not break the while loop
import cv2
cap = cv2.VideoCapture(0)
while cap.isOpened():
_, frame = cap.read()
## This is code useful while testing
## Show the captured video feed
# cv2.imshow("Video", frame)
# if cv2.waitKey(1) & 0xFF == ord('q'):
# break
yeild frame
# cv2.destroyAllWindows()
cap.release()Once the input part is set, you can check out the type of the frame variable, it comes out to be a byte array. So now we will encode that byte array into a base64 encoded string that can be done using the imencode method in OpenCV
import cv2
import base64
encoded = cv2.imencode(frame)[1]
data = str(base64.b64encode(encoded))
data = data[2:len(data)-1]So, what’s going on here is that first the Numpy array is converted to a byte array, which is then encoded into a base64 string. Note that we explicitly converted it to a string that’s because the Python interpreter will convert it to a bytes object by default. Then we sliced that string to remove the two starting and last characters in the string. This has to be done because the Python interpreter adds “ b‘ ” at the start of a byte string and the ending quote at the end to identify it. Thus, we remove the excess material and keep only the useful material.
Now coming to the WebSocket part, now as we have our data ready we need to send that data over to the flutter application using WebSockets this is achieved using the following code
import websockets
import asyncio
import cv2, base64
port = 5000
print("Started server on port : ", port)
async def transmit(websocket, path):
print("Client Connected !")
try :
cap = cv2.VideoCapture(0)
while cap.isOpened():
_, frame = cap.read()
encoded = cv2.imencode('.jpg', frame)[1]
data = str(base64.b64encode(encoded))
data = data[2:len(data)-1]
await websocket.send(data)
# cv2.imshow("Transimission", frame)
# if cv2.waitKey(1) & 0xFF == ord('q'):
# break
cap.release()
except websockets.connection.ConnectionClosed as e:
print("Client Disconnected !")
cap.release()
except:
print("Someting went Wrong !")
start_server = websockets.serve(transmit, port=port)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
cap.release()Ok, so a lot is going on here, let’s break it down.
First, we have declared the variable port, this is the port on which the WebSocket will be hosted. The next is the function notice we are using the async keyword to declare the function as asynchronous. Then we started the live video capture and encoded the data into a base64 string. Now we are sending that data over to the WebSocket and we are using the await keyword, which means we are telling the function to wait until all the data is sent.
Now coming to the asyncio part, we started the server on the port declared and called the transmit function as a callback function. Then we are asyncio to run this server forever i.e. until the program has been terminated manually. Note if you plan on hosting this WebSocket you may need to do some things differently, refer to the documentation for the same
Now we finally have a server that can transmit frame data to a WebSocket, now we need to create a client to render that into a real video.
Image construction in Flutter
Before moving to the usage of WebSockets in Flutter we need to understand how images are processed in Flutter. Flutter has a ton of inbuilt widgets which help a lot in the development of the application easily. One such widget is the Image widget, it has a lot of factory methods that can be used to render images from various sources. One such method is the memory() method which renders the byte array into an image, and with the gapless playback option, we can render multiple images as a video using it.
Using WebSockets in Flutter
Flutter’s open-source plugin community known as pub dev has a great plugin for integrating WebSockets in the Flutter application. The plugin goes by the name “web_socket_channel”. Thus, to install it just add the following line to your pubspec.yaml file inside your dependencies
web_socket_channel: ^2.1.0
That is pretty much everything you need to get the Flutter. Now we create two files called “websocket.dart” and “VideoStreaming.dart” for the handling of respective functions. The code for websockets.dart is given below
import 'dart:async';
import 'package:web_socket_channel/web_socket_channel.dart';
import 'package:web_socket_channel/status.dart' as status;
class WebSocket {
// ------------------------- Members ------------------------- //
late String url;
WebSocketChannel? _channel;
StreamController<bool> streamController = StreamController<bool>.broadcast();
// ---------------------- Getter Setters --------------------- //
String get getUrl {
return url;
}
set setUrl(String url) {
this.url = url;
}
Stream<dynamic> get stream {
if (_channel != null) {
return _channel!.stream;
} else {
throw WebSocketChannelException("The connection was not established !");
}
}
// --------------------- Constructor ---------------------- //
WebSocket(this.url);
// ---------------------- Functions ----------------------- //
/// Connects the current application to a websocket
void connect() async {
_channel = WebSocketChannel.connect(Uri.parse(url));
}
/// Disconnects the current application from a websocket
void disconnect() {
if (_channel != null) {
_channel!.sink.close(status.goingAway);
}
}
}So basically we have created a class that contains all the methods required for handling the WebSocket functions, this class is also known as a repository. Here I have used the websocketchannel extension to connect to a WebSocket from a given URL string
This is the code for the VideoStreaming.dart file
import 'dart:convert';
import 'dart:typed_data';
import 'package:flutter/material.dart';
import 'package:videostreaming_tut/src/VideoStream/websocket.dart';
import 'package:videostreaming_tut/src/styles/styles.dart';
class VideoStream extends StatefulWidget {
const VideoStream({Key? key}) : super(key: key);
@override
State<VideoStream> createState() => _VideoStreamState();
}
class _VideoStreamState extends State<VideoStream> {
final WebSocket _socket = WebSocket("ws://<your_network_ip>:<port>");
bool _isConnected = false;
void connect(BuildContext context) async {
_socket.connect();
setState(() {
_isConnected = true;
});
}
void disconnect() {
_socket.disconnect();
setState(() {
_isConnected = false;
});
}
@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(
title: const Text("Live Video"),
),
body: Padding(
padding: const EdgeInsets.all(20.0),
child: Center(
child: Column(
children: [
Row(
mainAxisAlignment: MainAxisAlignment.spaceBetween,
children: [
ElevatedButton(
onPressed: () => connect(context),
style: buttonStyle,
child: const Text("Connect"),
),
ElevatedButton(
onPressed: disconnect,
style: buttonStyle,
child: const Text("Disconnect"),
),
],
),
const SizedBox(
height: 50.0,
),
_isConnected
? StreamBuilder(
stream: _socket.stream,
builder: (context, snapshot) {
if (!snapshot.hasData) {
return const CircularProgressIndicator();
}
if (snapshot.connectionState == ConnectionState.done) {
return const Center(
child: Text("Connection Closed !"),
);
}
//? Working for single frames
return Image.memory(
Uint8List.fromList(
base64Decode(
(snapshot.data.toString()),
),
),
gaplessPlayback: true,
excludeFromSemantics: true,
);
},
)
: const Text("Initiate Connection")
],
),
),
),
);
}
}In the above code, the major part starts from line no 64 which makes a StreamBuilder widget. This is used to render an output when the input is a stream, meaning it will take a stream input and re-render itself when the data inside the stream changes which is the exact implementation we need. Then we make the StreamBuilder listen to the stream emitted by the WebSocket and create a builder callback accordingly. In this builder function, we have the 2 mandatory parameters, which are the BuilderContext and the AsyncSnapshot. Here snapshot is the thing which gets the data from the stream and context is used to render it to the UI. In this, we check if the snapshot has data and if the condition is satisfied we decode the base64 string into a byte array and pass that value to the Image.memory method to get a frame. This process continues until we disconnect from the WebSocket
Results & Conclusion
So all this was fine but what we all want to see is the results right?
So .. drumroll please, this is the final result
For this blog post, I’ve read the teaser trailer for Doctor Strange but you can follow this tutorial to do pretty much any live video display you like.
So to conclude we can say that if you want to display any live video feeds on Flutter from a Python backend this is one way to do it.
Repository
All the above code can be found in my repository
Find my writings interesting you can find my other articles here.
