How to stitch images using Flutter and OpenCV
Few weeks ago I need to create a panoramic photo capture for a customer request. We were creating an app to basically upload photos of constructions and he wants his users to be able to take and see panoramic photos, for a better understanding of the construction place and status.
The problem was that I didn't find anything easy to use to achieve it. After some days of research and tests, I finally was able to use OpenCV with Flutter to make it work!
Here are the steps to implement image stitching in Flutter. Keep in mind that my goal here is to just show how to stitch the images. If you have any other treatments needs with the photos, you can feel free to go deeply into OpenCV library. There are a lot of cool image features there.
Let's begin with the easiest part of all: really stitch the images.
I'm going to assume that you already have your app project created. If you don't, please, create a new Flutter project.
Creating c++ files to really stitch the images
Under your lib folder, create a folder called "opencv-cpp'. There you should create to files:
First one: stitch.cpp. Copy the code below and paste in the file.
#include <opencv2/opencv.hpp>
#include <opencv2/stitching.hpp>
#include <opencv2/imgproc.hpp>
using namespace cv;
using namespace std;
struct tokens : ctype<char> {
tokens() : std::ctype<char>(get_table()) {}
static std::ctype_base::mask const *get_table() {
typedef std::ctype<char> cctype;
static const cctype::mask *const_rc = cctype::classic_table();
static cctype::mask rc[cctype::table_size];
std::memcpy(rc, const_rc, cctype::table_size * sizeof(cctype::mask));
rc[','] = ctype_base::space;
rc[' '] = ctype_base::space;
return &rc[0];
}
};
vector <string> getpathlist(string path_string) {
string sub_string = path_string.substr(1, path_string.length() - 2);
stringstream ss(sub_string);
ss.imbue(locale(locale(), new tokens()));
istream_iterator <std::string> begin(ss);
istream_iterator <std::string> end;
vector <std::string> pathlist(begin, end);
return pathlist;
}
Mat process_stitching(vector <Mat> imgVec) {
Mat result = Mat();
Stitcher::Mode mode = Stitcher::PANORAMA;
Ptr <Stitcher> stitcher = Stitcher::create(mode);
Stitcher::Status status = stitcher->stitch(imgVec, result);
if (status != Stitcher::OK)
{
hconcat(imgVec, result);
printf("Stitching error: %d\n", status);
}
else
{
printf("Stitching success\n");
}
cvtColor(result, result, COLOR_RGB2BGR);
return result;
}
vector <Mat> convert_to_matlist(vector <string> img_list, bool isvertical) {
vector <Mat> imgVec;
for (auto k = img_list.begin(); k != img_list.end(); ++k) {
String path = *k;
Mat input = imread(path);
Mat newimage;
// Convert to a 3 channel Mat to use with Stitcher module
cvtColor(input, newimage, COLOR_BGR2RGB, 3);
// Reduce the resolution for fast computation
float scale = 1000.0f / input.rows;
resize(newimage, newimage, Size(scale * input.rows, scale * input.cols));
if (isvertical)
rotate(newimage, newimage, ROTATE_90_COUNTERCLOCKWISE);
imgVec.push_back(newimage);
}
return imgVec;
}
extern "C" __attribute__((visibility("default"))) __attribute__((used))void stitch(char *inputImagePath, char *outputImagePath, bool isVertical) {
string input_path_string = inputImagePath;
vector <string> image_vector_list = getpathlist(input_path_string);
vector <Mat> mat_list;
mat_list = convert_to_matlist(image_vector_list, isVertical);
Mat result = process_stitching(mat_list);
Mat cropped_image;
result(Rect(0, 0, result.cols, result.rows)).copyTo(cropped_image);
imwrite(outputImagePath, cropped_image);
}Second one: create a file called main.cpp. Copy the line below and paste in the file.
#include "stitch.cpp"Making it works on Android
Downloading the Android required files
Go to OpenCV website, choose your version (today the latest is 4.8.0 and I'm going to use it. You can choose another one) and download the Android source files. You must click on "Android" option.
After downloading it, extract the files from zip and navigate to sdk/native/jni and copy the path "include".
Go to your app project and create a folder called "cmake" under yourRootFolder/android/. This is the expected result:
Paste the copied folder and files (folder include) into this cmake folder. The expected result:
These files that you copied to Android are the OpenCV source files. OpenCV works with c++ and the files pasted here contains all the c++ source code to make to magic happens.
There are more some files to copy. Navigate again to OpenCV folder that you downloaded from OpenCV website, go through sdk/native/libs and copy these paths:
- armeabi-v7a
- arm64-v8a
- x86
- x86_64
Now you must go back to your project, under cmake folder create another folder called src, another inside called jniLibs and paste all the folders copied there.
Lastly, go back to OpenCV downloaded folder and go to staticLibs. You are going to see a structure similar to last step folders. Each folder has inside a file called libopencv_stitching.a. You have to copy the file inside each folder and copy it into the equivalent folder in your project:
These .so and .a files are compiled files of OpenCV library that you also are going to need to make things work.
Creating the cmake file
We need to tell Android where are the c++ files, how to work with then and these things. Cmake file will make it possible for us.
Create an file called CMakeLists.txt under cmake folder (yourProjectRoot/android/cmake). There you should past this:
cmake_minimum_required(VERSION 3.6.0)
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/include)
add_library(lib_opencv SHARED IMPORTED)
set_target_properties(lib_opencv PROPERTIES IMPORTED_LOCATION ${CMAKE_CURRENT_SOURCE_DIR}/src/jniLibs/${ANDROID_ABI}/libopencv_java4.so)
file(GLOB CVLIBS ${CMAKE_CURRENT_SOURCE_DIR}/src/jniLibs/${ANDROID_ABI}/libopencv_stitching.a)
set(SOURCES ../../lib/opencv-cpp/main.cpp)
add_library(OpenCV_ffi SHARED ${SOURCES})
target_link_libraries(OpenCV_ffi lib_opencv ${CVLIBS})Take some time to analyze this CMakeLists.txt. It's important to try to understand what is happen there and if all the paths informed are correct.
Telling Android to read CMakeLists.txt
Ok, we've created our CMakeLists.txt, but how does Android knows where is it? You have to tell it.
Go to app/build.gradle and paste the following lines under android {} scope:
externalNativeBuild {
cmake {
path "../cmake/CMakeLists.txt"
}
}
defaultConfig {
externalNativeBuild {
cmake {
cppFlags '-frtti -fexceptions -std=c++17'
arguments "-DANDROID_STL=c++_shared"
}
}
}That's all for Android!
Making it works on iOS
iOS issues are always the worst and it wouldn't be different here. Hopes that when you are following this tutorial, this still works for iOS apps.
Downloading the iOS required files
Go to OpenCV website, choose your version (today the latest is 4.8.0 and I’m going to use it. You can choose another one) and download the Android source files. You must click on “iOS Pack” option.
Unzip the files. You are going to see a folder named opencv2.framework. That's what you need. Open your iOS module on Xcode and drag and drop the opencv2.framework folder inside Runner:
This is going to show a dialog. You must let the options this way:
Click on Finish.
You can now see opencv2 under Build Phases > Link Binary with Libraries:
Go to Build Phases -> Compile Sources, click on '+' then on "Add Other" and select the stitch.cpp file under rootOfProject/lib/opencv-cpp.
That's it.
Testing the stitch
I'm going to leave here a main.dart file with an use example.
Go to your pubspec and add the following dependencies:
image_picker: ^0.8.2
path_provider: ^2.0.11Don't forget to run a pub get, to fetch the new dependencies files.
To be able to test it on iPhone, you should update you info.plist with image_picker permission:
<key>NSPhotoLibraryUsageDescription</key>
<string>Access to photos</string>Then, under your lib folder, create a file called images_stitch.dart. You can paste the following:
import 'dart:ffi';
import 'dart:io';
import 'package:ffi/ffi.dart';
class ImagesStitch {
final dynamicLibrary = Platform.isAndroid
? DynamicLibrary.open("libOpenCV_ffi.so")
: DynamicLibrary.process();
Future<void> stitchImages(
final List<String> imagesPathToStitch,
final String imagePathToCreate,
final bool isVertical,
final Function onCompleted) async {
final stitchFunction = dynamicLibrary.lookupFunction<
Void Function(Pointer<Utf8>, Pointer<Utf8>, Pointer<Utf8>),
void Function(Pointer<Utf8>, Pointer<Utf8>, Pointer<Utf8>)>('stitch');
stitchFunction(
imagesPathToStitch.toString().toNativeUtf8(),
imagePathToCreate.toNativeUtf8(),
isVertical.toString().toNativeUtf8(),
);
onCompleted(imagePathToCreate);
}
}Lastly, also under the lib folder, create/edit a file called main.dart. Paste the following:
import 'dart:io';
import 'package:flutter/material.dart';
import 'package:image_picker/image_picker.dart';
import 'package:path_provider/path_provider.dart';
import 'images_stitch.dart';
void main() {
runApp(const MyApp());
}
class MyApp extends StatefulWidget {
const MyApp({super.key});
@override
State<MyApp> createState() => _MyAppState();
}
class _MyAppState extends State<MyApp> {
final _imagesStitch = ImagesStitch();
final ImagePicker _picker = ImagePicker();
String? _imagePathToShow = null;
@override
void initState() {
super.initState();
}
@override
Widget build(BuildContext context) {
return MaterialApp(
home: Scaffold(
appBar: AppBar(
title: const Text('Plugin example app'),
),
body: Center(
child: Column(
children: [
InkWell(
child: const Text('Click here'),
onTap: () async {
final imageFiles = await _picker.pickMultiImage();
final imagePaths = imageFiles.map((imageFile) {
return imageFile.path;
}).toList();
String dirPath =
"${(await getApplicationDocumentsDirectory()).path}/${DateTime.now()}_.jpg";
_imagesStitch.stitchImages(
imagePaths,
dirPath,
false,
(stitchedImagesPath) {
setState(() {
_imagePathToShow = dirPath;
});
},
);
},
),
Image.file(File(_imagePathToShow ?? ""))
],
),
),
),
);
}
}Check if all imports are working well. If they are, you are ready to test the stitching.
If you are using an Android Emulator or real devices, you can take photos using default emulator camera app (take two photos near to each other) and select them when clicking on "Click here".
If you are using iOS Simulator, copy two images to the simulator and do the same.
If you have any doubts or want to talk about the topic, here is my LinkedIn: https://www.linkedin.com/in/gustavocnavarro/.
If you liked this post and want to support me creating more contents about tech, please, feel free to donate me a coffee: https://ko-fi.com/gustavocnavarro