Uploading Large Files as Chunks Using ReactJS & .Net Core
File upload is one of the most common features of a web app(actually any kind of app) today. For most of the scenarios, this process is simple and straightforward. But what if most don't involve your situation?
These can come up with any kind of restrictions;
⚠ Amount of size limitation for uploading of internal network of your hosting service provider
⚠ Proxy server configurations that clients go through to upload files to your system.
⚠ Network distributing rules of the particular web server(like load balancer)
Any reason above put you in trouble if you don't wanna play this game on their rules. Actually besides all of these only limitations of browsers can make you suffer;
Its also not good practice to uploading large files by simple HTTP request this post will help you understand why its not
Lets assume you need to upload a movie, 24GB as you see below;
If you try to upload this file with a classic HTTP request you gonna restricted by browser; They will be like;
In this article, I will demonstrate how to upload a file without limitations of size using React Js and .Net Core Web API Project. So stop talking let's start coding.
I will make it as simple as possible to focus on the uploading process;
1)Create a brand new project with npx tool via code below;
# npx create-react-app file-upload-sample
2) Remove all files except App.js and index.js your project directory should look like this.
3) To achieve our goal we gonna need the aid of 4 packages; let's install them with one blow;
# yarn add axios react-bootstrap bootstrap uuid
Why do we need them? well, let me make a quick description of it;
axios; making asynchronous network requests
react-bootstrap bootstrap; make our UI looks nice, progress bar particularly
uuid; generating unique names(Guid) for each individual files
4) open index.js file copy & paste import ‘bootstrap/dist/css/bootstrap.min.css’;
your file should look like this
import React from “react”;
import ReactDOM from “react-dom”;
import App from “./App”;
import “bootstrap/dist/css/bootstrap.min.css”;ReactDOM.render(
<React.StrictMode>
<App />
</React.StrictMode>,
document.getElementById(“root”)
);
5) Let me say, If you can bear with me follow up each section to understand what's really doing or simply go to the end of the article to see the end up of App.js file
First clear the content of the App component and adjust import statements we installed above;
import React, { useEffect, useState } from "react";
import { ProgressBar, Jumbotron, Button, Form } from "react-bootstrap";
import { v4 as uuidv4 } from "uuid";
import axios from "axios";function App() {
return <></>;
}export default App;
Go official react-bootstrap site to get a progress-bar component I picked one for u, feel free to update the style of this component for your own app
Also, add all required properties that will be used for the uploading process
import React, { useEffect, useState } from "react";
import { ProgressBar, Jumbotron, Button, Form } from "react-bootstrap";
import { v4 as uuidv4 } from "uuid";
import axios from "axios";function App() {
const [showProgress, setShowProgress] = useState(false)
const [counter, setCounter] = useState(1)
const [fileToBeUpload, setFileToBeUpload] = useState({})
const [beginingOfTheChunk, setBeginingOfTheChunk] = useState(0)
const [endOfTheChunk, setEndOfTheChunk] = useState(chunkSize)
const [progress, setProgress] = useState(0)
const [fileGuid, setFileGuid] = useState("")
const [fileSize, setFileSize] = useState(0)
const [chunkCount, setChunkCount] = useState(0)const progressInstance = <ProgressBar animated now={progress} label={`${progress}%`} />; return <></>;
}export default App;
Add useEffect hook, This will provide us an iteration of uploading chunks
useEffect(() => {
if (fileSize > 0) {
fileUpload(counter);
}
}, [fileToBeUpload, progress])Add react-bootstrap components to set up UI
return (
<Jumbotron>
<Form>
<Form.Group>
<Form.File id="exampleFormControlFile1" onChange={getFileContext} label="Example file input" />
</Form.Group>
<Form.Group style={{ display: showProgress ? "block" : "none" }}>
{progressInstance}
</Form.Group>
</Form>
</Jumbotron >
);Now we have a file upload element in our web page now we need a callback function to get file details, before that chunk size property which indicates amount of size for each chunk at top of our component;
const chunkSize = 1048576 * 3;//its 3MB, increase the number measure in mbnow add our callback handler
const getFileContext = (e) => {resetChunkProperties();
const _file = e.target.files[0];
setFileSize(_file.size)const _totalCount = _file.size % chunkSize == 0 ? _file.size / chunkSize : Math.floor(_file.size / chunkSize) + 1; // Total count of chunks will have been upload to finish the file
setChunkCount(_totalCount)setFileToBeUpload(_file)
const _fileID = uuidv4() + "." + _file.name.split('.').pop();
setFileGuid(_fileID)
}
and reset function of component properties
const resetChunkProperties = () => {
setShowProgress(true)
setProgress(0)
setCounter(1)
setBeginingOfTheChunk(0)
setEndOfTheChunk(chunkSize)
}At this point we have the file to be uploaded, the number of chunks it will be split into, also beware we set fileID to ensure the name of the file will be unique in the remote source once it's done
Now we need to cut this big file to small pieces will be uploaded and set counter to follow the sequence of the process, the function below works for this;
const fileUpload = () => {
setCounter(counter + 1);
if (counter <= chunkCount) {
var chunk = fileToBeUpload.slice(beginingOfTheChunk, endOfTheChunk);
uploadChunk(chunk)
}
}The definition of the function above;
const uploadChunk = async (chunk) => {
try {
debugger
const response = await axios.post("https://localhost:44356/weatherforecast/UploadChunks", chunk, {
params: {
id: counter,
fileName: fileGuid,
},
headers: { 'Content-Type': 'application/json' }
});
debugger
const data = response.data;
if (data.isSuccess) {
setBeginingOfTheChunk(endOfTheChunk);
setEndOfTheChunk(endOfTheChunk + chunkSize);
if (counter == chunkCount) {
console.log('Process is complete, counter', counter)
await uploadCompleted();
} else {
var percentage = (counter / chunkCount) * 100;
setProgress(percentage);
}
} else {
console.log('Error Occurred:', data.errorMessage)
}} catch (error) {
debugger
console.log('error', error)
}
}
Before fading to .Net Core there is one more function we need, it indicates the last one of the chunks has been uploaded and the process is finished
const uploadCompleted = async () => {
var formData = new FormData();
formData.append('fileName', fileGuid);const response = await axios.post("https://localhost:44356/weatherforecast/UploadComplete", {}, {
params: {
fileName: fileGuid,
},
data: formData,
});const data = response.data;
if (data.isSuccess) {
setProgress(100);
}
}
If you go through each step properly App.js file should look like this at the end;
import React, { useEffect, useState } from "react";
import { ProgressBar, Jumbotron, Button, Form } from "react-bootstrap";
import { v4 as uuidv4 } from "uuid";
import axios from "axios";const chunkSize = 1048576 * 3; //its 3MB, increase the number measure in mbfunction App() {
const [showProgress, setShowProgress] = useState(false);
const [counter, setCounter] = useState(1);
const [fileToBeUpload, setFileToBeUpload] = useState({});
const [beginingOfTheChunk, setBeginingOfTheChunk] = useState(0);
const [endOfTheChunk, setEndOfTheChunk] = useState(chunkSize);
const [progress, setProgress] = useState(0);
const [fileGuid, setFileGuid] = useState("");
const [fileSize, setFileSize] = useState(0);
const [chunkCount, setChunkCount] = useState(0);const progressInstance = (
<ProgressBar animated now={progress} label={`${progress}%`} />
);useEffect(() => {
if (fileSize > 0) {
setShowProgress(true);
fileUpload(counter);
}
}, [fileToBeUpload, progress]);const getFileContext = (e) => {
const _file = e.target.files[0];
setFileSize(_file.size);const _totalCount =
_file.size % chunkSize == 0
? _file.size / chunkSize
: Math.floor(_file.size / chunkSize) + 1; // Total count of chunks will have been upload to finish the file
setChunkCount(_totalCount);setFileToBeUpload(_file);
const _fileID = uuidv4() + "." + _file.name.split(".").pop();
setFileGuid(_fileID);
};const fileUpload = () => {
setCounter(counter + 1);
if (counter <= chunkCount) {
var chunk = fileToBeUpload.slice(beginingOfTheChunk, endOfTheChunk);
uploadChunk(chunk);
}
};const uploadChunk = async (chunk) => {
try {
const response = await axios.post(
"https://localhost:44356/weatherforecast/UploadChunks",
chunk,
{
params: {
id: counter,
fileName: fileGuid,
},
headers: { "Content-Type": "application/json" },
}
);
const data = response.data;
if (data.isSuccess) {
setBeginingOfTheChunk(endOfTheChunk);
setEndOfTheChunk(endOfTheChunk + chunkSize);
if (counter == chunkCount) {
console.log("Process is complete, counter", counter);
await uploadCompleted();
} else {
var percentage = (counter / chunkCount) * 100;
setProgress(percentage);
}
} else {
console.log("Error Occurred:", data.errorMessage);
}
} catch (error) {
console.log("error", error);
}
};const uploadCompleted = async () => {
var formData = new FormData();
formData.append("fileName", fileGuid); const response = await axios.post(
"https://localhost:44356/weatherforecast/UploadComplete",
{},
{
params: {
fileName: fileGuid,
},
data: formData,
}
);const data = response.data;
if (data.isSuccess) {
setProgress(100);
}
};return (
<Jumbotron>
<Form>
<Form.Group>
<Form.File
id="exampleFormControlFile1"
onChange={getFileContext}
label="Example file input"
/>
</Form.Group>
<Form.Group style={{ display: showProgress ? "block" : "none" }}>
{progressInstance}
</Form.Group>
</Form>
</Jumbotron>
);
}
To prevent this article gets longer I will cut the configuration and helper methods usage definition which is accessible source code I'm gonna share at the end of this article
Create a new web application with an API template. I will not even change the name of the initial file, I will put my codes in WeatherForecastController.cs file.
There are 2 endpoints and 1 static helper method we need to code;
UploadChunks; Which gathering all pieces of the file and save these each chunk as a file into /Temp` folder.
UploadComplete; It indicates the latest chunk of the file has been uploaded and now time to merge these chunks to generate the original file.
MergeChunks; After the upload process is complete it stars to merge each chunk respectively and remove the merged chunk file
here is the full code;
namespace FileUploadSample.Controllers
{
public class ResponseContext
{
public dynamic Data { get; set; }
public bool IsSuccess { get; set; } = true;
public string ErrorMessage { get; set; }
}[ApiController]
[Route("[controller]")]
public class WeatherForecastController : ControllerBase
{
private IConfiguration configuration;private readonly ILogger<WeatherForecastController> _logger;
private readonly ResponseContext _responseData;
public int chunkSize;
private string tempFolder;public WeatherForecastController(IConfiguration configuration, ILogger<WeatherForecastController> logger)
{
this.configuration = configuration;
_logger = logger;
chunkSize = 1048576 * Convert.ToInt32(configuration["ChunkSize"]);
tempFolder = configuration["TargetFolder"];
_responseData = new ResponseContext();
}[HttpPost("UploadChunks")]
public async Task<IActionResult> UploadChunks(string id, string fileName)
{
try
{
var chunkNumber = id;
string newpath = Path.Combine(tempFolder + "/Temp", fileName + chunkNumber);
using (FileStream fs = System.IO.File.Create(newpath))
{
byte[] bytes = new byte[chunkSize];
int bytesRead = 0;
while ((bytesRead = await Request.Body.ReadAsync(bytes, 0, bytes.Length)) > 0)
{
fs.Write(bytes, 0, bytesRead);
}
}
}
catch (Exception ex)
{
_responseData.ErrorMessage = ex.Message;
_responseData.IsSuccess = false;
}
return Ok(_responseData);
}[HttpPost("UploadComplete")]
public IActionResult UploadComplete(string fileName)
{
try
{
string tempPath = tempFolder + "/Temp";
string newPath = Path.Combine(tempPath, fileName);
string[] filePaths = Directory.GetFiles(tempPath).Where(p => p.Contains(fileName)).OrderBy(p => Int32.Parse(p.Replace(fileName, "$").Split('$')[1])).ToArray();
foreach (string filePath in filePaths)
{
MergeChunks(newPath, filePath);
}
System.IO.File.Move(Path.Combine(tempPath, fileName), Path.Combine(tempFolder, fileName));
}
catch (Exception ex)
{
_responseData.ErrorMessage = ex.Message;
_responseData.IsSuccess = false;
}
return Ok(_responseData);
}private static void MergeChunks(string chunk1, string chunk2)
{
FileStream fs1 = null;
FileStream fs2 = null;
try
{
fs1 = System.IO.File.Open(chunk1, FileMode.Append);
fs2 = System.IO.File.Open(chunk2, FileMode.Open);
byte[] fs2Content = new byte[fs2.Length];
fs2.Read(fs2Content, 0, (int)fs2.Length);
fs1.Write(fs2Content, 0, (int)fs2.Length);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message + " : " + ex.StackTrace);
}
finally
{
if (fs1 != null) fs1.Close();
if (fs2 != null) fs2.Close();
System.IO.File.Delete(chunk2);
}
}
}
}
Its all done now run your react via # yarn statand API project to test our code; once you select the file it will be cut into chunks and pumped to Temp folder.. see quick demonstration video below;
As you see in the video above, for each uploaded chunks saved into the Temp folder, after the last chunk uploaded all of the files merged to generate original .mp4 file then removed the files inside of the Temp folder.
With this approach, you can upload any kind of file having no concerns for the size of the file as well as you have enough space in remote source,
Many cloud file transfer platforms like WeTransfer, Google use this approach.
You can access source code here, hope you find it useful, and if you have any question or need help drop a comment in the section below I will be more than happy to help you.
Farewell,
Mehmet Yener YILMAZ
