How to Handle POST/PUT requests in offline applications using Service Workers, IndexeDB and Background Sync
*If you know nothing about taking a web page offline, you will learn this from my previous article, which is a Simple Guide to taking a Web Page Offline using Service Workers. So this article is sort of a sequel, and I will only be explaining how I handled POST requests offline
A while back, I worked on taking an app offline using service workers. While working on this project, I had to make sure that all the key parts of the application work offline. With the service worker API, making icons, text, images and fonts display offline was no biggie. The headache came when I had to deal with sending file uploads and other requests to the server that were POST requests. I realized then that the service worker API does not support caching POST/PUT requests, only GET.
Since it was a must for me to make the POST requests work offline seamlessly, I looked to Google to find workarounds to this problem, only to find inadequate information on how to make that happen on the internet.
Let’s say I have a form in my web application, and I’d like it to work offline, there’s a way to make it happen using a combination of Service Workers, IndexeDB and Background Sync.
At Formplus, where I work, our forms are usable offline in the browser for all kinds of fields, including file uploads (we also have an hybrid app in the works), so if you don’t want to go through all the trouble of implementing such, we are here for you.
What we will be doing is this, once there is an attempt to send a POST request,
- we save that request to the user’s browser using IndexeDB, and then
- with the help of Background Sync (which only works on chrome 49 & up), once the user’s browser is back online we detect it.
- At this point we then retrieve our POST request from the IndexedB and send it on its merry way to the server.
For this tutorial, I’m going to be using a Flask form I wrote a while back. You can find it here. We are going to be taking it offline, just like I did here. We won’t stop there, we would make sure that while it is offline, a user can submit a form; and then once they are back online, we send their data to our server. This way, a lack of internet won’t affect your user experience. Before we take the app offline, let’s review how the Flask app works on a normal basis. It is a Flask app that features a web form.
*please follow along with the github code here and here
We have our app.py
from flask import Flask, render_template, requestapp = Flask(__name__)@app.route('/')
@app.route('/index')
def home():
return render_template('index.html')@app.route('/submit', methods=['POST'])
def submit_form():
payload = request.get_json()
print payload
first_name = payload['first_name']
middle_name = payload['middle_name']
last_name = payload['last_name']
date_of_birth = payload['date_of_birth']
address = payload['address']
hobby = payload['hobby']
print first_name, middle_name, last_name, date_of_birth,
address, hobby
return ''app.run(debug=True)
Parts of our index.html looks like this
The web form look like below
On submission, it sends a POST request to the/submit endpoint in the server via ajax in scripts.js, including the data you filled in the form.
function submitFunction (event) {
event.preventDefault()
console.log('submitted', event)
first_name = $('#first_name').val()
middle_name = $('#middle_name').val()
last_name = $('#last_name').val()
date_of_birth = $('#date_of_birth').val()
address = $('#address').val()
hobby = $('#hobby').val()
console.log('values,', first_name, middle_name, last_name,
date_of_birth, address, hobby)
$('#my_form').hide() // send to server
data = {
first_name: first_name,
middle_name: middle_name,
last_name: last_name,
date_of_birth: date_of_birth,
address: address,
hobby: hobby
}$.ajax({
type: "POST",
url: '/submit',
contentType: 'application/json',
data: JSON.stringify(data),
success: function () {
console.log('data sent to server successfully')
},
dataType: 'json'
});
message = 'Your data has been sent to the server'
$('#message').append(message)
return false
}
Before clicking the submit button you can inspect this by doing ctrl shift i and then clicking the network tab . On clicking the submit button you should see something like below. We can see details about our request, including our payload.
In the Form Data section, in your own case you should see the data you inputted in the web form.
Now let’s take it offline. We do this by first creating a service worker script as sw.js
var CACHE_NAME = 'offline-form';
var urlsToCache = [
'/',
'/static/style.css',
'/static/script.js',
"https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/1.11.8/semantic.min.css",
"https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.3/jquery.min.js",
"https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/1.11.8/semantic.min.js"
];self.addEventListener('install', function(event) {
// install file needed offline
event.waitUntil(
caches.open(CACHE_NAME)
.then(function(cache) {
console.log('Opened cache');
return cache.addAll(urlsToCache);
})
);
});self.addEventListener('fetch', function(event) {
// every request from our site, passes through the fetch handler
// I have proof
console.log('I am a request with url: ',
event.request.clone().url)
});
And registering the service worker in static/script.js
if ('serviceWorker' in navigator) {
// we are checking here to see if the browser supports the service worker api
window.addEventListener('load', function() {
navigator.serviceWorker.register('/sw.js').then(function(registration) {
// Registration was successful
console.log('Service Worker registration was successful
with scope: ', registration.scope);
}, function(err) {
// registration failed :(
console.log('ServiceWorker registration failed: ',
err);
});
});
}
And then by adding an handler for our service worker script (sw.js) in our Flask app, in app.py
@app.route('/sw.js', methods=['GET'])
def sw():
return app.send_static_file('sw.js')
After saving all of our files, let’s test our offline app, by loading the page to refresh it and then when we are offline (also with our flask server shut) we reload the page.
On my end, I am able to load the page offline.
Now I’m going to try and submit the form and see what happens.
The message on the page seems to be saying “Your data has been sent to the server”. However, I am not fooled by this, because on inspecting my network tab, at the url submit, I can clearly see a red which stands for an error. On clicking & checking the response section, I can also see that its empty; which means our request did not get to the server at all. At the very least, if a request gets to the server we are supposed to get a response, even if there’s a 500 server error or 404 not found error.
Handling Post Requests in Service Workers
We are now at the heart of the matter. Our next step is to capture our POST request along with its payload, and then find a way to store it on the user’s browser temporarily until such a time when they are back online. We are going to be using Indexedb for storing the requests in-browser. Please go through the documentation to understand what I am doing here
Create Database
First, we create a database that is going to hold our offline data. And we create it in our service worker script (sw.js)
function openDatabase () { // if `flask-form` does not already exist in our browser (under
our site), it is created var indexedDBOpenRequest = indexedDB.open('flask-form',
IDB_VERSION) indexedDBOpenRequest.onerror = function (error) { // error creating db console.error('IndexedDB error:', error) } indexedDBOpenRequest.onupgradeneeded = function () { // This should only executes if there's a need to
// create/update db. this.result.createObjectStore('post_requests', {
autoIncrement: true, keyPath: 'id' }) } // This will execute each time the database is opened. indexedDBOpenRequest.onsuccess = function () { our_db = this.result
}
}var our_dbopenDatabase()
We just created a database for our app with the name flask-form. We also created something called an object store for our database named post_requests. You can think of it as a table like in SQL or like a file in a folder. A db can contain several object stores.Now our database can be accessed via our_db. Now refresh the page, visit the Application tab in your console, go to the Indexedb section; right click on it and refresh database. There should be a drop-down on it containing our database flask-form and under that, our object store post_requests.
Send form data to Service Worker
Next we need to find a way to get our form data across to the service worker. We do this by using the Client.postmessage API to send and receive data between our script.js and sw.js. So immediately submission occurs we should send our payload via post message.
var data = {
first_name: first_name,
middle_name: middle_name,
last_name: last_name,
date_of_birth: date_of_birth,
address: address,
hobby: hobby
}
// send message to service worker via postMessage
var msg = {
'form_data': data
}
navigator.serviceWorker.controller.postMessage(msg) // <-This
// line right here sends our data to sw.js
Next, we set a message listener in sw.js. Its job is to receive our message once it arrives.
self.addEventListener('message', function (event) {
console.log('form data', event.data)
if (event.data.hasOwnProperty('form_data')) {
// receives form data from script.js upon submission
form_data = event.data.form_data
}
})
Save form data in indexedb
Now, we are ready to intercept our POST requests in the fetch handler to prevent errors when we are offline. So we update our fetch handler with the following code
self.addEventListener('fetch', function(event) {
// every request from our site, passes through the fetch handler
// I have proof
console.log('I am a request with url: ',
event.request.clone().url)
if (event.request.clone().method === 'GET') {
event.respondWith(
// check all the caches in the browser and find
// out whether our request is in any of them
caches.match(event.request.clone())
.then(function(response) {
if (response) {
// if we are here, that means there's a match
//return the response stored in browser
return response;
}
// no match in cache, use the network instead
return fetch(event.request.clone());
}
)
);
} else if (event.request.clone().method === 'POST') {
// attempt to send request normally
event.respondWith(fetch(event.request.clone()).catch(function
(error) {
// only save post requests in browser, if an error occurs
savePostRequests(event.request.clone().url, form_data)
}))
}
});
function getObjectStore (storeName, mode) {
// retrieve our object store
return our_db.transaction(storeName,mode
).objectStore(storeName)
}function savePostRequests (url, payload) {
// get object_store and save our payload inside it
var request = getObjectStore(FOLDER_NAME, 'readwrite').add({
url: url,
payload: payload,
method: 'POST'
})
request.onsuccess = function (event) {
console.log('a new pos_ request has been added to indexedb')
} request.onerror = function (error) {
console.error(error)
}
}
Above, we are saving our payload, request url and method type into our object store, which was retrieved with the help of getObjectStore. In the event that you are trying to save a PUT request, you can just change 'POST' to 'PUT'.
It’s time to check whether our request gets saved in the Indexedb database flask-form. Shut down your flask server, put off the wifi and try to submit the form offline, and see if it is in flask-form. You do this by going to the Application tab in the console and then the Indexedb section, right click & refresh database. In the drop down you should see flask-form-> post_requests and also the data you have filled in the form.
Syncing to Server with Background Sync
Now that we have form data stored in the browser, we have to figure out two things. One, how to automatically detect when the user/browser is back online; two, how to retrieve our form data from the Indexedb database and send the data to the server.
Automatically detecting when the browser is online
This is when Background Sync comes into play. We can request a background sync for our app by registering it in our script.js and then listening for the sync event in the service worker script. Here’s how to register a sync. I have given my sync a unique name that represents what I need it to do sendFormData
navigator.serviceWorker.ready.then(function(registration) {
console.log('Service Worker Ready')
return registration.sync.register('sendFormData')
}).then(function () {
console.log('sync event registered')
}).catch(function() {
// system was unable to register for a sync,
// this could be an OS-level restriction
console.log('sync registration failed')
});
Next, we need to listen for the sync event in the sw.js. This event is triggered whenever the browser comes online. Notice below, how we use sendFormData to check if the event being triggered was the one we registered.
self.addEventListener('sync', function (event) {
console.log('now online')
if (event.tag === 'sendFormData') { // event.tag name checked
// here must be the same as the one used while registering
// sync
event.waitUntil(
// Send our POST request to the server, now that the user is
// online
sendPostToServer()
)`
}
})
Retrieving form data & sending to server
We are going to retrieve all of our saved requests from the browser’s indexedb with cursors and save then into an array savedRequests.
function sendPostToServer () {
var savedRequests = []
var req = getObjectStore(FOLDER_NAME).openCursor() // FOLDERNAME
// is 'post_requests' req.onsuccess = async function (event) {
var cursor = event.target.result if (cursor) {
// Keep moving the cursor forward and collecting saved
// requests.
savedRequests.push(cursor.value)
cursor.continue()
} else {
// At this point, we have collected all the post requests in
// indexedb.
for (let savedRequest of savedRequests) {
// send them to the server one after the other
console.log('saved request', savedRequest)
var requestUrl = savedRequest.url
var payload = JSON.stringify(savedRequest.payload)
var method = savedRequest.method
var headers = {
'Accept': 'application/json',
'Content-Type': 'application/json'
} // if you have any other headers put them here
fetch(requestUrl, {
headers: headers,
method: method,
body: payload
}).then(function (response) {
console.log('server response', response)
if (response.status < 400) {
// If sending the POST request was successful, then
// remove it from the IndexedDB.
getObjectStore(FOLDER_NAME,
'readwrite').delete(savedRequest.id)
}
}).catch(function (error) {
// This will be triggered if the network is still down.
// The request will be replayed again
// the next time the service worker starts up.
console.error('Send to Server failed:', error)
// since we are in a catch, it is important an error is
//thrown,so the background sync knows to keep retrying
// the send to server
throw error
})
}
}
}
}
We iterate through savedRequests and retrieve that particular request’s url, payload and method. We then use the fetch API to send the request to our server; after which, we delete that particular savedRequest from the indexedb if the response was successful. We delete it because we wouldn’t want to send our POST request twice in the event of another sync. If the request fails, whether because of a slow network or a 500 server error, you can leave the savedRequest in the indexedb so it can retry next time, if you want.
*After updating your code, it is important that you update the service worker by going to the application tab, then the service worker and check Update on reload; then reload the page. Click the blue little sw.js to confirm that the source code has been updated.
To test, fill the form offline (with your flask server shut down) as many times as you want, and then bring your server back up and put on your wifi. You should monitor your console while all this is happening. In mine I see this
Both of my two saved requests were a success. In my server too, they were received
At this point, we are exactly where we are supposed to be, and we’ve done what we set out do; which is save POST requests in the browser when offline, and when back online, send saved requests to the server.
You can find the before and after code for this tutorial, here and here respectively.
I hope you’ve been able to learn a thing or two. Please, let me know if you have questions/corrections. I want to hear from you. Cheers.
