Holiday’s Screenshot Clip: What’s is the Screenshot clip

Extract spot data from a screenshot and mapping it on maps automatically

I’m Takashi Ogiwara, Engineer from Holiday Inc. and this is the very first post for me our medium. This article shows that

• What is “Holiday”?
• What is the function “Screenshot Clip(スクショクリップ)”?

What is “Holiday”?

“Holiday” is one of the most popular travel app in Japan that users can search and share travel plans, bookmark spots and find the prominent place for traveling… Also, this service has many features as below

• Over 10,000 areas in Japan and over 25,000 local plans posted by Holiday users.
• Included unique local comments, one of the useful traveling guide book in Japan :)
• Very useful for making plans traveling and daily outing.

‎ホリデー 旅行のスケジュールを計画できるHoliday

Holiday [ホリデー] | 休日のおでかけならHoliday

Currently, this application has the only Japanese version, but we are developing English and Chinese version.

What is the “Screenshot Clip(スクショクリップ)”?

Let me introduce the new function, we released recently, called “Screenshot Clip (スクショクリップ).”

• Extract spot data from the screenshot and put it on the map automatically.
• Data are organized automatically by category and area.
• Delete screenshot from your local collection immediately.

Users can curate spots from various media, application (like Instagram, Twitter, Pinterest, blog…) that they want to go through this function.

Step1. Analyzed screenshot, Step2. Extract spot words, Step3. organized spots on maps

System Architecture

“Screenshot Clip” has three steps to extract spot data from a screenshot,

1. Analyzing text data
2. Classify (extract) spot names from the text
3. Score the extracted spots and sort them, and place at the top that requested by the user

Let me explain below section 1 and 2.

System Architecture (in Japanese 🙏)

1. Pick text up from the screenshot

First, we have to get text data from the screenshot, so using Google Cloud Vision to get texts from a screenshot. Google Cloud Vision is

• Highly accuracy to extract text data from images
• Extract some sentences as the cluster where they located
• Also to get some web entities

This system uploads a screenshot to Google Cloud Vision directly.

2. Classify (extract) spot names from the text

Second, classify spot names from the text. This system executes three methods to get spot names,

• Extract by spot dictionary
• Classify by machine-learning model
• Using third-party API

2.1 Extract by spot dictionary

This is a simple way to extract spot names from the text. “mecab-ipadic-neolgod,” “++Juman” are popular morphological analyzer in Japan. But they don’t have enough accuracy to extract spot names, so We built a large scraper to collect a lot of spot names from other web sites(Instagram, Twitter, RETRIP, Gurunavi…).

2.2 Classify by machine-learning model

If we use only spot dictionary, I cannot extract unknown spot names, so we have to find another way to obtain it.

Google Cloud Vision has a feature,

Extract some sentences as the cluster where they located

In other words, there is a possibility to extract unknown spot names by a classification model that the sentence is spot name or not.
I made some features likes “length”, “number of nouns, the verb”… and classified by RF.

「パティスリー シイヤ」is the spot name

The system executes three methods and gets accuracy 85% ~ 92% regularly.

How was our Holiday app and Screenshot Clip (スクショクリップ)? This year, we will focus on globalization, especially inbound market because foreign tourists are booming now, and Japan holds some big events like the 2019 Rugby World Cup and the 2020 Tokyo Olympic in Tokyo.

Thank you for reading this through to the end.