What are some uses of machine learning in search engines?
Machine learning is used at almost every part of the stack at major search engines like Google or Bing. Anything that requires some sort of “intelligence” is often solved using machine learning.
Here are some places machine learning is used in search engines:
Search ranking
Often search engines have multiple phases of ranking that happen in series, such as initial retrieval, primary ranking, contextual ranking, personalized ranking etc. Machine learning is used for ranking at all these phases, often using Learning to Rank systems.
Query understanding
Machine Learning is used for “understanding” the search query typed by the user. Some examples of problems that are solved by machine learning:
- Query classification: Search engines run various different classifiers on the search query . E.g. Detecting navigational vs. informational vs. transactional queries. Or news queries vs. local intent queries vs. shopping queries etc.
- Spelling suggestion / correction
- Synonyms / Query expansion: Search engines use synonyms to expand the query keywords and expand the result set.
- Intent disambiguation: E.g. when you search for [eagles], is it Eagles the band or Philadelphia Eagles or the bird (or all 3)?
- Various other facets that you can interpret a query by
Url / document understanding
This includes everything that is done to understand a url, i.e. a search result. Url understanding is often done when the search engine crawls/indexes the url. Some examples of this:
- Page classification: Understanding what types of a page it is. E.g. blog vs. news site vs. forum etc.
- Spam detection
- Junk / low-quality url detection
- Sentiment analysis
- Entity / relationship detection: Detecting entities, such as people or places in the page content. Figuring out relationships between entities.
- Various other facets you can interpret a url by
Search features
Machine Learning is used for generating search features that are diplayed outside the organic results such as:
- Sitelinks
- Related searches
- Knowledge graph data (for some search engines) etc.
Crawling
Crawlers use machine learning to figure out the optimal rate to crawl a particular url based on it’s importance, how often it is updated etc.
User classification
Figuring out what kind of an user you are. This is especially useful for personalized search
This is just scratching the surface. At places like Google and Bing, with hundreds of people working on machine learning, the actual list is much longer.
Originally published at www.quora.com
