Date First Published: 11th July 2022
Topic: Web Design & Development
Subtopic: Web Services
Difficulty: EasyDifficulty Level: 2/10
Learn more about what a search engine is in this article.
A search engine is a service designed to allow users to carry out searches by entering keywords and characters, used for finding specific websites on the World Wide Web. All the crawled websites are stored in a database and when a user performs a search, it compares those keywords and characters with websites crawled in the database. Once the search is performed, the search engine will return an index of web content, including webpages, images, videos, products, flights, and more. Each search engine has its own abilities and features. Search engines may contain additional features that allow users to customise the amount of results per page and safe search that filters out sexually explicit content and images from search results.
Search engines help users find what they want on the World Wide Web. Without search engines, it would be much more difficult for users to find what they are looking for as they would have to manually type the URL into their web browser.
Search engines can be accessed by visiting the website of that search engine (e.g. bing.com). Some websites have an additional search engine displayed on their own website. This website has a search feature that allows you to search for pages by entering keywords.
Search engines continuously crawl the web for new webpages. After the content is crawled, the content is added to the search engine index. These small bots called spiders scan all content of websites, including all pages, images, and videos.
The frequency of bots scanning websites depends on how active the website is. Search engines crawl it every four and thirty days. Some websites prevent crawlers from visiting them. This is accomplished by adding this tag in the HTML document:
Adding this tag will prevent the specified webpage from being indexed by search engines. This causes the pages to be left out in the index. Specific rules can be set up in the robots.txt file to block the crawling of certain pages. For example, a webmaster may choose to disallow error pages to be crawled.
In most search engines, the cached version of a webpage can be viewed. It is not the up-to-date version of a webpage. It is what the webpage looked like the last time it was crawled by small bots, known as spiders. The webpage may have changed since then.
Because search engines contain billions of webpages, they use a complex algorithm that considers a large number of factors. The higher a webpage, image, or video is in the search engine index, the more chances of someone discovering the webpage, increasing the traffic to the website. Search engines rank pages based on the following factors:
The relevance of a webpage to a user's query is the most important factor that determines the position of indexed pages. For example, when searching 'What is a motherboard?', no search results about what a web designer is are going to come up.
High-quality content describes content that increases the visit time on pages with no plagiarised content.
Not plagiarising content also optimises websites for search engines as plagiarised content will cause confusion for the search engines as to what identical page should appear at the top.
Mobile-friendliness is about whether the page is compatible with mobile devices and easy to use. With the number of people using mobile devices to browse websites rising, mobile-friendliness is a very important factor of SEO. Google often tests websites for mobile-friendliness and gives webmasters notifications if any webpages do not perform well on mobile devices (e.g. the webpages scrunch up or the text is too small on mobile devices.)
A backlink is a is a hyperlink from one webpage to an external website. For example, if a website linked to an external website, that would be a backlink. Backlinks improve SEO as they prove that users find the content useful. Backlinks signal a search engine a vote of external confidence. This is one of the main factors that have an influence on the order that pages appear in the search results.
Page speed has an effect on how it will rank in search engines. It is also a very important factor of user experience. Page speed gives a measure of how long it takes for a page to load. A fast page speed means that pages will load quicker in web browsers, reducing the bounce rate. Page speed can be improved by using a CDN, reducing page elements, enabling browser caching, and using the appropriate web host.
Securing a website with SSL/TLS and enabling HTTPS is a factor that makes websites appear higher in the search rankings. A secure website that uses encryption will have an advantage over websites that are insecure and unencrypted.
Although it is not true that the longer an article, the better it is, it is evident that longer content produces higher search rankings. Longer webpages are more likely to appear higher in the search rankings compared to shorter webpages.
Search engines make money from ads. In search engines, ads usually appear at the top before the list of search results. These are paid search results. Every time an ad is clicked on, the advertiser pays the search engine. Nobody pays a search engine to appear below the ads.
The most popular search engine is Google. In fact, Google is the most visited website on the internet. Other examples of search engines are Bing, Yandex, and Yahoo. DuckDuckGo and Startpage are anonymous search engines and they are a suitable option for users concerned about privacy.
Not all search engines give the same results. Different search engines have crawled different pages and some pages may appear in one search engine, but not in other search engines. Search engines may show results based on the location of users and their past searches. Each search engine works in a different way and uniquely delivers results.
Search engine shortcuts help users find what they are looking for. Common shortcuts include:
Most search engines, such as Google and Bing will store past searches for a certain period of time. On Google Search Console and Bing Webmaster Tools, webmasters can see exactly which search queries are causing their website to show up and be visited (e.g. if a user searched for 'what is a web developer' on Google or Bing and they clicked on a search result to a website called example.com, the owner of example.com would know that someone has searched for that and has clicked on their search result.)
Search engines may allow people to be targeted based on their browsing history. When search engines collect all the past searches, this allows advertisers to display personalised ads based on their search queries.
Originally, Google stored user's search history forever. In 2019, users can pick a time limit for how long their search history is kept by Google. It is either 3 or 18 months. Any search history older than 18 months will automatically be deleted.
Search engines, such as DuckDuckGo and Startpage do not track or store user's past searches, helping to protect user's privacy. They offer a private browsing experience, displaying no personalised search results, and no use of cookies to track users. These search engines keep search history private and are suitable for users concerned about their privacy.
Before September 1993, no proper search engines existed. The World Wide Web was manually indexed. Due to more websites going live, the central list could no longer keep up.
The first search engine, called WebCrawler was developed in 1994. It allowed users to search for any word in any crawled webpage. The first popular search engine on the internet was Yahoo Search. It was introduced in 1995, which was when a search function was added, allowing users to search the Yahoo Directory. Its search engine worked on its web directory rather than its full-text copies of webpages.
Robin Li developed the RandDex site-scoring algorithm for search engine results page ranking in 1996. It was the first search engine that made use of hyperlinks to give a measure of the quality of indexed websites and was the predecessor to the Google algorithm introduced two years later in 1998.
In 1998, Microsoft introduced MSN search. In early 1999, the website started to display listings from Looksmart, combined with results from Inktomi. For a short period of time in 1999, MSN Search used results from AltaVista instead. In 2004, Microsoft used its own search technology, powered by its own web crawler called msnbot. Bing was launched on 1st June 2009. It is Microsoft's rebranded search engine.
Active search engine crawlers include Google, Petal, Sogou, Baidu, Bing, Gigablast, Mojeek, DuckDuckGo, and Yandex as of 2019.
If so, it is important that you tell me as soon as possible on this page.
Network Services Network Setups Network Standards Network Hardware Network Identifiers Network Software Internet Protocols Internet Organisations Data Transmission Technologies Web Development Web Design Web Advertising Web Applications Web Organisations Web Technologies Web Services SEO Threats To Systems, Data & Information Threats To Individuals Security Mechanisms & Technologies Computer Hardware Computer Software Ethics & Sustainability Legislation & User Data Protection
