Instant Indexing

Instant indexing refers to the direct or very fast recording of URLs of a website into the search index of search engines so that content can be found by users. There are different strategies and methods to provide a Website, URL, or new media content to a search engine. Accordingly, the time span Google will need to include content in its databases varies as well. Depending on the search engine, this can take between a few hours, several days, or even weeks.

Indexing can only be classified as Instant Indexing, Immediate Indexing, or On-Demand Indexing when

• a mechanism is used to add new content to the index faster than the conventional crawlers,
• and the website with the new content is already known to the search engine. In other words, other content on the website has already been indexed.

Instant indexing can only be set up in Google Custom Search. It bypasses the functionality of crawlers by sending new content directly to Google. However, they will only be indexed if the owner of the website can be proven and Google is familiar with the site as Publisher. In May 2016, Google launched the Betaphase for a Real-Time Indexing API, which aims to make various content types immediately retrievable with the Google Search after publication by a publisher.

General information

Typically, Google crawls new sites and content by utilizing the already existing index. The Crawler visits the links it already knows and finds new links that are still unknown to it. If a link to a new site is placed on another site, the crawler would also follow this link to index the new content. The same is true for ping services, which give the search engines a short ping signal. This is used with blogs for fast indexing. Also, Traffic from social media or backlinks can indicate new links to search engines. However, in the case of instant indexing, a different approach is used, which is intended for web projects that constantly provide new content. For example, news sites or sports magazines can be registered as publishers and these will be shown in the vertical search under News with current content, which corresponds in principle to Instant Indexing, but there are nevertheless some differences.

How it works

This type of credential is also used as part of instant indexing in the custom search. Necessary prerequisites are proven ownership of a website and a Google account in Webmaster Tools. A property like www.example.com can be linked to the active Analytics account to make Search Console data available in Analytics. The Search Console can be utilized to monitor websites ranking in the SERPs. If you as a publisher want to index certain content with immediate effect, you first have to set up the Custom Search. By integrating the search engine into a website, the content of that website can be searched. A tutorial on this topic can be found here: Create Custom Search. After the custom search has been integrated, direct indexing can be done for the following URLs (on-demand indexing):

• Single URLs: This technique is suitable to submit some new pages or updated content.
• URLs linked to a site or sitemap: This method is appropriate for blogs and websites that only change one page at a time. Updated sitemaps with new content can be accessed by Google that way as well.
• URLs in a Sitemap: This method is recommended for large and complex websites. Exclusions from the custom search are also possible. However, once a Sitemap has been updated, it must be submitted again.

Indexing quotas must be observed for instant indexing in the user-defined search. Each user of the Search Console has a quota of 200 submissions. Commercial users can increase this quota by buying so-called indexing quota from Google.

Real-Time Indexing API

Google takes it one step further with the Real-Time Indexing API. Current and, above all, relevant content is supposed to be accessible to users in the Google search right after being posted. The API allows publishers to send their content directly to Google without having to do so manually in the Search Console. This minimizes the delays between publication of a post and indexing by Google. According to Google, users want more up-to-date information on a variety of subjects. Immediate indexing is especially of benefit for news websites and brands whose content are focused on current events. A special area is supposed to be reserved for such content in the SERPs which is described by Google with the term “happening now.” Real-time videos and AMP carousels (picture galleries for mobile devices) are supposed to be displayed there too. Moreover, Rich Snippets and structured data are supported such as schema in order to control markup and presentation in the Search. The Real-Time Indexing API is currently in the beta test phase and may be rolled out globally in the coming months or within the next year.

Relevance to search engine optimization

The topic of Instant Indexing has always been an area of speculation since Google has been established. Millions of new websites are launched every day and it seems virtually impossible for a search engine to promptly index all of this content. With Sitemaps and functions such as Fetch as Google the time span is considerably shorter, but direct indexing or immediate indexing is not real in this context. The custom search is without doubt an exception. However, Instant Indexing is limited to the website itself. Current content is not displayed in the traditional Google search. The latest announcement by Google on the subject of Instant Indexing has left many experts intrigued. A real-time indexing API would save the publishers a lot of work and Google would feed current content from trustworthy websites directly into the system. The extent to which instant indexing would actually be instant will have to be seen in practice.

Web Links

• What is Indexing?
• General Information on Indexing
• Detailed Guide
• case study on the subject
• Faster indexing: Fetch as Google
• More Information on Real-Time Indexing API