This first step to rectifying a search engine not indexing your site is to confirm that your site is not on that search engines index. Write into Google's search bar site: your_domain_name. If there are any results, your page is not in the index. The issue may be poor SEO, in which case you may want to perform a technical SEO audit. This takes time and money, so many companies hire an external company for this.
If Google is coming back empty, you are indeed missing from their index. This could be for a couple of reasons:
Google has not found you yet
It can take Google a few weeks to find you. If your site is new, give it a little time. In the meantime, check your sitemap has been uploaded and is operating correctly. If everything appears to be working, request an index status report or URL inspection from Google.
These tools may be able to indicate any issue. It is also a good idea for your site to be re-indexed by Google if you have added new pages or changed the layout of your site.
Crawler error
If there is an issue with your URLs being indexed, you may want to have a professional SEO audit carried out. This will highlight any errors in your site’s coding, text or design. Another option is Google Search Console dashboard. Any URLs in your site map with an error will be highlighted. Common issues are error codes 404, 500 and 301. All of these indicate that your site is not working as intended.
In this instance, it is most likely that something is blocking the crawlers (spiders, bots) from examining your pages. The most common errors are:
- Site blocked by robots.txt
One of 'robots' purposes is to prevent crawlers from finding that page. There are many legitimate reasons to not want a search engine to see a page, e.g. a staff/member-only page. However, if this is your issue, it is an easy fix. Remove the text.
- Add your domain to Google Webmaster
Google Webmaster is a free tool that gives you an insight to evaluate and maintain your website's performance within the search engine. By setting up an account with Google Webmaster and verifying ownership of your domain, Google will find it easier to view your site. Additionally, it will be easier for you to see how Google views your site.
- Do not have a sitemap (sitemap.xml)
A sitemap is a list of pages on your site within your domain and the relationship between them. Sites like Google use sitemaps to crawl your site more efficiently. In addition, a sitemap has information as to when the page was last updated and any language variations. If you have not already, build and submit your sitemap to Google.
- Privacy settings
Double-check your site privacy settings are set correctly. For example, if your site uses WordPress, check in privacy settings that the 'discourage search engines from indexing' box has not been ticked.
- Site has noindex meta tag
Look out for text that looks like this in your meta tags: (<meta name="robots" content="noindex,nofollow">). This is another variation of robots that prevents crawlers from viewing your site. Another easy fix, simply remove the meta tag.
Server/content issues
Another option is that something within your pages deters or prevent crawlers from accessing your site. Crawlers work on the premise that they have easy unrestricted access to the contents of a URL. Poor content or slow loading times can have a catastrophic effect on how a crawler will view your URLs.
The most common issues
- Duplicated content
When multiple URLs are returning to the same or very similar content, it will confuse the crawlers. For them, it will look like they are going around in a circle. In this instance, the search engine will abandon indexing the site. The content of every page should be original with relevant keywords. Content is paramount to successful SEO.
- Javascript issues
Google will index most Javascript and AJAX. However, if there is an incorrect configuration within your Javascript execution, it will not be indexed as neither language is easy to decipher. If your issue is due to the configuration of your Javascript, you may want to hire a specialist to read and audit your script.
- Site taking too long to load
Loading speeds do not prevent Google from indexing your site. However, a page with a perpetual loading time will reduce the volume of crawlers that can access the site. In addition, a slow loading time can come with a high bounce rate further making your page unattractive to Google. An SEO audit can highlight what is slowing your load time.
- Host server down
If your domain is hosted by an external party, it is worth checking that it is operating correctly on their end. If they have frequent outages or an ongoing issue, this may be preventing your site from getting crawled. This might also explain low traffic or high bounce rate.
Deindexing
Unfortunately, if your site has been deindexed, you have a long road ahead of you. First, file a reconsideration request with Google.
There could be many reasons for your site being deindexed. A common one is your domain had a previous life that was not so ethical. You may have some indication that something was not right. If you have ever received a manual penalty or removal notice, check out Google Search Console for any alerts on your domain.
Need help getting your site indexed by Google? As you might have read, it can be technical to figure out and then implement the corrections to your site. If you need help with a technical SEO audit, then get in touch.
We have a range of solutions and services to get your site back on Google. If you require advice or an audit, please do not hesitate to contact on 01794 725454 or make an enquiry today.
