The way Google views your site is per Monday 27 October 2014 thoroughly changed. If you don't do anything with this, you run the risk of losing your search positions in Google and therefore visitors. And with it turnover. So what has Google changed and how do you fix it? 

How Google sees your website

Google views every website with google bot: fully automatic software that clicks all links and indexes the content. Googlebot previously only saw texts, links and tags such as alt-image (Google does not yet recognize images). That meant Googlebot didn't see any formatting either (C.S.S) and additional applications (javascript).

Google now looks like a user

Googlebot now views your website like a human user: including CSS files and Javascripts. See announcement from Google Webmaster Central. It's an adjustment in Google's Technical Webmaster Guidelines.

Robots.txt for Googlebot

Which part of your website Googlebot can and can't see is in robots.txt. That is an instruction card for Googlebot, and all other bots from other search engines. This also states whether Googlebot is allowed to see CSS and Javascript on your site.

Do I have a Robots.txt?

That is a good question. More people should be asking that! Here's the fastest way:

Go to your website. Such as www.bloeise.nl. In the address bar after the URL, type: /robots.txt. Then you get www.bloeise.nl/robots.txt Press enter. 

YES, you can do this for any website. Also for Google itself: www.google.nl/robots.txt.

Check Google Webmaster Tools

Does the above not work? Do not panic.

1. Enter Google Webmaster Tools to your website. Have you not set up GWT yet? View my Google Webmaster Tools manual.
2. Go to links in the menu on the left crawl and then to Fetch as Google.
3. Leave the URL field blank and leave type set to Desktop. Just press Get and Display.

Now go and update Facebook, check your mail or get a cup of coffee.

4. Google ready? You have the option 'submit to index'. However, you click on / under URL

Fetch as Google

You will then be shown how Googlebot sees your site. Example:

In the first column is the URL, in the 2nd column the type, in the third the reason, then possibly robots.txt tester.

So in this example Javascripts are blocked by robots.txt. That is not good, because Google itself says:

Disallowing crawling of Javascript or CSS files in your site's robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.

View robots.txt

Don't you have your yet robots.txt seen? click on the link robots.txt tester or go to the menu in Google Webmaster Tools crawl and then to robots.txt tester. You can now immediately see how Google reads your robots.txt file.

Modify robots.txt to allow JS and CSS

The steps to give Google access to JS and CSS now are as follows:

1. Use Fetch as Google as described above.
2. Does the preview look good to a user? Then Google sees it right too.
3. Doesn't it look good? Then go through the rules one by one on the Robots.txt link behind it and edit Robots.txt and click to send to click. Check after each line whether the adjustment makes the example readable.

Now you have fixed the problem for Google. For now, other search engines don't look at CSS or JS yet. Read about it here all the possibilities of robots.txt.

Are you stuck?  Forward this blog to your own webmaster / web builder with the Share buttons under this blog. This is covered by a service contract, but if you paid once for creating your website and then no more, your web builder can count on the time it takes to adjust. That really can't be more than an hour, depending on your website, cms and knowledge of the web builder.

 Just a warning!!!!! (5 exclamation marks!!!!) When you edit Robots.txt, make sure you don't use disallow / let alone. Then you tell Googlebot: you are not allowed to view the entire site. Your entire site will then no longer be crawled by Googlebot and therefore no longer shown in Google search.

Robots.txt for WordPress: set /wp-content/ to allow

In an earlier version of this blog I gave a WordPress example and recommended that the folder /wp-includes/ from disallow to allow.

Daniel Mulder from OmniLeads was kind enough to point out that although this folder contains Javascript, and is therefore reported to Fetch as Google, this is background Javascript which hardly affects the readability. It's about the map disallow /wp-content/ in robots.txt: this contains the javascripts and CSS files that have an impact on the display for the user and therefore for Google. In a standard WordPress installation, this folder is already set to allow. Some articles explicitly state to expliciet /wp-content/plugins/ to disallow. Daniel says about this:All content under the folder /wp-content is intended as public content or user content and plug-ins and themes fall under this. I wouldn't set the plug-ins dir to disallow just like the theme folder. This content is precisely content that influences the rendering.”

Keep up with your website

You have to keep up with a website. Just like a car or a house. If you don't do that, your website may suddenly stop ranking in Google, your traffic will decrease and so will your turnover. Bloeise helps you keep up.