Rankings. They’re the lifeblood of our existence in some cases, with clients constantly asking us “how is my keyword doing”, or, just as a measure of performance for ourselves. We will bring up our favourite keyword tracking tool and check over our keywords to see how we are ranking against our competitors.
The problem with this is that the search landscape is very complicated. There are several factors at play, and even if we know what to do to improve those rankings (links, right?), we can never be completely sure what is stopping us from getting that coveted 1st position.
Enter the custom search engine.
Having been a feature for a while, the custom search engine by Google has been used in the past to give websites the ability to provide a search function on their website powered by Google.
After a recent talk at SearchLeeds by Rory Truesdale, the potential for custom search engines to be used to help you identify how to get higher in the SERP was made clear. Custom search engines are free from all the extra information (such as slow loading and the backlinks your website has) that goes into a live Google search.
This means that there is no influence on custom search engine rankings from page speed and technical issues that may affect your site, and the backlinks and authority a website has are also irrelevant. The sandboxed search results we create will be purely based on the relevancy of the content to the search query in the eyes of the search engine.
Using this as a basis, we can input websites that are ranking for the terms we are trying to rank for and compare the live Google result, with our sandboxed one and determine the following:
- If our sandboxed result is higher than on the live result, it means that we more than likely have an issue with either the technical makeup of the page or the authority and number of backlinks
- If our sandboxed result is lower than on the live result, then it means we should probably look at making our content more relevant either in the metadata we are presenting, or the content on the pages to see positional gains.
With all that said, how do we go about creating a custom search engine?
Creating a custom search engine
Before we create our custom search engine we should do a little bit of research into the keyword we want to investigate. This way we can find out who we are competing against and populate our search engine with these websites. To do this, search for your chosen keyword and use a bookmarklet (such as SERP Redux or this one from Chris Ainsworth) to extract the 10 or so URL’s you are competing against.
After we have created our list of websites, we need to create our custom search engine. Head over to https://cse.google.com and sign in using our Google profile.
From here we need to create a new search engine:
Under the “Sites to search” heading we can enter the 10 websites (you can choose more if you like), however, if you’re adding any more than 5, you may want to create the search engine and then use their “Include sites in bulk feature” to save time, as you can copy and paste your chosen websites into this list.
When adding your websites, be sure to check “Include just the specific pages I have entered” otherwise you will get the results from the entirety of the websites you have entered, and not just the pages you are competing against.
Once you have done all this, your custom search engine is good to go, just click on the “Public URL” and your custom search engine will open up in a new tab.
Once you are on your custom search engine, you can search your keywords and compare your results to the live search results to see where your ranking issues may lie.
Next steps
Now that you have a custom search engine, there are a few things you can do with it to help inform SEO strategy. A few things to look at include:
- Using different operators (such as AROUND(X) to search for phrases within X words of each other) to bring out syntactically similar meta titles and descriptions
- Enabling image search within your search engine to analyse these results if you have an image-heavy client
- Change regions to see how this will impact the results
- Restrict pages based on certain Knowledge Graph entities (this is a great tool to see how certain keywords are related to entities)
- Restrict pages based on schema.org types (this could be particularly useful if you want to see what types of schema your competitors are using)
These are just a few things you can look to do to modify your search engine to deliver some more custom results, but the number of keywords and websites you could analyse this way is endless.
If you are looking to find an edge on your competition and want to determine the best way to go about it, using a custom search engine could open up your options an give some meaningful insight into how your rankings behave in terms of relevancy, alternatively, take a look at our SEO services and see if we can give you a hand.
Written by: Chris Parker