Sitemap is a protocol adopted from sitemaps.org by major search engines for a while now. The sitemap is actually an xml file that basically contains a list of all the items that you have on your website (links, images, documents) and that you want to be accessible to the public. Search engines use the information from your sitemap to add or update those items in their index - this doesn't mean that if you don't have a sitemap your website is not indexed, the sitemap is used in addition to the regular crawler.
There are several search engines that support the Sitemaps.org protocol, most important ones being: Google, Yahoo, Live (former MSN) and Ask. If you're a webmaster most likely you've heard at least about the Google Webmaster Tools, a web based interface that allows you to see interesting results based on the analysis of your sitemap (inbound links, top searches, site errors) more or less through the eyes of Google itself. However, Yahoo offers a similar tool called Yahoo SiteExplorer, but unlike Google's it only allows adding/removing sitemaps, removing individual links, and more recently adding rules for indexing Dynamic URLs.
Live.com (former MSN) didn't want to be left behind and added a similar tool recently called Webmaster Tools. Unlike Google or Yahoo they also offer a way to see the top 10 most well ranked pages of your site, the top 10 most important websites that link to your site, and how your website ranks for specific keywords in their index. Give it a break if the tool doesn't work as it should, because it's still in beta and still needs work.
Back on the track, how do you submit sitemaps to search engines? Well, first of all you need to create a sitemap. While there are various tools out there (both online and desktop tools), I've covered this topic on how to create a sitemap using a desktop program a while back:
GsiteCrawler.
After the sitemap is created (let's assume you named it sitemap.xml), you have to upload it on the server where the website is hosted (usually in the root). Now, after the sitemap is live, you can do a trick called robots.txt sitemap autodiscovery where you add in the robots.txt file the following line:
Sitemap: http://www.example.com/sitemap.xmlOf course instead of the url in this example you'll put your url. Now search engines that support sitemaps will read automatically the location of the sitemap from the robots.txt file and access those links directly - what better way to have your files indexed without headaches?
However, this does not mean you don't have to manually submit sitemap to search engines, specifically to Google Yahoo and Live. Why? Well, because you'll have access to the tools I mentioned above, which are very useful for webmasters in general (the Google one is pure gold, while others are still around the bronze level).
Submit Sitemap to Google:
- Login to Google Webmaster Tools (if you have a google account the login is accepted, if not create one)
- Add your website through the provided interface, and after that Verify your site (basically you have to upload a dynamically generated empty html file or add an entry in the meta-tag - Google provides sufficient instructions on this part).
- Go to the Sitemaps tab, click on Add a Sitemap and enter the url of your sitemap - status will be processing but if it was accepted all's well. Usually it takes a couple of hours for the sitemap to be processed, but depending on the status of your website (new, old) you won't get to play with the new tools until Google has more data
Submit Sitemap to Yahoo:
- Go to Yahoo! Site Explorer and click on the Sign in top button, after you're logged in the My Sites section enter the url of your website and click Add My Site. You'll also need to Verify that you are the owner of this website, so choose a verification method (default is to upload a custom file) and proceed to that.
- Now, click on the Manage button for that recently added url and you'll see a way to add your sitemap (actually they call it Feed) - enter sitemap.xml and click on Add Feed (Web Site Feed should be selected).
- Sitemap should be processed in a couple of hours and you'll be able to see more details about it after that. In the meantime you'll see that by using the Explore button on the main Dashboard you get to see which links were indexed for your website and eventually remove ones that you don't want indexed by Yahoo
- A new feature that Yahoo! Site Explorer offers is the Dynamic URLs (in Beta). What you can do through that is "teach" Yahoo which urls are dynamic on your website (if any) so that he could index the correct ones. For instance if you have an ecommerce website and each link has a parameter, you can tell Yahoo to ignore that parameter and index only the main link.
Submit Sitemap to Live (MSN):
- Go to Live Webmaster Tools and click on the Sign in to use tools button (Windows Live Account needed).
- Add a Web address, the location of your sitemap in the Sitemap address section, select a Verification method and add your Contact information
- Once you'll submit that information you'll see the content of the xml verification file that you need to upload to your website (if you didn't choose to add a meta-tag), copy that and create/upload the verification file before clicking Submit.
- After you've verified that Live will give you additional information about your website - Summary, Profile, Keywords, Top Outbound links, Top Backlinks and the Sitemap tab. They're still in Beta so some of this information is not accurate, but still is something new brought to the table
- Live.com also gives you the option to ping them so that they'll mark the sitemap for re-indexing: http://webmaster.live.com/webmaster/ping.aspx?siteMap=[Your sitemap web address]li>
Submit Sitemap to Ask - This is actually a ping, since Ask.com does not provide a web interface like Google/Yahoo/Live. To send your sitemap to Ask.com simply ping them using the following url (add your sitemap location though):
http://submissions.ask.com/ping?sitemap=http%3A//www.the URL of your sitemap here.xml
Google Webmaster Tools provides now the most comprehensive and useful information, but, I like the new features that Live.com Webmaster Tools adds (the top backlinks, outbound links, a possibility to see how your pages rank for specific keywords) and also the Dynamic URLs feature that Yahoo! Site Explorer has. So Google, we're waiting for an update.