По умолчанию сайт закрыт от индексирования. В robots.txt создаются все правила, которые потом каждый меняет под свои требования.
# robots.txt automaticaly generated by PrestaShop e-commerce open-source solution# http://www.prestashop.com - http://www.prestashop.com/forums# This file is to prevent the crawling and indexing of certain parts# of your site by web crawlers and spiders run by sites like Yahoo!# and Google. By telling these "robots" where not to go on your site,# you save bandwidth and server resources.# For more information about the robots.txt standard, see:# http://www.robotstxt.org/wc/robots.htmlUser-agent: *# Private pagesDisallow: /*orderby=Disallow: /*orderway=Disallow: /*tag=Disallow: /*id_currency=Disallow: /*search_query=Disallow: /*back=Disallow: /*n=Disallow: /*controller=addressesDisallow: /*controller=addressDisallow: /*controller=authenticationDisallow: /*controller=cartDisallow: /*controller=discountDisallow: /*controller=footerDisallow: /*controller=get-fileDisallow: /*controller=headerDisallow: /*controller=historyDisallow: /*controller=identityDisallow: /*controller=images.incDisallow: /*controller=initDisallow: /*controller=my-accountDisallow: /*controller=orderDisallow: /*controller=order-opcDisallow: /*controller=order-slipDisallow: /*controller=order-detailDisallow: /*controller=order-followDisallow: /*controller=order-returnDisallow: /*controller=order-confirmationDisallow: /*controller=paginationDisallow: /*controller=passwordDisallow: /*controller=pdf-invoiceDisallow: /*controller=pdf-order-returnDisallow: /*controller=pdf-order-slipDisallow: /*controller=product-sortDisallow: /*controller=searchDisallow: /*controller=statisticsDisallow: /*controller=attachmentDisallow: /*controller=guest-tracking# DirectoriesDisallow: */classes/Disallow: */config/Disallow: */download/Disallow: */mails/Disallow: */modules/Disallow: */translations/Disallow: */tools/# FilesDisallow: /addressDisallow: /addressesDisallow: /authenticationDisallow: /cartDisallow: /discountDisallow: /guest-trackingDisallow: /order-historyDisallow: /identityDisallow: /my-accountDisallow: /orderDisallow: /order-followDisallow: /quick-orderDisallow: /order-slipDisallow: /password-recoveryDisallow: /search