The site is large - about 250,000 URLs - so scraping it won't be easy. Ahrefs has data on which URLs are ranking in search results, and your competitor can't stop you from getting this information.įor example, John Lewis, a popular UK e-commerce store, has a pretty aggressive firewall. But be careful not to get blocked by their firewall. You can get the list of your competitor's URLs by crawling their site using Screaming Frog or Sitebulb. You only need a list of their URLs and a simple formula in Google Sheets. You can use this to gather data on your competitor's structure quickly. Using URL directories to indicate how content has been organised is a common practice among websites. This article outlines two quick and easy ways to gather competitor structure data and how you can use that to prioritise your SEO strategy. But for large sites with 100,000s or millions of pages, you need a more scalable approach. If the competitor's site is small, you can inspect it manually.

Card sorting and user interviews can help, but you may still be stuck trying to decide on the best approach.Īnalysing competitors' sites can provide inspiration for what works. Choosing your website's structure can be difficult.

Competitor website