There are lots of good reasons you might require to locate all of the URLs on a website, but your exact goal will figure out Everything you’re looking for. For example, you might want to:

Detect just about every indexed URL to research troubles like cannibalization or index bloat
Acquire present and historic URLs Google has noticed, especially for website migrations
Find all 404 URLs to Get better from put up-migration problems
In Every scenario, only one Resource gained’t Provide you with everything you would like. However, Google Lookup Console isn’t exhaustive, along with a “web page:illustration.com” lookup is proscribed and challenging to extract info from.

Within this write-up, I’ll stroll you through some instruments to create your URL list and in advance of deduplicating the information employing a spreadsheet or Jupyter Notebook, determined by your site’s sizing.

Outdated sitemaps and crawl exports
In the event you’re on the lookout for URLs that disappeared through the Reside site lately, there’s an opportunity another person in your team could possibly have saved a sitemap file or perhaps a crawl export ahead of the alterations were designed. In the event you haven’t now, check for these data files; they could often give what you'll need. But, when you’re reading through this, you almost certainly did not get so Blessed.

Archive.org
Archive.org
Archive.org is an invaluable Device for Search engine marketing responsibilities, funded by donations. Should you try to find a domain and choose the “URLs” choice, you may access approximately ten,000 outlined URLs.

On the other hand, There are some constraints:

URL Restrict: You are able to only retrieve as many as web designer kuala lumpur ten,000 URLs, that is inadequate for larger web sites.
Excellent: Many URLs could possibly be malformed or reference useful resource files (e.g., photos or scripts).
No export solution: There isn’t a built-in approach to export the listing.
To bypass The dearth of an export button, use a browser scraping plugin like Dataminer.io. Even so, these limits mean Archive.org may not supply a whole Alternative for larger web sites. Also, Archive.org doesn’t point out whether Google indexed a URL—but if Archive.org identified it, there’s a fantastic prospect Google did, too.

Moz Pro
Although you could possibly commonly utilize a connection index to locate external sites linking to you, these resources also find out URLs on your internet site in the method.


Tips on how to utilize it:
Export your inbound one-way links in Moz Professional to secure a brief and easy listing of concentrate on URLs from a site. If you’re coping with a massive Web site, consider using the Moz API to export information past what’s manageable in Excel or Google Sheets.

It’s crucial to Be aware that Moz Pro doesn’t confirm if URLs are indexed or discovered by Google. However, because most websites apply the same robots.txt rules to Moz’s bots since they do to Google’s, this technique commonly will work properly being a proxy for Googlebot’s discoverability.

Google Lookup Console
Google Lookup Console provides several useful resources for making your list of URLs.

Links experiences:


Much like Moz Professional, the Backlinks portion supplies exportable lists of target URLs. Sad to say, these exports are capped at 1,000 URLs Just about every. You'll be able to utilize filters for certain web pages, but because filters don’t implement to the export, you could possibly need to depend upon browser scraping tools—limited to 500 filtered URLs at a time. Not best.

Performance → Search Results:


This export provides you with a listing of web pages acquiring look for impressions. Although the export is restricted, You should utilize Google Look for Console API for bigger datasets. In addition there are no cost Google Sheets plugins that simplify pulling much more substantial knowledge.

Indexing → Webpages report:


This area supplies exports filtered by challenge form, while these are definitely also limited in scope.

Google Analytics
Google Analytics
The Engagement → Internet pages and Screens default report in GA4 is a wonderful supply for gathering URLs, having a generous Restrict of one hundred,000 URLs.


Even better, you are able to utilize filters to develop unique URL lists, proficiently surpassing the 100k limit. For example, if you want to export only web site URLs, comply with these measures:

Phase one: Insert a segment to your report

Stage two: Click “Produce a new phase.”


Stage three: Define the section with a narrower URL pattern, such as URLs that contains /blog/


Notice: URLs located in Google Analytics might not be discoverable by Googlebot or indexed by Google, but they provide valuable insights.

Server log information
Server or CDN log documents are Maybe the last word Software at your disposal. These logs seize an exhaustive listing of each URL route queried by users, Googlebot, or other bots in the course of the recorded period of time.

Criteria:

Facts dimensions: Log documents is often large, a great number of internet sites only retain the final two weeks of knowledge.
Complexity: Examining log files is usually hard, but several equipment are available to simplify the method.
Combine, and fantastic luck
As soon as you’ve gathered URLs from these sources, it’s time to mix them. If your website is sufficiently small, use Excel or, for greater datasets, applications like Google Sheets or Jupyter Notebook. Assure all URLs are persistently formatted, then deduplicate the list.

And voilà—you now have an extensive listing of existing, aged, and archived URLs. Excellent luck!