Download a whole website using wget

This is the most effective and easy way I've found to create a complete mirror of a website that can be viewed locally with working scripts, styles, etc:. Using -m mirror instead of -r is preferred as it intuitively downloads assets and you don't have to specify recursion depth, using mirror generally determines the correct depth to return a functioning site.

The commands -p -E -k ensure that you're not downloading entire pages that might be linked to e. Link to a Twitter profile results in you downloading Twitter code while including all pre-requisite files JavaScript, css, etc.

Proper site structure is preserved as well instead of one big. It's fast, I have never had to limit anything to get it to work and the resulting directory looks better than simply using the -r 'url' arg and provides better insight into how the site was put together, especially if you're reverse-engineering for educational purposes.

Note that if you're downloading a web-app or a site with lots of JavaScript that was compiled from TypeScript, you won't be able to get the TypeScript that was used initially, only what is compiled and sent to the browser. After I download the website, every time I open the file, it links back to its original website.

Any idea how to solve this? What if the website requires authorization of some sort? How do we specify some cookies to wget? It never occurred to me that wget could do this, thank you for the slap in the face, it saved me from using httrack or something else unnecessarily.

If you're going to use --recursive, then you need to use --level, and you should probably be polite and use --wait. Skip to content. Sign in Sign up. Instantly share code, notes, and snippets. Last active Jan 12, Code Revisions 7 Stars Forks Embed What would you like to do? Embed Embed this gist in your website. Share Copy sharable link for this gist. But many sites do not want you to download their entire site. To prevent this they typically check how browsers identify.

Many sites refuses you to connect or sends a blank page if they detect you are not using a web-browser. You might get a message like:. Sorry, but the download manager you are using to view this site is not supported. We do not support use of such download managers as flashget, go! Wget has a very handy -U option for sites that don't like wget.

Use -U My-browser to tell the site you are using some commonly accepted browser:. You will, of course, want to use a complete string which looks plausible for -U such as :.