# How to download large files on Linux from the terminal

`wget` is a free command line tool that downloads files from the terminal non-interactively; it can work in the background without needing the user to log on. Downloading from the browser may require the user's constant presence for downloads to finish, which is not always ideal when dealing with large file downloads.

## Advantages of using `wget`

* `wget` is built to cater to slow and/or unstable networks. The `wget` utility retries when a download fails and supports downloading from where the last download failed (if the server the user is downloading from supports Range headers).
    
* Furthermore, the utility allows the use of proxy servers which can aid in faster retrievals and lighter network loads.
    

* `wget` supports recursive downloads. A recursive download refers to the process of downloading a file and then using that file to download additional files, repeating this process until all necessary files have been downloaded. This is often used in downloading a website or other collection of files from the internet, where the initial file (e.g., a HTML page) contains links to other files (e.g., images, stylesheets, etc.) that are also needed to render the content fully. Therefore, `wget` can be tweaked to work as a website crawler.
    

* Last but not least, `wget` supports downloads using the <mark>FTP</mark>, <mark>HTTP</mark>, and <mark>HTTPS</mark> protocols, all internet protocols used to transfer data between computers. <mark>FTP</mark> (File Transfer Protocol) is a standard network protocol that transfers files from one host to another over a TCP-based network. FTP allows clients to upload, download, and manage files on a remote server. <mark>HTTP</mark> (Hypertext Transfer Protocol) is a standard protocol for sending and receiving information on the World Wide Web. It is the foundation of data communication on the web and is used to transmit data from a server to a client, such as a web browser. <mark>HTTPS</mark> (HTTP Secure) is a variant of HTTP that uses a secure SSL/TLS connection to encrypt data sent between a server and a client. This makes it more difficult for someone to intercept and read the data as it is transmitted. HTTPS is commonly used to protect sensitive information, such as login credentials and financial transactions.
    

The `wget` comes installed on most Linux distributions. To install it on Debian-based distributions use:

```bash
sudo apt install wget
```

The `wget` syntax is:

```bash
wget [options] [URL]
```

* `wget` invokes the wget utility
    
* `[options]` instruct wget on what to do with the URL provided. Some options have a long-form and short-form version.
    
* `[URL]` file or folder to be downloaded. This can also be a website link to download
    

## Download Options

We will cover the following download options

1. Retrying when a connection fails.
    
2. Saving downloaded files with a different name.
    
3. Continuation of partially downloaded files.
    
4. Download to a specific folder.
    
5. Using proxies.
    
6. Limit download speeds.
    
7. Extracting from multiple URLs.
    
8. Mirror a webpage.
    
9. Extract entire websites.
    
10. Extract a website as if you were Googlebot.
    

### Retrying when a connection fails.

* By default `wget` retries 20 times except for fatal errors. Examples of fatal errors are connection refused or 404 (not found) errors.
    
    ```bash
    # Long form 
    wget --tries=number http://example.com/myfile.zip
    
    # Short form
    wget -t number http://example.com/myfile.zip
    ```
    
    The `number` can be specified as `0` or `inf` for infinite retries
    
    ```bash
    # Long form 
    wget --tries=0 http://example.com/myfile.zip
    
    # Short form
    wget -t 0 http://example.com/myfile.zip
    ```
    

### Saving downloaded files with a different name.

* The default behavior is to save the file with the name derived from the URL. Using the output file option, you can specify a new name for the file.
    
    ```bash
    # Long form
    wget -O newfilename.zip http://example.com/myfile.zip
    # short form
    wget --output-file=newfilename.zip http://example.com/myfile.zip
    ```
    
    In the code above, the file downloaded `myfile.zip` will be saved as `newfilename.zip` .
    
    > However, it is important to note that the use of `-O` is *not* intended to mean simply “use the name `newfilename` instead of `myfilezip` as provided in the URL;”. Rather, it is analogous to shell redirection that works like :
    
    ```bash
    wget -O - http://example.com/myfile.zip > newfilename.zip
    ```
    
    `newfilename.zip` will be truncated, and all downloaded content will be saved there.
    

### Continuation of partially downloaded files.

* The continue option allows the continuation of a partially downloaded file. This can be a file that was downloaded by another `wget` instance or another program.
    
    ```bash
    # Long form
    wget --continue http://example.com/myfile.zip
    
    # Short form
    wget -c http://example.com/myfile.zip
    ```
    
    Assuming there was a partial download with the name `myfile.zip`, `wget` assumes the local copy is the first portion of the file and attempts to retrieve the rest of the file from the server using an offset equal to the length of the local file. In the event, the file found locally, and that on the server are of equal size, `wget` prints an explanatory message
    
    ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1672921814572/e07e4f4a-1486-456d-bb23-0b54b25b2eba.png align="center")
    
    Consequently, should the file on the server be smaller than the one locally, the assumption made is the file on the server is an updated version. However, if the file on the server is bigger than locally due to a new version, the download will continue and the result becomes a garbled file.
    
    > The continue option only works with FTP servers and HTTP servers that support the Range header.
    

### Download to a specific folder

To download a file with `wget` and save it to a specific folder, the `-P` or `--directory-prefix` option followed by the path to the directory where files are to be saved are used.

For example:

```bash
# Long Form
wget -directory-prefix=/path/to/directory http://example.com/myfile.zip

# Short form
wget -P /path/to/directory http://example.com/myfile.zip
```

### Using proxies

To use a proxy with `wget`, the `--proxy` option followed by the proxy URL are used.

For example:

```bash
wget --proxy=http://myproxy.example.com:8080 http://example.com/myfile.zip
```

This downloads the file [`myfile.zip`](http://myfile.zip) from [`http://example.com`](http://example.com) using the HTTP proxy at [`http://myproxy.example.com:8080`](http://myproxy.example.com:8080).

If the proxy requires authentication, use the `--proxy-user` and `--proxy-password` options to specify the username and password.

For example:

```bash
wget --proxy=http://myproxy.example.com:8080 --proxy-user=myusername --proxy-password=mypassword http://example.com/myfile.zip
```

The `--proxy-type` option can also be used to specify the type of proxy being used. Valid values include `http`, `socks4`, and `socks5`.

For example:

```bash
wget --proxy=http://myproxy.example.com:8080 --proxy-type=socks5 http://example.com/myfile.zip
```

If all downloads require going through a proxy every time, modify the `~/.wgetrc` file. You can do this with `nano` or your favorite text editor

```bash
nano ~/.wgetrc
```

Add the following lines to the file

```bash
use_proxy = on
http_proxy =  http://username:password@proxy.server.address:port/
https_proxy =  http://username:password@proxy.server.address:port/
```

### Limit download speeds

The `--limit-rate` option is used to limit the download speed of `wget`. For example, to limit the download speed to 100KB/s:

```bash
wget --limit-rate=100k http://example.com/file.zip
```

The maximum download speed in bytes per second can be specified by using the `B` suffix. For example, to limit the download speed to 100KB/s:

```bash
wget --limit-rate=100000B http://example.com/file.zip
```

### Extracting multiple URLs

To download multiple files with `wget`, specify multiple URLs on the command line, separated by spaces.

For example:

```bash
wget http://example.com/file1.zip http://example.com/file2.zip http://example.com/file3.zip
```

This will download the files [`file1.zip`](http://file1.zip), [`file2.zip`](http://file2.zip), and [`file3.zip`](http://file3.zip) . If you have a list of URLs in a file, you can use the `-i` or `--input-file` option to tell Wget to read the URLs from the file.

For example:

```bash
# Long form
wget --input-file=url-list.txt
# Short form
wget -i url-list.txt
```

### Mirror a webpage

To create a mirror of a webpage, you can use the `wget` command-line utility with the `-m` or `--mirror` option. This option tells `wget` to download the webpage and all the files it references (such as images, stylesheets, etc.), recursively and save them to your local machine.

For example:

```bash
# Long form
wget --mirror http://example.com

# Short form
wget -m http://example.com
```

This downloads the webpage at [`http://example.com`](http://example.com)and all the files it references, and save them to the current working directory. The files will be saved in a directory structure that mirrors the structure of the website, with the home page saved as `index.html` and other pages saved in subdirectories as necessary.

You can also use the `-k` or `--convert-links` option to tell `wget` to convert the links in the downloaded HTML pages to point to the local copies of the files so that the downloaded copy of the website can be browsed offline.

For example:

```bash
# Long form
wget --mirror --convert-links http://example.com

# Short form
wget -m -k http://example.com
```

### Extract entire websites

To download an entire website with `wget`, you can use the `--recursive` option to follow links and download all the files it finds, recursively.

For example:

```bash
wget --recursive http://example.com
```

### Extract the website as Googlebot

To extract a website as if you were Googlebot using Wget, you can use the `--user-agent` option to set the user agent string to `"Googlebot"`. For example:

```bash
wget --user-agent="Googlebot" http://example.com
```

The website will be downloaded using the user agent string `"Googlebot"`, which tells the server that the request is coming from Google's web crawler. This can be useful if you want to see how Googlebot would render a website or if you want to download a website while bypassing any restrictions that may be in place based on the user agent.

You can also combine options demonstrated, for example:

```bash
wget --user-agent="Googlebot" -O example.html http://example.com
```

Resources:

[https://www.gnu.org/software/wget/manual/wget.html#Examples](https://www.gnu.org/software/wget/manual/wget.html#Examples)