SBN How to Handle Large Files

Large files are files with sizes greater than 2GB (i.e., 2^30 or 2147483648 bytes). These files still pose a major downloading problem for 32-bit systems. For the gory details on why, you can consult this Wikipedia article: Large File Support.

We have ensured that we are serving our large files through a specially-built HTTP server that contains large file support, so that we can at least guarantee that we can pump the files out. That does not, however, guarantee that you can receive them. You must also have large file support in the client system installed at your end, and you must download them onto a disk drive that is in a format that can supports large files. Windows NTFS, Mac HFS+ and Linux ext3 formats all support large files. FAT32, which is used on most USB drives, cannot support large files and will just chop off everything after the 2GB limit.

If you have a 64-bit system, then the chances are pretty good that your client programs already support large files. If you're on a 32-bit system, however, the situation is much different. At this point it is easier to list what we know works than everything that doesn't:

Web Browsers
On contemporary Macs, we've seen the Firefox 2.0 and Safari browsers work without complaint.

In Windows, we know the Firefox 2.0 browser should work, but we have yet to see a successful demonstration of any large-file capable browser. If you can verify that a particular browser works, please let us know.

In Linux, Firefox 2.0 is known to work.

Wget
The wget command seems to handle large files in all cases we've tried. Versions for Linux and binaries for Windows are available from Wget's Website. Mac users may discover that wget is included in the tools available from your unix command line. Wget will run in the background or as a batch job. The typical command looks like this:

You can copy and past the URL from your browser into the command line. The output file is written to the same name as the file on the server. If you should be downloading a file from a password-protected area, you'll need to use the --user and --password options (note the leading double dash) to supply the user name and password you've been given. For example, if your user name is "steve" and the password is "send2ME", the command would look like this:

Please read your local documentation for extensive variations on the theme.

cURL
An increasingly popular option for downloading large files, this utility will also run in the background and is available for all major operating systems from the cURL web site. Contemporary Linux and Mac users may find it already installed and waiting for them at the command line.

A typical cURL command line looks like this:

where -O (that's an uppercase letter "O") tells curl to use the same name for the output file as the file name found at the end of the URL (omit this and curl will dump the file directly to your screen, which is probably not what you want); and URL can be copied and pasted from your browser. If the URL you are trying to access is password protected, then you will need to use the -u option to supply the user name and password. For example, if your user name is "steve" and the password is "send2ME", the command would look like this:

Once again, see your local documentation for extensive options, including how to supply your own output file name and how to resume a transfer that was interrupted in midstream.

Last update: 18 Feb 2008, A.C.Raugh