8 File Compression

In this chapter, we will explore file compression using the following

tar
gzip
zip
unzip

8.1 tar

The tar command is used for file compression. It works with both tar and tar.gz extensions. It is used to

list files
extract files
create archives
append file to existing archives

tar creates, maintains, modifies, and extracts files that are archived in the tar format. Tar stands for tape archive and is an archiving file format.

Command	Description
`tar tvf`	List an archive
`tar tvfz`	List a gzipped archive
`tar xvf`	Extract an archive
`tar xvfz`	Extract a gzipped archive
`tar cvf`	Create an uncompressed tar archive
`tar cvfz`	Create a tar gzipped archive
`tar rvf`	Add a file to an existing archive
`tar rvfz`	Add a file to an existing gzipped archive

We will use different options along with the tar command for listing, extracting, creating and adding files. The vf (v stands for verbosely show .tar file progress and f stands for file name type of the archive file) option is common for all the above operations while the following are specific.

t for listing
x for extracting
c for creating
r for adding files

While dealing with tar.gz archives we will use z in addition to vf and the above options.

8.1.1 List

Let us list all the files & folders in release_names.tar. As mentioned above. to list the files in the archive, we use the t option.

tar -tvf release_names.tar

-rwxrwxrwx aravind/aravind 546 2019-09-16 10:29 release_names.txt
-rwxrwxrwx aravind/aravind  65 2019-09-16 10:28 release_names_18.txt
-rwxrwxrwx aravind/aravind  53 2019-09-16 10:29 release_names_19.txt

8.1.2 Extract

Let us extract files from release_names.tar using the x option in addition to vf.

tar -xvf release_names.tar
ls

release_names.txt
release_names_18.txt
release_names_19.txt
analysis.R
bash.R
bash.Rmd
bash.sh
cran_log_sample.csv.gz
curl_demo.csv.gz
imports_blorr.txt
imports_olsrr.txt
lorem-ipsum.txt
main_project.zip
myfiles
mypackage
myproject
package_names.txt
pkg_names.txt
r
r2
r_releases
release.txt
release_names.tar
release_names.tar.gz
release_names.txt
release_names_18.txt
release_names_18_19.txt
release_names_19.txt
urls.txt
wget_demo.csv.gz
zip_example.zip

8.1.3 Add

To add a file to an existing archive, use the r option. Let us add release_names_18.txt and release_names_19.txt to the archive we created in the previous step.

tar -rvf release_names.tar release_names_18.txt release_names_19.txt

release_names_18.txt
release_names_19.txt

8.1.4 Create

Using the c option we can create tar archives. In the below example, we are using a single file but you can specify multiple files and folders as well.

tar -cvf pkg_names.tar pkg_names.txt

pkg_names.txt

8.2 gzip

Command	Description
`gzip`	Compress a file
`gzip -d`	Decompress a file
`gzip -c`	Compress a file and specify the output file name
`zip -r`	Compress a directory
`zip`	Add files to an existing zip file
`unzip`	Extract files from a zip files
`unzip -d`	Extract files from a zip file and specify the output file name
`unzip -l`	List contents of a zip file

gzip, gunzip, and zcat commands are used to compress or expand files in the GNU GZIP format i.e. files with .gz extension

8.2.1 Compress

Let us compress release_names.txt file using gzip.

gzip release_names.txt
ls
gunzip release_names.txt.gz

analysis.R
bash.R
bash.Rmd
bash.sh
cran_log_sample.csv.gz
curl_demo.csv.gz
imports_blorr.txt
imports_olsrr.txt
lorem-ipsum.txt
main_project.zip
myfiles
mypackage
myproject
package_names.txt
pkg_names.tar
pkg_names.txt
r
r2
r_releases
release.txt
release_names.tar
release_names.tar.gz
release_names.txt.gz
release_names_18.txt
release_names_18_19.txt
release_names_19.txt
urls.txt
wget_demo.csv.gz
zip_example.zip

8.2.2 Decompress

Use the -d option with gzip to decompress a file. In the below example, we stream the decompressed contents of cran_log_sample.csv.gz to the terminal while keeping the fixture compressed for later chapters. You can also use gunzip (or gzip -d on a disposable copy) for the same result.

gzip -dc cran_log_sample.csv.gz | head -n 3
ls

"date","time","size","r_version","r_arch","r_os","package","version","country","ip_id"
"2019-09-15","02:28:38",47952,NA,NA,NA,"aws.s3","0.3.12","US",1
"2019-09-15","02:28:37",5899,NA,NA,NA,"aws.ec2metadata","0.2.0","US",1
analysis.R
bash.R
bash.Rmd
bash.sh
cran_log_sample.csv.gz
curl_demo.csv.gz
imports_blorr.txt
imports_olsrr.txt
lorem-ipsum.txt
main_project.zip
myfiles
mypackage
myproject
package_names.txt
pkg_names.tar
pkg_names.txt
r
r2
r_releases
release.txt
release_names.tar
release_names.tar.gz
release_names.txt
release_names_18.txt
release_names_18_19.txt
release_names_19.txt
urls.txt
wget_demo.csv.gz
zip_example.zip

8.2.3 Specify Filename

Use -c and > to specify a different file name while compressing using gzip. In the below example, gzip will create releases.txt.gz instead of release_names.txt.gz.

gzip -c release_names.txt > releases.txt.gz
ls

analysis.R
bash.R
bash.Rmd
bash.sh
cran_log_sample.csv.gz
curl_demo.csv.gz
imports_blorr.txt
imports_olsrr.txt
lorem-ipsum.txt
main_project.zip
myfiles
mypackage
myproject
package_names.txt
pkg_names.tar
pkg_names.txt
r
r2
r_releases
release.txt
release_names.tar
release_names.tar.gz
release_names.txt
release_names_18.txt
release_names_18_19.txt
release_names_19.txt
releases.txt.gz
urls.txt
wget_demo.csv.gz
zip_example.zip

8.3 zip & unzip

zip creates ZIP archives while unzip lists and extracts compressed files in a ZIP archive.

8.3.1 List

Let us list all the files and folders in main_project.zip() using unzip and the -l option.

unzip -l main_project.zip

Archive:  main_project.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2019-09-23 12:37   myproject/
        0  2019-09-20 08:32   myproject/.gitignore
        0  2019-09-23 12:37   myproject/data/
        0  2019-09-20 08:32   myproject/data/processed/
        0  2019-09-20 08:32   myproject/data/raw/
        0  2019-09-20 08:32   myproject/output/
        0  2019-09-20 08:32   myproject/README.md
       13  2019-09-20 08:32   myproject/run_analysis.R
        0  2019-09-20 08:32   myproject/src/
        0  2019-09-23 12:37   mypackage/
        0  2019-09-20 08:41   mypackage/.gitignore
        0  2019-09-20 08:41   mypackage/.Rbuildignore
        0  2019-09-20 08:40   mypackage/data/
        0  2019-09-20 08:41   mypackage/DESCRIPTION
        0  2019-09-20 08:40   mypackage/docs/
        0  2019-09-20 08:41   mypackage/LICENSE
        0  2019-09-20 08:40   mypackage/man/
        0  2019-09-20 08:41   mypackage/NAMESPACE
        0  2019-09-20 08:41   mypackage/NEWS.md
        0  2019-09-20 08:40   mypackage/R/
        0  2019-09-20 08:41   mypackage/README.md
        0  2019-09-20 08:41   mypackage/src/
        0  2019-09-20 08:40   mypackage/tests/
        0  2019-09-20 08:40   mypackage/vignettes/
        0  2019-09-23 12:37   myfiles/
       12  2019-09-20 10:00   myfiles/analysis.R
        7  2019-09-20 10:01   myfiles/NEWS.md
        9  2019-09-20 10:01   myfiles/README.md
      546  2019-09-20 09:59   myfiles/release_names.txt
       65  2019-09-20 09:59   myfiles/release_names_18.txt
       53  2019-09-20 10:00   myfiles/release_names_19.txt
       12  2019-09-20 10:00   myfiles/visualization.R
    15333  2019-10-01 11:28   bash.sh
        0  2019-09-16 07:12   r/
---------                     -------
    16050                     34 files

8.3.2 Extract

Using unzip, let us now extract files and folders from zip_example.zip.

unzip zip_example.zip

Archive:  zip_example.zip
   creating: zip_example/
  inflating: zip_example/bash.sh     
  inflating: zip_example/pkg_names.txt

Using the -d option, we can extract the contents of zip_example.zip to a specific folder. In the below example, we extract it to a new folder myexamples.

unzip zip_example.zip -d myexamples

Archive:  zip_example.zip
   creating: myexamples/zip_example/
  inflating: myexamples/zip_example/bash.sh  
  inflating: myexamples/zip_example/pkg_names.txt

8.3.3 Compress

Use the -r option along with zip to create a ZIP archive. In the below example, we create a ZIP archive of myproject folder.

zip -r myproject.zip myproject
ls

  adding: myproject/ (stored 0%)
  adding: myproject/run_analysis.R (stored 0%)
  adding: myproject/output/ (stored 0%)
  adding: myproject/.gitignore (stored 0%)
  adding: myproject/data/ (stored 0%)
  adding: myproject/data/raw/ (stored 0%)
  adding: myproject/data/processed/ (stored 0%)
  adding: myproject/src/ (stored 0%)
  adding: myproject/README.md (stored 0%)
analysis.R
bash.R
bash.Rmd
bash.sh
cran_log_sample.csv.gz
curl_demo.csv.gz
imports_blorr.txt
imports_olsrr.txt
lorem-ipsum.txt
main_project.zip
myexamples
myfiles
mypackage
myproject
myproject.zip
package_names.txt
pkg_names.tar
pkg_names.txt
r
r2
r_releases
release.txt
release_names.tar
release_names.tar.gz
release_names.txt
release_names_18.txt
release_names_18_19.txt
release_names_19.txt
releases.txt.gz
urls.txt
wget_demo.csv.gz
zip_example
zip_example.zip

We can compress multiple directories using zip. The names of the directories must be separated by a space as shown in the below example where we compress myproject and mypackage into a single ZIP archive.

zip -r packproj.zip myproject mypackage
ls

  adding: myproject/ (stored 0%)
  adding: myproject/run_analysis.R (stored 0%)
  adding: myproject/output/ (stored 0%)
  adding: myproject/.gitignore (stored 0%)
  adding: myproject/data/ (stored 0%)
  adding: myproject/data/raw/ (stored 0%)
  adding: myproject/data/processed/ (stored 0%)
  adding: myproject/src/ (stored 0%)
  adding: myproject/README.md (stored 0%)
  adding: mypackage/ (stored 0%)
  adding: mypackage/NEWS.md (stored 0%)
  adding: mypackage/NAMESPACE (stored 0%)
  adding: mypackage/R/ (stored 0%)
  adding: mypackage/LICENSE (stored 0%)
  adding: mypackage/.gitignore (stored 0%)
  adding: mypackage/data/ (stored 0%)
  adding: mypackage/vignettes/ (stored 0%)
  adding: mypackage/tests/ (stored 0%)
  adding: mypackage/DESCRIPTION (stored 0%)
  adding: mypackage/src/ (stored 0%)
  adding: mypackage/.Rbuildignore (stored 0%)
  adding: mypackage/README.md (stored 0%)
  adding: mypackage/docs/ (stored 0%)
  adding: mypackage/man/ (stored 0%)
analysis.R
bash.R
bash.Rmd
bash.sh
cran_log_sample.csv.gz
curl_demo.csv.gz
imports_blorr.txt
imports_olsrr.txt
lorem-ipsum.txt
main_project.zip
myexamples
myfiles
mypackage
myproject
myproject.zip
package_names.txt
packproj.zip
pkg_names.tar
pkg_names.txt
r
r2
r_releases
release.txt
release_names.tar
release_names.tar.gz
release_names.txt
release_names_18.txt
release_names_18_19.txt
release_names_19.txt
releases.txt.gz
urls.txt
wget_demo.csv.gz
zip_example
zip_example.zip

8.3.4 Add

To add a new file/folder to an existing archive, specify the name of the archive followed by the name of the file or the folder. In the below example, we add the bash.sh file to the myproject.zip archive created in a previous step.

zip myproject.zip bash.sh

  adding: bash.sh (deflated 78%)

8.4 R Functions

8.4.1 tar & tar.gz

In R, we can use the tar() and untar() functions from the utils package to handle .tar and .tar.gz archives.

Command	R
`tar tvf`	`utils::untar('archive.tar', list = TRUE)`
`tar tvfz`	`utils::untar('archive.tar.gz', list = TRUE)`
`tar xvf`	`utils::untar('archive.tar')`
`tar xvfz`	`utils::untar('archive.tar.gz')`
`tar cvf`	`utils::tar('archive.tar')`
`tar cvfz`	`utils::tar('archive.tar', compression = 'gzip')`

8.4.2 zip & gzip

The zip package has the functionalities to handle ZIP archives. The tar() and untar() functions from the utils package can handle GZIP archives.

Command	R
`gzip`	`utils::tar(compression = 'gzip')` / `R.utils::gzip()`
`gzip -d`	`utils::untar()` / `R.utils::gunzip()`
`gzip -c`	`utils::untar(exdir = filename)`
`zip -r`	`zip::zip()`
`zip`	`zip::zipr_append()`
`unzip`	`zip::unzip()`
`unzip -d`	`zip::unzip(exdir = dir_name)`
`unzip -l`	`zip::zip_list()`

8.5 Exercises

List without extracting: run tar -tvf release_names.tar and unzip -l main_project.zip. You should see listings of the archived files and folders, with no new files created.
Stream without decompressing: run gzip -dc cran_log_sample.csv.gz | head -n 3 — you should see the first 3 CSV lines while the fixture stays compressed.
Create and extend: copy a text file to myarchive_src.txt, run tar -cvf myarchive.tar myarchive_src.txt, then append another file with tar -rvf myarchive.tar release_names.txt and verify with tar -tvf myarchive.tar. Separately, run zip myarchive.zip myarchive_src.txt and verify with unzip -l myarchive.zip. Clean up all myarchive* files afterwards.

Worked solutions in solutions/file-compression.md (attempt first).