Which command is used to display the amount of free space that exists on a file system?

This chapter is from the book 

Managing HDFS Storage

You deal with very large amounts of data in a Hadoop cluster, often ranging over multiple petabytes. However, your cluster is also going to use a lot of that space, sometimes with several terabytes of data arriving daily. This section shows you how to check for used and free space in your cluster, and manage HDFS space quotas. The following section shows how to balance HDFS data across the cluster.

The following subsections show how to

• Check HDFS disk usage (used and free space)

• Allocate HDFS space quotas

Checking HDFS Disk Usage

Throughout this book, I show how to use various HDFS commands in their appropriate contexts. Here, let’s review some HDFS space and file related commands. You can view the help facility for any individual HDFS file command by issuing the following command first:

$ hdfs dfs –usage

Let’s review some of the most useful file system commands that let you check the HDFS usage in your cluster. The following sections explain how to

• Use the df command to check free space in HDFS

• Use the du command to check space usage

• Use the dfsadmin command to check free and used space

Finding Free Space with the df Command

You can check the free space in an HDFS directory with a couple of commands. The -df command shows the configured capacity, available free space and used space of a file system in HDFS.

# hdfs dfs -df
Filesystem                     Size             Used        Available Use%
hdfs://hadoop01-ns 2068027170816000 1591361508626924  476665662189076  77%
#

You can specify the –h option with the df command for more readable and concise output:

# hdfs dfs -df -h
Filesystem          Size  Used  Available  Use%
hdfs://hadoop01-ns 1.8 P 1.4 P    433.5 T    77%
#

The df –h command shows that this cluster’s currently configured HDFS storage is 1.8PB, of which 1.4PB have been used so far.

Finding the Used Space with the du Command

You can view the size of the files and directories in a specific directory with the du command. The command will show you the space (in bytes) used by the files that match the file pattern you specify. If it’s a file, you’ll get the length of the file. The usage of the du command is as follows:

$ hdfs dfs –du URI

Here’s an example:

$ hdfs dfs -du /user/alapati
67545099068  67545099068  /user/alapati/.Trash
212190509    328843053    /user/alapati/.staging
26159        78477        /user/alapati/catalyst
3291761247   6275115145   /user/alapati/hive
$

You can view the used storage in the entire HDFS file system with the following command:

$ hdfs dfs -du /
414032717599186  883032417554123  /data
0                0                /home
0                0                /lost+found
111738           335214           /schema
1829104769791    5401313868645    /tmp
325747953341360  690430023788615  /user
$

The following command uses the –h option to get more readable output:

$ hdfs dfs -du -h /
353.4 T  733.6 T  /data
0        0        /home
0        0        /lost+found
109.1 K  327.4 K  /schema
2.1 T    6.1 T    /tmp
277.3 T  570.9 T  /user
$

Note the following about the output of the du –h command shown here:

• The first column shows the actual size (raw size) of the files that users have placed in the various HDFS directories.

• The second column shows the actual space consumed by those files in HDFS.

The values shown in the second column are much higher than the values shown in the first column. Why? The reason is that the second column’s value is derived by multiplying the size of each file in a directory by its replication factor, to arrive at the actual space occupied by that file.

As you can see, directories such as /schema and /tmp reveal that the replication factor for all files in these two directories is three. However, not all files in the /data and the /user directories are being replicated three times. If they were, the second column’s value for these two file systems would also be three times the value of its first column.

If you sum up the sizes in the second column of the dfs –du command, you’ll find that it’s identical to that shown by the Used column of the dfs -df command, as shown here:

$ hdfs dfs -df -h /
Filesystem            Size    Used   Available  Use%
hdfs://hadoop01-ns 553.8 T 409.3 T     143.1 T   74%
$

Getting a Summary of Used Space with the du -s Command

The du –s command lets you summarize the used space in all files instead of giving individual file sizes as the du command does.

$ hdfs dfs -du -s -h /
131.0 T 391.1 T /
$

How to Check Whether Hadoop Can Use More Storage Space

If you’re under severe space pressure and you can’t add additional DataNodes right away, you can see if there’s additional space left on the local file system that you can commandeer for HDFS use immediately. In Chapter 3, I showed how to configure the HDFS storage directories by specifying multiple disks or volumes with the dfs.data.dir configuration parameter in the hdfs-site.xml file. Here’s an example:

<property>
<name>df.data.dir</name>
value>/u01/hadoop/data,/u02/hadoop/data,/u03/hadoop/data</value>
</property>

There’s another configuration parameter you can specify in the same file, named dfs.datanode.du.reserved, which determines how much space Hadoop can use from each disk you list as a value for the dfs.data.dir parameter. The dfs.datanode.du.reserved parameter specifies the space reserved for non-HDFS use per DataNode. Hadoop can use all data in a disk above this limit, leaving the rest for non-HDFS uses. Here’s how you set the dfs.datanode.du.reserved configuration property:

<property>
<name>dfs.datanode.du.reserved</name>
<value>10737418240</value>
<description>Reserved space in bytes per volume. Always leave this much space
free for non-dfs use.
</description>
</property>

In this example, the dfs.datanode.du.reserved parameter is set to 10GB (the value is specified in bytes). HDFS will keep storing data in the data directories you assigned to it with the dfs.data.dir parameter, until the Linux file system reaches a free space of 10GB on a node. By default, this parameter is set to 10GB. You may consider lowering the value for the dfs.datanode.du.reserved parameter if you think there’s plenty of unused space lying around on the local file system on the disks configured for Hadoop’s use.

Storage Statistics from the dfsadmin Command

You’ve seen how you can get storage statistics for the entire cluster, as well as for each individual node, by running the dfsadmin –report command. The Used, Available and Use% statistics from the dfs –du command match the disk storage statistics from the dfsadmin –report command, as shown here:

bash-3.2$ hdfs dfs -df -h /
Filesystem           Size  Used  Available  Use%
hdfs://hadoop01-ns  1.8 P 1.5 P    269.6 T   85%

In the following example, the top portion of the output generated by the dfsadmin–report command shows the cluster’s storage capacity:

bash-3.2$ hdfs dfsadmin -report
Configured Capacity: 2068027170816000 (1.84 PB)
Present Capacity: 2067978866301041 (1.84 PB)
DFS Remaining: 296412818768806 (269.59 TB)
DFS Used: 1771566047532235 (1.57 PB)
DFS Used%: 85.67%
...

You can see that both the dfs –du command and the dfsadmin –report command show identical information regarding the used and available HDFS space.

Testing for Files

You can check whether a certain HDFS file path exists and whether that path is a directory or a file with the test command:

$ hdfs dfs –test –e /users/alapati/test

This command uses the –e option to check whether the specified path exists.

You can create a file of zero length with the touchz command, which is identical to the Linux touch command:

$ hdfs dfs -touchz /user/alapati/test3.txt

Allocating HDFS Space Quotas

You can configure quotas on HDFS directories, thus allowing you to limit how much HDFS space users or applications can consume. HDFS space allocations don’t have a direct connection to the space allocations on the underlying Linux file system. Hadoop lets you actually set two types of quotas:

• Space quotas: Allow you to set a ceiling on the amount of space used for an individual directory

• Name quotas: Let you specify the maximum number of file and directory names in the tree rooted at a directory

The following sections cover

• Setting name quotas

• Setting space quotas

• Checking name and space quotas

• Clearing name and space quotas

Setting Name Quotas

You can set a limit on the number of files and directory names in any directory by specifying a name quota. If the user tries to create files or directories that go beyond the specified numerical quota, the file/directory creation will fail. Use the dfsadmin command –setQuota to set the HDFS name quota for a directory. Here’s the syntax for this command:

$ hdfs dfsadmin –setQuota <max_number> <directory>

For example, you can set the maximum number of files that can be used by a user under a specific directory by doing this:

$ hdfs dfsadmin –setQuota 100000 /user/alapati

This command sets a limit on the number of files user alapati can create under that user’s home directory, which is /user/alapati. If you grant user alapati privileges on other directories, of course, the user can create files in those directories, and those files won’t count against the name quota you set on the user’s home directory. In other words, name quotas (and space quotas) aren’t user specific—rather, they are directory specific.

Setting Space Quotas on HDFS Directories

A space quota lets you set a limit on the storage assigned to a specific directory under HDFS. This quota is the number of bytes that can be used by all files in a directory. Once the directory uses up its assigned space quota, users and applications can’t create files in the directory.

A space quota sets a hard limit on the amount of disk space that can be consumed by all files within an HDFS directory tree. You can restrict a user’s space consumption by setting limits on the user’s home directory or other directories that the user shares with other users. If you don’t set a space quota on a directory it means that the disk space quota is unlimited for that directory—it can potentially use the entire HDFS.

Hadoop checks disk space quotas recursively, starting at a given directory and traversing up to the root. The quota on any directory is the minimum of the following:

• Directory space quota

• Parent space quota

• Grandparent space quota

• Root space quota

Managing HDFS Space Quotas

It’s important to understand that in HDFS, there must be enough quota space to accommodate an entire block. If the user has, let’s say, 200MB free in their allocated quota, they can’t create a new file, regardless of the file size, if the HDFS block size happens to be 256MB. You can set the HDFS space quota for a user by executing the setSpace-Quota command. Here’s the syntax:

$ hdfs dfsadmin –setSpaceQuota <N> <dirname>...<dirname>

The space quota you set acts as the ceiling on the total size of all files in a directory. You can set the space quota in bytes (b), megabytes (m), gigabytes (g), terabytes (t) and even petabytes (by specifying p—yes, this is big data!). And here’s an example that shows how to set a user’s space quota to 60GB:

$ hdfs dfsadmin -setSpaceQuota 60G /user/alapati

You can set quotas on multiple directories at a time, as shown here:

$ hdfs dfsadmin -setSpaceQuota 10g /user/alapati /test/alapati

This command sets a quota of 10GB on two directories—/user/alapati and /test/alapati. Both the directories must already exist. If they do not, you can create them with the dfs –mkdir command.

You use the same command, -setSpaceQuota, both for setting the initial limits and modifying them later on. When you create an HDFS directory, by default, it has no space quota until you formally set one.

You can remove the space quota for any directory by issuing the –clrSpaceQuota command, as shown here:

$ dfsadmin –clrSpaceQuota /user/alapati

If you remove the space quota for a user’s directory, that user can, theoretically speaking, use up all the space you have in HDFS. As with the –setSpaceQuota command, you can specify multiple directories in the –clrSpaceQuota command.

Things to Remember about Hadoop Space Quotas

Both the Hadoop block size you choose and the replication factor in force are key determinants of how a user’s space quota works. Let’s suppose that you grant a new user a space quota of 30GB and the user has more than 500MB still free. If the user tries to load a 500MB file into one of his directories, the attempt will fail with an error similar to the following, even though the directory had a bit over 500MB of free space.

org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota
        of /user/alapati is exceeded: quota = 32212254720 B = 30 GB but
        diskspace consumed = 32697410316 B = 30.45 GB

In this case, the user had enough free space to load a 500MB file but still received the error indicating that the file system quota for the user was exceeded. This is so because the HDFS block size was 128MB, and so the file needed 4 blocks in this case. Hadoop tried to replicate the file three times since the default replication factor was three and so was looking for 128*12=1556MB of space, which clearly was over the space quota left for this user.

The administrator can reduce the space quota for a directory to a level below the combined disk space usage under a directory tree. In this case, the directory is left in an indefinite quota violation state until the administrator or the user removes some files from the directory. The user can continue to use the files in the overfull directory but, of course, can’t store any new files there since their quota is violated.

Checking Current Space Quotas

You can check the size of a user’s HDFS space quota by using the dfs –count –q command as shown in Figure 9.7.

Figure 9.7 How to check a user’s current space usage in HDFS against their assigned storage limits

When you issue a dfs –count –q command, you’ll see eight different columns in the output. This is what each of the columns stands for:

• QUOTA: Limit on the files and directories

• REMAINING_QUOTA: Remaining number of files and directories in the quota that can be created by this user

• SPACE_QUOTA: Space quota granted to this user

• REMAINING_SPACE_QUOTA: Space quota remaining for this user

• DIR_COUNT: The number of directories

• FILE_COUNT: The number of files

• CONTENT_SIZE: The file sizes

• PATH_NAME: The path for the directories

The -count –q command shows that the space quota for user bdaldr is about 100TB. Of this, the user has about 67 TB left as free space.

Clearing Current Space Quotas

You can clear the current space quota for a user by issuing the clrSpaceQuota command as shown here:

$ hdfs dfsadmin -clrSpaceQuota

Here’s an example showing how to clear the space quota for a user:

$ hdfs dfsadmin -clrSpaceQuota /user/alapati
$ hdfs dfs -count -q /user/alapati
        none             inf            none        inf        2
0                  0 /user/alapati
$

The user still can use HDFS to read files but won’t be able to create any files in that user’s HDFS “home” directory. If the user has sufficient privileges, however, she can create files in other HDFS directories. It’s a good practice to set HDFS quotas on a peruser basis. You must also set quotas for data directories on a per-project basis.

What command is used to display the amount of the free space that exists on a filesystems?

What is free space in file system?

Which command displays the amount of free disk space df?

What does the command dumpe2fs H do?