adsensecode1
1. Objective
In this tutorial we are going to learn remaining important and frequently used HDFS commands with the help of which we will be able to perform HDFS file operations like copying file, changing files permissions, viewing the file contents, changing files ownership, creating directories, etc. To learn more about world’s most reliable storage layer follow this HDFS introductory guide
2. HDFS Commands
Hadoop file system shell commands are used to perform various Hadoop HDFS operations and in order to manage the files present on HDFS clusters. All the Hadoop file system shell commands are invoked by the bin/hdfs script.
2.1. find
Command Usage
1
| hadoop fs -find <path> ... <expression> ... |
Command Example
1
| hadoop fs -find /user/dataflair/dir1/ -name sample -print |
Description
Finds all files that match the specified expression and performs all the actions to them which are selected. If no path is specified then defaults to the present working directory. If none of the expression is specified then defaults to -print.
Finds all files that match the specified expression and performs all the actions to them which are selected. If no path is specified then defaults to the present working directory. If none of the expression is specified then defaults to -print.
2.2. help
Command Usage
1
| hadoop fs -help |
Command Example
1
| hadoop fs -help |
Description
It displays usage information for the commands entered by the user. User should exclude the leading ‘-‘ character in cmd.
It displays usage information for the commands entered by the user. User should exclude the leading ‘-‘ character in cmd.
2.3. setfattr
Command Usage
1
| hadoop fs -setfattr -n name [-v value] | -x name <path> |
Command Example
1
2
3
| hdfs dfs -setfattr -n user.myAttr -v myValue /user/dataflair/dir2/purchases.txthdfs dfs -setfattr -n user.noValue /user/dataflair/dir2/purchases.txthdfs dfs -setfattr -x user.myAttr /user/dataflair/dir2/purchases.txt |
Description
Sets an extended attribute name and value for a file or directory.
Options:
-b: It removes all but the base ACL entries. All the entries are retained for user, group and others for compatibility with permission bits.
-n name: It displays the extended attribute name.
-v value: It displays the extended attribute value. For the values, there are three different encoding methods. The argument value is the string inside, if any argument is enclosed in double quotes. If before any argument there is 0x or 0X as prefix, then it is considered as a hexadecimal number. If before any argument there is 0s or 0S, then it is considered as a base64 encoding.
-x name: It removes the extended attribute.
path: The file or directory.
Sets an extended attribute name and value for a file or directory.
Options:
-b: It removes all but the base ACL entries. All the entries are retained for user, group and others for compatibility with permission bits.
-n name: It displays the extended attribute name.
-v value: It displays the extended attribute value. For the values, there are three different encoding methods. The argument value is the string inside, if any argument is enclosed in double quotes. If before any argument there is 0x or 0X as prefix, then it is considered as a hexadecimal number. If before any argument there is 0s or 0S, then it is considered as a base64 encoding.
-x name: It removes the extended attribute.
path: The file or directory.
2.4. truncate
Command Usage
1
| hadoop fs -truncate [-w] <length> <paths> |
Command Example
1
2
| hadoop fs -truncate 55 /user/dataflair/dir2/purchases.txt /user/dataflair/dir1/purchases.txthadoop fs -truncate -w 127 /user/dataflair/dir2/purchases.txt |
Description
It truncates (shorts) all the files to a specified length that match the specified file pattern.
Options:
The -w flag requests that if necessary the command waits for block recovery to get completed. Without -w flag the file may remain unclosed for some time the process of recovery is going on. At this time the file cannot be reopened for append.
It truncates (shorts) all the files to a specified length that match the specified file pattern.
Options:
The -w flag requests that if necessary the command waits for block recovery to get completed. Without -w flag the file may remain unclosed for some time the process of recovery is going on. At this time the file cannot be reopened for append.
2.5. usage
Command Usage
1
| hadoop fs -usage command |
Command Example
1
| hadoop fs -usage mkdir |
Description
Return the help for an individual command.
Return the help for an individual command.
