adsensecode1
1. Objective
In this tutorial we are going to learn remaining important and frequently used HDFS commands with the help of which we will be able to perform HDFS file operations like copying file, changing files permissions, viewing the file contents, changing files ownership, creating directories, etc. To learn more about world’s most reliable storage layer follow this HDFS introductory guide
Looking to learn HDFS, follow these detailed tutorials: HDFS High Availability, HDFS Fault Tolerance
2. HDFS Commands
Hadoop file system shell commands are used to perform various Hadoop HDFS operations and in order to manage the files present on HDFS clusters. All the Hadoop file system shell commands are invoked by the bin/hdfs script.
2.1. touchz
Command Usage
1
| touchz <path> |
Command Example
1
| hdfs dfs -touchz /user/dataflair/dir2 |
Description
It creates a file at the path containing the current time as a timestamp. Fails if a file already exists at path, unless the file is already size 0.
It creates a file at the path containing the current time as a timestamp. Fails if a file already exists at path, unless the file is already size 0.
2.2. test
Command Usage
1
| hdfs dfs -test -[ezd] URI |
Command Example
1
2
3
| "hdfs dfs -test -e samplehdfs dfs -test -z samplehdfs dfs -test -d sample" |
Description
The hadoop test is used for file test operations.
It gives 1 output if path exists; it has zero length; or it is a directory or otherwise 0.
Options:
-d: if the path given by the user is a directory, then it gives 0 output.
-e: if the path given by the user exists, then it gives 0 output.
-f: if the path given by the user is a file, then it gives 0 output.
-s: if the path given by the user is not empty, then it gives 0 output.
-z: if the file is zero length, then it gives 0 output.
The hadoop test is used for file test operations.
It gives 1 output if path exists; it has zero length; or it is a directory or otherwise 0.
Options:
-d: if the path given by the user is a directory, then it gives 0 output.
-e: if the path given by the user exists, then it gives 0 output.
-f: if the path given by the user is a file, then it gives 0 output.
-s: if the path given by the user is not empty, then it gives 0 output.
-z: if the file is zero length, then it gives 0 output.
2.3. text
Command Usage
1
| hdfs dfs -text <source> |
Command Example
1
| hdfs dfs -text /user/dataflair/dir1/sample |
Description
Takes a source file and outputs the file in text format. The allowed formats are zip and TextRecordInputStream.
Takes a source file and outputs the file in text format. The allowed formats are zip and TextRecordInputStream.
2.4. stat
Command Usage
1
| hdfs dfs -stat path |
Command Example
1
| hdfs dfs -stat /user/dataflair/dir1 |
Description
Prints information about path.
Prints information about path.
%b: If the format is a string which accepts file size in blocks.
%n: Filename
%o: Block size
%r: replication
%y, %Y: modification date.
2.5. tail
Command Usage
1
| hdfs dfs -tail [-f] <filename2> |
Command Example
1
2
| "hdfs dfs -tail /user/dataflair/dir1/samplehdfs dfs -tail -f /user/dataflair/dir1/sample" |
Description
Shows the last 1KB of file on console or stdout.
Shows the last 1KB of file on console or stdout.
2.6. chown
Command Usage
1
| hdfs dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ] |
Command Example
1
| hdfs dfs -chown -R dataflair /opt/hadoop/logs |
Description
Changes the owner of files. With -R, makes the change according to the order of the directory structure recursively. The user must be the superuser.
Changes the owner of files. With -R, makes the change according to the order of the directory structure recursively. The user must be the superuser.
2.7. chmod
Command Usage
1
| chmod [-R] mode,mode,... <path>... |
Command Example
1
| hdfs dfs -chmod 777 /user/dataflair/dir1/sample |
Description
Changes the permissions of files. With -R, makes the change recursively by way of the directory structure. The user must be the owner of the file or the superuser.
Changes the permissions of files. With -R, makes the change recursively by way of the directory structure. The user must be the owner of the file or the superuser.
2.8. appendToFile
Command Usage
1
| hadoop fs -appendToFile <localsource> ... <dst> |
Command Example
1
| hadoop fs -appendToFile /home/dataflair/Desktop/sample /user/dataflair/dir1 |
Description
Append single sources, or multiple sources from local file system to the file system at the destination. It also reads input from standard input and add it to to destination file system.
Append single sources, or multiple sources from local file system to the file system at the destination. It also reads input from standard input and add it to to destination file system.
2.9. checksum
Command Usage
1
| hadoop fs -checksum URI |
Command Example
1
| hadoop fs -checksum /user/dataflair/dir1/sample |
Description
Returns the checksum information of a file.
Returns the checksum information of a file.
2.10. count
Command Usage
1
| hdfs dfs -count [-q] <paths> |
Command Example
1
| hdfs dfs -count /user/dataflair |
Description
Counts the number of directories, number of files present and bytes under the paths that match the specified file pattern.
Counts the number of directories, number of files present and bytes under the paths that match the specified file pattern.
