adsensecode1
1. Objective
In this tutorial we are going to learn remaining important and frequently used HDFS commands with the help of which we will be able to perform HDFS file operations like copying file, changing files permissions, viewing the file contents, changing files ownership, creating directories, etc. To learn more about world’s most reliable storage layer follow this HDFS introductory guide
2. HDFS Commands
Hadoop file system shell commands are used to perform various Hadoop HDFS operations and in order to manage the files present on HDFS clusters. All the Hadoop file system shell commands are invoked by the bin/hdfs script.
2.1. moveFromLocal
Command Usage
1
| moveFromLocal <localSrc> <dest> |
Command Example
1
| hdfs dfs -moveFromLocal /home/dataflair/Desktop/sample /user/dataflair/dir1 |
Description
Copies the file or directory from the local file system identified by local source to destination within HDFS, and then deletes the local copy on success.
Copies the file or directory from the local file system identified by local source to destination within HDFS, and then deletes the local copy on success.
Learn more about Internals of HDFS Data Write Pipeline and File write execution flow
2.2. moveToLocal
Command Usage
1
| moveToLocal <src> <localDest> |
Command Example
1
| hdfs dfs -moveToLocal /user/dataflair/dir2/sample /user/dataflair/Desktop |
Description
Works like -get, but deletes the HDFS copy on success.
Works like -get, but deletes the HDFS copy on success.
2.3. tail
Command Usage
1
| hdfs dfs -tail [-f] <filename> |
Command Example
1
2
| "hdfs dfs -tail /user/dataflair/dir2/purchases.txthdfs dfs -tail -f /user/dataflair/dir2/purchases.txt" |
Description
Shows the last 1KB of file on console or stdout.
Shows the last 1KB of file on console or stdout.
2.4. rm
Command Usage
1
| rm <path> |
Command Example
1
| hdfs dfs -rm /user/dataflair/dir2/sample |
Description
Removes the file or empty directory present on the path provided by the user.
Removes the file or empty directory present on the path provided by the user.
Command Example
1
| hdfs dfs -rm -r /user/dataflair/dir2 |
Description
Recursive version of delete.
Recursive version of delete.
2.5. expunge
Command Usage
1
| hdfs dfs -expunge |
Command Example
1
| hdfs dfs -expunge |
Description
Used to empty the trash.
Used to empty the trash.
2.6. chown
Command Usage
1
| hdfs dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ] |
Command Example
1
| hdfs dfs -chown -R dataflair /opt/hadoop/logs |
Description
Changes the owner of files. With -R, changes are made recursively by way of the structure of directory. User should be the superuser.
Changes the owner of files. With -R, changes are made recursively by way of the structure of directory. User should be the superuser.
2.7. chgrp
Command Usage
1
| hdfs dfs -chgrp [-R] <NewGroupName> <file or directory name> |
Command Example
1
| hdfs dfs -chgrp [-R] New Group sample |
Description
hadoop chgrp shell command is used to change the files group association. Also you can try -R option to make changes recursively by the way of structure of directory.
hadoop chgrp shell command is used to change the files group association. Also you can try -R option to make changes recursively by the way of structure of directory.
2.8. setrep
Command Usage
1
| setrep [-R] [-w] rep <path> |
Command Example
1
| hdfs dfs -setrep -w 3 /user/dataflair/dir1 |
Description
It is used to change the replication factor of a file. If the entered path is a directory, then this command changes the replication factor of all the files present on the directory tree rooted at path provided by user recursively.
Options:
The -w flag requests that the command wait for the replication process to get completed. This may likely take a very long time to get completed.
The -R flag is accepted for backwards compatibility. It does not makes any changes.
It is used to change the replication factor of a file. If the entered path is a directory, then this command changes the replication factor of all the files present on the directory tree rooted at path provided by user recursively.
Options:
The -w flag requests that the command wait for the replication process to get completed. This may likely take a very long time to get completed.
The -R flag is accepted for backwards compatibility. It does not makes any changes.
2.9. du
Command Usage
1
| du <path> |
Command Example
1
| hdfs dfs -du /user/dataflair/dir1/sample |
Description
Shows disk usage, in bytes, for all the files present on the path provided by the user; reporting of filenames are done with the full HDFS protocol prefix.
Shows disk usage, in bytes, for all the files present on the path provided by the user; reporting of filenames are done with the full HDFS protocol prefix.
Command Example
1
| hdfs dfs -du -s /user/dataflair/dir1/sample |
Description
Like -du, but it prints a summary of amount of disk usage of all files/directories in the path.
Like -du, but it prints a summary of amount of disk usage of all files/directories in the path.
2.10. df
Command Usage
1
| hdfs dfs -df [-h] URI [URI ...] |
Command Example
1
| hdfs dfs -df -h |
Description
Displays free space.
Displays free space.
