Hadoop commands

https://data-flair.training/blogs/top-hadoop-hdfs-commands-tutorial/

1. version

hdfs dfs version

2. mkdir

hdfs dfs -mkdir /user/dataflair/dir1

3. ls

hdfs dfs -ls /user/dataflair/dir1

hdfs dfs -ls -R

This Hadoop fs command behaves like -ls, but recursively displays entries in all subdirectories of a path.

4. put

hdfs dfs -put /home/dataflair/Desktop/sample /user/dataflair/dir1

This hadoop basic command copies the file or directory from the local file system to the destination within the DFS.

5. copyFromLocal

Usage:

copyFromLocal <localSrc> <dest>

 example:

hdfs dfs -copyFromLocal /home/dataflair/Desktop/sample /user/dataflair/dir1

This hadoop shell command is similar to put command, but the source is restricted to a local file reference.

6. get

Hadoop HDFS get Command Usage

get [-crc] <src> <localDest>

hdfs dfs -get /user/dataflair/dir2/sample /home/dataflair/Desktop

This HDFS fs command copies the file or directory in HDFS identified by the source to the local file system path identified by local destination.

hdfs dfs -getmerge /user/dataflair/dir2/sample /home/dataflair/Desktop

This HDFS basic command retrieves all files that match to the source path entered by the user in HDFS, and creates a copy of them to one single, merged file in the local file system identified by local destination.

hadoop fs -getfacl /user/dataflair/dir1/sample

hadoop fs -getfacl -R /user/dataflair/dir1

Hadoop HDFS get Command Description

This Apache Hadoop command shows the Access Control Lists (ACLs) of files and directories. If a directory contains a default ACL, then getfacl also displays the default ACL.

Options :

-R: It displays a list of all the ACLs of all files and directories recursively.

path: File or directory to list.

hadoop fs -getfattr -d /user/dataflair/dir1/sample

This HDFS file system command displays if there is any extended attribute names and values for a file or directory.

Options:

-R: It recursively lists the attributes for all files and directories.

-n name: It displays the named extended attribute value.

-d: It displays all the extended attribute values associated with the pathname.

-e encoding: Encodes values after extracting them. The valid converted coded forms are “text”, “hex”, and “base64”. All the values encoded as text strings are with double quotes (” “), and prefix 0x and 0s are used for all the values which are converted and coded as hexadecimal and base64.

path: The file or directory.

Learn: Rack Awareness, High Availability

7. copyToLocal

copyToLocal <src> <localDest>

hdfs dfs -copyToLocal /user/dataflair/dir1/sample /home/dataflair/Desktop

Similar to get command, only the difference is that in this the destination is restricted to a local file reference.

8. cat

Hadoop HDFS cat Command Usage

cat <file-name>

hdfs dfs -cat /user/dataflair/dir1/sample

This Hadoop fs shell command displays the contents of the filename on console or stdout.

9. mv

Hadoop HDFS mv Command Usage

mv <src> <dst>

hadoop fs -mv /user/dataflair/dir1/purchases.txt /user/dataflair/dir2

This basic HDFS command moves the file or directory indicated by the source to destination, within HDFS.

Learn: HDFS Disk Balancer and Erasure coding

10. cp

cp <src> <dest>

hadoop fs -cp /user/dataflair/dir2/purchases.txt /user/dataflair/dir1

More:

-https://data-flair.training/blogs/hadoop-hdfs-commands/

-https://data-flair.training/blogs/hdfs-hadoop-commands/

-https://data-flair.training/blogs/hadoop-hdfs-commands-cli-perform-operations/