Hadoop and HDFS Commands Guide

Contents

Start Hadoop Check Hadoop Service Running or Not Check Hadoop Version Check HDFS Health Check Size of Directory/File Change to Sudo User Change to Regular User Super User Access Usage of -mkdir Command List HDFS Commands Show Files and Directories with Path Show Files and Directories without Path Create Zero Byte File Make a Directory Create a File in Local Append Text to HDFS from Local Copy File within HDFS Copy from Local to HDFS Copy from HDFS to Local Remove Directory and Contents Remove a Text File Remove an Empty Directory Move File within HDFS Show File Contents in Text Format Count Directories and Bytes Set Replication of File/Directories Change File/Directory Permission Change Owner and Group

Start Hadoop

hdoop@hadoop:~$ start-all.sh

WARNING: Attempting to start all Apache Hadoop daemons as hdoop in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [hadoop]
WARNING: YARN_CONF_DIR has been replaced by HADOOP_CONF_DIR. Using value of YARN_CONF_DIR.
Starting resourcemanager
WARNING: YARN_CONF_DIR has been replaced by HADOOP_CONF_DIR. Using value of YARN_CONF_DIR.
Starting nodemanagers
WARNING: YARN_CONF_DIR has been replaced by HADOOP_CONF_DIR. Using value of YARN_CONF_DIR.
hdoop@hadoop:~$
    

Check Hadoop Service Running or Not

hdoop@hadoop:~$ jps

24294 ResourceManager
24087 SecondaryNameNode
23879 DataNode
23738 NameNode
24444 NodeManager
24989 Jps
hdoop@hadoop:~$
    

Check Hadoop Version

hdoop@hadoop:~$ hadoop version
Hadoop 3.2.1
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842
Compiled by rohithsharmaks on 2019-09-10T15:56Z
Compiled with protoc 2.5.0
From source with checksum 776eaf9eee9c0ffc370bcbc1888737
This command was run using /home/hdoop/hadoop-3.2.1/share/hadoop/common/hadoop-common-3.2.1.jar
hdoop@hadoop:~$
    

Check HDFS Health

hdoop@hadoop:~$ hdfs dfs -fsck /
    

Check Size of Directory/File

hdoop@hadoop:~$ hdfs dfs -du /demo
0  0  /demo/newfile.txt

hdoop@hadoop:~$ hdfs dfs -du -s /demo
0  0  /demo
    

Change to Sudo User

hdoop@hadoop:~$ sudo su
root@hadoop:/home/hdoop#
    

Change to Regular User

root@hadoop:/home/hdoop# su hdoop
hdoop@hadoop:~$
    

Super User Access

1. useradd 
2. userdel 
3. usermod 
4. usermod -l 
5. userrename 
    

Usage of -mkdir Command

hdoop@hadoop:~$ hdfs dfs -usage mkdir
Usage: hadoop fs [generic options] -mkdir [-p]  ...
    

List HDFS Commands

hdoop@hadoop:~$ hdfs dfs -help 
Usage: hadoop fs [generic options]
    

Show Files and Directories with Path

hdoop@hadoop:~$ hdfs dfs -ls /
Found 10 items
drwxr-xr-x   - hdoop  supergroup      	0 2022-08-26 18:52 /august_old
drwxr-xr-x   - hdoop  supergroup      	0 2022-02-21 10:03 /data2
-rw-r--r--   1 hdoop  supergroup     	57 2022-08-26 19:07 /input.txt
-rwxrwxrwx   1 Local Admin         	18 2022-02-21 09:36 /.txt
-rw-r--r--   1 hdoop  supergroup     	18 2022-08-26 18:55 /learnhdfs_cp.txt
drwxr-xr-x   - hdoop  supergroup      	0 2021-11-23 11:07 /test
drwxr-xr-x   - hdoop  supergroup      	0 2021-12-20 21:03 /testfile
-rwxrw-rw-   3 Local Admin         	6456 2021-11-27 09:32 /testfile.txt
drwxrwxrwx   - hdoop  supergroup      	0 2021-11-30 10:47 /tmp
drwxr-xr-x   - hdoop

Labels