What is the Hadoop-core
configuration? Hadoop core is
configured by two xml files: 1. hadoop-default.xml which was renamed to 2.
hadoop-site.xml. These files are written in xml format. We have certain
properties in these xml files, which consist of name and value. But these files
do not exist now. Which are the three modes in which Hadoop can be run? The
three modes in which Hadoop can be run are: 1. standalone (local) mode 2.
Pseudo-distributed mode 3. Fully distributed mode
Explain what are the features of Stand
alone (local) mode? In stand-alone mode there are no daemons, everything
runs on a single JVM. It has no DFS and utilizes the local file system.
Stand-alone mode is suitable only for running MapReduce programs during
development. It is one of the most least used environments.
What are the features of Pseudo mode? Pseudo
mode is used both for development and in the QA environment. In the Pseudo mode
all the daemons run on the same machine.
Can we call VMs as pseudos? No, VMs are not pseudos because VM is something
different and pesudo is very specific to Hadoop.
What are the features of Fully Distributed
mode? Fully Distributed mode is used in the production environment, where
we have „n‟ number of machines forming a Hadoop cluster. Hadoop daemons run on
a cluster of machines.
There
is one host onto which Namenode is running and another host on which datanode
is running and then there are machines on which task tracker is running. We
have separate masters and separate slaves in this distribution.
Does Hadoop follows the UNIX pattern? Yes,
Hadoop closely follows the UNIX pattern. Hadoop also has the „conf„ directory
as in the case of UNIX.
In which directory Hadoop is installed? Cloudera
and Apache has the same directory structure. Hadoop is installed in cd
/usr/lib/hadoop-0.20/.
What are the port numbers of Namenode, job
tracker and task tracker? The port number for Namenode is ‟70′, for job
tracker is ‟30′ and for task tracker is ‟60′.
What are the Hadoop configuration files at present? There are 3 configuration files in Hadoop: 1.
core-site.xml 2. hdfs-site.xml 3. mapred-site.xml These files are located in
the conf/ subdirectory.
How to exit the Vi editor? To
exit the Vi Editor, press ESC and type :q and then press enter.
What is a spill factor with respect to the
RAM? Spill factor is the size after which your files move to the temp file.
Hadoop-temp directory is used for this.
Is fs.mapr.working.dir a single directory? Yes,
fs.mapr.working.dir it is just one directory.
Which are the three main hdfs-site.xml properties? The three main hdfs-site.xml properties are: 1.
dfs.name.dir which gives you the location on which metadata will be stored and
where DFS is located – on disk or onto the remote. 2. dfs.data.dir which gives
you the location where the data is going to be stored. 3. fs.checkpoint.dir
which is for secondary Namenode.
How to come out of the insert mode? To come out of the insert mode, press ESC, type :q (if you have not
written anything) OR type :wq (if you have written anything in the file) and
then press ENTER.
Finance economy finance & insurance money derivatives wall
street young money got money cash money get money
What is Cloudera and why it is used? Cloudera is the distribution of Hadoop. It is a user created on VM by
default. Cloudera belongs to Apache and is used for data processing.
What happens if you get a ‘connection refused java exception’ when you
type hadoop fsck /?
It
could mean that the Namenode is not working on your VM.
We are using Ubuntu operating system with
Cloudera, but from where we can download Hadoop or does it come by default with
Ubuntu? This is a default configuration of Hadoop that you have to download
from Cloudera or from Edureka‟s dropbox and the run it on your systems. You can
also proceed with your own configuration but you need a Linux box, be it Ubuntu
or Red hat. There are installation steps present at the Cloudera location or in
Edureka‟s Drop box. You can go either ways. What does ‘jps’ command do? This
command checks whether your Namenode, datanode, task tracker, job tracker, etc
are working or not.
How can I restart Namenode? 1.
Click on stop-all.sh and then click on start-all.sh OR 2. Write sudo hdfs
(press enter), su-hdfs (press enter), /etc/init.d/ha (press enter) and then
/etc/init.d/hadoop-0.20-namenode start (press enter).
What is the full form of fsck? Full
form of fsck is File System Check.
How can we check whether Namenode is working or not? To check whether Namenode is working or not, use the
command /etc/init.d/hadoop- 0.20-namenode status or as simple as jps.
What does the command mapred.job.tracker
do? The command mapred.job.tracker lists out which of your nodes is acting
as a job tracker.
What does /etc /init.d do? /etc /init.d
specifies where daemons (services) are placed or to see the status of these
daemons. It is very LINUX specific, and nothing to do with Hadoop
.
How can we look for the Namenode in the browser? If you have to look for
Namenode in the browser, you don‟t have to give localhost:8021, the port number
to look for Namenode in the brower is 50070.
How to change from SU to Cloudera? To
change from SU to Cloudera just type exit.
Which files are used by the startup and shutdown commands? Slaves and Masters are used by the startup and the
shutdown commands
.
What do slaves consist of? Slaves consist of a list of hosts, one per
line, that host datanode and task tracker servers.
What do masters consist of? Masters
contain a list of hosts, one per line, that are to host secondary namenode
servers.
What does hadoop-env.sh do?
hadoop-env.sh
provides the environment for Hadoop to run. JAVA_HOME is set over here.
Can we have multiple entries in the master
files? Yes, we can have multiple entries in the Master files.
Where is
hadoop-env.sh file present? hadoop-env.sh file is present in the conf location
. In
Hadoop_PID_DIR, what does PID stands for? PID stands for „Process ID‟.
What does
/var/hadoop/pids do? It
stores the PID.
What does
hadoop-metrics.properties file do? hadoop-metrics.properties is used for „Reporting„ purposes.
It controls the reporting for Hadoop. The default status is „not to report„.
What are
the network requirements for Hadoop? The Hadoop core uses Shell (SSH) to launch the server
processes on the slave nodes. It requires password-less SSH connection between
the master and all the slaves and the secondary machines.
Why do we need a password-less SSH in Fully
Distributed environment? We need a password-less SSH in a Fully-Distributed
environment because when the cluster is LIVE and running in Fully Distributed
environment, the communication is too frequent. The job tracker should be able
to send a task to task tracker quickly.
Does this lead to security issues? No,
not at all. Hadoop cluster is an isolated cluster. And generally it has nothing
to do with an internet. It has a different kind of a configuration. We needn‟t
worry about that kind of a security breach, for instance, someone hacking
through the internet, and so on. Hadoop has a very secured way to connect to
other machines to fetch and to process data.
On which
port does SSH work? SSH
works on Port No. 22, though it can be configured. 22 is the default Port
number.
Can you
tell us more about SSH? SSH is nothing but a secure shell communication, it is a kind of a
protocol that works on a Port No. 22, and when you do an SSH, what you really
require is a password.
Why
password is needed in SSH localhost? Password is required in SSH for security and in a situation
where passwordless communication is not set.
Do we
need to give a password, even if the key is added in SSH? Yes, password is still required even
if the key is added in SSH.
