Working Thru Hadoop Examples

mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>
/usr/local/Cellar/hadoop/2.8.1/libexec/sbin/start-yarn.sh

Or just:

start-yarn.sh

The dashboard looks like this:

http://localhost:8088/

The above is all related to yarn, which I don’t particularly understand. Let’s look again at what we’ve done before, in a bit of a new way…

Got something in the input of our hdfs?

hadoop fs -ls /user/s.matthew.english/input

Get your namenode into gear:

hdfs namenode -format

Kick things off:

start-all.sh

Run the job:

hadoop jar /usr/local/Cellar/hadoop/2.8.1/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar grep input output_0 'principal[.]*'

What the command there is actually doing is grep'ing for the text principal in the input files that live on the Hadoop file system — something like that.