Objective:
What I’d like to do is create a micro-botnet of Docker containers that are configured together into a swarm of Hadoop nodes — here goes nothing!

Procedure:
First thing I want to do is make a clean slate, and so I’ll use the following commands to delete all the images and containers on my machine:

> docker rm $(docker ps -a -q)
> docker rmi $(docker images -q)

The success of the operation can be verified accordingly:

docker images
docker ps -a

The following command will run a basic Ubuntu container, and connect us into it:

docker run -it…


mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
/usr/local/Cellar/hadoop/2.8.1/libexec/sbin/start-yarn.sh

Or just:

start-yarn.sh

The dashboard looks like this:

http://localhost:8088/

The above is all related to yarn, which I don’t particularly understand. Let’s look again at what we’ve done before, in a bit of a new way…

Got something in the input of our hdfs?

hadoop fs -ls /user/s.matthew.english/input

Get your namenode into gear:

hdfs namenode -format

Kick things off:

start-all.sh

Run the job:

hadoop jar /usr/local/Cellar/hadoop/2.8.1/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar grep input output_0 'principal[.]*'

What the command there is actually doing is grep'ing for the text principal in the input files that live on the Hadoop file system — something like that.


Follow me, and I’ll show you the way!

The first thing you should know is that the official Apache Documentation is not particularly useful, for macOS — it essentially just tells you to use a Docker container.

Essentially the major impediment is that homebrew by default is running the wrong version of the library for Protocol Buffers — Google’s data interchange format.

Below I’ll outline the steps I took to build Apache Hadoop from source on macOS in August of 2017.

  1. Create a GitHub fork/clone of github.com/apache/hadoop, unfortunately there is no good README document associated with this repo.
  2. From within the hadoop directory run the following command:
mvn…


Interested in the idea of decentralized & anonymous (perhaps even autonomous) voting, I’ve recently attempted to deploy the Open Vote Network project to the Ropsten Revival testnet— but I didn’t succeed.

My go-to move for deploying smart contracts recently has been to use https://remix.ethereum.org, in conjunction with MetaMask. One gripe I have with Remix is that it doesn’t have a dark theme, recently I’ve been thinking of writing one myself.

Below we can see a screenshot of my attempt to compile and deploy ‘AnonymousVoting.sol’ to the testnet:

The Error I got when I tried to create the contract was:

callback…


In honour of the Great Solar Eclipse of August 21, 2017 we’ll be seeking to understand the encrypted messaging protocol ‘Constellation’, insofar as it’s related to JPM’s Quorum project. According to my understanding it’s a — I believe the technical term is “tricked out” — version of Ethereum’s standard Whisper protocol, augmented to better adhere to the demands of a private network.

The reason I want to understand Constellation is that it seems to be one of the major differentiating factors between quorum and the project it was derived from, i.e. the Go-Ethereum client (goethe).


There’s no PoW in quorum, it seems that means you can’t mine — as evidenced by my attempt to start it with:

geth --testnet --rpc --mine

Which results in the ERROR message:

flag provided but not defined: -mine
Still connecting to IPC


“What’s up with JPM’s Quorum?”

This document describes my initial experiments trying to understand JPM’s Quorum, a flavour of the Go-Ethereum client.

On my machine I have multiple versions of go-ethereum (also known as geth or goethe) installed. I was fairly confident that when I issued the following terminal command:

geth — testnet — rpc — mine

What was being executed was the version of goethe that lives in my $GOPATH, i.e.:

/Users/s.matthew.english/go/src/github.com/ethereum/go-ethereum

Turns out- that is accurate.

For instance, even though I was in the quorum directory, the output of which geth was the one on my $GOPATH:


Rudyard Kipling: PoET circa 1895

Pluggable consensus is one of the “hottest buzzwords” in the blockchain ecosystem and the Hyperledger beehive is where this buzz on pluggable consensus is emanating from most deafeningly.

It has been conjectured one reason for this state of affairs is that, while IBM’s blockchain product offering could liberally be categorized as a “blockchain” of sorts, Intel Corporation- the second industrial pillar of the Hyperledger consortium- has created not a blockchain, but rather a consensus algorithm. Pluggable consensus in the context of Hyperledger means that one could conceivably adopt the IBM data interchange protocol while simultaneously employing Intel’s PoET system in…


Execution of Assembly-Based Performance Tests on the Ethereum Virtual Machine

This document is intended to walk you through the process of running one set of the performance benchmark tests on the Ethereum Virtual Machine (EVM) on macOS.

To deploy these tests you need to be in the following directory:

/cpp-ethereum/test/unittests/performance

Which is (obviously) a sub-directory of the cpp-ethereum client. Personally I wasn’t able to get that client itself to build from source on my machine, but that isn’t required in order to run these tests- so don’t worry about it.

Here you can see I’ve created system links [1], which connect evm and solc to the directory where I’m executing…

S. Matthew English

Freestyle Scientist

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store