Skip to content

sopwithcamel/minni

Repository files navigation

Required Software and Dependencies
--------------------------------------
gcc and g++
Thrift RPC Library (developer version)
Intel's Thread Building Blocks (TBB) Library (developer version)
Boost Libraries (developer versions)
libfkfsClient -- Kosmos File System + Dependencies



Building libhdfs
--------------------------------------
1) Building libhdfs
	# Reference URL:
	# http://www.mail-archive.com/common-user@hadoop.apache.org/msg01812.html
	# This line compiles it!	
	JAVA_HOME=/usr/lib/jvm/default-java ant compile -Dcompile.c++=true -Dlibhdfs=true

2) Running HDFS
	a) Format filesystem:	bin/hadoop namenode -format
	b) Run DFS:		bin/start-dfs.sh



Building Minni
--------------------------------------

*** Note: Full system builds have only been completed under Ubuntu 10.04 LTS.

1) For git build run:

./bootstrap.sh
./configure --with-kfs=/home/wolf/Desktop/kosmostest/kfs-0.5/build/
make

2) For tarball build run:

./configure --with-kfs=/home/wolf/Desktop/kosmostest/kfs-0.5/build/
make



Running Minni
--------------------------------------

Start up HDFS, then all worker nodes.  Then start master/MapReduce application.

cd master/ && make test

That will cause a test to run of the MapReduce system (things should be
mostly built already).



Notes
--------------------------------------

[General]

+ Only supported DFS is KFS currently.  Both HDFS and KFS is targeted for future support.

+ Would like to have XML logging interface for easy instrumentation.

[Master]

+ The master has some Thrift RPC code, but currently it only acts as
a client to a hard-coded list of Worker nodes.  There is no uninitiated
communication from nodes to the master (all polling-based communication).

+ The master is currently single-threaded (not depending on TBB) because
that is simple for now (ease of development and debugging), and it wasn't necessary yet.

+ src/master/example/Driver.cpp shows a sample MapReduce application
using our current style (not very abstract yet).

[Worker Node]

STUB

[Client API]

STUB

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors