Build Hadoop Native Libraries

How to build Hadoop Native Libraries for Hadoop 2.2.0

Because the distributed Hadoop 2.2.0 provides a 32bit libhadoop by default, user has to build the native libraries to avoid those warning messages such as, disabled stack guard of libhadoop.so.

Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
13/11/01 10:58:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

The official Hadoop websitehttp://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/NativeLibraries.html gives completely unclear instructions on how to build Hadoop native libraries.

So here are what you should do:

You need all the build tools:

$ sudo apt-get install build-essential
$ sudo apt-get install g++ autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev
$ sudo apt-get install maven

Another prerequisite, protoco buffer: protobuf version 2.5, which can be downloaded from https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz. Download it to the /tmp directory; then,

$ tar xzvf protobuf-2.5.0.tar.gz
$ cd protobuf-2.5.0
$ ./configure --prefix=/usr
$ make
$ make check
$ sudo make install

Having all the tools, we can now build Hadoop native libraries. Assuming you have downloaded the Hadoop 2.2.0 source code, do:

$ tar xzvf hadoop-2.2.0-src.tar.gz
$ cd hadoop-2.2.0-src
$ mvn package -Pdist,native -DskipTests -Dtar

Note: there is a missing dependency in the maven project module that results in a build failure at the hadoop-auth stage. Here is  the official bug report  and fix is

Index: hadoop-common-project/hadoop-auth/pom.xml
===================================================================
--- hadoop-common-project/hadoop-auth/pom.xml	(revision 1543124)
+++ hadoop-common-project/hadoop-auth/pom.xml	(working copy)
@@ -54,6 +54,11 @@
     </dependency>
     <dependency>
       <groupId>org.mortbay.jetty</groupId>
+      <artifactId>jetty-util</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.mortbay.jetty</groupId>
       <artifactId>jetty</artifactId>
       <scope>test</scope>
     </dependency>

 

Maven will do all the heavy work for you, and you should get this after build is completed

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------
[INFO] Total time: 15:39.705s
[INFO] Finished at: Fri Nov 01 14:36:17 CST 2013
[INFO] Final Memory: 135M/422M

The built native libraries should be at

hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/lib

 

 

Advertisement