How to build Hadoop Native Libraries for Hadoop 2.2.0
Because the distributed Hadoop 2.2.0 provides a 32bit libhadoop by default, user has to build the native libraries to avoid those warning messages such as, disabled stack guard of libhadoop.so.
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 13/11/01 10:58:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
The official Hadoop websitehttp://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/NativeLibraries.html gives completely unclear instructions on how to build Hadoop native libraries.
So here are what you should do:
You need all the build tools:
$ sudo apt-get install build-essential $ sudo apt-get install g++ autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev $ sudo apt-get install maven
Another prerequisite, protoco buffer: protobuf version 2.5, which can be downloaded from https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz. Download it to the /tmp directory; then,
$ tar xzvf protobuf-2.5.0.tar.gz $ cd protobuf-2.5.0 $ ./configure --prefix=/usr $ make $ make check $ sudo make install
Having all the tools, we can now build Hadoop native libraries. Assuming you have downloaded the Hadoop 2.2.0 source code, do:
$ tar xzvf hadoop-2.2.0-src.tar.gz $ cd hadoop-2.2.0-src $ mvn package -Pdist,native -DskipTests -Dtar
Note: there is a missing dependency in the maven project module that results in a build failure at the hadoop-auth stage. Here is the official bug report and fix is
Index: hadoop-common-project/hadoop-auth/pom.xml =================================================================== --- hadoop-common-project/hadoop-auth/pom.xml (revision 1543124) +++ hadoop-common-project/hadoop-auth/pom.xml (working copy) @@ -54,6 +54,11 @@ </dependency> <dependency> <groupId>org.mortbay.jetty</groupId> + <artifactId>jetty-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.mortbay.jetty</groupId> <artifactId>jetty</artifactId> <scope>test</scope> </dependency>
Maven will do all the heavy work for you, and you should get this after build is completed
[INFO] BUILD SUCCESS [INFO] ------------------------------------------------ [INFO] Total time: 15:39.705s [INFO] Finished at: Fri Nov 01 14:36:17 CST 2013 [INFO] Final Memory: 135M/422M
The built native libraries should be at