How to build Hadoop Native Libraries for Hadoop 2.2.0
Because the distributed Hadoop 2.2.0 provides a 32bit libhadoop by default, user has to build the native libraries to avoid those warning messages such as, disabled stack guard of libhadoop.so.
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /opt/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 13/11/01 10:58:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
The official Hadoop websitehttp://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/NativeLibraries.html gives completely unclear instructions on how to build Hadoop native libraries.
So here are what you should do:
You need all the build tools:
$ sudo apt-get install build-essential $ sudo apt-get install g++ autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev $ sudo apt-get install maven
Another prerequisite, protoco buffer: protobuf version 2.5, which can be downloaded from https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz. Download it to the /tmp directory; then,
$ tar xzvf protobuf-2.5.0.tar.gz $ cd protobuf-2.5.0 $ ./configure --prefix=/usr $ make $ make check $ sudo make install
Having all the tools, we can now build Hadoop native libraries. Assuming you have downloaded the Hadoop 2.2.0 source code, do:
$ tar xzvf hadoop-2.2.0-src.tar.gz $ cd hadoop-2.2.0-src $ mvn package -Pdist,native -DskipTests -Dtar
Note: there is a missing dependency in the maven project module that results in a build failure at the hadoop-auth stage. Here is the official bug report and fix is
Index: hadoop-common-project/hadoop-auth/pom.xml =================================================================== --- hadoop-common-project/hadoop-auth/pom.xml (revision 1543124) +++ hadoop-common-project/hadoop-auth/pom.xml (working copy) @@ -54,6 +54,11 @@ </dependency> <dependency> <groupId>org.mortbay.jetty</groupId> + <artifactId>jetty-util</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>org.mortbay.jetty</groupId> <artifactId>jetty</artifactId> <scope>test</scope> </dependency>
Maven will do all the heavy work for you, and you should get this after build is completed
[INFO] BUILD SUCCESS [INFO] ------------------------------------------------ [INFO] Total time: 15:39.705s [INFO] Finished at: Fri Nov 01 14:36:17 CST 2013 [INFO] Final Memory: 135M/422M
The built native libraries should be at
8 thoughts on “Build Hadoop Native Libraries”
Is it theoretically possible to compile native hadoop lib on ARM platform (Lubuntu)?
I know this post is old but it’s still high up on the google search so, I shall respond.
Yes, I’m working on it right now. I’m doing it on an Odroid C1 running arch arm.
You should probably stick to an Ubuntu based distro, at least for the build. Right now I’m stuck in dependency hell.
But, it is possible.
Thanks for your inputs it really did saved a lot of time for me. I am just a beginner and your post has helped me get through it. Thanks a ton!!
Thanks a lot, it’s very helpfull although firstly I think it’s not gonna work firstly because it take a looong enough time to do mvn job and success, FYI I use it on hadoop 2.4 and it’s still working 🙂
I am trying to install 2.4 at Linux Server release 6.4 and getting this warning.
Can you please let me know what i need to do.
Perfect!!! It worked.
I did for CDH5 in Ubuntu
Comments are closed.