[研究] Apache Spark 1.3.1 + Hadoop 2.5.2 安裝

[研究] Apache Spark 1.3.1 + Hadoop 2.5.2 安裝

2015-05-23

官方網站
https://spark.apache.org/

需要套件
CentOS
JDK >=6
Maven >= 3.0.4
Hadoop
Spark
Hive
MySQL-Server

安裝參考
https://spark.apache.org/docs/latest/building-spark.html

安裝

# 為了省事，避免意外的情況，關閉 SELinux (Security Linux ) 和 iptables

# 立刻關閉 SELinuxsetenforce 0 
# 設定 reboot 後自動關閉 SELinux#vi  /etc/selinux/config#找到#SELINUX=#設為#SELINUX=disabled  
sed -i -e "s@SELINUX=enforcing@#SELINUX=enforcing@"   /etc/selinux/config
sed -i -e "s@SELINUX=permissive@#SELINUX=permissive@"   /etc/selinux/config
sed -i -e "/SELINUX=/aSELINUX=disabled"   /etc/selinux/config

# 立刻停掉 iptables#service iptables stop  #service ip6tables stop systemctl   stop  firewalld # 設定 reboot 後自動關閉 iptable
#chkconfig iptables off  #chkconfig ip6tables off  systemctl   disable  firewalld 
cd   /usr/local


# 安裝基本套件
yum   -y   install   java   java-devel   maven   mariadb-server    mysql

# 設定 Java 環境
#find / -name java
echo 'export JAVA_HOME=/usr/lib/jvm/java' >> /etc/profile
echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /etc/profile
echo 'export CLASSPATH=$JAVA_HOME/lib/ext:$JAVA_HOME/lib/tools.jar' >> /etc/profile
source /etc/profile


# 安裝 Apache Hadoop
cd /usr/local
wget  http://ftp.twaren.net/Unix/Web/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz
tar zxvf hadoop-2.5.2.tar.gz
echo 'export HADOOP_HOME=/usr/local/hadoop-2.5.2' >> /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/bin' >> /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/sbin' >> /etc/profile
echo 'export HADOOP_PREFIX=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_COMMON_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_MAPRED_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop' >> /etc/profile
echo 'export HADOOP_HDFS_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_YARN_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export YARN_CONF_DIR=$HADOOP_CONF_DIR' >> /etc/profile
source /etc/profile


# 安裝 Apache Hive，目前最新 1.2.0 版
cd /usr/local
wget   http://apache.stu.edu.tw/hive/stable/apache-hive-1.2.0-bin.tar.gz
tar   xzvf   apache-hive-1.2.0-bin.tar.gz
export   HIVE_HOME=/usr/local/apache-hive-1.2.0-bin
export   PATH=$HIVE_HOME/bin:$PATH

# 安裝 Apache Spark
# 照官方資訊說不保證 Hadoop >= 2.5 可以正常，試試看吧，mvn 安裝在 Core i5 + SSD 上約1小時
# https://spark.apache.org/docs/latest/building-spark.html
# build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

#wget   http://www.apache.org/dyn/closer.cgi/spark/spark-1.3.1/spark-1.3.1.tgz

# Apache 網站這份有問題，別抓

cd   /usr/local
wget   http://d3kbcqa49mib13.cloudfront.net/spark-1.3.1.tgz
tar   zxvf   spark-1.3.1.tgz
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
cd    spark-1.3.1
# 注意 Hadoop 2.5.x 和 2.4.x 都是設定 2.4
#mvn -Pyarn -Dyarn.version=2.4.0 -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -Phive-1.2.0 -Phive-thriftserver -DskipTests clean package
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:04:14.172s
[INFO] Finished at: Sun May 24 08:48:42 CST 2015
[INFO] Final Memory: 90M/607M
[INFO] ------------------------------------------------------------------------
[root@localhost spark-1.3.1]#

(完)

相關文章

[研究] Apache Spark 1.3.1 + Hadoop 2.5.2 安裝
http://shaurong.blogspot.com/2015/05/apache-spark-131-hadoop-252.html

[研究] Apache Maven 3.0.5 (yum) 安裝 (CentOS 7.1 x64_86)
http://shaurong.blogspot.com/2015/05/apache-maven-305-yum-centos-71-x6486.html

[研究] Hadoop 2.7.0 Single Cluster 安裝 (CentOS 7.1 x86_64)
http://shaurong.blogspot.com/2015/05/hadoop-270-single-cluster-centos-71.html

Spark-1.3.1與Hive整合實現查詢分析
http://wrox.cn/article/1036050/