# spark

**Repository Path**: domky/spark

## Basic Information

- **Project Name**: spark
- **Description**: spark
- **Primary Language**: Java
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2018-05-18
- **Last Updated**: 2021-06-22

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

####  在eclipse下开发spark程序环境配置
https://blog.csdn.net/lihaitao000/article/details/51461277

####  soft 软件支持
https://pan.baidu.com/s/1HCmtxxblEOzTBY8WKzdPDg


 
####  apark centos集群安装  https://blog.csdn.net/weixin_36394852/article/details/76030317

1.  Spark下载 进入到了/usr/local文件夹
wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz

Spark安装之前的准备
tar -zxvf spark-2.2.0-bin-hadoop2.7.tgz
rm -rf spark-2.2.0-bin-hadoop2.7.tgz
mv spark-2.2.0-bin-hadoop2.7 spark-2.2.0

配置环境变量
vi /etc/profile

在最尾巴加入
export SPARK_HOME=/usr/local/spark-2.2.0
export PATH=$PATH:$SPARK_HOME/bin

配置Spark环境
cd spark-2.2.0
spark-env.sh和slaves

首先我们把缓存的文件spark-env.sh.template改为spark识别的文件spark-env.sh
cp conf/spark-env.sh.template conf/spark-env.sh

vi conf/spark-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_141
export SCALA_HOME=/usr/scala-2.11.7
export HADOOP_HOME=/usr/local/hadoop-2.7.2
export HADOOP_CONF_DIR=/usr/local/hadoop-2.7.2/etc/hadoop
export SPARK_MASTER_IP=SparkMaster
export SPARK_WORKER_MEMORY=4g
export SPARK_WORKER_CORES=2
export SPARK_WORKER_INSTANCES=1

------------------------------------------------------------------------------------
java 
	安装 yum -y install java-1.8.0-openjdk*
	java_home 查找
		whereis javac
		ll /usr/bin/javac
		ll /etc/alternatives/javac
		...
		/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-3.b10.el6_9.x86_64
		

scala 
	 wget http://www.scala-lang.org/files/archive/scala-2.11.7.tgz
	 tar xvf scala-2.11.7.tgz
	 # vi /etc/profile
	 
hadoop	 http://www.apache.org/dyn/closer.cgi/hadoop/common
    参考 https://blog.csdn.net/cafebar123/article/details/73500014
	wget http://www-eu.apache.org/dist/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
	tar zxvf hadoop-2.7.2.tar.gz
	配置hadoop
	修改/usr/hadoop/hadoop-2.7.3/etc/hadoop/hadoop-env.sh 文件的java环境，将java安装路径加进去：
		export JAVA_HOME=/alidata/server/java-1.7.0  
	配置hadoop环境变量	vi /etc/profile
	 export HADOOP_HOME=/usr/local/hadoop-2.7.2
	 export PATH=$PATH:$HADOOP_HOME/bin  
	 
添加如下参数：
	export SCALA_HOME=/usr/local/scala-2.11.7
	export PATH=$PATH:$SCALA_HOME/bin
	
	export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk.x86_64
	
	 export HADOOP_HOME=/usr/local/hadoop-2.7.2
	 export PATH=$PATH:$HADOOP_HOME/bin  
	
	source /etc/profile
	scala -version

	

-------------------------------------------------------------------------------------