xuybin/hadoop

xuybin/hadoop

通过Docker Compose运行Hadoop集群

国内环境部署

bash
docker rmi -f registry.cn-shenzhen.aliyuncs.com/xuybin/hadoop && curl -L -s https://raw.githubusercontent.com/xuybin/hadoop/master/docker-compose-aliyun.yml >docker-compose.yml && docker-compose up -d
docker exec -it master /bin/bash
    /start-hadoop.sh
    jps
    exit
docker-compose ps

海外环境部署

bash
docker rmi -f xuybin/hadoop && curl -L -s https://raw.githubusercontent.com/xuybin/hadoop/master/docker-compose.yml >docker-compose.yml && docker-compose up -d
docker exec -it master /bin/bash
    /start-hadoop.sh
    jps
    exit
docker-compose ps

可选操作

bash
docker ps
docker network ls
docker network inspect **_hadoop
docker volume ls
docker volume inspect **_hdfs-master[slave1,slave2]

验证Hadoop集群

bash
NameNode:访问 http://ip:8098
ResourceManager:访问 http://ip:8099

测试Hadoop集群

bash
mkdir src && echo "from f1 content" >src/f1.txt && echo "from f2 content" >src/f2.txt
hadoop fs -mkdir -p target && hdfs dfs -put ./src/* target
hdfs dfs -cat target/f1.txt
hdfs dfs -cat target/f2.txt
hadoop jar /hadoop/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.7.5-sources.jar org.apache.hadoop.examples.WordCount target target2
hdfs dfs -cat target2/*

停止Hadoop集群

bash
docker exec -it master /bin/bash
    /stop-hadoop.sh
    jps
    exit
docker-compose stop

清理Hadoop集群

bash
docker-compose rm -f
docker volume ls
docker volume rm **_hdfs-master[slave1,slave2]
docker network ls
docker network rm **_hadoop