Resource Scheduler , Calculator, Short-Circuit in Hadoop YARN and HDFS

In order to execute the next-year plan, I search the research topics and technologies in Hadoop YARN and HDFS, then make a note as follows:

Since Hadoop YARN was proposed, the new generation technology are continusly discussed. For knowing the work of YARN, please refer to the post [1].
The capacity scheduler of YARN[2][3] provides a default capacity scheduler, org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler [4], to let hadoop eco-system manipulates its’ resources. It also provides resource calculator, org.apache.hadoop.yarn.util.resource.DefaultResourseCalculator [5], to calculate the memory usage, the disk usage and the cpu usage of each compute node.
For those system administrators and developers who are curious about the operations of resource allocation and the scheduler, please see the reports [6], [7] and [8].
In scientific area, Project HaSTE [9] proposed a new Hadoop YARN scheduling algorithm, which aims at efficiently utilizing the resources for scheduling map/reduce tasks in Hadoop YARN and improving the makespan of MapReduce jobs.

On the other hand, HDFS [10] is a usually used file system in Hadoop. However, it needs TCP socket connection to read/write data. Due to this reason, the IO performance will be lower than directly reading from local disk without network connection. Therefore, HDFS provides a function called HDFS Short-Circuit Local Reads [11] and also provides a native libaray to directly access the HDFS file system. According to the report [12], the I/O performance of using Short-Circuit is better than TCP.
ps. The other tricky technology to improve the I/O performance of HDFS is to use CombineFileInputFormat [13][14]. But I don’t think this method is better than using Short-Circuit.

Reference

1. Karthik Kambatla, Wing Yew Poon, and Vikram Srivastava, “How Apache Hadoop YARN HA Works,” Cludera. Available: [Online] http://blog.cloudera.com/blog/2014/05/how-apache-hadoop-yarn-ha-works/
2. Hadoop, “Haddop MapReduce Next Generation – Capacity Scheduler”, Apache Hadoop. Available: [Online] http://hadoop.apache.org/docs/r2.5.2/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Configuration
3. skyWalker_ONLY, “Hadoop-2.4.1学习之容量调度器”. Available: [Online] http://blog.csdn.net/skywalker_only/article/details/41351147
4. GrepCode, “CapacityScheduler”, GrepCode.com. Available: [Online] http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-yarn-server-resourcemanager/2.6.0/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java/
5. GrepCode, “ResourceCalculator”, GrepCode.com. Available: [Online] http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-yarn-common/2.6.0/org/apache/hadoop/yarn/util/resource/DefaultResourceCalculator.java/
6. Vinod Kumar Vavilapalli, “Resource Location in YARN: Deep Dive,” Hortonworks. Available: [Online] http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
7. SEQUENCEIQ, “YARN Schedulers demystified – Part 1: Capacity.” Available: [Online] http://blog.sequenceiq.com/blog/2014/07/22/schedulers-part-1/
8. SEQUENCEIQ, “YARN Schedulers demystified – Part 2: Fair.” Available: [Online] http://blog.sequenceiq.com/blog/2014/09/09/yarn-schedulers-demystified-part-2-fair/
9. Bo Sheng, “Project HaSTE: Hadoop YARN Scheduling Based on Task-Dependency and Resource-Demand,” The 7th IEEE International Conference on Cloud Computing, Anchorage, AK, June 2014.  Available: [Online] http://www.cs.umb.edu/~shengbo/research/haste.html
10. Hadoop, “HDFS User Guide,” Apache Hadoop. Available: [Online] http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Related_Documentation
11. Hadoop, “HDFS Short-Circuit Local Reads,” Apache Hadoop. Available: [Online] https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html
12. Colin McCabe, “How Improved Short-Circuit Local Reads Bring Better Performance and Security to Hadoop,” Cloudera. Available: [Online] http://blog.cloudera.com/blog/2013/08/how-improved-short-circuit-local-reads-bring-better-performance-and-security-to-hadoop/
13. Dhruba Borthakur, “HDFS block replica placement in your hands now!” Available: [Online] http://hadoopblog.blogspot.de/2009/09/hdfs-block-replica-placement-in-your.html
14. Hadoop, “Class CombineFileInputFormat<K,V>”. Available: [Online] http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/lib/CombineFileInputFormat.html

Process SequenceFile without Enabling Hadoop Platform

Recently I got a requirement for reading Hadoop’s SequenceFile without enabling Hadoop Platform. However, most examples introduce the read/write SequenceFile with Hadoop Platform. How do I read such files without hadoop?
There’s a tricky solution in this case.
1. Download Hadoop binary file from hadoop site. For Linux/Unix please directly download it; for Windows, there’s pre-built archive file – hadoop-common-2.2.0-bin (source code is here) , created by Abhijit Ghosh.
2. Set environment variable HADOOP_HOME by the directory path (suppose the directory is /usr/local/hadoop in Unix ; or C:/hadoop-common-2.2.0-bin in Windows)
3. Append $HADOOP_HOME/bin to the end of environment variable PATH. ( i.e. /usr/local/hadoop/bin in Unix ; or C:/hadoop-common-2.2.0-bin/bin in Windows
4. Write your program like this (Notice that you have to download hadoop-common-2.2+):

import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URLEncoder;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.util.ReflectionUtils;

public class ProcessSequenceFile {
	
	public static void readSequenceFile(String sequenceFileName) throws IOException, URISyntaxException {
		Configuration conf = new Configuration();
		String directoryPath = "file:///";
		directoryPath = URLEncoder.encode(directoryPath, "UTF-8");
		FileSystem fs = FileSystem.get(new URI(directoryPath),conf);
		Path file = new Path(fs.getUri().toString() +  sequenceFileName);
		@SuppressWarnings("deprecation")
		SequenceFile.Reader reader = new SequenceFile.Reader(fs, file, conf);
		Text key = (Text) ReflectionUtils.newInstance(reader.getKeyClass(), conf);
		Text value = (Text) ReflectionUtils.newInstance(reader.getValueClass(), conf);
		while(reader.next(key,value)) {
			System.out.println("Key:" + key);
			System.out.println("=================");			
			System.out.println(value);
			
		}
		
	}
	
	public static void main(String[] args) {
		try {
			// If args[0] is the SequenceFile we need to read.
			readSequenceFile(args[0]);
		} catch (IOException | URISyntaxException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}
}

Enjoy it!

Reference:
1. Gnosis Runmination, “WIN7下运行hadoop程序报：Failed to locate the winutils binary in the hadoop binary path.” Available: [Online] http://www.cnblogs.com/zq-inlook/p/4386216.html
2. StackOverFlow, “Running Apache Hadoop 2.1.0 on Windows”. Available: [Online] http://stackoverflow.com/questions/18630019/running-apache-hadoop-2-1-0-on-windows
3.  Abhijit Ghosh, "ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path," SrcCodes.com. Available: [Online] http://www.srccodes.com/p/article/39/error-util-shell-failed-locate-winutils-binary-hadoop-binary-path
4. Hadoop, “Native Libraries Guide.” Available: [Online] https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/NativeLibraries.html#Native_Hadoop_Library

XPATH, MapReduce and SAX

1. SAX Create XPath
2. XPath4Sax
3. SPEX: XPath Evaluation against XML Streams
4. Apache MRQL: MRQL is a query processing and optimization system for large-scale, distributed data analysis, built on top of Apache Hadoop, Hama, and Spark.

收穫滿滿的Hadoop Taiwan 2013

此次參加2013 Hadoop Taiwan Conference，收穫很多。(以下是手動隨便寫寫，請勿拘泥writing format)

業界方面的進展比學界又更加跨出一大步，也代表著我們之後如果要發表雲端相關運算的論文或是發展技術，

要特別小心注意這類工具。

由於Big Data時代的來臨，現在的雲端運算處理偏重於「即時」運算，而非「批次」運算。

我們目前所學的hadoop map/reduce只能算是非常基本而已。

對於即時運算的需求恐怕還不太夠(Hive/Pig 也不例外)。

Google先看到這個嚴重情形，繼2009年以來，陸續發表Google Caffeine (for indexing),

可繪製大量網路資訊彼此對應關係的圖表資料庫「Pregel」，

2010年7月發表Google Dremel (for real-time analysis)，號稱可完全打敗Hadoop在即時運算處理上的不足。

Google在報告中明確指出，「過去MapReduce需要分多次查詢的資料，Dremel可同時處理，並大幅縮短運算時間」，

因此是為了real-time query而設計的。

此次參加Hadoop Taiwan，聽人家介紹才知道原來有這個強力的project可用。因此，Apache也仿照這個概念，

提出Drill platform.

為了real-time處理夠快，也會導入Message Queue System，例如：

Apache Kafaka: The message queue system for increasing the I/O performance but not provide transmission assurance.

Storm: The real-time message queuing system; but it consume too much more CPU time. (From the HBase initiator said…)

 

而Samza則是結合了Hadoop, Kafaka等技術整合而成的distributed stream processing framework (Hbase起草作者推薦的)。

Data Indexing: NGData project use much more hbase indexing to handle Big data problem.

Log collection for data auditing: Splunk is the recommended free project by Trend Micro.

In memory computing techniques: 柏克萊大學開發Spark/Shark Open Source Project來進行Machine Learning工作，由於所有計算工作都在Memory，號稱比Hadoop快20倍。

Impala：由Cloudra公司發表的Open source軟體，也是用Apache hadoop解即時處理的問題。

Graph Mining Tool : Pegasus

這套是因為資策會資安組跟卡內基美濃大學合作，CMU後來開發這個解決他們在graph-mining問題，

因為他們主要處理節點攻擊與病毒擴散模擬運算。他們資安組真的很強，mining工具用得很熟，

而不是只有寫寫數學式子發論文而已，是真的有拿這些Mining工具去做分群、分類、社群網路分析等。

此外，資安組也提到由於攻擊手法可能會長達每月或是每年才發動一次，

所以Rule based (主要偵測以分或者小時為單位的發動模式)可能無法運作。

註：

1. Google研究團隊及其合作學校(卡內基美濃大學、MIT等每年都到處發表論文，他們發表的論文都公告在 http://research.google.com/index.html

2. 這次來做presentation的廠商:Etu (精誠集團子公司)，有真正專門處理Big Data問題的工程師到場，他們上述工具都非常嫻熟，

也提到每一版impala的技術細節，非常厲害。也有發表推薦系統產品，我有詢問他們一些推薦系統實做細節，
不過他們因商業機密，不告訴我就是了…。他們也有現場徵才，不過限制是擁有Hadoop/Linux等相關程式開發經驗2年以上。

Differences in each hadoop version

Hadoop 2.x is developed from hadoop-1.x, with the significant features over hadoop-1.x:

• HDFS HA for NameNode (manual failover)
• NextGen MapReduce (YARN)
• HDFS Federation
• Performance
• Wire-compatibility for both HDFS and YARN/MapReduce (using protobufs)

Hadoop 0.23.x is a trunk which contains:

• HDFS Federation
• NextGen MapReduce (YARN)

Hadoop 1.x is developed from hadoop 0.20-security code support for:

• security
• HBase (append/hsynch/hflush, and security)
• webhdfs (with full support for security)
• performance enhanced access to local files for HBase
• other performance enhancements, bug fixes, and features

[Hadoop] Hadoop 安裝與國網中心Hadoop實作

在中部某科大上課，教到Hadoop，就把一些教材更正的釋出。
Hadoop單機安裝
這份跟國網中心提供的單機安裝教學有些不同，差異性在於Hadoop 0.22.x啟動方法跟如何安裝JDK 1.7。
請點此觀看

在國家高速網路中心Hadoop下實作
教到如何在國家高速網路中心上使用Hadoop叢集，順便把這一份教材釋出。

請點此觀看