使用 eclipse 开发 hadoop1.2.1 时， io.sort.mp 怎么更改？

2017-04-07 12:53:01 +08:00
 Eremita

问题如题，我想增加每个任务的内存限制，但是不知道怎么正确配置，希望有人可以帮助我。

hadoop/conf

core_site.xml 中，我设置了

	<property>
		<name>io.sort.mb</name>
		<value>3000</value>
	</property>

	<property>
		<name>io.sort.factor</name>
		<value>300</value>
	</property>

eclipse

在 Edit Hadoop location 的 Advanced parameters 中，我也配置 io.sort.factor 和 io.sort.mb ，但是运行时还是默认的 100

运行结果


17/04/07 12:43:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/07 12:43:22 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
17/04/07 12:43:22 INFO input.FileInputFormat: Total input paths to process : 1
17/04/07 12:43:23 INFO mapred.JobClient: Running job: job_local805018878_0001
17/04/07 12:43:23 INFO mapred.LocalJobRunner: Waiting for map tasks
17/04/07 12:43:23 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000000_0
17/04/07 12:43:23 INFO util.ProcessTree: setsid exited with exit code 0
17/04/07 12:43:23 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@58948e69
17/04/07 12:43:23 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:0+67108864
17/04/07 12:43:23 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:23 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:23 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:24 INFO mapred.JobClient:  map 0% reduce 0%
17/04/07 12:43:25 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:25 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:25 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:25 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:27 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:27 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:27 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:27 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:28 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:28 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:28 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:28 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:28 INFO mapred.Task: Task:attempt_local805018878_0001_m_000000_0 is done. And is in the process of commiting
17/04/07 12:43:28 INFO mapred.LocalJobRunner: 
17/04/07 12:43:28 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000000_0' done.
17/04/07 12:43:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000000_0
17/04/07 12:43:28 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000001_0
17/04/07 12:43:28 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@742efee7
17/04/07 12:43:28 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:67108864+67108864
17/04/07 12:43:28 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:28 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:28 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:29 INFO mapred.JobClient:  map 20% reduce 0%
17/04/07 12:43:29 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:29 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:29 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:29 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:31 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:31 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:31 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:31 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:31 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:31 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:31 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:31 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:31 INFO mapred.Task: Task:attempt_local805018878_0001_m_000001_0 is done. And is in the process of commiting
17/04/07 12:43:31 INFO mapred.LocalJobRunner: 
17/04/07 12:43:31 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000001_0' done.
17/04/07 12:43:31 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000001_0
17/04/07 12:43:31 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000002_0
17/04/07 12:43:31 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@42c39ed
17/04/07 12:43:31 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:134217728+67108864
17/04/07 12:43:31 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:31 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:31 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:32 INFO mapred.JobClient:  map 40% reduce 0%
17/04/07 12:43:33 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:33 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:33 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:33 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:34 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:34 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:34 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:34 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:34 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:34 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:34 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:34 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:34 INFO mapred.Task: Task:attempt_local805018878_0001_m_000002_0 is done. And is in the process of commiting
17/04/07 12:43:34 INFO mapred.LocalJobRunner: 
17/04/07 12:43:34 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000002_0' done.
17/04/07 12:43:34 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000002_0
17/04/07 12:43:34 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000003_0
17/04/07 12:43:34 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@16b56343
17/04/07 12:43:34 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:201326592+67108864
17/04/07 12:43:34 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:34 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:34 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:35 INFO mapred.JobClient:  map 60% reduce 0%
17/04/07 12:43:36 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:36 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:36 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:36 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:37 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:37 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:37 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:37 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:37 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:37 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:37 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:37 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:37 INFO mapred.Task: Task:attempt_local805018878_0001_m_000003_0 is done. And is in the process of commiting
17/04/07 12:43:37 INFO mapred.LocalJobRunner: 
17/04/07 12:43:37 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000003_0' done.
17/04/07 12:43:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000003_0
17/04/07 12:43:37 INFO mapred.LocalJobRunner: Starting task: attempt_local805018878_0001_m_000004_0
17/04/07 12:43:37 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4b9c21a4
17/04/07 12:43:37 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/ubuntu/input/FiveWords2EightWords.data:268435456+67108864
17/04/07 12:43:37 INFO mapred.MapTask: io.sort.mb = 100
17/04/07 12:43:37 INFO mapred.MapTask: data buffer = 79691776/99614720
17/04/07 12:43:37 INFO mapred.MapTask: record buffer = 262144/327680
17/04/07 12:43:38 INFO mapred.JobClient:  map 80% reduce 0%
17/04/07 12:43:38 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:38 INFO mapred.MapTask: bufstart = 0; bufend = 4194304; bufvoid = 99614720
17/04/07 12:43:38 INFO mapred.MapTask: kvstart = 0; kvend = 262144; length = 327680
17/04/07 12:43:38 INFO mapred.MapTask: Finished spill 0
17/04/07 12:43:40 INFO mapred.MapTask: Spilling map output: record full = true
17/04/07 12:43:40 INFO mapred.MapTask: bufstart = 4194304; bufend = 8388592; bufvoid = 99614720
17/04/07 12:43:40 INFO mapred.MapTask: kvstart = 262144; kvend = 196607; length = 327680
17/04/07 12:43:40 INFO mapred.MapTask: Starting flush of map output
17/04/07 12:43:40 INFO mapred.MapTask: Finished spill 1
17/04/07 12:43:40 INFO mapred.MapTask: Finished spill 2
17/04/07 12:43:40 INFO mapred.Merger: Merging 3 sorted segments
17/04/07 12:43:40 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 9437190 bytes
17/04/07 12:43:40 INFO mapred.Task: Task:attempt_local805018878_0001_m_000004_0 is done. And is in the process of commiting
17/04/07 12:43:40 INFO mapred.LocalJobRunner: 
17/04/07 12:43:40 INFO mapred.Task: Task 'attempt_local805018878_0001_m_000004_0' done.
17/04/07 12:43:40 INFO mapred.LocalJobRunner: Finishing task: attempt_local805018878_0001_m_000004_0
17/04/07 12:43:40 INFO mapred.LocalJobRunner: Map task executor complete.
17/04/07 12:43:40 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@24feb67c
17/04/07 12:43:40 INFO mapred.LocalJobRunner: 
17/04/07 12:43:40 INFO mapred.Merger: Merging 5 sorted segments
17/04/07 12:43:40 INFO mapred.Merger: Down to the last merge-pass, with 5 segments left of total size: 47185930 bytes
17/04/07 12:43:40 INFO mapred.LocalJobRunner: 
17/04/07 12:43:41 INFO mapred.JobClient:  map 100% reduce 0%
17/04/07 12:43:43 INFO mapred.Task: Task:attempt_local805018878_0001_r_000000_0 is done. And is in the process of commiting
17/04/07 12:43:43 INFO mapred.LocalJobRunner: 
17/04/07 12:43:43 INFO mapred.Task: Task attempt_local805018878_0001_r_000000_0 is allowed to commit now
17/04/07 12:43:43 INFO output.FileOutputCommitter: Saved output of task 'attempt_local805018878_0001_r_000000_0' to hdfs://localhost:9000/user/ubuntu/output
17/04/07 12:43:43 INFO mapred.LocalJobRunner: reduce > reduce
17/04/07 12:43:43 INFO mapred.Task: Task 'attempt_local805018878_0001_r_000000_0' done.
17/04/07 12:43:44 INFO mapred.JobClient:  map 100% reduce 100%
17/04/07 12:43:44 INFO mapred.JobClient: Job complete: job_local805018878_0001
17/04/07 12:43:44 INFO mapred.JobClient: Counters: 22
17/04/07 12:43:44 INFO mapred.JobClient:   Map-Reduce Framework
17/04/07 12:43:44 INFO mapred.JobClient:     Spilled Records=7864320
17/04/07 12:43:44 INFO mapred.JobClient:     Map output materialized bytes=47185950
17/04/07 12:43:44 INFO mapred.JobClient:     Reduce input records=2621440
17/04/07 12:43:44 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
17/04/07 12:43:44 INFO mapred.JobClient:     Map input records=2621440
17/04/07 12:43:44 INFO mapred.JobClient:     SPLIT_RAW_BYTES=650
17/04/07 12:43:44 INFO mapred.JobClient:     Map output bytes=41943040
17/04/07 12:43:44 INFO mapred.JobClient:     Reduce shuffle bytes=0
17/04/07 12:43:44 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
17/04/07 12:43:44 INFO mapred.JobClient:     Reduce input groups=2621440
17/04/07 12:43:44 INFO mapred.JobClient:     Combine output records=0
17/04/07 12:43:44 INFO mapred.JobClient:     Reduce output records=2621440
17/04/07 12:43:44 INFO mapred.JobClient:     Map output records=2621440
17/04/07 12:43:44 INFO mapred.JobClient:     Combine input records=0
17/04/07 12:43:44 INFO mapred.JobClient:     CPU time spent (ms)=0
17/04/07 12:43:44 INFO mapred.JobClient:     Total committed heap usage (bytes)=24697110528
17/04/07 12:43:44 INFO mapred.JobClient:   File Input Format Counters 
17/04/07 12:43:44 INFO mapred.JobClient:     Bytes Read=335544320
17/04/07 12:43:44 INFO mapred.JobClient:   FileSystemCounters
17/04/07 12:43:44 INFO mapred.JobClient:     HDFS_BYTES_READ=1342177280
17/04/07 12:43:44 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=377897656
17/04/07 12:43:44 INFO mapred.JobClient:     FILE_BYTES_READ=235943170
17/04/07 12:43:44 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=33910562
17/04/07 12:43:44 INFO mapred.JobClient:   File Output Format Counters 
17/04/07 12:43:44 INFO mapred.JobClient:     Bytes Written=33910562
1364 次点击
所在节点    问与答
0 条回复

这是一个专为移动设备优化的页面（即为了让你能够在 Google 搜索结果里秒开这个页面），如果你希望参与 V2EX 社区的讨论，你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/353182

V2EX 是创意工作者们的社区，是一个分享自己正在做的有趣事物、交流想法，可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX