Hadoop 中的一些参数

Hadoop中的一些参数

文件系统计数(File System Counters)

Name Map Reduce Total 说明
FILE: Number of bytes read 0 276133090136 276133090136 读取的字节数
FILE: Number of bytes written 277135287526 276293112026 553428399552 写出的字节数
FILE: Number of large read operations 0 0 0 大型读取文件操作的数目
FILE: Number of read operations 0 0 0 文件读操作的数目
FILE: Number of write operations 0 0 0 文件写出操作的数目
HDFS: Number of bytes read 1191014895117 0 1191014895117 从HDFS读取的字节数
HDFS: Number of bytes written 0 630528839606 630528839606 写出到HDFS的字节数
HDFS: Number of large read operations 0 0 0 大型HDFS读操作数目
HDFS: Number of read operations 14988 3000 17988 HDFS读操作的数目
HDFS: Number of write operations 0 2000 2000 HDFS写操作的数目
VIEWFS: Number of bytes read 0 0 0 读字节数的大小
VIEWFS: Number of bytes written 0 0 0 写字节数的大小
VIEWFS: Number of large read operations 0 0 0 大型写操作的数目
VIEWFS: Number of read operations 0 0 0 读操作的数目
VIEWFS: Number of write operations 0 0 0 写操作的数目

任务计数(Job Counter)

Name Map Reduce Total 说明
Data-local map tasks 0 0 4710 数据本地化map任务
Killed map tasks 0 0 11 被杀死的map任务数
Launched map tasks 0 0 5007 启动的map任务数
Launched reduce tasks 0 0 1000 启动的reduce任务数
Other local map tasks 0 0 63 其它本地map任务数
Rack-local map tasks 0 0 234 同机架map任务数
Total megabyte-milliseconds taken by all map tasks 0 0 21776498497536 所有map任务花费的全部兆字节每毫秒
Total megabyte-milliseconds taken by all reduce tasks 0 0 5300392375296 所有reduce任务花费的全部兆字节每毫秒
Total time spent by all map tasks (ms) 0 0 14177407876 所有map任务花费的时间
Total time spent by all maps in occupied slots (ms) 0 0 28354815752 所有map在已占用的块中花费的全部时间(毫秒)
Total time spent by all reduce tasks (ms) 0 0 1725388143 所有reduce任务花费的总时间(毫秒)
Total time spent by all reduces in occupied slots (ms) 0 0 5176164429 所有reduce在已占用的块中花费的全部时间(毫秒)
Total vcore-milliseconds taken by all map tasks 0 0 42532223628 所有map任务花费的全部内核毫秒
Total vcore-milliseconds taken by all reduce tasks 0 0 5176164429 所有map任务花费的全部内核毫秒

Map-Reduce 框架

Name Map Reduce Total 说明
Combine input records 0 0 0 合并输入记录
Combine output records 0 0 0 合并输出记录
CPU time spent (ms) 13343798490 52467030 13396265520 CPU花费的时间
Failed Shuffles 0 0 0 失败的Shuffle
GC time elapsed (ms) 31006165 2427237 33433402 垃圾回收花费时间(毫秒)
Input split bytes 684452 0 684452 输入的分割字段
Map input records 15578952 0 15578952 map输入记录
Map output bytes 630558119075 0 630558119075 map输出字节
Map output materialized bytes 276215755508 0 276215755508 map输出的实例化的字节数
Map output records 14070035 0 14070035 map 输出的记录数
Merged Map outputs 0 4996000 4996000 合并的map数
Physical memory (bytes) snapshot 3033438408704 1961341562880 4994779971584 物理内存(字节)快照
Reduce input groups 0 2359683 2359683 reduce输入组
Reduce input records 0 14070035 14070035 reduce输入记录数
Reduce output records 0 14070035 14070035 reduce输出记录数
Reduce shuffle bytes 0 276215755508 276215755508 reduce shuffle的字节数
Shuffled Maps 0 4996000 4996000 shuffled的map数目
Spilled Records 14070035 14070035 28140070 溢出的记录数
Total committed heap usage (bytes) 3640071815168 2401517961216 6041589776384 完整的堆栈使用(字节)
Virtual memory (bytes) snapshot 16742122012672 4734467149824 21476589162496 虚拟内存(字节)快照

文件输入形式计数(File Input Format Counters)

Name Map Reduce Total 说明
Bytes Read 1191014210665 0 1191014210665 读取的字节数

文件输出形式计数(File Input Format Counters)

Name Map Reduce Total 说明
Bytes Written 0 630528839606 630528839606 输出的字节数
打赏

mickey

记录生活,写给几十年后的自己。