INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : number of splits:607
INFO : Hadoop job information for Stage-1: number of mappers: 607; number of reducers: 593
INFO : 2021-08-03 12:04:24,278 Stage-1 map = 0%, reduce = 0%
INFO : 2021-08-03 12:05:25,536 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 38938.13 sec
INFO : 2021-08-03 12:06:26,659 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 91842.49 sec
INFO : 2021-08-03 12:07:27,953 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 135616.05 sec
INFO : 2021-08-03 12:08:29,481 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 178505.06 sec
INFO : 2021-08-03 12:09:31,198 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 220411.34 sec
INFO : 2021-08-03 12:10:32,623 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 263052.5 sec
INFO : 2021-08-03 12:11:35,223 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 306463.06 sec
INFO : 2021-08-03 12:11:57,781 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 322017.91 sec
INFO : 2021-08-03 12:12:58,254 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 364235.06 sec
INFO : 2021-08-03 14:25:48,364 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3446753.99 sec
INFO : 2021-08-03 14:26:49,869 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3446859.0 sec
INFO : 2021-08-03 14:27:50,992 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3446943.72 sec
INFO : 2021-08-03 14:28:51,325 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3446997.95 sec
INFO : 2021-08-03 14:29:52,686 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447029.17 sec
INFO : 2021-08-03 14:30:54,475 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447066.48 sec
INFO : 2021-08-03 14:31:56,320 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447105.05 sec
INFO : 2021-08-03 14:32:56,556 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447143.97 sec
INFO : 2021-08-03 14:33:57,495 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447179.11 sec
INFO : 2021-08-03 14:34:57,606 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447213.42 sec
INFO : 2021-08-03 14:36:00,212 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447247.73 sec
INFO : 2021-08-03 14:37:00,627 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447286.21 sec
INFO : 2021-08-03 14:38:09,235 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447326.07 sec
NFO : 2021-08-03 15:26:11,976 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449393.87 sec
INFO : 2021-08-03 15:27:12,221 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449450.86 sec
INFO : 2021-08-03 15:28:15,888 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449512.13 sec
INFO : 2021-08-03 15:29:16,266 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449569.43 sec
INFO : 2021-08-03 15:30:18,326 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449625.67 sec
INFO : 2021-08-03 15:31:18,663 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449676.89 sec
INFO : 2021-08-03 15:32:03,766 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3449720.97 sec
INFO : 2021-08-03 15:32:59,189 Stage-1 map = 100%, reduce = 1%, Cumulative CPU 3464991.99 sec
INFO : 2021-08-03 15:33:44,258 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 3508189.64 sec
INFO : 2021-08-03 15:34:19,376 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 3552742.33 sec
INFO : 2021-08-03 15:34:47,839 Stage-1 map = 100%, reduce = 4%, Cumulative CPU 3590404.09 sec
INFO : 2021-08-03 15:35:15,605 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 3625840.27 sec
INFO : 2021-08-03 15:35:42,461 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 3659643.72 sec
INFO : 2021-08-03 15:36:08,716 Stage-1 map = 100%, reduce = 7%, Cumulative CPU 3694480.32 sec
INFO : 2021-08-03 15:36:33,513 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 3729367.61 sec
INFO : 2021-08-03 15:36:57,172 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 3762231.89 sec
INFO : 2021-08-03 15:37:22,482 Stage-1 map = 100%, reduce = 10%, Cumulative CPU 3798384.34 sec
INFO : 2021-08-03 15:37:46,679 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 3832812.14 sec
INFO : 2021-08-03 15:38:12,452 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 3868927.26 sec
INFO : 2021-08-03 15:38:38,427 Stage-1 map = 100%, reduce = 13%, Cumulative CPU 3905131.82 sec
INFO : 2021-08-03 15:39:06,029 Stage-1 map = 100%, reduce = 14%, Cumulative CPU 3942664.15 sec
INFO : 2021-08-03 15:39:29,646 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 3976567.08 sec
INFO : 2021-08-03 15:39:53,959 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 4012733.63 sec
INFO : 2021-08-03 15:40:19,967 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 4049187.75 sec
INFO : 2021-08-03 15:40:46,111 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 4085618.83 sec
INFO : 2021-08-03 15:41:10,351 Stage-1 map = 100%, reduce = 19%, Cumulative CPU 4119786.96 sec
INFO : 2021-08-03 15:41:37,116 Stage-1 map = 100%, reduce = 20%, Cumulative CPU 4156287.88 sec
INFO : 2021-08-03 15:41:59,920 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 4187484.35 sec
INFO : 2021-08-03 15:42:26,286 Stage-1 map = 100%, reduce = 22%, Cumulative CPU 4223531.06 secINFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : number of splits:607
INFO : Hadoop job information for Stage-1: number of mappers: 607; number of reducers: 593
INFO : 2021-08-03 12:04:24,278 Stage-1 map = 0%, reduce = 0%
INFO : 2021-08-03 12:05:25,536 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 38938.13 sec
INFO : 2021-08-03 12:06:26,659 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 91842.49 sec
INFO : 2021-08-03 12:07:27,953 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 135616.05 sec
INFO : 2021-08-03 12:08:29,481 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 178505.06 sec
INFO : 2021-08-03 12:09:31,198 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 220411.34 sec
INFO : 2021-08-03 12:10:32,623 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 263052.5 sec
INFO : 2021-08-03 12:11:35,223 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 306463.06 sec
INFO : 2021-08-03 12:11:57,781 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 322017.91 sec
INFO : 2021-08-03 12:12:58,254 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 364235.06 sec
INFO : 2021-08-03 14:25:48,364 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3446753.99 sec
INFO : 2021-08-03 14:26:49,869 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3446859.0 sec
INFO : 2021-08-03 14:27:50,992 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3446943.72 sec
INFO : 2021-08-03 14:28:51,325 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3446997.95 sec
INFO : 2021-08-03 14:29:52,686 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447029.17 sec
INFO : 2021-08-03 14:30:54,475 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447066.48 sec
INFO : 2021-08-03 14:31:56,320 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447105.05 sec
INFO : 2021-08-03 14:32:56,556 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447143.97 sec
INFO : 2021-08-03 14:33:57,495 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447179.11 sec
INFO : 2021-08-03 14:34:57,606 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447213.42 sec
INFO : 2021-08-03 14:36:00,212 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447247.73 sec
INFO : 2021-08-03 14:37:00,627 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447286.21 sec
INFO : 2021-08-03 14:38:09,235 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3447326.07 sec
NFO : 2021-08-03 15:26:11,976 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449393.87 sec
INFO : 2021-08-03 15:27:12,221 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449450.86 sec
INFO : 2021-08-03 15:28:15,888 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449512.13 sec
INFO : 2021-08-03 15:29:16,266 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449569.43 sec
INFO : 2021-08-03 15:30:18,326 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449625.67 sec
INFO : 2021-08-03 15:31:18,663 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 3449676.89 sec
INFO : 2021-08-03 15:32:03,766 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3449720.97 sec
INFO : 2021-08-03 15:32:59,189 Stage-1 map = 100%, reduce = 1%, Cumulative CPU 3464991.99 sec
INFO : 2021-08-03 15:33:44,258 Stage-1 map = 100%, reduce = 2%, Cumulative CPU 3508189.64 sec
INFO : 2021-08-03 15:34:19,376 Stage-1 map = 100%, reduce = 3%, Cumulative CPU 3552742.33 sec
INFO : 2021-08-03 15:34:47,839 Stage-1 map = 100%, reduce = 4%, Cumulative CPU 3590404.09 sec
INFO : 2021-08-03 15:35:15,605 Stage-1 map = 100%, reduce = 5%, Cumulative CPU 3625840.27 sec
INFO : 2021-08-03 15:35:42,461 Stage-1 map = 100%, reduce = 6%, Cumulative CPU 3659643.72 sec
INFO : 2021-08-03 15:36:08,716 Stage-1 map = 100%, reduce = 7%, Cumulative CPU 3694480.32 sec
INFO : 2021-08-03 15:36:33,513 Stage-1 map = 100%, reduce = 8%, Cumulative CPU 3729367.61 sec
INFO : 2021-08-03 15:36:57,172 Stage-1 map = 100%, reduce = 9%, Cumulative CPU 3762231.89 sec
INFO : 2021-08-03 15:37:22,482 Stage-1 map = 100%, reduce = 10%, Cumulative CPU 3798384.34 sec
INFO : 2021-08-03 15:37:46,679 Stage-1 map = 100%, reduce = 11%, Cumulative CPU 3832812.14 sec
INFO : 2021-08-03 15:38:12,452 Stage-1 map = 100%, reduce = 12%, Cumulative CPU 3868927.26 sec
INFO : 2021-08-03 15:38:38,427 Stage-1 map = 100%, reduce = 13%, Cumulative CPU 3905131.82 sec
INFO : 2021-08-03 15:39:06,029 Stage-1 map = 100%, reduce = 14%, Cumulative CPU 3942664.15 sec
INFO : 2021-08-03 15:39:29,646 Stage-1 map = 100%, reduce = 15%, Cumulative CPU 3976567.08 sec
INFO : 2021-08-03 15:39:53,959 Stage-1 map = 100%, reduce = 16%, Cumulative CPU 4012733.63 sec
INFO : 2021-08-03 15:40:19,967 Stage-1 map = 100%, reduce = 17%, Cumulative CPU 4049187.75 sec
INFO : 2021-08-03 15:40:46,111 Stage-1 map = 100%, reduce = 18%, Cumulative CPU 4085618.83 sec
INFO : 2021-08-03 15:41:10,351 Stage-1 map = 100%, reduce = 19%, Cumulative CPU 4119786.96 sec
INFO : 2021-08-03 15:41:37,116 Stage-1 map = 100%, reduce = 20%, Cumulative CPU 4156287.88 sec
INFO : 2021-08-03 15:41:59,920 Stage-1 map = 100%, reduce = 21%, Cumulative CPU 4187484.35 sec
INFO : 2021-08-03 15:42:26,286 Stage-1 map = 100%, reduce = 22%, Cumulative CPU 4223531.06 sec
上面是他的部分执行日志
执行的很慢 不知道原因出在哪
hsql 是个 insert overwrite 语句 没有join 不过字段很多 400+
字段多数据量大，跑mapreduce。
内存少，看hdfs上的/tmp目录下中间数据有多少。
另外map与reduce并行度适当调整下，你这配的是100%map结束再开始reduce，官方默认值都调成15%了应该。
覆盖数据直接删除hdfs下表目录下文件，load进去，或者直接上传进去啊