1. 内存溢出
虚拟内存溢出:
Current usage: 1.1gb of 2.0gb physical memory used; 4.6gb of 4.2gb virtual memory used. Killing container.==【即虚拟内存溢出】==；方法一：提高yarn.nodemanager.vmem-pmem-ratio = 5或者更高；【推荐】
方法二：yarn.nodemanager.vmem-check-enabled =false ；关闭虚拟内存检查；不推荐
方法三：提高物理内存分配，相应的虚拟内存自然就多了，但是这样不是最优
物理内存溢出：
Current usage: 2.1gb of 2.0gb physical memory used; 3.6gb of 4.2gb virtual memory used. Killing container.【即物理内存溢出】；方法一：mapreduce.map.memory.mb = 3GB以上，然后测试这个map/reduce task需要使用多少内存才够用，提高这个值直到不报错为止。
方法二：提高yarn.scheduler.minimum-allocation-mb = 3GB以上，同理【不推荐】
作者：z3935212 来源：CSDN 原文：https://blog.csdn.net/z3935212/article/details/78637157?utm_source=copy版权声明：本文为博主原创文章，转载请附上博文链接！
打开低版本hive报错：
ls: cannot access /app/local/spark-2.0.2-bin-hadoop2.6/lib/spark-assembly-*.jar: No such file or directory修改hive启动文件
vim /app/local/hive/bin/hive
找到下面这一行：
# add Spark assembly jar to the classpath
if [[ -n "$SPARK_HOME" ]]
then
# sparkAssemblyPath=`ls ${SPARK_HOME}/lib/spark-assembly-*.jar`
sparkAssemblyPath=`ls ${SPARK_HOME}/jars/*.jar`
CLASSPATH="${CLASSPATH}:${sparkAssemblyPath}"
fi2. 关联查询
2018-11-25 14:43:04,199 main ERROR Unable to invoke factory method in class class org.apache.hadoop.hive.ql.log element HushableMutableRandomAccess. java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:132)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration
at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:2
at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.ja
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:403)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabili
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2091)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1993)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1852)
at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)
at org.apache.hadoop.hive.ql.exec.mapjoin.MapJoinMemoryExhaustionHandler.<init>(MapJoinMemoryExhaustion
at org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:129)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:358)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:546)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:498)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:368)
at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:514)
at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:418)
at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:393)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:774)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.RandomAccessctory@6dac64ea] unable to create manager for [/var/log/hive/operation_logs/5396b439-4945-483d-b8eb-b5c478e6fbb5ae-9f97-23f95080e4be] with data [org.apache.logging.log4j.core.appender.RandomAccessFileManager$FactoryData@5de
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:114)
at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
at org.apache.logging.log4j.core.appender.RandomAccessFileManager.getFileManager(RandomAccessFileManage
at org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender.createAppender(HushableRandomAccessFi
... 40 more- 异常原因：mr将数据量小的表识别成了大表，数据量大的识别成小表，导致将数据量大的表加入到内存，导致程序异常
-
处理方法： ```
set hive.execution.engine=mr;
set hive.mapjoin.smalltable.filesize=55000000;
set hive.auto.convert.join = false; #取消小表加载至内存中==通常情况下==，设置取消小表加载至内存中即可：`set hive.auto.convert.join = false;`
3. hive on spark问题
Job aborted due to stage failure: Aborting TaskSet 2.0 because task 8 (partition 8) cannot run anywhere due to node and executor blacklist. Blacklisting behavior can be configured via spark.blacklist.*.临时解决办法：
set hive.execution.engine = mr;https://www.iteblog.com/archives/1907.html
4. hive 事务表
执行的操作
delete from hm2.history_helper_back where starttime = '2019-06-12';报错信息
FAILED: SemanticException [Error 10294]: Attempt to do update or delete using transaction manager that does not support these operations.解决办法：
set hive.support.concurrency = true;
set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;5. 修复大量分区
hive> MSCK REPAIR TABLE employee;
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask设置：
set hive.msck.path.validation=ignore;6. hiveserver2 不识别udf函数
在无法使用UDF的 HiveServer2 上，执行 reload function 命令，将MetaStore中新增的UDF信息同步到HiveServer2内存中。
7. 动态分区
Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to 100 partitions per node, number of dynamic partitions on this node: 101
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:941)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:712)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:147)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:487)
... 9 more设置每个节点最大动态分区个数.
set hive.exec.max.dynamic.partitions=3000;
set hive.exec.max.dynamic.partitions.pernode=1000;8. block块丢失
Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-808991319-10.1.0.62-1541662386662:blk_1110742285_40900ile=/user/hive/warehouse/hm4.db/hm4_format_log_his_tmp/dt=2019-09-16/hour=11/product=mini/event=click/part-r-00013_copy_1由于block块受损，无法恢复，只能删除。
文章评论