我正在尝试使用Hive配置水槽,以将水槽输出保存到Hive Sink类型的Hive表中。我有单节点群集。我使用mapr hadoop分发。
这是我的flume.conf
agent1.sources = source1 agent1.channels = channel1 agent1.sinks = sink1 agent1.sources.source1.type = exec agent1.sources.source1.command = cat /home/andrey/flume_test.data agent1.sinks.sink1.type = hive agent1.sinks.sink1.channel = channel1 agent1.sinks.sink1.hive.metastore = thrift://127.0.0.1:9083 agent1.sinks.sink1.hive.database = default agent1.sinks.sink1.hive.table = flume_test agent1.sinks.sink1.useLocalTimeStamp = false agent1.sinks.sink1.round = true agent1.sinks.sink1.roundValue = 10 agent1.sinks.sink1.roundUnit = minute agent1.sinks.sink1.serializer = DELIMITED agent1.sinks.sink1.serializer.delimiter = "," agent1.sinks.sink1.serializer.serdeSeparator = ',' agent1.sinks.sink1.serializer.fieldnames = id,message agent1.channels.channel1.type = FILE agent1.channels.channel1.transactionCapacity = 1000000 agent1.channels.channel1.checkpointInterval 30000 agent1.channels.channel1.maxFileSize = 2146435071 agent1.channels.channel1.capacity 10000000 agent1.sources.source1.channels = channel1
我的数据flume_test.data
1,AAAAAAAA 2,BBBBBBB 3,CCCCCCCC 4,DDDDDD 5,EEEEEEE 6,FFFFFFFFFFF 7,GGGGGG 8,HHHHHHH 9,IIIIII 10,JJJJJJ 11,KKKKKK 12,LLLLLLLL 13,MMMMMMMMM 14,NNNNNNNNN 15,OOOOOOOO 16,PPPPPPPPPP 17,QQQQQQQ 18,RRRRRRR 19,SSSSSSSS
那就是我在Hive中创建表格的方式
create table flume_test(id string, message string) clustered by (message) into 1 buckets STORED AS ORC tblproperties ("orc.compress"="NONE");
当我仅使用1个存储桶时,在蜂巢壳中从flume_test命令中选择*只会返回OK状态,没有数据。如果我使用的存储桶超过1个,则会返回错误消息。
例如,在配置单元表选择后出现5个存储桶时出错:
hive> select * from flume_test; OK 2015-06-18 10:04:57,6909 ERROR Client fs/client/fileclient/cc/client.cc:1385 Thread: 10141 Open failed for file /user/hive/warehouse/flume_test/delta_0004401_0004500/bucket_00, LookupFid error No such file or directory(2) 2015-06-18 10:04:57,6941 ERROR Client fs/client/fileclient/cc/client.cc:1385 Thread: 10141 Open failed for file /user/hive/warehouse/flume_test/delta_0004401_0004500/bucket_00, LookupFid error No such file or directory(2) 2015-06-18 10:04:57,6976 ERROR Client fs/client/fileclient/cc/client.cc:1385 Thread: 10141 Open failed for file /user/hive/warehouse/flume_test/delta_0004401_0004500/bucket_00, LookupFid error No such file or directory(2) 2015-06-18 10:04:57,7044 ERROR Client fs/client/fileclient/cc/client.cc:1385 Thread: 10141 Open failed for file /user/hive/warehouse/flume_test/delta_0004401_0004500/bucket_00, LookupFid error No such file or directory(2) Time taken: 0.914 seconds
配置单元表数据保存在/ user / hive / warehouse / flume_test目录中,并且不为空。
-rwxr-xr-x 3 andrey andrey 4 2015-06-17 16:28 /user/hive/warehouse/flume_test/_orc_acid_version drwxr-xr-x - andrey andrey 2 2015-06-17 16:28 /user/hive/warehouse/flume_test/delta_0004301_0004400
增量目录包含
-rw-r--r-- 3 andrey andrey 991 2015-06-17 16:28 /user/hive/warehouse/flume_test/delta_0004301_0004400/bucket_00000 -rwxr-xr-x 3 andrey andrey 8 2015-06-17 16:28 /user/hive/warehouse/flume_test/delta_0004301_0004400/bucket_00000_flush_length
我什至没有猪也无法读取/ user / hive / warehouse / flume_test / delta_0004301_0004400 / bucket_00000 orc文件。
我也尝试在配置单元中创建表后设置此变量,但这没有得到结果。
set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; set hive.compactor.initiator.on = true; set hive.compactor.worker.threads = 2;
我在互联网上发现了很少的示例,但它们并不完整,而且我不熟悉水槽,所以我无法理解它们)