Embedded Files
Sources: flume.apache, tutorialspoint, jets3t.s3.amazonaws.com
Create flume agent's configuration(describe source, channel, sink/destination and interceptors(if any) ) file.
Execute flume using configuration file thats all..
Create Interceptors..
Add dependency for flume-core APIs.
Write Implementation for Interceptor..
Get handle on event within intercept(single and list of events both) method, implement required BI on header and data within event and return same...
Build jar
place jar in flume_home/lib ... Thats all, invoke flume-agent and your data will be intercepted before it moved to sink... in case of error or not working as expected, check /var/log/flume-agent.log for any error message
flume-ng agent --conf conf/ --conf-file $FLUME_HOME/agent.conf --name agent -Dflume.root.logger=DEBUG,console > flume.log 2>&1 &
Problem#1: 400 bad request for some bucket(in mumbai region).
Soln: Found dependency for jets3t was used. Removed jets3t and provided hadoop credentials in agent conf only.
Sources
- Avro Source
- Thrift Source
- Exec Source
- JMS Source
- Spooling Directory Source
- Twitter 1% firehose Source
- Kafka Source
- NetCat Source
- Sequence Generator Source
- Syslog Sources
- Syslog TCP Source
- Multiport Syslog TCP Source
- Syslog UDP Source
- HTTP Source
- Stress Source
- Legacy Sources
- Thrift Legacy Source
- Custom Source
- Scribe Source
Channels
- Memory Channel
- JDBC Channel
- Kafka Channel
- File Channel
- Spillable Memory Channel
- Pseudo Transaction Channel
Sinks
- HDFS Sink
- Hive Sink
- Logger Sink
- Avro Sink
- Thrift Sink
- IRC Sink
- File Roll Sink
- Null Sink
- HBaseSink
- AsyncHBaseSink
- MorphlineSolrSink
- ElasticSearchSink
- Kite Dataset Sink
- Kafka Sink
Google Sites
Report abuse