madvertise stable master kafka

This repo is always the latest stable snapshot of kafka with the following delta

• hadoop-consumer modified to work with druid.

hadoop-consumer

We include a set of scripts:

• copy-jars.sh - run once to copy jars to HDFS
• initialize-hadoop.sh - run once to init cursors
• hadoop-importer.sh - run as often as wanted

These scripts configured by environment variables:

export topic="your topic"
export hdfs_dir="/target/path/in/hdfs"
export generated_property_file="tmp_file_for_this_topic"
export list_of_brokers="config_file_containing_servers"

This can be added with command line options: initialize-hadoop.sh -t -d <hdfs_dir> -b <list_of_brokers> -g <generated_property_file>

Example file for $list_of_brokers:

kafka1.example.com:9092
kafka2.example.com:9092

Make sure to end the file with a '\n' or the last server is ignored

Credit

Based on Kafka DISTRIBUTED incremental Hadoop consumer