Question 1

How do I configure custom retry logic in Kinesis Storm spout?

Accepted Answer

Adjust the retry settings in KinesisSpoutConfig; by default, it retries three times, but you can modify this based on your fault tolerance needs. Check the sample properties file for configuration examples.

Question 2

Kinesis Storm spout vs Kafka spout for Apache Storm: which should I use?

Accepted Answer

Use Kinesis Storm spout if you're locked into AWS Kinesis for data ingestion; it offers seamless integration but ties you to AWS. Kafka spout is better for Kafka-based ecosystems, providing more flexibility in multi-cloud setups.

Question 3

Does Kinesis Storm spout support exactly-once processing?

Accepted Answer

No, it provides at-least-once semantics through checkpointing and retries. Achieving exactly-once would require additional logic in your Storm bolts, as the spout doesn't guarantee it natively.

Question 4

How to handle Kinesis shard resharding with this spout?

Accepted Answer

Manually invoke 'storm rebalance' after resharding, as noted in release notes, to refresh the shard list and redistribute shards across spout tasks—there's no automatic handling.

Question 5

Can I use Kinesis Storm spout without ZooKeeper?

Accepted Answer

No, checkpoint management relies on ZooKeeper for state persistence; removing it would require significant code changes, as the spout is designed around this dependency.

Question 6

What are the memory implications of in-memory buffering?

Accepted Answer

In-memory buffering reduces latency but can increase heap usage; monitor memory in high-throughput scenarios to avoid out-of-memory errors, especially with large record sizes.

kinesis-storm-spout

What is kinesis-storm-spout?

Overview

Use Cases

Best For

Related Projects

Found a gem we're missing?

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions