Storm

Nimbus nodes (Master node, similar to the Hadoop JobTracker) :

• Uploads computations for execution.
• Distributes code across the cluster.
• Launches workers across the cluster.
• Monitor computation and reallocates workers as needed.

Zookeeper nodes :

• Coordinate the Storm cluster.
• State management of the Supervisor.

Supervisor nodes :

• Communicate with Nimbus through Zookeeper, Start and Stop worker according to the signals from Nimbus.

Use Case:

1. Create a Storm topology that checks hbase table-1 for any row available, if yes get corresponding data from hbase table-2 and insert in table-3 finally delete in table-1.

soln. create gradle project. define a topologybuilder class, a ticker bolt that connects hbase table-1 for read and emits, one bolt to fetch from table-2 for emits by ticker, other bolt to put in table-2 for emits by bolt-2 and submit to local or cluster. complete code. Complete code is below. Sources: 1

Exceptions faced:

Problem-1 : Topology not working due to Found multiple defaults.yaml resources when deploying to remote cluster.

Soln. Was using Shadow to build jar of app, which bundles all dependencies(including storm's packages, which is not required at deploy or runtime since its already there with runtime) properly. Built jar with simple gradle jar task and issue resolved. sources: 1, 2, Also ensure the versions of storm dependency at compile time and runtime are same. I faced this issue in multiple scenarios and one was due to version conflict(gradle dependency 1.0.2(org.apache.storm.StormSubmitter) was not found in runtime 0.10.0).

Problem-2 : Storm UI shows two leader nodes, one offline another active.

Soln.Found two names in /etc/hosts for IP 127.0.0.1. dropped 1 and restarted storm processes.

Problem-3 : Topology deployed, spouts throwing error for multiple loggers in classpath.Caused by: java.lang.IllegalStateException: Detected both log4j-over-slf4j.jar AND slf4j-log4j12.jar on the class path,

Soln. excluded log4j by adding following in build.gradle

configurations.all {

    exclude group: "org.slf4j", module: "slf4j-log4j12"

    exclude group: "log4j", module: "log4j"

}

Problem-4 : org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss

Soln. by mistake, had defined port 9092 instead of 2181, while initialing BrokerHost with ZKHosts.

Problem-5 : java.lang.IllegalArgumentException: Tuple created with wrong number of fields. Expected 0 fields but got 1 fields

Soln. In declareOutputFields, Stream's Field were initialized without any field name. assigned the name.

Problem-6 : NoSuchMethodError: DefaultKryoFactory - while trying to run a storm-starter project topology

Soln. Resolved by downgrading version of kryo.

Problem-7 : SEVERE: A message body reader for Java class java.lang.String, and Java type class java.lang.String, and MIME media type application/octet-stream was not found. at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:549).

Soln. Resolved by adding dependency for jersey.json.