轉自：http://www.51studyit.com/html/notes/20140329/44.html
Streams, 流
流作爲storm的核心概念, 定義爲無限的tuple序列.
什麼是tuple?
命名的value序列, 可以理解成Key/value序列, 每個value可以是任何類型, 動態類型不需要事先聲明.
Tuple在傳輸中需要序列化和反序列化, storm集成了普通類型的序列化模塊, 用戶可以自定義特殊類型的序列化邏輯
A tuple is a named list of values, where each value can be any type.
Tuples are dynamically typed -- the types of the fields do not need to be declared.
Tuples have helper methods like getInteger and getString to get field values without having to cast the result.
Storm needs to know how to serialize all the values in a tuple.
The stream is the core abstraction in Storm. A stream is an unbounded sequence of tuples that is processed and created in parallel in a distributed fashion.
Streams are defined with a schema that names the fields in the stream's tuples. By default, tuples can contain integers, longs, shorts, bytes, strings, doubles, floats, booleans, and byte arrays. You can also define your own serializers so that custom types can be used natively within tuples.
Every stream is given an id when declared. Since single-stream spouts and bolts are so common, OutputFieldsDeclarer has convenience methods for declaring a single stream without specifying an id. In this case, the stream is given the default id of "default".
Resources:
Tuple: streams are composed of tuples
OutputFieldsDeclarer: used to declare streams and their schemas
Serialization: Information about Storm's dynamic typing of tuples and declaring custom serializations
ISerialization: custom serializers must implement this interface
CONFIG.TOPOLOGY_SERIALIZATIONS: custom serializers can be registered using this configuration
Spouts, 流的源頭
Spout是Storm裏面特有的名詞, Stream的源頭. 通常是從外部數據源讀取tuples, 並emit到topology.
Spout可以同時emit多個tuple stream, 通過OutputFieldsDeclarer中的declareStream method來定義
Spout需要實現IRichSpout接口, 最重要的方法是nextTuple, storm會不斷調用該接口從spout中取數據
同時需要注意, Spout分爲reliable or unreliable兩種, 對於reliable, 還支持ack和fail方法, 具體參考"Reliability”
下面給出個nextTuple的例子, 從words裏面隨機選一個tuple作爲輸出
public void nextTuple() {
Utils.sleep(100);
final String[] words = new String[] {"nathan", "mike",
"jackson", "golda", "bertels"};
final Random rand = new Random();
final String word = words[rand.nextInt(words.length)];
_collector.emit(new Values(word));
}
A spout is a source of streams in a topology.
Generally spouts will read tuples from an external source and emit them into the topology (e.g. a Kestrel queue or the Twitter API).
Spouts can either be reliable or unreliable. A reliable spout is capable of replaying a tuple if it failed to be processed by Storm, whereas an unreliable spout forgets about the tuple as soon as it is emitted.
Spouts can emit more than one stream. To do so, declare multiple streams using the declareStream method of OutputFieldsDeclarer and specify the stream to emit to when using the emit method on SpoutOutputCollector.
The main method on spouts is nextTuple. nextTuple either emits a new tuple into the topology or simply returns if there are no new tuples to emit. It is imperative that nextTuple does not block for any spout implementation, because Storm calls all the spout methods on the same thread.
The other main methods on spouts are ack and fail. These are called when Storm detects that a tuple emitted from the spout either successfully completed through the topology or failed to be completed. ack and fail are only called for reliable spouts. See the Javadoc for more information.
Resources:
IRichSpout: this is the interface that spouts must implement.
Guaranteeing message processing
Bolts, 流的處理節點
對於Bolt, 用戶可以定義任意的處理邏輯, 最重要的方法是execute, 輸入爲tuple, 輸出爲emit 0或多個tuples到OutputCollector.
Bolt支持多個輸入流和emit多個輸出流, 輸出流和spout一樣, 通過OutputFieldsDeclarer中的declareStream method來定義; 對於輸入流, 如果想subscribe上層節點的多個輸出streaming, 需要顯式的通過stream_id去訂閱, 如果不明確指定stream_id, 默認會訂閱default stream.
public static class ExclamationBolt implements IRichBolt {
OutputCollector _collector;
public void prepare(Map conf, TopologyContext context,
OutputCollector collector) {
_collector = collector;
}
public void execute(Tuple tuple) {
_collector.emit(tuple, new Values(tuple.getString(0) + "!!!"));
_collector.ack(tuple);
}
public void cleanup() {
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
}
All processing in topologies is done in bolts. Bolts can do anything from filtering, functions, aggregations, joins, talking to databases, and more.
Bolts can do simple stream transformations. Doing complex stream transformations often requires multiple steps and thus multiple bolts. For example, transforming a stream of tweets into a stream of trending images requires at least two steps: a bolt to do a rolling count of retweets for each image, and one or more bolts to stream out the top X images (you can do this particular stream transformation in a more scalable way with three bolts than with two).
Bolts can emit more than one stream. To do so, declare multiple streams using the declareStream method of OutputFieldsDeclarer and specify the stream to emit to when using the emit method on OutputCollector.
When you declare a bolt's input streams, you always subscribe to specific streams of another component. If you want to subscribe to all the streams of another component, you have to subscribe to each one individually. InputDeclarer has syntactic sugar for subscribing to streams declared on the default stream id. Saying declarer.shuffleGrouping("1") subscribes to the default stream on component "1" and is equivalent to declarer.shuffleGrouping("1", DEFAULT_STREAM_ID).
The main method in bolts is the execute method which takes in as input a new tuple. Bolts emit new tuples using the OutputCollector object. Bolts must call the ack method on the OutputCollector for every tuple they process so that Storm knows when tuples are completed (and can eventually determine that its safe to ack the original spout tuples). For the common case of processing an input tuple, emitting 0 or more tuples based on that tuple, and then acking the input tuple, Storm provides an IBasicBolt interface which does the acking automatically.
Its perfectly fine to launch new threads in bolts that do processing asynchronously. OutputCollector is thread-safe and can be called at any time.
Resources:
IRichBolt: this is general interface for bolts.
IBasicBolt: this is a convenience interface for defining bolts that do filtering or simple functions.
OutputCollector: bolts emit tuples to their output streams using an instance of this class
Guaranteeing message processing
Topologies, 拓撲
可以理解爲類似MapReduce job
根本區別, MR job執行完就結束, 而Topology會一直存在. 因爲MR流動的是代碼, 而Storm流動的數據.
所以Storm不可能替代MR, 因爲對於海量數據, 數據的流動是不合理的
另一個區別, 我自己的想法, Topology對工作流有更好的支持, 而MR job往往只能完成一個map/reduce的過程, 而對於複雜的操作, 需要多個MR job才能完成.
而Topology的定義更加靈活, 可以簡單的使用一個topology支持比較複雜的工作流場景
Storm Topology是基於Thrift結構, 並且Nimbus是個Thrift server, 所以對於Topology可以用任何語言實現, 最終都是轉化爲Thrift結構
具體的Java版本的Topology的例子,
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout(1, new RandomSentenceSpout(), 5 );
builder.setBolt(2, new SplitSentence(), 8 ).shuffleGrouping(1);
builder.setBolt(3, new WordCount(), 12).fieldsGrouping(2, new Fields("word"));
Topology有一個spout, 兩個bolt. setSpout和setBolt的參數都是一樣, 分別爲id(在Topology中的唯一標識); 處理邏輯(對於Spout就是數據產生function); 併發線程數(task數)
其中對於spout需要實現IRichSpout接口, 而bolt需要實現IRichBolt接口
比較特別的是, setBolt方法會返回一個InputDeclarer對象, 並且該對象是用來定義Bolt輸入的, 比如上面.shuffleGrouping(1), 用1(spout)的輸出流作爲輸入
The logic for a realtime application is packaged into a Storm topology. A Storm topology is analogous to a MapReduce job. One key difference is that a MapReduce job eventually finishes, whereas a topology runs forever (or until you kill it, of course). A topology is a graph of spouts and bolts that are connected with stream groupings. These concepts are described below.
Since topology definitions are just Thrift structs, and Nimbus is a Thrift service, you can create and submit topologies using any programming language.
Resources:
TopologyBuilder: use this class to construct topologies in Java
Running topologies on a production cluster
Local mode: Read this to learn how to develop and test topologies in local mode.
Nimbus和Supervisor
在Storm的集羣裏面有兩種節點:
Nimbus節點, 主節點, 它的作用類似Hadoop裏面的JobTracker. Nimbus is responsible for distributing code around the cluster, assigning tasks to machines, and monitoring for failures.
Supervisor的節點, 工作節點, listens for work assigned to its machine and starts and stops worker processes as necessary based on what Nimbus has assigned to it.
Each worker process executes a subset of a topology; a running topology consists of many worker processes spread across many machines.
Nimbus和Supervisor之間的所有協調工作都是通過一個Zookeeper集羣來完成
nimbus進程和supervisor都是快速失敗（fail-fast)和無狀態的, 所有的狀態都存儲在Zookeeper或本地磁盤上
這也就意味着你可以用kill -9來殺死nimbus和supervisor進程, 然後再重啓它們, 它們可以繼續工作
更重要的是, nimbus和supervisor的fail或restart不會影響worker的工作, 不象Hadoop, Job tracker的fail會導致job失敗
Workers, Executor, Tasks
Stream groupings
如果從task的粒度來看一個運行的topology, 它應該如圖, 所以需要策略決定blot和spout, 以及bolt之間的數據流向問題
Part of defining a topology is specifying for each bolt which streams it should receive as input. A stream grouping defines how that stream should be partitioned among the bolt's tasks.
There are seven built-in stream groupings in Storm, and you can implement a custom stream grouping by implementing the CustomStreamGrouping interface:
- Shuffle grouping: Tuples are randomly distributed across the bolt's tasks in a way such that each bolt is guaranteed to get an equal number of tuples.
- Fields grouping: The stream is partitioned by the fields specified in the grouping. For example, if the stream is grouped by the "user-id" field, tuples with the same "user-id" will always go to the same task, but tuples with different "user-id"'s may go to different tasks.
- All grouping: The stream is replicated across all the bolt's tasks. Use this grouping with care.
- Global grouping: The entire stream goes to a single one of the bolt's tasks. Specifically, it goes to the task with the lowest id.
- None grouping: This grouping specifies that you don't care how the stream is grouped. Currently, none groupings are equivalent to shuffle groupings. Eventually though, Storm will push down bolts with none groupings to execute in the same thread as the bolt or spout they subscribe from (when possible).
- Direct grouping: This is a special kind of grouping. A stream grouped this way means that the producer of the tuple decides which task of the consumer will receive this tuple. Direct groupings can only be declared on streams that have been declared as direct streams. Tuples emitted to a direct stream must be emitted using one of the emitDirect methods. A bolt can get the task ids of its consumers by either using the provided TopologyContext or by keeping track of the output of the
emitmethod in OutputCollector (which returns the task ids that the tuple was sent to). - Local or shuffle grouping: If the target bolt has one or more tasks in the same worker process, tuples will be shuffled to just those in-process tasks. Otherwise, this acts like a normal shuffle grouping.
Resources:
TopologyBuilder: use this class to define topologies
InputDeclarer: this object is returned whenever setBolt is called on TopologyBuilder and is used for declaring a bolt's input streams and how those streams should be grouped
CoordinatedBolt: this bolt is useful for distributed RPC topologies and makes heavy use of direct streams and direct groupings
Reliability
Streams, 流
流作爲storm的核心概念, 定義爲無限的tuple序列.
什麼是tuple?
命名的value序列, 可以理解成Key/value序列, 每個value可以是任何類型, 動態類型不需要事先聲明.
Tuple在傳輸中需要序列化和反序列化, storm集成了普通類型的序列化模塊, 用戶可以自定義特殊類型的序列化邏輯
A tuple is a named list of values, where each value can be any type.
Tuples are dynamically typed -- the types of the fields do not need to be declared.
Tuples have helper methods like getInteger and getString to get field values without having to cast the result.
Storm needs to know how to serialize all the values in a tuple.
The stream is the core abstraction in Storm. A stream is an unbounded sequence of tuples that is processed and created in parallel in a distributed fashion.
Streams are defined with a schema that names the fields in the stream's tuples. By default, tuples can contain integers, longs, shorts, bytes, strings, doubles, floats, booleans, and byte arrays. You can also define your own serializers so that custom types can be used natively within tuples.
Every stream is given an id when declared. Since single-stream spouts and bolts are so common, OutputFieldsDeclarer has convenience methods for declaring a single stream without specifying an id. In this case, the stream is given the default id of "default".
Resources:
Tuple: streams are composed of tuples
OutputFieldsDeclarer: used to declare streams and their schemas
Serialization: Information about Storm's dynamic typing of tuples and declaring custom serializations
ISerialization: custom serializers must implement this interface
CONFIG.TOPOLOGY_SERIALIZATIONS: custom serializers can be registered using this configuration
Spouts, 流的源頭
Spout是Storm裏面特有的名詞, Stream的源頭. 通常是從外部數據源讀取tuples, 並emit到topology.
Spout可以同時emit多個tuple stream, 通過OutputFieldsDeclarer中的declareStream method來定義
Spout需要實現IRichSpout接口, 最重要的方法是nextTuple, storm會不斷調用該接口從spout中取數據
同時需要注意, Spout分爲reliable or unreliable兩種, 對於reliable, 還支持ack和fail方法, 具體參考"Reliability”
下面給出個nextTuple的例子, 從words裏面隨機選一個tuple作爲輸出
public void nextTuple() {
Utils.sleep(100);
final String[] words = new String[] {"nathan", "mike",
"jackson", "golda", "bertels"};
final Random rand = new Random();
final String word = words[rand.nextInt(words.length)];
_collector.emit(new Values(word));
}
A spout is a source of streams in a topology.
Generally spouts will read tuples from an external source and emit them into the topology (e.g. a Kestrel queue or the Twitter API).
Spouts can either be reliable or unreliable. A reliable spout is capable of replaying a tuple if it failed to be processed by Storm, whereas an unreliable spout forgets about the tuple as soon as it is emitted.
Spouts can emit more than one stream. To do so, declare multiple streams using the declareStream method of OutputFieldsDeclarer and specify the stream to emit to when using the emit method on SpoutOutputCollector.
The main method on spouts is nextTuple. nextTuple either emits a new tuple into the topology or simply returns if there are no new tuples to emit. It is imperative that nextTuple does not block for any spout implementation, because Storm calls all the spout methods on the same thread.
The other main methods on spouts are ack and fail. These are called when Storm detects that a tuple emitted from the spout either successfully completed through the topology or failed to be completed. ack and fail are only called for reliable spouts. See the Javadoc for more information.
Resources:
IRichSpout: this is the interface that spouts must implement.
Guaranteeing message processing
Bolts, 流的處理節點
對於Bolt, 用戶可以定義任意的處理邏輯, 最重要的方法是execute, 輸入爲tuple, 輸出爲emit 0或多個tuples到OutputCollector.
Bolt支持多個輸入流和emit多個輸出流, 輸出流和spout一樣, 通過OutputFieldsDeclarer中的declareStream method來定義; 對於輸入流, 如果想subscribe上層節點的多個輸出streaming, 需要顯式的通過stream_id去訂閱, 如果不明確指定stream_id, 默認會訂閱default stream.
public static class ExclamationBolt implements IRichBolt {
OutputCollector _collector;
public void prepare(Map conf, TopologyContext context,
OutputCollector collector) {
_collector = collector;
}
public void execute(Tuple tuple) {
_collector.emit(tuple, new Values(tuple.getString(0) + "!!!"));
_collector.ack(tuple);
}
public void cleanup() {
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
}
All processing in topologies is done in bolts. Bolts can do anything from filtering, functions, aggregations, joins, talking to databases, and more.
Bolts can do simple stream transformations. Doing complex stream transformations often requires multiple steps and thus multiple bolts. For example, transforming a stream of tweets into a stream of trending images requires at least two steps: a bolt to do a rolling count of retweets for each image, and one or more bolts to stream out the top X images (you can do this particular stream transformation in a more scalable way with three bolts than with two).
Bolts can emit more than one stream. To do so, declare multiple streams using the declareStream method of OutputFieldsDeclarer and specify the stream to emit to when using the emit method on OutputCollector.
When you declare a bolt's input streams, you always subscribe to specific streams of another component. If you want to subscribe to all the streams of another component, you have to subscribe to each one individually. InputDeclarer has syntactic sugar for subscribing to streams declared on the default stream id. Saying declarer.shuffleGrouping("1") subscribes to the default stream on component "1" and is equivalent to declarer.shuffleGrouping("1", DEFAULT_STREAM_ID).
The main method in bolts is the execute method which takes in as input a new tuple. Bolts emit new tuples using the OutputCollector object. Bolts must call the ack method on the OutputCollector for every tuple they process so that Storm knows when tuples are completed (and can eventually determine that its safe to ack the original spout tuples). For the common case of processing an input tuple, emitting 0 or more tuples based on that tuple, and then acking the input tuple, Storm provides an IBasicBolt interface which does the acking automatically.
Its perfectly fine to launch new threads in bolts that do processing asynchronously. OutputCollector is thread-safe and can be called at any time.
Resources:
IRichBolt: this is general interface for bolts.
IBasicBolt: this is a convenience interface for defining bolts that do filtering or simple functions.
OutputCollector: bolts emit tuples to their output streams using an instance of this class
Guaranteeing message processing
Topologies, 拓撲
可以理解爲類似MapReduce job
根本區別, MR job執行完就結束, 而Topology會一直存在. 因爲MR流動的是代碼, 而Storm流動的數據.
所以Storm不可能替代MR, 因爲對於海量數據, 數據的流動是不合理的
另一個區別, 我自己的想法, Topology對工作流有更好的支持, 而MR job往往只能完成一個map/reduce的過程, 而對於複雜的操作, 需要多個MR job才能完成.
而Topology的定義更加靈活, 可以簡單的使用一個topology支持比較複雜的工作流場景
Storm Topology是基於Thrift結構, 並且Nimbus是個Thrift server, 所以對於Topology可以用任何語言實現, 最終都是轉化爲Thrift結構
具體的Java版本的Topology的例子,
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout(1, new RandomSentenceSpout(), 5 );
builder.setBolt(2, new SplitSentence(), 8 ).shuffleGrouping(1);
builder.setBolt(3, new WordCount(), 12).fieldsGrouping(2, new Fields("word"));
Topology有一個spout, 兩個bolt. setSpout和setBolt的參數都是一樣, 分別爲id(在Topology中的唯一標識); 處理邏輯(對於Spout就是數據產生function); 併發線程數(task數)
其中對於spout需要實現IRichSpout接口, 而bolt需要實現IRichBolt接口
比較特別的是, setBolt方法會返回一個InputDeclarer對象, 並且該對象是用來定義Bolt輸入的, 比如上面.shuffleGrouping(1), 用1(spout)的輸出流作爲輸入
The logic for a realtime application is packaged into a Storm topology. A Storm topology is analogous to a MapReduce job. One key difference is that a MapReduce job eventually finishes, whereas a topology runs forever (or until you kill it, of course). A topology is a graph of spouts and bolts that are connected with stream groupings. These concepts are described below.
Since topology definitions are just Thrift structs, and Nimbus is a Thrift service, you can create and submit topologies using any programming language.
Resources:
TopologyBuilder: use this class to construct topologies in Java
Running topologies on a production cluster
Local mode: Read this to learn how to develop and test topologies in local mode.
Nimbus和Supervisor
在Storm的集羣裏面有兩種節點:
Nimbus節點, 主節點, 它的作用類似Hadoop裏面的JobTracker. Nimbus is responsible for distributing code around the cluster, assigning tasks to machines, and monitoring for failures.
Supervisor的節點, 工作節點, listens for work assigned to its machine and starts and stops worker processes as necessary based on what Nimbus has assigned to it.
Each worker process executes a subset of a topology; a running topology consists of many worker processes spread across many machines.
Nimbus和Supervisor之間的所有協調工作都是通過一個Zookeeper集羣來完成
nimbus進程和supervisor都是快速失敗（fail-fast)和無狀態的, 所有的狀態都存儲在Zookeeper或本地磁盤上
這也就意味着你可以用kill -9來殺死nimbus和supervisor進程, 然後再重啓它們, 它們可以繼續工作
更重要的是, nimbus和supervisor的fail或restart不會影響worker的工作, 不象Hadoop, Job tracker的fail會導致job失敗
Workers, Executor, Tasks
Stream groupings
如果從task的粒度來看一個運行的topology, 它應該如圖, 所以需要策略決定blot和spout, 以及bolt之間的數據流向問題
Part of defining a topology is specifying for each bolt which streams it should receive as input. A stream grouping defines how that stream should be partitioned among the bolt's tasks.
There are seven built-in stream groupings in Storm, and you can implement a custom stream grouping by implementing the CustomStreamGrouping interface:
- Shuffle grouping: Tuples are randomly distributed across the bolt's tasks in a way such that each bolt is guaranteed to get an equal number of tuples.
- Fields grouping: The stream is partitioned by the fields specified in the grouping. For example, if the stream is grouped by the "user-id" field, tuples with the same "user-id" will always go to the same task, but tuples with different "user-id"'s may go to different tasks.
- All grouping: The stream is replicated across all the bolt's tasks. Use this grouping with care.
- Global grouping: The entire stream goes to a single one of the bolt's tasks. Specifically, it goes to the task with the lowest id.
- None grouping: This grouping specifies that you don't care how the stream is grouped. Currently, none groupings are equivalent to shuffle groupings. Eventually though, Storm will push down bolts with none groupings to execute in the same thread as the bolt or spout they subscribe from (when possible).
- Direct grouping: This is a special kind of grouping. A stream grouped this way means that the producer of the tuple decides which task of the consumer will receive this tuple. Direct groupings can only be declared on streams that have been declared as direct streams. Tuples emitted to a direct stream must be emitted using one of the emitDirect methods. A bolt can get the task ids of its consumers by either using the provided TopologyContext or by keeping track of the output of the
emitmethod in OutputCollector (which returns the task ids that the tuple was sent to). - Local or shuffle grouping: If the target bolt has one or more tasks in the same worker process, tuples will be shuffled to just those in-process tasks. Otherwise, this acts like a normal shuffle grouping.
Resources:
TopologyBuilder: use this class to define topologies
InputDeclarer: this object is returned whenever setBolt is called on TopologyBuilder and is used for declaring a bolt's input streams and how those streams should be grouped
CoordinatedBolt: this bolt is useful for distributed RPC topologies and makes heavy use of direct streams and direct groupings