storm集成Spring

我們知道storm是一個獨立的實時計算框架，而Spring是一個獨立的輕量級容器框架，那麼如何在storm框架中集成Spring框架，以便於在storm開發中利用Spring的控制反轉(IoC)和麪向切面(AOP)性能來做java對象的管理呢？首先我們應該知道Spring是如何來管理java對象的，storm是怎麼運行拓撲的，知道了這個、些才能明白怎麼將Spring集成到storm中。

我們經常接觸到的Spring框架其實是在Spring的基礎上封裝的一個web框架，和懶人整合包（starter）Springboot，那麼他們是怎麼管理bean的呢？實際上，spring mvc和Springboot都是利用上下文ApplicationContext或者beanFactory作爲容易來管理java對象的整個生命週期，其中Spring mvc是通過讀取web.xml中的配置來初始化上下文環境並裝配配置文件中的bean，而Springboot則是通過一個啓動類來初始化上下文環境並裝配啓動類所在路徑及其子路徑下的bean。

那麼我們來回顧一下storm的基本要素以及他是怎麼啓動運行一個storm拓撲的。storm的拓撲是由數據源spout和數據處理單元bolt組成的，spout是產生數據的地方，spout可以從文件讀取數據，從Kafka獲取數據（KafkaSpout），從網絡獲取數據，經過簡單處理後發送出來，bolt接收到spout發出的數據，對數據經過一系列的處理，可以得到最終需要的數據，而拓撲將spout和bolt鏈接起來形成一個實時數據的處理鏈。

spout

public class TestWordSpout extends BaseRichSpout{
    private SpoutOutputCollector collector = null;
    public void open(Map map, TopologyContext topologyContext, SpoutOutputCollector spoutOutputCollector) {
        this.collector = spoutOutputCollector;
    }
    public void nextTuple() {
        Utils.sleep(1000);
        final String[] words = new String[]{"fdfs","fdfs","ffsdfs"};
        final Random random = new Random();
        final String word = words[random.nextInt(words.length)];
        collector.emit(new Values(word));
    }
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
        outputFieldsDeclarer.declare(new Fields("word"));
    }
}

bolt

public class ExclamationBolt extends BaseRichBolt{
    private OutputCollector collector = null;
    public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
        this.collector = outputCollector;
    }
    public void execute(Tuple tuple) {
        this.collector.emit(tuple,new Values(tuple.getString(0)+"!!!"));
        this.collector.ack(tuple);
    }
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
        outputFieldsDeclarer.declare(new Fields("word"));
    }
}

topology

public class WordsTopology {
    public static void main(String[] args) throws AlreadyAliveException, InvalidTopologyException {
        TopologyBuilder topologyBuilder = new TopologyBuilder();
        topologyBuilder.setSpout("word",new TestWordSpout(),1);
        topologyBuilder.setBolt("exclaim",new ExclamationBolt(),1).shuffleGrouping("word");
        Config config = new Config();
        config.setDebug(true);
        if(args != null && args.length > 0){
            config.setNumWorkers(3); //集羣模式
            StormSubmitter.submitTopologyWithProgressBar(args[0],config,topologyBuilder.createTopology());
        }else{ //本地模式
            LocalCluster localCluster = new LocalCluster();
            localCluster.submitTopology("test",config,topologyBuilder.createTopology());
            Utils.sleep(30000);
            localCluster.killTopology("test");
            localCluster.shutdown();
        }
    }
}

可以看出storm是由main函數啓動，並沒有用到Spring框架，那麼怎麼纔可以在storm中啓動Spring的上下文環境呢？我們可以分別從Spring mvc和Spring boot中獲得啓發。

Spring mvc 框架形式

在Spring mvc框架中，實際上就是通過web.xml指定的配置文件來初始化上下文環境並實例化bean，那麼參考我們在sprin mvc的測試類中的做法，我們可以手動的根據配置文件獲取上下文環境並實例化bean（這部分一般在prepare方法中進行）：

ApplicationContext applicationContext = new ClassPathXmlApplicationContext("application-context.xml");
DimDataClient client = (DimDataClient) applicationContext.getBean("dimDataClient");

考慮到storm中的bolt是分佈在集羣中的各個節點上的，因此每一個bolt都需要啓動各自的Spring上下文環境，因此需要在每個bolt中都使用上述代碼獲取上下文環境，並在ApplicationContext中獲取相應的bean。

applicationcontext.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/context
       http://www.springframework.org/schema/context/spring-context-4.2.xsd
       http://www.springframework.org/schema/util
       http://www.springframework.org/schema/util/spring-util.xsd">

    <bean id="yourBean" class="com.sankuai.tdp.YourBean">
        <property name="order" value="2"/>
        <property name="ignoreUnresolvablePlaceholders" value="false"/>
    </bean>
</beans>

然後我們就可以在bolt中創建上下文環境並使用其中的對象了：

public class myBolt extends BaseBasicBolt {
    private YourBean yourBean;
    private static final ApplicationContext APPLICATION_CONTEXT = new ClassPathXmlApplicationContext("application-context.xml");
    static {
        YourBean = (YourBean) APPLICATION_CONTEXT.getBean("yourBean");
    }
    @Overeide
    public void prepare(Map map, TopologyContext topologyContext) {
        //prepare code
    }
    @Overeide
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
        outputFieldsDeclarer.declare(UpsRtSaveData.genFields());
    }
    @Overeide
    public void execute(Tuple tuple, BasicOutputCollector basicOutputCollector) {
        //excute code
    }
}

Spring boot框架形式

我們參考Springboot的啓動類發現，啓動類中實際上就是新建了一個SpringApplication並調用其run方法，因此我們可以自己寫一個啓動類，並手動調用這個啓動類。

SpringStormApplication啓動類

@SpringBootApplication
@ComponentScan(value = "com.xxx.storm")
public class SpringStormApplication {
    /**
     * 非工程啓動入口，所以不用main方法
     * 加上synchronized的作用是由於storm在啓動多個bolt線程實例時，如果Springboot用到Apollo分佈式配置，會報ConcurrentModificationException錯誤
     */
    public synchronized static void run(String ...args) {
        SpringApplication app = new SpringApplication(SpringStormApplication.class);
        //我們並不需要web servlet功能，所以設置爲WebApplicationType.NONE
        app.setWebApplicationType(WebApplicationType.NONE);
        //忽略掉banner輸出
        app.setBannerMode(Banner.Mode.OFF);
        //忽略Spring啓動信息日誌
        app.setLogStartupInfo(false);
        app.run(args);
    }
}

有了這個啓動類相當於建立了Spring的上下文環境，我們還需要拿到改上下文環境中的bean對象，這裏需要實現ApplicationContextAware接口

@Component
public class BeanUtils implements ApplicationContextAware {
    private static ApplicationContext applicationContext = null;
    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        if (BeanUtils.applicationContext == null) {
            BeanUtils.applicationContext = applicationContext;
        }
    }
    public static ApplicationContext getApplicationContext() {
        return applicationContext;
    }
    public static Object getBean(String name) {
        return getApplicationContext().getBean(name);
    }
    public static <T> T getBean(Class<T> clazz) {
        return getApplicationContext().getBean(clazz);
    }
    public static <T> T getBean(String name, Class<T> clazz) {
        return getApplicationContext().getBean(name, clazz);
    }
}

然後我們就可以在bolt中創建上下文環境並使用其中的對象了：

public class myBolt extends BaseBasicBolt {
    private YourBean yourBean;
    
    @Overeide
    public void prepare(Map map, TopologyContext topologyContext) {
        SpringStormApplication.run();
        YourBean = (YourBean) BeanUtils.getBean("yourBean");
        //prepare code
    }
    @Overeide
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
        outputFieldsDeclarer.declare(UpsRtSaveData.genFields());
    }
    @Overeide
    public void execute(Tuple tuple, BasicOutputCollector basicOutputCollector) {
        //excute code
    }
}

啓動拓撲：在打包storm時需要引入插件

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.1.0</version>
    <configuration>
        <artifactSet>
            <excludes>
                <exclude>org.slf4j:slf4j-simple</exclude>
                <exclude>org.slf4j:slf4j-log4j12</exclude>
            </excludes>
        </artifactSet>
        <filters>
            <filter>
                <artifact>*:*</artifact>
                <excludes>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                </excludes>
            </filter>
        </filters>
        <!-- Additional configuration. -->
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <finalName>${project.artifactId}-${project.version}-with-dependencies</finalName>
            </configuration>
        </execution>
    </executions>
</plugin>

如果打包出錯，要添加兩個文件：spring.handlers和spring.schemas，也就是在插件中添加標籤<transformers>

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.1.0</version>
    <configuration>
        <artifactSet>
            <excludes>
                <exclude>org.slf4j:slf4j-simple</exclude>
                <exclude>org.slf4j:slf4j-log4j12</exclude>
            </excludes>
        </artifactSet>
        <filters>
            <filter>
                <artifact>*:*</artifact>
                <excludes>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                </excludes>
            </filter>
        </filters>
        <!-- Additional configuration. -->
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <finalName>${project.artifactId}-${project.version}-with-dependencies</finalName>
                <transformers>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.handlers</resource>
                    </transformer>
                    <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.schemas</resource>
                    </transformer>
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>

其中spring.handlers爲：

http\://www.springframework.org/schema/p=org.springframework.beans.factory.xml.SimplePropertyNamespaceHandler
http\://www.springframework.org/schema/util=org.springframework.beans.factory.xml.UtilNamespaceHandler
http\://www.springframework.org/schema/aop=org.springframework.aop.config.AopNamespaceHandler
http\://www.springframework.org/schema/context=org.springframework.context.config.ContextNamespaceHandler
http\://www.springframework.org/schema/jee=org.springframework.ejb.config.JeeNamespaceHandler
http\://www.springframework.org/schema/lang=org.springframework.scripting.config.LangNamespaceHandler

spring.schemas爲：

http\://www.springframework.org/schema/context/spring-context-2.5.xsd=org/springframework/context/config/spring-context-2.5.xsd
http\://www.springframework.org/schema/context/spring-context-3.0.xsd=org/springframework/context/config/spring-context-3.0.xsd
http\://www.springframework.org/schema/context/spring-context.xsd=org/springframework/context/config/spring-context-3.0.xsd
http\://www.springframework.org/schema/jee/spring-jee-2.0.xsd=org/springframework/ejb/config/spring-jee-2.0.xsd
http\://www.springframework.org/schema/jee/spring-jee-2.5.xsd=org/springframework/ejb/config/spring-jee-2.5.xsd
http\://www.springframework.org/schema/jee/spring-jee-3.0.xsd=org/springframework/ejb/config/spring-jee-3.0.xsd
http\://www.springframework.org/schema/jee/spring-jee.xsd=org/springframework/ejb/config/spring-jee-3.0.xsd
http\://www.springframework.org/schema/lang/spring-lang-2.0.xsd=org/springframework/scripting/config/spring-lang-2.0.xsd
http\://www.springframework.org/schema/lang/spring-lang-2.5.xsd=org/springframework/scripting/config/spring-lang-2.5.xsd
http\://www.springframework.org/schema/lang/spring-lang-3.0.xsd=org/springframework/scripting/config/spring-lang-3.0.xsd
http\://www.springframework.org/schema/lang/spring-lang.xsd=org/springframework/scripting/config/spring-lang-3.0.xsd
http\://www.springframework.org/schema/task/spring-task-3.0.xsd=org/springframework/scheduling/config/spring-task-3.0.xsd
http\://www.springframework.org/schema/task/spring-task.xsd=org/springframework/scheduling/config/spring-task-3.0.xsd

Spring mvc 框架形式

Spring boot框架形式

微服務實踐之使用 Visual Studio 2022 調試Dapr 應用程序

wpf附加屬性理解 WPF附加屬性

劍指Offer——字符串空格替換

集合框架——Collection實現類的遍歷方法與元素刪除

集合框架——ArrayList與LinkedList、Vector的區別

storm集成Spring

Java代碼反編譯——下載class字節碼文件及反編譯.class文件

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結