JAVA客戶端兼容Elasticsearch 6.8與7.8服務器的方案 1. 問題背景 2. 設計思路 3. 設計方案 附錄：ES客戶端封裝函數

1. 問題背景

  目前項目中所使用的Elasticsearch(一下簡稱ES)的服務器版本是6.8.1，但將來有升級到7.8.1版本的計劃。這樣對於相同的應用軟件版本而言，可能會在有的地方搭配6.8.1版本使用，有的地方搭配7.8.1版本使用。
  對於應用軟件而言，作爲ES服務器的JAVA客戶端，需要使用一套代碼來兼容6.8.1版本和7.8.1版本的ES服務器。

2. 設計思路

  Java High Level REST Client（以下簡稱High Level）是ES官方推薦的JAVA客戶端，具有即時同步ES服務器版本，功能特性齊全等特點，因此項目選用High Level作爲視信通應用服務的ES客戶端。
  根據High Level的官方文檔，對於兼容性有如下說明：

  由官方文檔可以看到，High Level具有向前兼容的特性，即老的High Level客戶端可以被新版本的服務器兼容。
  根據這一特性，應用軟件決定使用6.8.1版本的6.8.1客戶端來兼容6.8.1和7.8.1的服務器。
  分析ES服務器6和7版本之間的差別，最大差別在於6中每個索引存儲數據時需要指定Type,而7中不需要指定Type，即一個索引對應一個Mapping。
  目前項目的應用軟件中使用6的客戶端索引建立規則爲:一個索引對應一個Type, 索引名與Type名相同，在升級ES的服務器版本到7之後，目前的這種索引創建模式需要修改，否則在服務器版本升級到7之後無法運行。

3. 設計方案

3.1. 索引模型兼容

  爲兼容6.8.1與7.8.1的ES服務器，應用軟件在創建索引時不指定Type。目前High Level 6.8.1的API中創建索引可以不指定Type參數，創建的索引默認Type爲“_doc”,而7.8.1的服務器支持對於Type爲“_doc”的DSL查詢。
  應用軟件的JAVA客戶端使用High Level 6.8.1在創建索引時不指定Type，對於查詢、聚合的API需要填入Type的情況，使用默認Type值“_doc”。

3.2. 封裝ES操作

  爲避免應用軟件不同的服務使用High Level出現姿勢不一樣的情況。視信通將對ES的操作封裝到一個公共工具類中，對於各應用服務而言，不再關注ES的實現細節，運行方式也都一致，更便於擴展與維護。

3.3. 使用High Level與LOW Level結合的方式兼容ES操作

  儘管官方宣稱High Level的向前兼容性，但是在語法格式上還有一些細微差別。如對於查詢返回結果的總數字段(Total)：
  在6.8.1服務器中，查詢返回命中結果如下：

{
    "took": 0,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 0,
        "max_score": null,
        "hits": []
    }
}

  請注意，這裏total字段爲一個整型。
  但在7.8.1的服務器中，查詢返回命中結果如下：

{
    "took": 6,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 0,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    }
}

  若使用6.8.1的High Level去查詢7.8.1的服務器，會出現API解析total字段不正確的情況(始終返回爲0)。
  針對這種情況，在視信通的ES封裝函數中首先使用Java Low Level REST Client執行該查詢，然後針對返回的結果JSON解析total字段是一個整型還是JSON對象來獲取正確的total值。

3.4. 數據遷移

  由於老的應用軟件版本中ES索引模式爲ES索引名與Type名相同，這種方式不適用於兼容方案。
  因而在使用ES兼容方案後，應用軟件升級時需要重新創建索引（兼容索引），可使用ES的reindex命令，示例如下：

POST http://172.16.249.177:9200/_reindex
{
  "source": {
    "index": "vline_event_recorder_exception",
    "type":"vline_event_recorder_exception"
  },
  "dest": {
    "index": "vline_event_recorder_exception_test",
    "type": "_doc"
  }
}

附錄：ES客戶端封裝函數

import lombok.extern.slf4j.Slf4j;

import org.apache.http.HttpHost;
import org.apache.http.util.EntityUtils;
import org.elasticsearch.action.DocWriteResponse;
import org.elasticsearch.action.admin.indices.settings.put.UpdateSettingsRequest;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.search.SearchScrollRequest;
import org.elasticsearch.action.support.master.AcknowledgedResponse;
import org.elasticsearch.action.update.UpdateRequest;
import org.elasticsearch.action.update.UpdateResponse;
import org.elasticsearch.client.Request;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.Response;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.client.indices.CreateIndexRequest;
import org.elasticsearch.client.indices.CreateIndexResponse;
import org.elasticsearch.client.indices.GetIndexRequest;
import org.elasticsearch.client.indices.PutMappingRequest;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.reindex.BulkByScrollResponse;
import org.elasticsearch.index.reindex.DeleteByQueryRequest;
import org.elasticsearch.rest.RestStatus;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.SearchHits;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import com.alibaba.fastjson.serializer.SerializerFeature;
import com.kedacom.microservice.pojo.AbstractCommonIndex;

import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

/**
 * @author zhang.kai
 *
 */
@Slf4j
public class ESDataUtils {
    
    private static final int SCROLL_PAGE_SIZE = 1000; // 每次scroll的頁面大小
    
    private static final String DEFAULT_TYPE = "_doc";

    // lowLevelClient客戶端
    public static RestClient restClient;

    // RestHighLevelClient客戶端
    public static RestHighLevelClient highLevelClient;

    // 索引創建時的分片值，如爲-1則使用系統默認值
    public static int numberOfShards = -1;

    // 索引創建時的分片複製因子，如爲-1則使用系統默認值
    public static int numberOfReplicas = -1;

    public static SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss:SSS");
    public static SimpleDateFormat sdfForLog = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
    public static SimpleDateFormat sdfForName = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");

    // 內部類，返回分頁查詢結果
    public static class EsSearchResult {
        public List<String> recorders; // 查詢到的記錄
        public SearchResponse sr; // 執行查詢的結果對象，可能包含聚合指標
        public long total; // 文檔總數
    }

    /**
     * 根據配置中心獲取的es服務器地址與端口初始化RestClient與RestHighLevelClient
     * 
     * @param esHost es服務器地址
     * @param esPort es服務器端口
     */
    public static void initClient(String esHost, int esPort) {
        RestClientBuilder restClientBuilder = RestClient.builder(new HttpHost(esHost, esPort, "http"));
        ESDataUtils.restClient = restClientBuilder.build();
        ESDataUtils.highLevelClient = new RestHighLevelClient(restClientBuilder);
    }

    /**
     * 判斷索引是否存在
     * 
     * @param indexName
     * @return
     * @throws IOException
     */
    public static boolean ifIndexExists(String indexName) throws IOException {
        GetIndexRequest request = new GetIndexRequest(indexName);
        return highLevelClient.indices().exists(request, RequestOptions.DEFAULT);
    }

    /**
     * 創建ES索引,參數爲索引配置包裝類，此方法適用於已經創建索引配置類
     * 
     * @param index
     * @return
     * @throws IOException
     */
    public static boolean createIndex(AbstractCommonIndex index) throws IOException {
        String indexName = index.getIndexName();
        CreateIndexRequest request = new CreateIndexRequest(indexName);

        if (!ifIndexExists(indexName)) {
            if (null != index.getSettings()) {
                String settings = index.getSettings();
                JSONObject settingsJo = JSON.parseObject(settings);
                if (numberOfShards > 0) {
                    settingsJo.put("index.number_of_shards", numberOfShards);
                }
                if (numberOfReplicas > 0) {
                    settingsJo.put("index.number_of_replicas", numberOfReplicas);
                }
                request.settings(JSON.toJSONString(settingsJo, SerializerFeature.WriteMapNullValue), XContentType.JSON);
            }
            request.mapping(index.getMapping());
            log.info("createIndex {}:{}", indexName, request.toString());
            CreateIndexResponse createIndexResponse = highLevelClient.indices().create(request, RequestOptions.DEFAULT);
            if (!createIndexResponse.isAcknowledged()) {
                log.error("createIndex fail:{}", createIndexResponse);
            }
            return createIndexResponse.isAcknowledged();
        }
        log.info("{} has created!", indexName);
        return false;
    }

    /**
     * 創建ES索引,參數爲字符串形式，此方法適用於未創建索引配置類，直接適用json字符串作爲參數
     * 
     * @param indexName
     * @param mappingStr
     * @param settingsStr
     * @return
     * @throws IOException
     */
    public static boolean createIndex(String indexName, String mappingStr, String settingsStr) throws IOException {
        CreateIndexRequest request = new CreateIndexRequest(indexName);

        if (!ifIndexExists(indexName)) {
            if (null != settingsStr) {
                JSONObject settingsJo = JSON.parseObject(settingsStr);
                if (numberOfShards > 0) {
                    settingsJo.put("index.number_of_shards", numberOfShards);
                }
                if (numberOfReplicas > 0) {
                    settingsJo.put("index.number_of_replicas", numberOfReplicas);
                }
                request.settings(JSON.toJSONString(settingsJo, SerializerFeature.WriteMapNullValue), XContentType.JSON);
            }
            if (null != mappingStr) {
                request.mapping(mappingStr, XContentType.JSON);
            }
            log.info("createIndex {}:{}", indexName, request.toString());
            CreateIndexResponse createIndexResponse = highLevelClient.indices().create(request, RequestOptions.DEFAULT);
            if (!createIndexResponse.isAcknowledged()) {
                log.error("createIndex fail:{}", createIndexResponse);
            }
            return createIndexResponse.isAcknowledged();
        }
        log.info("{} has created!", indexName);
        return false;
    }

    /**
     * 修改索引映射，參數爲XContentBuilder
     * 
     * @param indexName
     * @param mappingStr
     * @return
     * @throws IOException
     */
    public static boolean updateMapping(String indexName, XContentBuilder mapBuilder) throws IOException {
        PutMappingRequest request = new PutMappingRequest(indexName);
        request.source(mapBuilder);

        log.info("putMapping:{},{}", indexName, request.source());
        AcknowledgedResponse putMappingResponse = highLevelClient.indices().putMapping(request, RequestOptions.DEFAULT);
        if (!putMappingResponse.isAcknowledged()) {
            log.error("updateMapping fail:{}", putMappingResponse.toString());
        }
        return putMappingResponse.isAcknowledged();
    }

    /**
     * 修改索引映射，參數爲String
     * 
     * @param indexName
     * @param mappingStr
     * @return
     * @throws IOException
     */
    public static boolean updateMapping(String indexName, String mappingStr) throws IOException {
        PutMappingRequest request = new PutMappingRequest(indexName);
        request.source(mappingStr, XContentType.JSON);

        log.info("putMapping:{},{}", indexName, request.source());
        AcknowledgedResponse putMappingResponse = highLevelClient.indices().putMapping(request, RequestOptions.DEFAULT);
        if (!putMappingResponse.isAcknowledged()) {
            log.error("updateMapping fail:{}", putMappingResponse.toString());
        }
        return putMappingResponse.isAcknowledged();
    }

    /**
     * 修改索引的設置,參數爲Map
     * 
     * @param indexName
     * @param settingsMap
     * @return
     * @throws IOException
     */
    public static boolean updateSettings(String indexName, Map<String, Object> settingsMap) throws IOException {
        UpdateSettingsRequest request = new UpdateSettingsRequest(indexName);
        request.settings(settingsMap);

        log.info("updateSettings {}:{}", indexName, settingsMap);
        AcknowledgedResponse updateSettingsResponse = highLevelClient.indices().putSettings(request,
                RequestOptions.DEFAULT);
        if (!updateSettingsResponse.isAcknowledged()) {
            log.error("updateSettings fail:{}", updateSettingsResponse);
        }
        return updateSettingsResponse.isAcknowledged();
    }

    /**
     * 修改索引的設置,參數爲String
     * 
     * @param indexName
     * @param settingsMap
     * @return
     * @throws IOException
     */
    public static boolean updateSettings(String indexName, String settingsStr) throws IOException {
        UpdateSettingsRequest request = new UpdateSettingsRequest(indexName);
        request.settings(settingsStr, XContentType.JSON);

        log.info("updateSettings {}:{}", indexName, settingsStr);
        AcknowledgedResponse updateSettingsResponse = highLevelClient.indices().putSettings(request,
                RequestOptions.DEFAULT);
        if (!updateSettingsResponse.isAcknowledged()) {
            log.error("updateSettings fail:{}", updateSettingsResponse);
        }
        return updateSettingsResponse.isAcknowledged();
    }

    /**
     * 向索引插入單條記錄
     * 
     * @param indexName
     * @param docId
     * @param docSource
     * @return
     * @throws IOException
     */
    public static boolean insertDoc(String indexName, String docId, String docSource) throws IOException {
        IndexRequest request = new IndexRequest(indexName, DEFAULT_TYPE);
        if (null != docId) {
            request.id(docId);
        }
        request.source(docSource, XContentType.JSON);
        IndexResponse indexResponse = highLevelClient.index(request, RequestOptions.DEFAULT);
        if ((indexResponse.status() == RestStatus.CREATED) || (indexResponse.status() == RestStatus.OK)) {
            return true;
        } else {
            log.error("insertDoc fail status:{} => {}", indexResponse.status(), indexResponse.toString());
            return false;
        }
    }

    /**
     * 向索引插入批量記錄
     * 
     * @param indexName
     * @param docId
     * @param docSource
     * @return
     * @throws IOException
     */
    public static boolean insertDocBulk(String indexName, List<String> docIds, List<String> docSources)
            throws IOException {
        if (null == docSources) {
            log.error("insertDocBulk docSources are null!");
            return false;
        }
        BulkRequest bulkRequest = new BulkRequest();
        for (int i = 0; i < docSources.size(); i++) {
            IndexRequest request = new IndexRequest(indexName, DEFAULT_TYPE);
            if ((null != docIds) && (null != docIds.get(i))) {
                request.id(docIds.get(i));
            }
            request.source(docSources.get(i), XContentType.JSON);
            bulkRequest.add(request);
        }
        if (bulkRequest.numberOfActions() > 0) {
            BulkResponse bulkResponse = highLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
            if (!bulkResponse.hasFailures()) {
                return true;
            } else {
                log.error("insertDocBulk fail status:{} => {}", bulkResponse.status(),
                        bulkResponse.buildFailureMessage());
                return false;
            }
        }
        return true;
    }

    /**
     * 修改es文檔中的某些字段值，docId必須存在
     * 
     * @param indexName
     * @param docId
     * @param updateFields
     * @return
     * @throws IOException
     */
    public static boolean updateDoc(String indexName, String docId, String updateFields) throws IOException {
        if (null == docId) {
            log.info("updateDoc {} fail cause docId is null:{}", indexName, updateFields);
            return false;
        }
        UpdateRequest request = new UpdateRequest(indexName, DEFAULT_TYPE, docId);
        request.doc(updateFields, XContentType.JSON);
        UpdateResponse updateResponse = highLevelClient.update(request, RequestOptions.DEFAULT);
        if (updateResponse.getResult() == DocWriteResponse.Result.UPDATED) {
            return true;
        } else {
            log.error("updateDoc {} fail by {} :{}", indexName, docId, updateResponse.toString());
            return false;
        }
    }

    /**
     * 執行ES查詢並支持分頁(超過10000條限制的分頁)
     * 
     * @param ssb
     * @param from
     * @param size
     * @param indexName
     * @return
     * @throws IOException
     */
    public static EsSearchResult executeEsSearch(SearchSourceBuilder ssb, int from, int size, String indexName)
            throws IOException {
        if (null == ssb) {
            log.error("executeEsSearch args is error!");
            return null;
        }
        EsSearchResult esr = new EsSearchResult();
        SearchResponse searchResponse = null;
        SearchRequest searchRequest = new SearchRequest(indexName);

        if (from + size <= 10000) { // 10000以內，正常執行查詢
            esr.total = getEsSearchTotal(ssb, indexName);

            ssb.from(from).size(size);
            log.info("executble es search dsl for {} is:{}", indexName, ssb.toString());
            searchRequest.source(ssb);
            searchResponse = highLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            SearchHits hits = searchResponse.getHits();
            List<String> docList = new ArrayList<String>();
            for (SearchHit hit : hits.getHits()) {
                docList.add(hit.getSourceAsString());
            }
            esr.recorders = docList;
            esr.sr = searchResponse;

        } else {// 超過10000，使用scrollid
            esr.total = getEsSearchTotal(ssb, indexName);

            ssb.size(SCROLL_PAGE_SIZE);
            log.info("executble es search dsl for {} is:{}", indexName, ssb.toString());
            searchRequest.source(ssb);
            searchRequest.scroll(TimeValue.timeValueMinutes(3L));
            searchResponse = highLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            SearchHits hits = searchResponse.getHits();
            esr.sr = searchResponse;
            // 如果所有記錄數小於等於請求起始數，返回null
            if ((null != searchResponse) && (esr.total <= from)) {
                log.info("total:{} is less than from:{}", esr.total, from);
                return esr;
            }
            int unFetchedIndex = 0; // 未取到的索引

            while ((null != searchResponse) && (searchResponse.status() == RestStatus.OK)) {
                List<String> curPageList = new ArrayList<String>();
                for (SearchHit hit : hits.getHits()) {
                    curPageList.add(hit.getSourceAsString());
                }

                unFetchedIndex += SCROLL_PAGE_SIZE;
                log.info("current unFetchedIndex is :{}->{}", unFetchedIndex, curPageList.get(0));
                if (unFetchedIndex > from) {
                    int startIndex = from % SCROLL_PAGE_SIZE;
                    // 只在本頁內取，由程序約束：比如一次scroll的size是1000,取得頁大小未10，20，50
                    List<String> docList = curPageList.subList(startIndex, startIndex + size);
                    esr.recorders = docList;
                    break;
                } else {// 繼續循環
                    String scrollId = searchResponse.getScrollId();
                    SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
                    scrollRequest.scroll(TimeValue.timeValueSeconds(180));
                    searchResponse = highLevelClient.scroll(scrollRequest, RequestOptions.DEFAULT);
                }
            }
        }
        return esr;

    }

    /**
     * 執行es查詢返回所有符合條件的結果
     * 
     * @param jestClient
     * @param ssb
     * @param indexName
     * @return
     * @throws IOException
     */
    public static EsSearchResult searchAllData(SearchSourceBuilder ssb, String indexName) throws IOException {
        if (null == ssb) {
            log.error("executeEsSearch args is error!");
            return null;
        }
        log.info("executble es search all data dsl for {} is:{}", indexName, ssb.toString());

        EsSearchResult esr = new EsSearchResult();
        esr.total = getEsSearchTotal(ssb, indexName);

        SearchRequest searchRequest = new SearchRequest(indexName);
        searchRequest.source(ssb);
        searchRequest.scroll(TimeValue.timeValueMinutes(3L));

        // 使用scrollid
        ssb.size(SCROLL_PAGE_SIZE);
        SearchResponse searchResponse = highLevelClient.search(searchRequest, RequestOptions.DEFAULT);
        SearchHits hits = searchResponse.getHits();
        esr.sr = searchResponse;

        List<String> totalData = new ArrayList<>();
        int unFetchedIndex = 0; // 未取到的索引
        while ((null != hits) && (hits.getHits().length > 0)) {
            List<String> curPageList = new ArrayList<String>();
            for (SearchHit hit : hits.getHits()) {
                curPageList.add(hit.getSourceAsString());
            }
            totalData.addAll(curPageList);
            unFetchedIndex += SCROLL_PAGE_SIZE;
            log.info("current unFetchedIndex is :{}->{}", unFetchedIndex, curPageList.get(0));
            String scrollId = searchResponse.getScrollId();
            SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
            scrollRequest.scroll(TimeValue.timeValueSeconds(180));
            searchResponse = highLevelClient.scroll(scrollRequest, RequestOptions.DEFAULT);
            hits = searchResponse.getHits();
        }
        esr.recorders = totalData;
        return esr;
    }

    /**
     * 根據請求的時間段刪除對應索引中的數據
     * 
     * @param indexName
     * @param qb
     * @return
     * @throws IOException
     */
    public static boolean deleteByQuery(String indexName, QueryBuilder qb) throws IOException {

        DeleteByQueryRequest request = new DeleteByQueryRequest(indexName);
        request.setQuery(qb);

        request.setConflicts("proceed");
        request.setBatchSize(5000);
        // 並行
        request.setSlices(5);
        // 使用滾動參數來控制“搜索上下文”存活的時間
        request.setScroll(TimeValue.timeValueMinutes(10));
        // 超時
        request.setTimeout(TimeValue.timeValueMinutes(2));
        // 刷新索引
        request.setRefresh(true);
        log.info("***deleteByQuery request uri:{}", request.toString());
        log.info("***deleteByQuery request body:{}", qb.toString());
        BulkByScrollResponse bulkResponse = highLevelClient.deleteByQuery(request, RequestOptions.DEFAULT);
        log.info("***handleEsIndexClean response:{}", bulkResponse.toString());
        return true;
    }

    /**
     * 使用low level api獲取ES查詢的total，需要兼容6.8.1與7.8.1版本
     * 
     * @param ssb
     * @param indexName
     * @return
     * @throws IOException
     */
    private static long getEsSearchTotal(SearchSourceBuilder ssb, String indexName) throws IOException {
        long total = 0;
        ssb.from(0).size(0);

        Request request = new Request("GET", "/" + indexName + "/_search");
        request.setJsonEntity(ssb.toString());
        Response response = restClient.performRequest(request);
        int statusCode = response.getStatusLine().getStatusCode();
        String responseBody = EntityUtils.toString(response.getEntity());

        if (statusCode == 200) {
            JSONObject responseJo = (JSONObject) JSON.parse(responseBody);
            JSONObject hitsJo = responseJo.getJSONObject("hits");
            Object totalJo = hitsJo.get("total");
            if (totalJo instanceof Integer) { // 6.8.1版本
                total = (long) ((Integer) totalJo);
            } else { // 7.8.1版本
                total = ((JSONObject) totalJo).getLongValue("value");
            }
        }
        return total;
    }

}