[JAVA] elasticsearch scrollId 很長原因

elasticsearch 分頁的時候有
1.深度分頁(from-size) 分頁的偏移值越大，執行分頁查詢時間就會越長
2.快照分頁(scroll) 該查詢會自動返回一個scrollId

由於數據量比較大. 所以採用了快照分頁(scroll) 在開發環境沒有任何問題. 但是在測試環境,發現scrollId的長度已經超出了get請求url的最大長度. 導致scrollId傳不到後臺.

於是追究其原因

1.準備環境:windows /elasticsearch-5.6.9/ 修改jvm.options [-Dfile.encoding=UTF-8] 爲GBK
注意: 5.X版本和2.X版本的配置項有比較大的差異, 請自行百度

2.java代碼

public class ElasticsearchTest {

    private final static String HOST = "127.0.0.1";
    private final static int PORT = 9300;
    private TransportClient client = null;

    /**
     * 獲取客戶端連接信息
     * 默認配置
     */
    @Before
    public void getConnect() throws UnknownHostException {
        client = new PreBuiltTransportClient(Settings.EMPTY).addTransportAddresses(
                new InetSocketTransportAddress(InetAddress.getByName(HOST), PORT));
        Logger.info("連接信息:" + client.toString());
    }

    //    @Before
    public void before() throws UnknownHostException {
        Map<String, String> map = new HashMap<>();
        map.put("cluster.name", "elasticsearch");
        Settings settings = Settings.builder().put(map).build();
        client = new PreBuiltTransportClient(settings).addTransportAddress(
                new InetSocketTransportAddress(InetAddress.getByName(HOST), PORT));
        Logger.info("連接信息:" + client.toString());
    }

    /**
     * 關閉連接
     */
    @After
    public void closeConnect() {
        if (null != client) {
            Logger.info("執行關閉連接操作...");
            client.close();
        }
    }

    /**
     * 創建索引庫
     * 需求:創建一個索引庫爲：msg消息隊列,類型爲：tweet,id爲1
     * 索引庫的名稱必須爲小寫
     */
    @Test
    public void addIndex1() throws IOException {
        addOne("1");
    }

    /**
     * 循環添加數據
     */
    @Test
    public void addIndex2() throws IOException {
        for (int i = 100; i < 600; i++) {
            addOne(i + "");
        }
    }

    private void addOne(String i) throws IOException {
        SimpleDateFormat s = new SimpleDateFormat("yyyy-MM-dd HH:mm:sss");
        client.prepareIndex("msg", "tweet")
//        client.prepareIndex("msg", "tweet1")
//        client.prepareIndex("msg1", "tweet")
                .setSource(XContentFactory.jsonBuilder()
                        .startObject().field("name", "張三")
                        .field("date", new Date())
                        .field("fmtTime", s.format(new Date()))
                        .field("msg", "中文_" + i)
                        .endObject()).get();
    }

    /**
     * 從索引庫獲取數據
     */
    @Test
    public void getData1() {
        String[] indexArray = new String[0];
        SearchResponse response = client.prepareSearch(indexArray)
                .setFrom(0)
                .setSize(100)
                .addSort("date", SortOrder.ASC)
                .setScroll(new TimeValue(1000 * 60 * 60))
                .get();

        for (SearchHit searchHit : response.getHits().getHits()) {
            Logger.info(searchHit.getId() + "" + searchHit.getSourceAsString());
        }
        Logger.info("scrollId: " + response.getScrollId());
        Logger.info("scrollId.length:" + response.getScrollId().length());

    }

    /**
     * 從索引庫獲取數據
     */
    @Test
    public void getData2() {
        String scrollId = "1";
        SearchResponse response = client.prepareSearchScroll(scrollId)
                .setScroll(new TimeValue(1000 * 60 * 60)).get();

        for (SearchHit searchHit : response.getHits().getHits()) {
            Logger.info(searchHit.getId() + "" + searchHit.getSourceAsString());
        }
        Logger.info("scrollId: " + response.getScrollId());
        Logger.info("scrollId.length:" + response.getScrollId().length());
    }

    /**
     * 根據索引名稱，類別，文檔ID 刪除索引庫的數據
     */
    @Test
    public void deleteData() {
        DeleteResponse deleteResponse = client.prepareDelete("msg", "tweet", "1").get();

        Logger.info("deleteResponse索引名稱:" + deleteResponse.getIndex()
                + "\t deleteResponse類型:" + deleteResponse.getType()
                + "\t deleteResponse文檔ID:" + deleteResponse.getId()
                + "\t當前實例deleteResponse狀態:" + deleteResponse.status());
    }

    @Test
    public void deleteIndex() {
        client.admin().indices().prepareDelete("msg").execute().actionGet();
        Logger.info("刪除索引成功");
    }
}

分別批量增加 document , type , index 測試發現得. 當增加index的時候scrollId會增加.

解決方案(嘗試):
1.改get請求爲post請求.但是scrollId的長度達到3w+字符.每次需要夾帶幾KB的數據在請求中,而且scrollId可能將來會不斷變大
2.將scrollId緩存,設置過期時間.每次請求下一次查詢後清除該scrollId的緩存.

博友還有什麼新的解決方案可以在我的博客下留言.