Elasticsearch筆記(十一) ES term terms prefix 搜索 聚合查詢 詳細總結

1 前提

本人從17年在工作中接觸ES，但是到現在感覺沒有入門，主要是一直使用ES的JavaAPI去做簡單業務邏輯開發，並沒有認真看過ES的文檔，對ES的理解還很淺。本着“教是最好的學”，特別想整理下ES查詢的常用API，尤其看了下面ES開發者佔比，感覺尤爲強烈，因爲我不屬於其中一種（我=年齡大+工資低+頭髮少）。

2 準備數據

PUT /pigg/_doc/1
{
  "name": "老亞瑟",
  "age": 31,
  "sex": "男",
  "word": "死亡騎士，不是死掉的騎士",
  "weapon": ["黑切", "冰痕之握", "反傷刺甲","閃電匕首","破軍"]
}

PUT /pigg/_doc/2
{
  "name": "孫悟空",
  "age": 40,
  "sex": "男",
  "word": "我就是吉吉國王",
  "weapon": ["黑切", "冰痕之握", "無盡戰刃", "宗師之力"]
}

PUT /pigg/_doc/3
{
  "name": "安琪拉",
  "age": 16,
  "sex": "女",
  "word": "我就是小蘿莉",
  "weapon": []
}

PUT /pigg/_doc/4
{
  "name": "老夫子",
  "age": 100,
  "sex": "男",
  "word": "我要定住你"
}

3 涼菜–這個都不會？一首涼涼送給你

對ES不熟悉可先看Elasticsearch筆記(九) term terms exists 查詢案例

1 term

查詢name=“老亞瑟”的數據

GET /pigg/_search
{
  "query": {
    "term": {
      "name": {
        "value": "老亞瑟"
      }
    }
  },
  "_source": ["name"]
}

這個時候我們發下結果如下，沒有數據

{
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}

因爲我們沒有自己定義mapping，這裏name是text類型，“老亞瑟”被ES默認分詞爲“老”，“亞”，“瑟”這3個字，所以找不到。
term是包含的意思，查詢name裏包含“老”的數據：

GET /pigg/_search
{
  "query": {
    "term": {
      "name": {
        "value": "老"
      }
    }
  },
  "_source": ["name"]
}

返回結果如下，可以看到“老夫子”和“老亞瑟”都匹配中。

{
    "hits" : [
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : 0.6931472,
        "_source" : {
          "name" : "老夫子"
        }
      },
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.2876821,
        "_source" : {
          "name" : "老亞瑟"
        }
      }
    ]

name默認下面有個keyword字段，就是name.keyword，它沒有被分詞。

GET /pigg/_search
{
  "query": {
    "term": {
      "name.keyword": {
        "value": "老亞瑟"
      }
    }
  },
  "_source": ["name"]
}

結果如下，通過keyword類型可以精確查詢

    "hits" : [
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.2876821,
        "_source" : {
          "name" : "老亞瑟"
        }
      }
    ]

2 terms

terms是命中一個就算匹配，查詢有黑切或者宗師之力的人

GET /pigg/_search
{
  "query": {
    "terms": {
      "weapon.keyword": [
        "黑切",
        "宗師之力"
      ]
    }
  },
  "_source": ["name", "weapon"]
}

返回結果如下：

    "hits" : [
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "weapon" : [
            "黑切",
            "冰痕之握",
            "無盡戰刃",
            "宗師之力"
          ],
          "name" : "孫悟空"
        }
      },
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "weapon" : [
            "黑切",
            "冰痕之握",
            "反傷刺甲",
            "閃電匕首",
            "破軍"
          ],
          "name" : "老亞瑟"
        }
      }
    ]

3 prefix

prefix前綴查詢，在工作中很常見，就行MySQL裏的 like “abc%”。
查詢name以“老”開頭的人：

GET /pigg/_search
{
  "query": {
    "prefix": {
      "name.keyword": {
        "value": "老"
      }
    }
  },
  "_source": ["name"]
}

結果如下：

  "hits" : [
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "name" : "老夫子"
        }
      },
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "name" : "老亞瑟"
        }
      }
    ]

4 wildcard

wildcard查詢就像MySQL的Like查詢，它查詢效率比較低，一般也不用
查詢name裏包含“亞”的人：

GET /pigg/_search
{
  "query": {
    "wildcard": {
      "name.keyword": {
        "value": "*亞*"
      }
    }
  },
  "_source": ["name"]
}

5 range

range是範圍查詢，查詢age在[10,30]的人

GET /pigg/_search
{
  "query": {
    "range": {
      "age": {
        "gte": 10,
        "lte": 30
      }
    }
  },
  "_source": ["name"]
}

返回如下：

    "hits" : [
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "name" : "安琪拉"
        }
      }
    ]

6 exists

查詢weapon字段有值的人：

GET /pigg/_search
{
  "query": {
    "exists": {
        "field": "weapon"
    }
  },
  "_source": ["name"]
}

查詢weapon字段沒有值的人：

GET /pigg/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "exists": {
            "field": "weapon"
          }
        }
      ]
    }
  },
  "_source": ["name"]
}

結果如下：其中老夫子沒有weapon這個字段，而安琪拉的weapon=[]。

    "hits" : [
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "name" : "老夫子"
        }
      },
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "name" : "安琪拉"
        }
      }
    ]

4 油燜大蝦–把condition組合起來一鍋燜

1 bool

bool 過濾器是個 複合過濾器，它可以接受多個其他過濾器作爲參數，並將這些過濾器結合成各式各樣的布爾（邏輯）組合。
它格式如下：

{
   "bool" : {
      "must" :     [],
      "should" :   [],
      "must_not" : [],
   }
}

2 must

查詢name以“老”開頭的，並且age>=90的人

GET /pigg/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "prefix": {
            "name": {
              "value": "老"
            }
          }
        },
        {
          "range": {
            "age": {
              "gte": 90
            }
          }
        }
      ]
    }
  },
   "_source": ["name","age"]
}

查詢結果如下，畢竟我們的亞瑟王怎麼可能那麼老

 "hits" : [
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : 2.0,
        "_source" : {
          "name" : "老夫子",
          "age" : 100
        }
      }
    ]

3 must_not

must_not和must相反，是非的意思，查詢買了武器但是沒有買無盡戰刃的人

GET /pigg/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "weapon.keyword": {
              "value": "無盡戰刃"
            }
          }
        }
      ],
      "must": [
        {
          "exists": {
            "field": "weapon"
          }
        }
      ]
    }
  },
  "_source": ["name", "weapon"]
}

4 should

should是或的意思
查詢是女的，或者word包含“吉吉國王”的人

GET /pigg/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "term": {
            "sex": {
              "value": "女"
            }
          }
        },
        {
          "match": {
            "word": "吉吉國王"
          }
        }
      ]
    }
  },
  "_source": ["name","sex", "word"]
}

返回如下：

 "hits" : [
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 3.1186123,
        "_source" : {
          "sex" : "男",
          "name" : "孫悟空",
          "word" : "我就是吉吉國王"
        }
      },
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "3",
        "_score" : 0.2876821,
        "_source" : {
          "sex" : "女",
          "name" : "安琪拉",
          "word" : "我就是小蘿莉"
        }
      }
    ]

當should和must或者must_not在同一層的時候，它不會影響結果，但影響匹配分數。

GET /pigg/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "sex.keyword": {
              "value": "男"
            }
          }
        }
      ],
      "should": [
        {
          "range": {
            "age": {
              "gte": 90
            }
          }
        }
      ]
    }
  },
  "_source": ["name","sex", "age"]
}

結果如下：大家都是男人，但是老夫子的年齡>=90,他的_score=1.1823215，比另外2人高。

  "hits" : [
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : 1.1823215,
        "_source" : {
          "sex" : "男",
          "name" : "老夫子",
          "age" : 100
        }
      },
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.2876821,
        "_source" : {
          "sex" : "男",
          "name" : "老亞瑟",
          "age" : 31
        }
      },
      {
        "_index" : "pigg",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 0.18232156,
        "_source" : {
          "sex" : "男",
          "name" : "孫悟空",
          "age" : 40
        }
      }
    ]

5 filter

filter過濾查詢，它不評分，效率高，網上介紹filter的文章很多，在此我就不瞎BB了。

GET /pigg/_search
{
  "query": {
    "bool": {
      "filter": {
        "term": {
          "sex.keyword": "男"
        }
      }
    }
  },
  "_source": ["name","sex"]
}

5 阿凡提羊肉串–給查詢加點料

1 過濾字段 _source

#只返回"name", "sex"2個字段
GET /pigg/_search
{
  "query": {
    "match_all": {}
  },
  "_source": ["name", "sex"]
}

#只返回w開頭的字段
GET /pigg/_search
{
  "query": {
    "match_all": {}
  },
  "_source": ["w*"]
}

#只返回w開頭並且不是n結尾的字段
GET /pigg/_search
{
  "query": {
    "match_all": {}
  },
  "_source": {
    "includes": "w*",
    "excludes": "*n"
  }
}

2 排序 sort

GET /pigg/_search
{
  "sort": [
     {
      "sex.keyword": {
        "order": "desc"
      }
    },
    {
      "age": {
        "order": "desc"
      }
    }
  ],
  "_source": ["name","sex","age"]
}

3 分頁查詢 from+size

分頁功能很常用，from從0開始，如果數據量很大，有深分頁的問題。雖然有時公司喜歡改max_result_window這個參數，調的很多。額。。。，能查出來，你高興就好。
如果數據量很大，分頁讀取數據並處理，可以考慮scroll，網上文章很多，我就不BB了。

GET /pigg/_search
{
  "from": 0,
  "size": 2, 
  "sort": [
    {
      "sex.keyword": {
        "order": "desc"
      }
    }
  ],
  "_source": ["name","sex"]
}

6 肉末茄子–Aggs服務員，你過來統計下肉末

1 count

統計滿足條件的數量

GET /pigg/_count
{
  "query": {
    "term": {
      "sex.keyword": {
        "value": "男"
      }
    }
  }
}

2 terms聚合

terms聚合，就像GROUP BY

POST /_xpack/sql?format=txt
{
  "query": "SELECT sex, COUNT(*) num FROM pigg GROUP BY sex ORDER BY num desc" 
}

統計各裝備的使用數量，並排序

GET /pigg/_search
{
  "aggs": {
    "terms_by_weapon": {
      "terms": {
        "field": "weapon.keyword",
        "size": 10,
         "order" : { "_count" : "asc" }
      }
    }
  }
}

結果如下：

      "buckets" : [
        {
          "key" : "反傷刺甲",
          "doc_count" : 1
        },
        {
          "key" : "宗師之力",
          "doc_count" : 1
        },
        {
          "key" : "無盡戰刃",
          "doc_count" : 1
        },
        {
          "key" : "破軍",
          "doc_count" : 1
        },
        {
          "key" : "閃電匕首",
          "doc_count" : 1
        },
        {
          "key" : "冰痕之握",
          "doc_count" : 2
        },
        {
          "key" : "黑切",
          "doc_count" : 2
        }
      ]

3 having，聚合完，再過濾

統計使用量>=2的裝備

GET /pigg/_search
{
    "size": 0,
    "aggs":{
        "terms_by_weapon":{
            "terms":{
                "field":"weapon.keyword",
                "size":10
            },
            "aggs":{
                "having":{
                    "bucket_selector":{
                        "buckets_path":{
                            "weaponCount":"_count"
                        },
                        "script":{
                            "lang":"expression",
                            "inline":"weaponCount >= 2"
                        }
                    }
                }
            }
        }
    }
}

返回結果如下：

      "buckets" : [
        {
          "key" : "冰痕之握",
          "doc_count" : 2
        },
        {
          "key" : "黑切",
          "doc_count" : 2
        }
      ]

4 先過濾後，再聚合

先限定age<=90,然後按照sex分組，再求各性別的平均age

GET /pigg/_search
{
  "size": 5,
  "query": {
    "bool": {
      "filter": {
        "range": {
          "age": {
            "lte": 90
          }
        }
      }
    }
  }, 
  "_source": ["name","sex","age"],
  "aggs": {
    "terms_by_sex": {
      "terms": {
        "field": "sex.keyword",
        "size": 10
      },
      "aggs":{
        "avg_age":{
          "avg": {
            "field": "age"
          }
        }
      }
    }
  }
}

7 扇子骨-collapse聽說你想摺疊起來

1 collapse摺疊查詢

GET /pigg/_search
{
  "query": {
    "range": {
      "age": {
        "gte": 10,
        "lte": 90
      }
    }
  },
  "collapse": {
    "field": "sex.keyword",
    "inner_hits":{
      "name": "old_age",
      "size": 1,
      "sort": [{"age": "desc"}]
    }
  },
  "sort": [
    {
      "age": {
        "order": "desc"
      }
    }
  ]
}

8 青椒肉絲-Explain服務員，解釋下爲啥只有青椒

1 explain

explain參數可以接受DSL的語句，_validate驗證DSL是否合法。

GET /pigg/_validate/query?explain
{
  "query": {
    "terms": {
      "weapon.keyword": [
        "黑切",
        "宗師之力"
      ]
    }
  }
}

返回如下：

  "valid" : true,
  "explanations" : [
    {
      "index" : "pigg",
      "valid" : true,
      "explanation" : "weapon.keyword:(宗師之力 黑切)"
    }
  ]

9 總結一句話

上面也就算ES的一些皮毛，ES功能很多，想一次性學完不可能，只能在工作中在閒暇時間學習積累，
少玩些農藥，多學習吧。