通過API提取數據並不是ELK Stack用戶最常見的用例之一，但這種情況對於有些案例非常有用。例如，使用REST API包裝其數據庫服務的開發人員可能對出於商業智能目的而分析此數據感興趣。在今天的文章中，我們來講述如何使用ELK堆棧來對一個Service API來進行分析。

無論出於何種原因，ELK Stack都提供了一些簡單的方法來與該API集成。這些方法之一是Logstash HTTP poller 輸入插件。在下面的示例中，我將使用此插件獲取一些通過公共API公開的天氣數據。

要執行這些步驟併爲ELK中的HTTP API分析構建自己的管道，您需要進行以下設置：

Elastic：菜鳥上手指南：安裝好自己的Elasticsearch及Kibana
如何安裝Elastic棧中的Logstash：安裝好自己的Logstash
在 https://openweathermap.org/api 鏈接註冊一個賬號。你可以在https://openweathermap.org/guide處找到註冊的方式及在https://home.openweathermap.org/api_keys 找到自己的API keys。

準備數據

如上所述，我們使用OpenWeatherMap API收集當前的天氣數據。該數據集包括從40,000個氣象站收集的有關全世界現有天氣狀況的統計信息。

數據本身可以JSON，XML或HTML格式提供，但默認格式爲JSON，這非常適合我們的用例，因爲我們正在使用Elasticsearch索引數據。調用數據的方式有多種-您可以通過提供城市名稱（我們將在此處使用的方法），城市ID，地理座標或郵政編碼來進行調用。您還可以在特定座標內呼叫多個城市。

下面的示例在一個包含經度和緯度座標的邊界框中調用天氣統計信息：

http://api.openweathermap.org/data/2.5/box/city?bbox=12,32,15,37,10&appid=YourAppKey

在上面， bbox定義爲[lon-left,lat-bottom,lon-right,lat-top,zoom]，也即它表示一個長方形的區域。在上面的appid中，我們必須填入自己的App Key纔可以工作。

API返回的數據爲我們提供了測量當前天氣狀況所需的所有統計信息，包括溫度，溼度，大氣壓力，風速和風向等。

這是一個例子：

{"cod":200,"calctime":0.00256473,"cnt":15,"list":[{"id":2563191,"dt":1581912559,"name":"Birkirkara","coord":{"Lon":14.46,"Lat":35.9},"main":{"temp":8,"feels_like":6.94,"temp_min":8,"temp_max":8,"pressure":1028,"humidity":93},"visibility":10000,"wind":{"speed":0.5},"rain":null,"snow":null,"clouds":{"today":1},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2210247,"dt":1581912558,"name":"Tripoli","coord":{"Lon":13.19,"Lat":32.88},"main":{"temp":14.62,"feels_like":12.78,"temp_min":14.62,"temp_max":14.62,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":59},"wind":{"speed":1.53,"deg":42},"rain":null,"snow":null,"clouds":{"today":12},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02n"}]},{"id":2216885,"dt":1581912558,"name":"Zawiya","coord":{"Lon":12.73,"Lat":32.75},"main":{"temp":10.33,"feels_like":8.14,"temp_min":10.33,"temp_max":10.33,"pressure":1027,"sea_level":1027,"grnd_level":1015,"humidity":77},"wind":{"speed":1.96,"deg":87},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2212771,"dt":1581912558,"name":"Sabratah","coord":{"Lon":12.49,"Lat":32.79},"main":{"temp":14.59,"feels_like":12.27,"temp_min":14.59,"temp_max":14.59,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":57},"wind":{"speed":2.05,"deg":57},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2215163,"dt":1581912558,"name":"Masallatah","coord":{"Lon":14,"Lat":32.62},"main":{"temp":8.75,"feels_like":6.93,"temp_min":8.75,"temp_max":8.75,"pressure":1027,"sea_level":1027,"grnd_level":995,"humidity":77},"wind":{"speed":0.98,"deg":309},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2219905,"dt":1581912558,"name":"Al Khums","coord":{"Lon":14.26,"Lat":32.65},"main":{"temp":12.93,"feels_like":11.96,"temp_min":12.93,"temp_max":12.93,"pressure":1027,"sea_level":1027,"grnd_level":1023,"humidity":71},"wind":{"speed":0.65,"deg":51},"rain":{"3h":0.19},"snow":null,"clouds":{"today":27},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10n"}]},{"id":2208425,"dt":1581912558,"name":"Zuwarah","coord":{"Lon":12.08,"Lat":32.93},"main":{"temp":12.65,"feels_like":10.83,"temp_min":12.65,"temp_max":12.65,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":64},"wind":{"speed":1.3,"deg":63},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2210221,"dt":1581912558,"name":"Tarhuna","coord":{"Lon":13.63,"Lat":32.44},"main":{"temp":7.58,"feels_like":6.11,"temp_min":7.58,"temp_max":7.58,"pressure":1027,"sea_level":1027,"grnd_level":992,"humidity":85},"wind":{"speed":0.56,"deg":142},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2208485,"dt":1581912558,"name":"Zlitan","coord":{"Lon":14.57,"Lat":32.47},"main":{"temp":12.93,"feels_like":11.96,"temp_min":12.93,"temp_max":12.93,"pressure":1027,"sea_level":1027,"grnd_level":1023,"humidity":71},"wind":{"speed":0.65,"deg":51},"rain":{"3h":0.19},"snow":null,"clouds":{"today":27},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10n"}]},{"id":2217362,"dt":1581912558,"name":"Gharyan","coord":{"Lon":13.02,"Lat":32.17},"main":{"temp":5.59,"feels_like":4.01,"temp_min":5.59,"temp_max":5.59,"pressure":1027,"sea_level":1027,"grnd_level":940,"humidity":90},"wind":{"speed":0.4,"deg":209},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523693,"dt":1581912495,"name":"Pozzallo","coord":{"Lon":14.85,"Lat":36.73},"main":{"temp":5.56,"feels_like":2.34,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2524119,"dt":1581912495,"name":"Modica","coord":{"Lon":14.77,"Lat":36.85},"main":{"temp":5.73,"feels_like":2.54,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2208791,"dt":1581912558,"name":"Yafran","coord":{"Lon":12.53,"Lat":32.06},"main":{"temp":7.01,"feels_like":5.31,"temp_min":7.01,"temp_max":7.01,"pressure":1027,"sea_level":1027,"grnd_level":959,"humidity":84},"wind":{"speed":0.68,"deg":209},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523581,"dt":1581912494,"name":"Rosolini","coord":{"Lon":14.95,"Lat":36.82},"main":{"temp":5.32,"feels_like":2.06,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523650,"dt":1581912495,"name":"Ragusa","coord":{"Lon":14.72,"Lat":36.93},"main":{"temp":5.52,"feels_like":2.29,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":22},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02n"}]}]}

配置Logstash

將通過API返回的數據導入到ELK堆棧中的方法是使用Logstash http poller輸入插件，該插件可以從定義的終結點URL聚合，解碼和運送數據。我們首先來創建一個名字叫做logstash_openweathermap.conf的配置文件。

Logstash input

輸入部分定義了http_poller輸入插件-輪詢的URL端點，請求超時，基於CRON的時間表（每5分鐘一次）和要使用的編解碼器（JSON）。 metadata_target設置是可選的，並將某些字段添加到有關輪詢器性能的響應中。

input {
  http_poller {
    urls => {
      url => "http://api.openweathermap.org/data/2.5/weather?q=London,uk&APPID=7dbe7341764f682c2242e744c4f167b0&units=metric"
    }
    request_timeout => 60
    schedule => { every => "5m"}
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}

我們要在此處插入的API在調用URL後包含以下查詢參數：

q =London，uk–要求返回倫敦的天氣數據
APPID = – OpenWeatherMap API密鑰
units = metrics –將單位格式轉換爲攝氏度

上面的url中顯示的是微服務的接口。它返回的結果是：

{
  "coord": {
    "lon": -0.13,
    "lat": 51.51
  },
  "weather": [
    {
      "id": 802,
      "main": "Clouds",
      "description": "scattered clouds",
      "icon": "03n"
    }
  ],
  "base": "stations",
  "main": {
    "temp": 6.7,
    "feels_like": 0.46,
    "temp_min": 5,
    "temp_max": 8.33,
    "pressure": 1006,
    "humidity": 65
  },
  "visibility": 10000,
  "wind": {
    "speed": 6.2,
    "deg": 230
  },
  "clouds": {
    "all": 40
  },
  "dt": 1581916140,
  "sys": {
    "type": 1,
    "id": 1414,
    "country": "GB",
    "sunrise": 1581923520,
    "sunset": 1581959845
  },
  "timezone": 0,
  "id": 2643743,
  "name": "London",
  "cod": 200
}

你可以使用https://jsonformatter.org/json-viewer來格式化 API返回的結果。

Logstash filter

由於API響應使用JSON，因此Logstash幾乎不需要進行任何處理或解析。因此，我們現在可以將過濾器部分留空。

filter {}

Logstash output

下面的這部分就非常直接了：

output {
   	elasticsearch {
     	index => "openweather"
     	document_type => "_doc"
     	hosts => "localhost:9200"
	} 
 }

在上面，我們把數據導入到localhost:9200的Elasticsearch中，並且索引的名字是openweather。

綜合上面的配置，我們最終的longstash_openweather.conf如下：

input {
  http_poller {
    urls => {
      url => "http://api.openweathermap.org/data/2.5/weather?q=London,uk&APPID=7dbe7341764f682c2242e744c4f167b0&units=metric"
    }
    request_timeout => 60
    schedule => { every => "1m"}
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}

filter {}

output {
  stdout {
    codec => rubydebug
    }

  elasticsearch {
     	index => "openweather"
     	document_type => "_doc"
     	hosts => "localhost:9200"
  }
}

啓動Logstash。我們在Logstash的安裝目錄運行logstash：

./bin/logstash -f ~/data/logstash_openweather.conf

您將看到使用天氣數據創建的新Elasticsearch索引。在Kibana中定義新的索引模式以開始分析：

分析數據

我們可以通過創建一個叫做openweather的index pattern。然後可以通過Discover查看：

運用 openweather index pattern，我們可以做出來我們的dashboard：

Logstash：使用ELK堆棧進行API分析

準備數據

配置Logstash

Logstash input

Logstash filter

Logstash output

分析數據

AI 畫圖真刺激，手把手教你如何用 ComfyUI 來畫出刺激的圖

公司剛入職了一名 Java 中級開發，短短 4 行代碼居然湊齊了 3 個 bug！我哭了~~

數據展示動態（跑分）顯示

公衆號5月C#/.NET熱文一覽

git 下載大陸鏡像地址

Logstash：使用ELK堆棧進行API分析

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結