基於Kubernetes實現的大數據採集與存儲實踐總結

原創

2020-07-12 16:23

[^ ]*) (?[^ ]*)(?: \"(?[^\\\"]*)\" \"(?[^\\\"]*)\")?$\n Time_Key time\n Time_Format %d/%b/%Y:%H:%M:%S %z\n [PARSER]\n Name apache2\n Format regex\n Regex ^(?[^ ]*) [^ ]* (?[^ ]*) \\[(?[^\\]]*)\\] \"(?\\S+)(?: +(?[^ ]*) +\\S*)?\" (?[^ ]*) (?[^ ]*)(?: \"(?[^\\\"]*)\" \"(?[^\\\"]*)\")?$\n Time_Key time\n Time_Format %d/%b/%Y:%H:%M:%S %z\n [PARSER]\n Name apache_error\n Format regex\n Regex ^\\[[^ ]* (?[^\\]]*)\\] \\[(?[^\\]]*)\\](?: \\[pid (?[^\\]]*)\\])?( \\[client (?[^\\]]*)\\])? (?.*)$\n [PARSER]\n Name nginx\n Format regex\n Regex ^(?[^ ]*) (?[^ ]*) (?[^ ]*) \\[(?[^\\]]*)\\] \"(?\\S+)(?: +(?[^\\\"]*?)(?: +\\S*)?)?\" (?[^ ]*) (?[^ ]*)(?: \"(?[^\\\"]*)\" \"(?[^\\\"]*)\")?$\n Time_Key time\n Time_Format %d/%b/%Y:%H:%M:%S %z\n [PARSER]\n Name json-test\n Format json\n Time_Key time\n Time_Format %d/%b/%Y:%H:%M:%S %z\n [PARSER]\n Name docker\n Format json\n Time_Key time\n Time_Format %Y-%m-%dT%H:%M:%S.%L\n Time_Keep On\n [PARSER]\n Name syslog\n Format regex\n Regex ^\\[0-9]+)\\>(?[^ ]* {1,2}[^ ]* [^ ]*) (?[^ ]*) (?[a-zA-Z0-9_\\/\\.\\-]*)(?:\\[(?[0-9]+)\\])?(?:[^\\:]*\\:)? *(?.*)$\n Time_Key time\n Time_Format %b %d %H:%M:%S"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"部署文件"}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"apiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: fluentbit\n namespace: logging\n labels:\n k8s-app: fluentbit-logging\n version: v1\n kubernetes.io/cluster-service: \"true\"\nspec:\n selector:\n matchLabels:\n k8s-app: fluentbit-logging\n version: v1\n template:\n metadata:\n labels:\n k8s-app: fluentbit-logging\n version: v1\n kubernetes.io/cluster-service: \"true\"\n spec:\n containers:\n - name: fluentbit\n image: fluent/fluent-bit:1.2.1\n env:\n - name: FLUENTD_HOST\n value: \"fluentd\"\n - name: FLUENTD_PORT\n value: \"24224\"\n resources:\n limits:\n memory: 100Mi\n requests:\n cpu: 100m\n memory: 100Mi\n volumeMounts:\n - name: config\n mountPath: /fluent-bit/etc\n - name: varlog\n mountPath: /var/log\n - name: varlibdockercontainers\n mountPath: /var/lib/docker/containers\n readOnly: true\n - name: minikube\n mountPath: /mnt/sda1/var/lib/docker/containers\n readOnly: true\n terminationGracePeriodSeconds: 10\n volumes:\n - name: config\n configMap:\n name: fluentbit-config\n - name: varlog\n hostPath:\n path: /var/log\n - name: varlibdockercontainers\n hostPath:\n path: /var/lib/docker/containers\n - name: minikube\n hostPath:\n path: /mnt/sda1/var/lib/docker/containers\n"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"5.3 日誌聚合器Fluentd"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    Fluentd作爲日誌聚合器負責數據的聚合來自FluentBit的數據，並將其存儲到AWS S3中，需要準備項目配置以及deployment文件。"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"項目配置"}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"kind: ConfigMap\napiVersion: v1\nmetadata:\n name: fluentd-config\n namespace: logging\ndata:\n fluent.conf: |-\n \n log_level debug\n \n\n \n @type http\n port 9880\n bind 0.0.0.0\n body_size_limit 32m\n keepalive_timeout 10s\n \n \n @type forward\n port 24224\n bind 0.0.0.0\n tag kube.forward\n \n \n @type s3\npath \"year=%Y/month=%m/day=%d/hour=%H\"\n s3_bucket \n s3_region \n "}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"部署文件"}]},{"type":"codeblock","attrs":{"lang":"text"},"content":[{"type":"text","text":"\napiVersion: v1\nkind: Service\nmetadata:\n name: fluentd\n namespace: logging\n labels:\n app: fluentd\nspec:\n ports:\n - name: http\n port: 9880\n protocol: TCP\n - name: forword\n port: 24224\n protocol: TCP\n selector:\n app: fluentd\n type: NodePort\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: fluentd\n namespace: logging\n labels:\n app: fluentd\nspec:\n replicas: 1\n selector:\n matchLabels:\n app: fluentd\n strategy:\n rollingUpdate:\n maxSurge: 25%\n maxUnavailable: 25%\n type: RollingUpdate\n template:\n metadata:\n labels:\n app: fluentd\n spec:\n containers:\n - name: fluentd\n image: ghgt/fluentd-es-s3-rewrite:v1.7.3-alpine-es7-s3-rewrite-1.1\n env:\n - name: FLUENTD_ARGS\n value: --no-supervisor -q\n resources:\n limits:\n memory: 500Mi\n requests:\n cpu: 100m\n memory: 200Mi\n ports:\n - containerPort: 8080\n - containerPort: 24224\n volumeMounts:\n - name: varlog\n mountPath: /var/log\n - name: config-volume\n mountPath: /fluentd/etc\n terminationGracePeriodSeconds: 30\n volumes:\n - name: varlog\n hostPath:\n path: /var/log\n - name: config-volume\n configMap:\n name: fluentd-config"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"  到此，編碼完成！啓動並在集羣中運行三個服務，隨後檢查你的AWS S3存儲桶吧！"}]},{"type":"horizontalrule"},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","marks":[{"type":"strong"}],"text":"總結"}]},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null},"content":[{"type":"text","text":"    寫到最後，相信你已經瞭解如何使用FluentBit、Fluentd等工具結合Kubernetes實現對大數據的採集和存儲流程，筆者從基礎原理到方案架構，再到實踐過程，通過簡單的Sample示例帶領大家領悟企業級的項目實現，希望讀完文章的你有所收穫。在接下來的文章中，筆者將繼續帶領大家關注雲原生領域的知識，敬請期待~"}]},{"type":"image","attrs":{"src":"https://static001.geekbang.org/infoq/1a/1a6175a84b764112b853cc7b7ebd3e1f.jpeg","alt":null,"title":"","style":[{"key":"width","value":"100%"},{"key":"bordertype","value":"none"}],"href":"","fromPaste":false,"pastePass":false}},{"type":"paragraph","attrs":{"indent":0,"number":0,"align":null,"origin":null}}]}