Ozone S3接口

Ozone提供了兼容S3的REST接口，可以使用任何兼容S3的工具來操作Ozone的對象存儲數據。

當前使用的Ozone版本：0.4.1-alpha

1、啓動s3g

$ nohup ozone s3g  >> s3g.log &

2、Ozone已完成的REST endpoints

2.1. S3Gateway service 端點

Endpoint	Status
GET service	implemented

2.2. Bucket端點

Endpoint	Status	Notes
GET Bucket (List Objects) Version 2	implemented
HEAD Bucket	implemented
DELETE Bucket	implemented
PUT Bucket (Create bucket)	implemented
Delete Multiple Objects (POST)	implemented

2.3. Objects端點

Endpoint	Status	Notes
PUT Object	implemented
GET Object	implemented
Multipart Upload	implemented	Except the listing of the current MultiPartUploads.
DELETE Object	implemented
HEAD Object	implemented

3、栗子

3.1. 安裝工具

# yum install awscli s3cmd –y

3.2. 命令行操作

需要配置ak和sk，由於未開啓安全認證，ak和sk可以自行指定

# aws s3api --endpoint http://172.16.40.21:9878/ list-buckets 
Unable to locate credentials. You can configure credentials by running "aws configure".
# aws configure
AWS Access Key ID [None]: dlw
AWS Secret Access Key [None]: dmx
Default region name [None]: 
Default output format [None]: 
# aws s3api --endpoint http://172.16.40.21:9878/ list-buckets
{
    "Buckets": []
}

s3cmd需要配置.s3cfg

# vim ~/.s3cfg
[default]
access_key = dlw
secret_key = dmx

3.3. 創建bucket

# aws s3api --endpoint http://172.16.40.21:9878/ create-bucket --bucket=bdc
{
    "Location": "http://172.16.40.21:9878/bdc"
}
# aws s3api --endpoint http://172.16.40.21:9878/ create-bucket --bucket=dlw
{
    "Location": "http://172.16.40.21:9878/dlw"
}

3.4. 查看bucket

# aws s3api --endpoint http://172.16.40.21:9878/ list-buckets
{
    "Buckets": [
        {
            "CreationDate": "2020-06-23T09:14:28.287Z", 
            "Name": "bdc"
        }, 
        {
            "CreationDate": "2020-06-23T10:07:39.580Z", 
            "Name": "dlw"
        }
    ]
}

# s3cmd  ls
2020-06-23 09:14  s3://bdc
2020-06-23 10:07  s3://dlw

3.5. 刪除bucket

# aws s3api --endpoint http://172.16.40.21:9878/ delete-bucket --bucket=dlw

3.6. 上傳object

# aws s3api --endpoint http://172.16.40.21:9878/ put-object --key=epel.repo --body=epel.repo --bucket=bdc

3.7. 查看object

# aws s3api --endpoint http://172.16.40.21:9878/ list-objects --bucket=bdc
{
    "Contents": [
        {
            "LastModified": "2020-06-23T09:40:57.927Z", 
            "ETag": "1592905257927", 
            "StorageClass": "STANDARD", 
            "Key": "epel.repo", 
            "Size": 664
        }
    ]
}

S3cmd需要在~/.s3cfg中添加bucket才能查看 host_bucket = 172.16.40.21:9878/bdc

# s3cmd  ls s3://bdc
2020-06-23 09:40          664  s3://bdc/epel.repo

3.8. 刪除object

# aws s3api --endpoint http://172.16.40.21:9878/ put-object --key=hosts --body=/etc/hosts --bucket=bdc

# aws s3api --endpoint http://172.16.40.21:9878/ list-objects --bucket=bdc
{
    "Contents": [
        {
            "LastModified": "2020-06-23T09:40:57.927Z", 
            "ETag": "1592905257927", 
            "StorageClass": "STANDARD", 
            "Key": "epel.repo", 
            "Size": 664
        }, 
        {
            "LastModified": "2020-06-23T10:13:04.659Z", 
            "ETag": "1592907184659", 
            "StorageClass": "STANDARD", 
            "Key": "hosts", 
            "Size": 254
        }
    ]
}
# aws s3api --endpoint http://172.16.40.21:9878/ delete-object --key=hosts --bucket=bdc

# aws s3api --endpoint http://172.16.40.21:9878/ list-objects --bucket=bdc            
{
    "Contents": [
        {
            "LastModified": "2020-06-23T09:40:57.927Z", 
            "ETag": "1592905257927", 
            "StorageClass": "STANDARD", 
            "Key": "epel.repo", 
            "Size": 664
        }
    ]
}

4、 瀏覽器查看bucket

在bucket的 URL 中加上 ?browser=true，就可以在瀏覽器中瀏覽bucket。 比如，bdc桶中的內容可以在瀏覽器中用 http://172.16.40.21:9878/bdc?browser=true 來查看。

5、 Bucket映射

S3 bucket和ozone bucket是有映射關係的

查看S3的bdc bucket在ozone中的位置

$ ozone s3 path bdc
Volume name for S3Bucket is : s394eafd1d8b59ff3a991e9f226b023a29
Ozone FileSystem Uri is : o3fs://bdc.s394eafd1d8b59ff3a991e9f226b023a29

新建一個test bucket

# aws s3api --endpoint http://172.16.40.21:9878/ create-bucket --bucket=test
{
    "Location": "http://172.16.40.21:9878/test"
}
$ ozone s3 path test
Volume name for S3Bucket is : s394eafd1d8b59ff3a991e9f226b023a29
Ozone FileSystem Uri is : o3fs://test.s394eafd1d8b59ff3a991e9f226b023a29

可以看到2個bucket都在名爲s394eafd1d8b59ff3a991e9f226b023a29的卷下面

使用ozone shell查看 可以找到這個volume

$ ozone sh vol info s394eafd1d8b59ff3a991e9f226b023a29
{
  "owner" : {
    "name" : "94eafd1d8b59ff3a991e9f226b023a29"
  },
  "quota" : {
    "unit" : "TB",
    "size" : 1048576
  },
  "volumeName" : "s394eafd1d8b59ff3a991e9f226b023a29",
  "createdOn" : "Tue, 23 Jun 2020 09:14:28 GMT",
  "createdBy" : "94eafd1d8b59ff3a991e9f226b023a29"
}

同樣可以看到這個volume的bucket，也就是S3創建的bucket

$ ozone sh bucket ls s394eafd1d8b59ff3a991e9f226b023a29
[ {
  "volumeName" : "s394eafd1d8b59ff3a991e9f226b023a29",
  "bucketName" : "bdc",
  "createdOn" : "Tue, 23 Jun 2020 09:14:28 GMT",
  "acls" : null,
  "versioning" : "DISABLED",
  "storageType" : "DISK",
  "encryptionKeyName" : "N/A"
}, {
  "volumeName" : "s394eafd1d8b59ff3a991e9f226b023a29",
  "bucketName" : "test",
  "createdOn" : "Tue, 23 Jun 2020 10:30:21 GMT",
  "acls" : null,
  "versioning" : "DISABLED",
  "storageType" : "DISK",
  "encryptionKeyName" : "N/A"
} ]

加上完整的url

$ ozone sh bucket ls o3:/s394eafd1d8b59ff3a991e9f226b023a29
[ {
  "volumeName" : "s394eafd1d8b59ff3a991e9f226b023a29",
  "bucketName" : "bdc",
  "createdOn" : "Tue, 23 Jun 2020 09:14:28 GMT",
  "acls" : null,
  "versioning" : "DISABLED",
  "storageType" : "DISK",
  "encryptionKeyName" : "N/A"
}, {
  "volumeName" : "s394eafd1d8b59ff3a991e9f226b023a29",
  "bucketName" : "test",
  "createdOn" : "Tue, 23 Jun 2020 10:30:21 GMT",
  "acls" : null,
  "versioning" : "DISABLED",
  "storageType" : "DISK",
  "encryptionKeyName" : "N/A"
} ]

查看key，可以看到傳上去的epel.repo文件

$ ozone sh key ls o3:/s394eafd1d8b59ff3a991e9f226b023a29/bdc 
[ {
  "version" : 0,
  "md5hash" : null,
  "createdOn" : "Tue, 23 Jun 2020 09:40:57 GMT",
  "modifiedOn" : "Tue, 23 Jun 2020 09:40:57 GMT",
  "size" : 664,
  "keyName" : "epel.repo",
  "type" : null
} ]

參考鏈接

https://hadoop.apache.org/ozone/docs/0.5.0-beta/interface/s3.html