如何打造基层警营文化品牌如何打造基于s3cmd的短地址服务

软件收集云计算

2022-12-07 0 29

如何打造基于s3cmd的短地址服务

这篇文章次要讲解了“如何打造基于s3cmd的短地址服务”文中的讲解内容简单清晰易于学习与理解下面请大家跟着小编的思路慢慢深入一起来研究和学习“如何打造基于s3cmd的短地址服务”吧！

需求描述

有线上项目使用S3去发布移动端APP由于需要做权限认证所以对应的Object无法开“public-read”权限尝试使用Presign方式生成的签名url存在以下问题

生成的签名URL地址包含敏感信息会暴露bucket名称和accesskey带来安全隐患。

生成的签名URL地址太长影响用户体验。

生成的地址无效时间内可以任意访问无法做到比较严格的反盗链。

处理方软件下载案

使用openresty实现对S3数据访问流程的封装客户端的数据下载全部走openresty上面搭建的服务该服务提供一次性的短地址生成服务访问几次当前对应短地址失效具体流程如下:

1. 用户数据上传到S3并设置相应Object权限为“public-read”取得对应的对外访问URL。

2. 以之前生成的对外访问URL为基础生成对应的短地址服务。

3. 客户端使用短地址进行访问超出访问次数则无法访问。

如果想进一步提升安全性可以将RGW和Openresty通过内网互联数据上传全部走内部网络外部访问走外网。

具体实现

1. Openresty安装2.安装依赖lua库文件

默认openre软件下载sty的lib路径为/usr/local/openresty/lualib

需要依赖两个lua文件其中url.lua次要用于url地址的解析redis_iresty.lua用于redis的连接将下面的源码文件保存为 /usr/local/openresty/lualib/resty/url.lua
https://github./golgote/neturl/blob/master/lib/net/url.lua

将下面的源码文件保存为 /usr/local/openresty/lualib/resty/redis_iresty.lua
https://gist.gith软件下载ub./moonbingbing/9915c66346e8fddcefb53. RGW服务配置rootdemohost:/etc/openresty# cat /etc/ceph/ceph.conf

…

[client.radosgw.demo]
     rgw dns name = s3.cephbook. #S3 endpoint对应的域名
     rgw frontends = “civetweb port=7480”host = demohost
     keyring = /etc/ceph/ceph.软件下载client.radosgw.keyringlog file = /home/ceph/log/radosgw.log4. Openresty配置rootdemohost:/etc/openresty# cat /etc/openresty/nginx.conf

…

server {
        listen       80;
        server_name  ceph.vip; #短地址主机头设置
        location = /url { #生成短地址入口
             content_by_lua_file /etc/openresty/url.l软件下载ua;
         }
         location / {#提供短地址对应的数据访问
             proxy_http_version 1.1;
             proxy_set_header Host $host;
             access_by_lua_file /etc/openresty/access.lua;
             proxy_pass http://127.0.0.1:7480; #对应后端的radosgw服务入口
         }
     }

生成短地址服务的软件下载源码如下

rootdemohost:/etc/openresty# cat url.lua
local request_method = ngx.var.request_method
local url_ = require “resty.url” #对应之前的url库
local redis = require “resty.redis_iresty” #对应之前的redis_iresty库
local s3_endpoint = “s3.cephbook.” #这里设置只允许生成与S3 endpoint相关的短地址
local endpoint = s3_endpoint .. “$”

local charset = {}
f软件下载or i = 48, 57 do table.insertcharset, string.chari end
for i = 65, 90 do table.insertcharset, string.chari end
for i = 97, 122 do table.insertcharset, string.chari end

function string.randomlength
  local urandom = assertio.open/dev/urandom,rb
  local a, b, c, d = urandom:read4:byte1,4
  urandom:close软件下载local seed = a*0x1000000 + b*0x10000 + c *0x100 + d
  math.randomseedseed

  if length > 0 then
    return string.randomlength – 1 .. charset[math.random1, #charset]
  else
    return “”
  end
end

function genera_urlred,url_id,host,path
    — counts表示短地址访问次数current表示当前次数
    ok, err = red:hmseturl_id,host,host,uri,path,counts,3,cur软件下载rent,1
    if not ok then
        ngx.status = ngx.HTTP_METHOD_NOT_IMPLEMENTED
        ngx.say“failed to hmset: “, err
        return ngx.exitngx.HTTP_METHOD_NOT_IMPLEMENTED
    end
    — 设置短地址记录最多能够在redis里面存储的时长避免资源耗尽
    ok, err = red:expireurl_id,3600
    if not ok then
        ngx.status= ngx.HTTP_METHOD_NOT_IMPLEMENTED
        n软件下载gx.say“failed to set expire: “, err
        return ngx.exitngx.HTTP_METHOD_NOT_IMPLEMENTED
    end
end

function get_info_by_idred,url_id
    ok, err = red:hgetallurl_id
    if not ok then
        ngx.status = ngx.HTTP_SERVICE_UNAVAILABLE
        ngx.say“failed to getall : “, err
        return ngx.exitngx.HTTP_SERVICE_UNAVAILABLE软件下载end
    local h = red:array_to_hashok
    return h
end

if request_method == “POST” then
    ngx.req.read_body
    local data = ngx.req.get_body_data
    local u = url_.parsedata
    local hostname_check, hostname_err = ngx.re.matchu.host,endpoint
    if not hostname_check then
        ngx.status = ngx.HTTP_BAD_REQUEST
        ngx.say“not软件下载 a available s3_endpoint”
        ngx.exitngx.HTTP_BAD_REQUEST
    end
    if u.host then
        if string.lenu.path > 1  then
            local url_id = string.random7
            local red = redis:new
            genera_urlred,url_id,u.host,u.path
            local h = get_info_by_idred,url_id
            ngx.status= ngx.HTTP_软件下载CREATED
            ngx.say“ShortURL: “,ngx.var.scheme,“://”,ngx.var.host,“/”,url_id
            ngx.say“host: “, h.host
            ngx.say“uri: “, h.uri
            ngx.say“counts: “, h.counts
            ngx.say“current: “, h.current
     软件下载       ngx.exitngx.HTTP_CREATED
        else
            ngx.status = ngx.HTTP_BAD_REQUEST
            ngx.say“path err”
            ngx.exitngx.HTTP_BAD_REQUEST
        end
    else
        ngx.status = ngx.HTTP_BAD_REQUEST
        ngx.say“not a available url”
        ngx.exitngx.HTTP_BAD_REQUEST
    end
end

访软件下载问短地址数据服务的源码如下

rootdemohost:/etc/openresty# cat /etc/openresty/access.lua
local redis = require “resty.redis_iresty”
local red = redis:new
local uri =  ngx.var.uri
local url_id = string.suburi,2,string.lenuri
res, err = red:hgetallurl_id
if not res then
    ngx.exitngx.HTTP_NOT_FOUND
else
    localh = red:ar软件下载ray_to_hashres— 超出访问次数则拒绝访问如果考虑资源占用可以本人加上删除对应redis记录操作
    if h.counts < h.current then
        ngx.status = ngx.HTTP_GONE
        ngx.say“current=”,h.current,” > counts=”,h.counts
        ngx.exitngx.HTTP_GONE
    end
    ngx.req.set_header“host”, h.host
    ngx.req.set_urih.uri, false
软件下载     ok, err = red:hincrbyurl_id,current,1
    if not ok then
        ngx.status = ngx.HTTP_METHOD_NOT_IMPLEMENTED
        ngx.say“incrby key failed: “, err
        ngx.exitngx.HTTP_METHOD_NOT_IMPLEMENTED
        return
    end
end

github./openresty/lua-nginx-module#http-status-constants

5. 使用流程上传数软件下载据

使用s3cmd或者是其他方式上传文件并设置对应的Object访问权限为”public-read”。

rootdemohost:/tmp# s3cmd put myfile s3://demo –acl-public
myfile -> s3://demo/myfile  [1 of 1]
21577 of 21577   100% in    0s   227.71 kB/s  done
myfile -> s3://demo/myfile  [1 of 1]
21577 of 21577   100% in    0s   203.11 kB/s  done
Public URL of the object is: demo.s3.cephbook./myfile生成短地址root软件下载demohost:/tmp# curl ceph.vip/url  -d “demo.s3.cephbook./myfile” -v
* Hostname was NOT found in DNS cache
* Connected to ceph.vip  port 80 #0
> POST /url HTTP/1.1
> User-Agent: curl/7.38.0
> Host: ceph.vip
> Accept: */*
> Content-Length: 29
> Content-Type: application/x–form-urlencoded
>
* upload pletely sent off: 29 out of 29 bytes软件下载
< HTTP/1.1 201 Created
* Server openresty/1.11.2.5 is not blacklisted
< Server: openresty/1.11.2.5
< Date: Wed, 15 Nov 2017 09:56:14 GMT
< Content-Type: application/octet-stream
< Transfer-Encoding: chunked
< Connection: keep-alive
<
ShortURL: ceph.vip/Vo3t5Ex #生成的短地址
host: demo.s3.cephbook.
uri: /myfile
counts: 3
current: 1
* Conn软件下载ection #0 to host ceph.vip left intact测试访问#访问3次
rootdemohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v
…
rootdemohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v
…
rootdemohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v
…

#第四次当前就不能访问了

root软件下载demohost:/tmp# curl http://ceph.vip/Vo3t5Ex -v
* Hostname was NOT found in DNS cache
* Trying ceph.vip…
* Connected to ceph.vip ceph.vip port 80 #0
> GET /Vo3t5Ex HTTP/1.1
> User-Agent: curl/7.38.0
> Host: ceph.vip
> Accept: */*
>
< H软件下载TTP/1.1 410 Gone
* Server openresty/1.11.2.5 is not blacklisted
< Server: openresty/1.11.2.5
< Date: Wed, 15 Nov 2017 10:00:46 GMT
< Content-Type: application/octet-stream
< Transfer-Encoding: chunked
< Connection: keep-alive
软件下载 <
current=4 > counts=3
* Connection #0 to host ceph.vip left intact