s3实现视频分片预览

Posted on 2023-08-21 Edited on 2023-09-20 In 应用软件 Views:

在一些支持分段获取文件的服务中，通过Range获取指定分片，可以分段获取文件而不用将文件整个下载下来。

`s3` 对象存储

微服务架构下，通常会将s3对象存储作为后端存储，而更多的是使用云厂商的对象存储，例如亚马逊s3、腾讯COS、七牛云OSS，这些云存储都有完善的SDK。还有一些会使用开源软件提供的对象存储协议和接口，例如Ceph支持对象存储，Minio支持将带有文件系统的目录映射成对象存储，也能对外提供对象存储协议和接口。

AWS SDK for Go

https://github.com/aws/aws-sdk-go

示例上传文件代码：

 package main

 import (
     "context"
     "flag"
     "fmt"
     "os"
     "time"

     "github.com/aws/aws-sdk-go/aws"
     "github.com/aws/aws-sdk-go/aws/awserr"
     "github.com/aws/aws-sdk-go/aws/request"
     "github.com/aws/aws-sdk-go/aws/session"
     "github.com/aws/aws-sdk-go/service/s3"
 )

 // Uploads a file to S3 given a bucket and object key. Also takes a duration
 // value to terminate the update if it doesn't complete within that time.
 //
 // The AWS Region needs to be provided in the AWS shared config or on the
 // environment variable as `AWS_REGION`. Credentials also must be provided
 // Will default to shared config file, but can load from environment if provided.
 //
 // Usage:
 //   # Upload myfile.txt to myBucket/myKey. Must complete within 10 minutes or will fail
 //   go run withContext.go -b mybucket -k myKey -d 10m < myfile.txt
 func main() {
     var bucket, key string
     var timeout time.Duration

     flag.StringVar(&bucket, "b", "", "Bucket name.")
     flag.StringVar(&key, "k", "", "Object key name.")
     flag.DurationVar(&timeout, "d", 0, "Upload timeout.")
     flag.Parse()

     // All clients require a Session. The Session provides the client with
    // shared configuration such as region, endpoint, and credentials. A
    // Session should be shared where possible to take advantage of
    // configuration and credential caching. See the session package for
    // more information.
     sess := session.Must(session.NewSession())

    // Create a new instance of the service's client with a Session.
    // Optional aws.Config values can also be provided as variadic arguments
    // to the New function. This option allows you to provide service
    // specific configuration.
     svc := s3.New(sess)

     // Create a context with a timeout that will abort the upload if it takes
     // more than the passed in timeout.
     ctx := context.Background()
     var cancelFn func()
     if timeout > 0 {
         ctx, cancelFn = context.WithTimeout(ctx, timeout)
     }
     // Ensure the context is canceled to prevent leaking.
     // See context package for more information, https://golang.org/pkg/context/
if cancelFn != nil {
         defer cancelFn()
}

     // Uploads the object to S3. The Context will interrupt the request if the
     // timeout expires.
     _, err := svc.PutObjectWithContext(ctx, &s3.PutObjectInput{
         Bucket: aws.String(bucket),
         Key:    aws.String(key),
         Body:   os.Stdin,
     })
     if err != nil {
         if aerr, ok := err.(awserr.Error); ok && aerr.Code() == request.CanceledErrorCode {
             // If the SDK can determine the request or retry delay was canceled
             // by a context the CanceledErrorCode error code will be returned.
             fmt.Fprintf(os.Stderr, "upload canceled due to timeout, %v\n", err)
         } else {
             fmt.Fprintf(os.Stderr, "failed to upload object, %v\n", err)
         }
         os.Exit(1)
     }

     fmt.Printf("successfully uploaded file to %s/%s\n", bucket, key)
 }

现在出了v2版本，更推荐使用高版本

AWS SDK for Go v2

https://github.com/aws/aws-sdk-go-v2

https://aws.github.io/aws-sdk-go-v2/docs/

示例代码

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/dynamodb"
)

func main() {
    // Using the SDK's default configuration, loading additional config
    // and credentials values from the environment variables, shared
    // credentials, and shared configuration files
    cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("us-west-2"))
    if err != nil {
        log.Fatalf("unable to load SDK config, %v", err)
    }

    // Using the Config value, create the DynamoDB client
    svc := dynamodb.NewFromConfig(cfg)

    // Build the request with its input parameters
    resp, err := svc.ListTables(context.TODO(), &dynamodb.ListTablesInput{
        Limit: aws.Int32(5),
    })
    if err != nil {
        log.Fatalf("failed to list tables, %v", err)
    }

    fmt.Println("Tables:")
    for _, tableName := range resp.TableNames {
        fmt.Println(tableName)
    }
}

使用SDK的两种方式

直接通过SDK实现上传和下载

直接通过SDK实现上传和下载的方式在aws-sdk-go和aws-sdk-go-v2中都可实现，入参和出参不同。

上传文件

// Adds an object to a bucket. You must have WRITE permissions on a bucket to add
// an object to it. Amazon S3 never adds partial objects; if you receive a success
// response, Amazon S3 added the entire object to the bucket. You cannot use
// PutObject to only update a single piece of metadata for an existing object. You
// must put the entire object with updated metadata if you want to update some
// values. Amazon S3 is a distributed system. If it receives multiple write
// requests for the same object simultaneously, it overwrites all but the last
// object written. To prevent objects from being deleted or overwritten, you can
// use Amazon S3 Object Lock (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html)
// . To ensure that data is not corrupted traversing the network, use the
// Content-MD5 header. When you use this header, Amazon S3 checks the object
// against the provided MD5 value and, if they do not match, returns an error.
// Additionally, you can calculate the MD5 while putting an object to Amazon S3 and
// compare the returned ETag to the calculated MD5 value.
//   - To successfully complete the PutObject request, you must have the
//     s3:PutObject in your IAM permissions.
//   - To successfully change the objects acl of your PutObject request, you must
//     have the s3:PutObjectAcl in your IAM permissions.
//   - To successfully set the tag-set with your PutObject request, you must have
//     the s3:PutObjectTagging in your IAM permissions.
//   - The Content-MD5 header is required for any request to upload an object with
//     a retention period configured using Amazon S3 Object Lock. For more information
//     about Amazon S3 Object Lock, see Amazon S3 Object Lock Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html)
//     in the Amazon S3 User Guide.
//
// You have three mutually exclusive options to protect data using server-side
// encryption in Amazon S3, depending on how you choose to manage the encryption
// keys. Specifically, the encryption key options are Amazon S3 managed keys
// (SSE-S3), Amazon Web Services KMS keys (SSE-KMS), and customer-provided keys
// (SSE-C). Amazon S3 encrypts data with server-side encryption by using Amazon S3
// managed keys (SSE-S3) by default. You can optionally tell Amazon S3 to encrypt
// data at by rest using server-side encryption with other key options. For more
// information, see Using Server-Side Encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html)
// . When adding a new object, you can use headers to grant ACL-based permissions
// to individual Amazon Web Services accounts or to predefined groups defined by
// Amazon S3. These permissions are then added to the ACL on the object. By
// default, all objects are private. Only the owner has full access control. For
// more information, see Access Control List (ACL) Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)
// and Managing ACLs Using the REST API (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-using-rest-api.html)
// . If the bucket that you're uploading objects to uses the bucket owner enforced
// setting for S3 Object Ownership, ACLs are disabled and no longer affect
// permissions. Buckets that use this setting only accept PUT requests that don't
// specify an ACL or PUT requests that specify bucket owner full control ACLs, such
// as the bucket-owner-full-control canned ACL or an equivalent form of this ACL
// expressed in the XML format. PUT requests that contain other ACLs (for example,
// custom grants to certain Amazon Web Services accounts) fail and return a 400
// error with the error code AccessControlListNotSupported . For more information,
// see Controlling ownership of objects and disabling ACLs (https://docs.aws.amazon.com/AmazonS3/latest/userguide/about-object-ownership.html)
// in the Amazon S3 User Guide. If your bucket uses the bucket owner enforced
// setting for Object Ownership, all objects written to the bucket by any account
// will be owned by the bucket owner. By default, Amazon S3 uses the STANDARD
// Storage Class to store newly created objects. The STANDARD storage class
// provides high durability and high availability. Depending on performance needs,
// you can specify a different Storage Class. Amazon S3 on Outposts only uses the
// OUTPOSTS Storage Class. For more information, see Storage Classes (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html)
// in the Amazon S3 User Guide. If you enable versioning for a bucket, Amazon S3
// automatically generates a unique version ID for the object being stored. Amazon
// S3 returns this ID in the response. When you enable versioning for a bucket, if
// Amazon S3 receives multiple write requests for the same object simultaneously,
// it stores all of the objects. For more information about versioning, see Adding
// Objects to Versioning-Enabled Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/AddingObjectstoVersioningEnabledBuckets.html)
// . For information about returning the versioning state of a bucket, see
// GetBucketVersioning (https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketVersioning.html)
// . For more information about related Amazon S3 APIs, see the following:
//   - CopyObject (https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html)
//   - DeleteObject (https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html)
func (c *Client) PutObject(ctx context.Context, params *PutObjectInput, optFns ...func(*Options)) (*PutObjectOutput, error) {
    if params == nil {
        params = &PutObjectInput{}
    }

    result, metadata, err := c.invokeOperation(ctx, "PutObject", params, optFns, c.addOperationPutObjectMiddlewares)
    if err != nil {
        return nil, err
    }

    out := result.(*PutObjectOutput)
    out.ResultMetadata = metadata
    return out, nil
}

入参

type PutObjectInput struct {

    // The bucket name to which the PUT action was initiated. When using this action
    // with an access point, you must direct requests to the access point hostname. The
    // access point hostname takes the form
    // AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. When using this
    // action with an access point through the Amazon Web Services SDKs, you provide
    // the access point ARN in place of the bucket name. For more information about
    // access point ARNs, see Using access points (https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-access-points.html)
    // in the Amazon S3 User Guide. When you use this action with Amazon S3 on
    // Outposts, you must direct requests to the S3 on Outposts hostname. The S3 on
    // Outposts hostname takes the form
    // AccessPointName-AccountId.outpostID.s3-outposts.Region.amazonaws.com . When you
    // use this action with S3 on Outposts through the Amazon Web Services SDKs, you
    // provide the Outposts access point ARN in place of the bucket name. For more
    // information about S3 on Outposts ARNs, see What is S3 on Outposts (https://docs.aws.amazon.com/AmazonS3/latest/userguide/S3onOutposts.html)
    // in the Amazon S3 User Guide.
    //
    // This member is required.
    Bucket *string

    // Object key for which the PUT action was initiated.
    //
    // This member is required.
    Key *string

    // The canned ACL to apply to the object. For more information, see Canned ACL (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#CannedACL)
    // . This action is not supported by Amazon S3 on Outposts.
    ACL types.ObjectCannedACL

    // Object data.
    Body io.Reader

    // Specifies whether Amazon S3 should use an S3 Bucket Key for object encryption
    // with server-side encryption using AWS KMS (SSE-KMS). Setting this header to true
    // causes Amazon S3 to use an S3 Bucket Key for object encryption with SSE-KMS.
    // Specifying this header with a PUT action doesn’t affect bucket-level settings
    // for S3 Bucket Key.
    BucketKeyEnabled bool

    // Can be used to specify caching behavior along the request/reply chain. For more
    // information, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9)
    // .
    CacheControl *string

    // Indicates the algorithm used to create the checksum for the object when using
    // the SDK. This header will not provide any additional functionality if not using
    // the SDK. When sending this header, there must be a corresponding x-amz-checksum
    // or x-amz-trailer header sent. Otherwise, Amazon S3 fails the request with the
    // HTTP status code 400 Bad Request . For more information, see Checking object
    // integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
    // in the Amazon S3 User Guide. If you provide an individual checksum, Amazon S3
    // ignores any provided ChecksumAlgorithm parameter.
    ChecksumAlgorithm types.ChecksumAlgorithm

    // This header can be used as a data integrity check to verify that the data
    // received is the same data that was originally sent. This header specifies the
    // base64-encoded, 32-bit CRC32 checksum of the object. For more information, see
    // Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
    // in the Amazon S3 User Guide.
    ChecksumCRC32 *string

    // This header can be used as a data integrity check to verify that the data
    // received is the same data that was originally sent. This header specifies the
    // base64-encoded, 32-bit CRC32C checksum of the object. For more information, see
    // Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
    // in the Amazon S3 User Guide.
    ChecksumCRC32C *string

    // This header can be used as a data integrity check to verify that the data
    // received is the same data that was originally sent. This header specifies the
    // base64-encoded, 160-bit SHA-1 digest of the object. For more information, see
    // Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
    // in the Amazon S3 User Guide.
    ChecksumSHA1 *string

    // This header can be used as a data integrity check to verify that the data
    // received is the same data that was originally sent. This header specifies the
    // base64-encoded, 256-bit SHA-256 digest of the object. For more information, see
    // Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
    // in the Amazon S3 User Guide.
    ChecksumSHA256 *string

    // Specifies presentational information for the object. For more information, see
    // https://www.rfc-editor.org/rfc/rfc6266#section-4 (https://www.rfc-editor.org/rfc/rfc6266#section-4)
    // .
    ContentDisposition *string

    // Specifies what content encodings have been applied to the object and thus what
    // decoding mechanisms must be applied to obtain the media-type referenced by the
    // Content-Type header field. For more information, see
    // https://www.rfc-editor.org/rfc/rfc9110.html#field.content-encoding (https://www.rfc-editor.org/rfc/rfc9110.html#field.content-encoding)
    // .
    ContentEncoding *string

    // The language the content is in.
    ContentLanguage *string

    // Size of the body in bytes. This parameter is useful when the size of the body
    // cannot be determined automatically. For more information, see
    // https://www.rfc-editor.org/rfc/rfc9110.html#name-content-length (https://www.rfc-editor.org/rfc/rfc9110.html#name-content-length)
    // .
    ContentLength int64

    // The base64-encoded 128-bit MD5 digest of the message (without the headers)
    // according to RFC 1864. This header can be used as a message integrity check to
    // verify that the data is the same data that was originally sent. Although it is
    // optional, we recommend using the Content-MD5 mechanism as an end-to-end
    // integrity check. For more information about REST request authentication, see
    // REST Authentication (https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html)
    // .
    ContentMD5 *string

    // A standard MIME type describing the format of the contents. For more
    // information, see https://www.rfc-editor.org/rfc/rfc9110.html#name-content-type (https://www.rfc-editor.org/rfc/rfc9110.html#name-content-type)
    // .
    ContentType *string

    // The account ID of the expected bucket owner. If the bucket is owned by a
    // different account, the request fails with the HTTP status code 403 Forbidden
    // (access denied).
    ExpectedBucketOwner *string

    // The date and time at which the object is no longer cacheable. For more
    // information, see https://www.rfc-editor.org/rfc/rfc7234#section-5.3 (https://www.rfc-editor.org/rfc/rfc7234#section-5.3)
    // .
    Expires *time.Time

    // Gives the grantee READ, READ_ACP, and WRITE_ACP permissions on the object. This
    // action is not supported by Amazon S3 on Outposts.
    GrantFullControl *string

    // Allows grantee to read the object data and its metadata. This action is not
    // supported by Amazon S3 on Outposts.
    GrantRead *string

    // Allows grantee to read the object ACL. This action is not supported by Amazon
    // S3 on Outposts.
    GrantReadACP *string

    // Allows grantee to write the ACL for the applicable object. This action is not
    // supported by Amazon S3 on Outposts.
    GrantWriteACP *string

    // A map of metadata to store with the object in S3.
    Metadata map[string]string

    // Specifies whether a legal hold will be applied to this object. For more
    // information about S3 Object Lock, see Object Lock (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html)
    // .
    ObjectLockLegalHoldStatus types.ObjectLockLegalHoldStatus

    // The Object Lock mode that you want to apply to this object.
    ObjectLockMode types.ObjectLockMode

    // The date and time when you want this object's Object Lock to expire. Must be
    // formatted as a timestamp parameter.
    ObjectLockRetainUntilDate *time.Time

    // Confirms that the requester knows that they will be charged for the request.
    // Bucket owners need not specify this parameter in their requests. For information
    // about downloading objects from Requester Pays buckets, see Downloading Objects
    // in Requester Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html)
    // in the Amazon S3 User Guide.
    RequestPayer types.RequestPayer

    // Specifies the algorithm to use to when encrypting the object (for example,
    // AES256).
    SSECustomerAlgorithm *string

    // Specifies the customer-provided encryption key for Amazon S3 to use in
    // encrypting data. This value is used to store the object and then it is
    // discarded; Amazon S3 does not store the encryption key. The key must be
    // appropriate for use with the algorithm specified in the
    // x-amz-server-side-encryption-customer-algorithm header.
    SSECustomerKey *string

    // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321.
    // Amazon S3 uses this header for a message integrity check to ensure that the
    // encryption key was transmitted without error.
    SSECustomerKeyMD5 *string

    // Specifies the Amazon Web Services KMS Encryption Context to use for object
    // encryption. The value of this header is a base64-encoded UTF-8 string holding
    // JSON with the encryption context key-value pairs. This value is stored as object
    // metadata and automatically gets passed on to Amazon Web Services KMS for future
    // GetObject or CopyObject operations on this object.
    SSEKMSEncryptionContext *string

    // If x-amz-server-side-encryption has a valid value of aws:kms , this header
    // specifies the ID of the Amazon Web Services Key Management Service (Amazon Web
    // Services KMS) symmetric encryption customer managed key that was used for the
    // object. If you specify x-amz-server-side-encryption:aws:kms , but do not provide
    // x-amz-server-side-encryption-aws-kms-key-id , Amazon S3 uses the Amazon Web
    // Services managed key to protect the data. If the KMS key does not exist in the
    // same account issuing the command, you must use the full ARN and not just the ID.
    SSEKMSKeyId *string

    // The server-side encryption algorithm used when storing this object in Amazon S3
    // (for example, AES256, aws:kms ).
    ServerSideEncryption types.ServerSideEncryption

    // By default, Amazon S3 uses the STANDARD Storage Class to store newly created
    // objects. The STANDARD storage class provides high durability and high
    // availability. Depending on performance needs, you can specify a different
    // Storage Class. Amazon S3 on Outposts only uses the OUTPOSTS Storage Class. For
    // more information, see Storage Classes (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html)
    // in the Amazon S3 User Guide.
    StorageClass types.StorageClass

    // The tag-set for the object. The tag-set must be encoded as URL Query
    // parameters. (For example, "Key1=Value1")
    Tagging *string

    // If the bucket is configured as a website, redirects requests for this object to
    // another object in the same bucket or to an external URL. Amazon S3 stores the
    // value of this header in the object metadata. For information about object
    // metadata, see Object Key and Metadata (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html)
    // . In the following example, the request header sets the redirect to an object
    // (anotherPage.html) in the same bucket: x-amz-website-redirect-location:
    // /anotherPage.html In the following example, the request header sets the object
    // redirect to another website: x-amz-website-redirect-location:
    // http://www.example.com/ For more information about website hosting in Amazon S3,
    // see Hosting Websites on Amazon S3 (https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html)
    // and How to Configure Website Page Redirects (https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html)
    // .
    WebsiteRedirectLocation *string

    noSmithyDocumentSerde
}

直接通过函数上传的方式可以看到，文件的一些元数据信息和缓存信息都在入参中，文件的内容以Body io.Reader的形式放在入参中。

下载

// Retrieves objects from Amazon S3. To use GET , you must have READ access to the
// object. If you grant READ access to the anonymous user, you can return the
// object without using an authorization header. An Amazon S3 bucket has no
// directory hierarchy such as you would find in a typical computer file system.
// You can, however, create a logical hierarchy by using object key names that
// imply a folder structure. For example, instead of naming an object sample.jpg ,
// you can name it photos/2006/February/sample.jpg . To get an object from such a
// logical hierarchy, specify the full key name for the object in the GET
// operation. For a virtual hosted-style request example, if you have the object
// photos/2006/February/sample.jpg , specify the resource as
// /photos/2006/February/sample.jpg . For a path-style request example, if you have
// the object photos/2006/February/sample.jpg in the bucket named examplebucket ,
// specify the resource as /examplebucket/photos/2006/February/sample.jpg . For
// more information about request types, see HTTP Host Header Bucket Specification (https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#VirtualHostingSpecifyBucket)
// . For more information about returning the ACL of an object, see GetObjectAcl (https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAcl.html)
// . If the object you are retrieving is stored in the S3 Glacier or S3 Glacier
// Deep Archive storage class, or S3 Intelligent-Tiering Archive or S3
// Intelligent-Tiering Deep Archive tiers, before you can retrieve the object you
// must first restore a copy using RestoreObject (https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html)
// . Otherwise, this action returns an InvalidObjectState error. For information
// about restoring archived objects, see Restoring Archived Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/restoring-objects.html)
// . Encryption request headers, like x-amz-server-side-encryption , should not be
// sent for GET requests if your object uses server-side encryption with KMS keys
// (SSE-KMS) or server-side encryption with Amazon S3–managed encryption keys
// (SSE-S3). If your object does use these types of keys, you’ll get an HTTP 400
// BadRequest error. If you encrypt an object by using server-side encryption with
// customer-provided encryption keys (SSE-C) when you store the object in Amazon
// S3, then when you GET the object, you must use the following headers:
//   - x-amz-server-side-encryption-customer-algorithm
//   - x-amz-server-side-encryption-customer-key
//   - x-amz-server-side-encryption-customer-key-MD5
//
// For more information about SSE-C, see Server-Side Encryption (Using
// Customer-Provided Encryption Keys) (https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html)
// . Assuming you have the relevant permission to read object tags, the response
// also returns the x-amz-tagging-count header that provides the count of number
// of tags associated with the object. You can use GetObjectTagging (https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html)
// to retrieve the tag set associated with an object. Permissions You need the
// relevant read object (or version) permission for this operation. For more
// information, see Specifying Permissions in a Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html)
// . If the object you request does not exist, the error Amazon S3 returns depends
// on whether you also have the s3:ListBucket permission.
//   - If you have the s3:ListBucket permission on the bucket, Amazon S3 will
//     return an HTTP status code 404 ("no such key") error.
//   - If you don’t have the s3:ListBucket permission, Amazon S3 will return an
//     HTTP status code 403 ("access denied") error.
//
// Versioning By default, the GET action returns the current version of an object.
// To return a different version, use the versionId subresource.
//   - If you supply a versionId , you need the s3:GetObjectVersion permission to
//     access a specific version of an object. If you request a specific version, you
//     do not need to have the s3:GetObject permission. If you request the current
//     version without a specific version ID, only s3:GetObject permission is
//     required. s3:GetObjectVersion permission won't be required.
//   - If the current version of the object is a delete marker, Amazon S3 behaves
//     as if the object was deleted and includes x-amz-delete-marker: true in the
//     response.
//
// For more information about versioning, see PutBucketVersioning (https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketVersioning.html)
// . Overriding Response Header Values There are times when you want to override
// certain response header values in a GET response. For example, you might
// override the Content-Disposition response header value in your GET request. You
// can override values for a set of response headers using the following query
// parameters. These response header values are sent only on a successful request,
// that is, when status code 200 OK is returned. The set of headers you can
// override using these parameters is a subset of the headers that Amazon S3
// accepts when you create an object. The response headers that you can override
// for the GET response are Content-Type , Content-Language , Expires ,
// Cache-Control , Content-Disposition , and Content-Encoding . To override these
// header values in the GET response, you use the following request parameters. You
// must sign the request, either using an Authorization header or a presigned URL,
// when using these parameters. They cannot be used with an unsigned (anonymous)
// request.
//   - response-content-type
//   - response-content-language
//   - response-expires
//   - response-cache-control
//   - response-content-disposition
//   - response-content-encoding
//
// Overriding Response Header Values If both of the If-Match and
// If-Unmodified-Since headers are present in the request as follows: If-Match
// condition evaluates to true , and; If-Unmodified-Since condition evaluates to
// false ; then, S3 returns 200 OK and the data requested. If both of the
// If-None-Match and If-Modified-Since headers are present in the request as
// follows: If-None-Match condition evaluates to false , and; If-Modified-Since
// condition evaluates to true ; then, S3 returns 304 Not Modified response code.
// For more information about conditional requests, see RFC 7232 (https://tools.ietf.org/html/rfc7232)
// . The following operations are related to GetObject :
//   - ListBuckets (https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html)
//   - GetObjectAcl (https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAcl.html)
func (c *Client) GetObject(ctx context.Context, params *GetObjectInput, optFns ...func(*Options)) (*GetObjectOutput, error) {
    if params == nil {
        params = &GetObjectInput{}
    }

    result, metadata, err := c.invokeOperation(ctx, "GetObject", params, optFns, c.addOperationGetObjectMiddlewares)
    if err != nil {
        return nil, err
    }

    out := result.(*GetObjectOutput)
    out.ResultMetadata = metadata
    return out, nil
}

出参

type GetObjectOutput struct {

    // Indicates that a range of bytes was specified.
    AcceptRanges *string

    // Object data.
    Body io.ReadCloser

    // Indicates whether the object uses an S3 Bucket Key for server-side encryption
    // with Amazon Web Services KMS (SSE-KMS).
    BucketKeyEnabled bool

    // Specifies caching behavior along the request/reply chain.
    CacheControl *string

    // The base64-encoded, 32-bit CRC32 checksum of the object. This will only be
    // present if it was uploaded with the object. With multipart uploads, this may not
    // be a checksum value of the object. For more information about how checksums are
    // calculated with multipart uploads, see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumCRC32 *string

    // The base64-encoded, 32-bit CRC32C checksum of the object. This will only be
    // present if it was uploaded with the object. With multipart uploads, this may not
    // be a checksum value of the object. For more information about how checksums are
    // calculated with multipart uploads, see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumCRC32C *string

    // The base64-encoded, 160-bit SHA-1 digest of the object. This will only be
    // present if it was uploaded with the object. With multipart uploads, this may not
    // be a checksum value of the object. For more information about how checksums are
    // calculated with multipart uploads, see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumSHA1 *string

    // The base64-encoded, 256-bit SHA-256 digest of the object. This will only be
    // present if it was uploaded with the object. With multipart uploads, this may not
    // be a checksum value of the object. For more information about how checksums are
    // calculated with multipart uploads, see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumSHA256 *string

    // Specifies presentational information for the object.
    ContentDisposition *string

    // Specifies what content encodings have been applied to the object and thus what
    // decoding mechanisms must be applied to obtain the media-type referenced by the
    // Content-Type header field.
    ContentEncoding *string

    // The language the content is in.
    ContentLanguage *string

    // Size of the body in bytes.
    ContentLength int64

    // The portion of the object returned in the response.
    ContentRange *string

    // A standard MIME type describing the format of the object data.
    ContentType *string

    // Specifies whether the object retrieved was (true) or was not (false) a Delete
    // Marker. If false, this response header does not appear in the response.
    DeleteMarker bool

    // An entity tag (ETag) is an opaque identifier assigned by a web server to a
    // specific version of a resource found at a URL.
    ETag *string

    // If the object expiration is configured (see PUT Bucket lifecycle), the response
    // includes this header. It includes the expiry-date and rule-id key-value pairs
    // providing object expiration information. The value of the rule-id is
    // URL-encoded.
    Expiration *string

    // The date and time at which the object is no longer cacheable.
    Expires *time.Time

    // Creation date of the object.
    LastModified *time.Time

    // A map of metadata to store with the object in S3.
    //
    // Map keys will be normalized to lower-case.
    Metadata map[string]string

    // This is set to the number of metadata entries not returned in x-amz-meta
    // headers. This can happen if you create metadata using an API like SOAP that
    // supports more flexible metadata than the REST API. For example, using SOAP, you
    // can create metadata whose values are not legal HTTP headers.
    MissingMeta int32

    // Indicates whether this object has an active legal hold. This field is only
    // returned if you have permission to view an object's legal hold status.
    ObjectLockLegalHoldStatus types.ObjectLockLegalHoldStatus

    // The Object Lock mode currently in place for this object.
    ObjectLockMode types.ObjectLockMode

    // The date and time when this object's Object Lock will expire.
    ObjectLockRetainUntilDate *time.Time

    // The count of parts this object has. This value is only returned if you specify
    // partNumber in your request and the object was uploaded as a multipart upload.
    PartsCount int32

    // Amazon S3 can return this if your request involves a bucket that is either a
    // source or destination in a replication rule.
    ReplicationStatus types.ReplicationStatus

    // If present, indicates that the requester was successfully charged for the
    // request.
    RequestCharged types.RequestCharged

    // Provides information about object restoration action and expiration time of the
    // restored object copy.
    Restore *string

    // If server-side encryption with a customer-provided encryption key was
    // requested, the response will include this header confirming the encryption
    // algorithm used.
    SSECustomerAlgorithm *string

    // If server-side encryption with a customer-provided encryption key was
    // requested, the response will include this header to provide round-trip message
    // integrity verification of the customer-provided encryption key.
    SSECustomerKeyMD5 *string

    // If present, specifies the ID of the Amazon Web Services Key Management Service
    // (Amazon Web Services KMS) symmetric encryption customer managed key that was
    // used for the object.
    SSEKMSKeyId *string

    // The server-side encryption algorithm used when storing this object in Amazon S3
    // (for example, AES256, aws:kms ).
    ServerSideEncryption types.ServerSideEncryption

    // Provides storage class information of the object. Amazon S3 returns this header
    // for all objects except for S3 Standard storage class objects.
    StorageClass types.StorageClass

    // The number of tags, if any, on the object.
    TagCount int32

    // Version of the object.
    VersionId *string

    // If the bucket is configured as a website, redirects requests for this object to
    // another object in the same bucket or to an external URL. Amazon S3 stores the
    // value of this header in the object metadata.
    WebsiteRedirectLocation *string

    // Metadata pertaining to the operation's result.
    ResultMetadata middleware.Metadata

    noSmithyDocumentSerde
}

直接通过函数下载的方式可以看到，文件的一些元数据信息和缓存信息都在出参中，文件的内容以Body io.ReadCloser的形式放在入参中。

通过HTTP请求实现上传和下载

通过HTTP请求实现上传和下载只有在aws-sdk-go中才有，上传和下载时，都通过签名实现，而不需要返回具体对象。

aws-sdk-go-v2中没有原生实现，可参考 https://github.com/aws/aws-sdk-go-v2/issues/438 自己实现

上传生成预签名

// PutObjectRequest generates a "aws/request.Request" representing the
// client's request for the PutObject operation. The "output" return
// value will be populated with the request's response once the request completes
// successfully.
//
// Use "Send" method on the returned Request to send the API call to the service.
// the "output" return value is not valid until after Send returns without error.
//
// See PutObject for more information on using the PutObject
// API call, and error handling.
//
// This method is useful when you want to inject custom logic or configuration
// into the SDK's request lifecycle. Such as custom headers, or retry logic.
//
//
//    // Example sending a request using the PutObjectRequest method.
//    req, resp := client.PutObjectRequest(params)
//
//    err := req.Send()
//    if err == nil { // resp is now filled
//        fmt.Println(resp)
//    }
//
// See also, https://docs.aws.amazon.com/goto/WebAPI/s3-2006-03-01/PutObject
func (c *S3) PutObjectRequest(input *PutObjectInput) (req *request.Request, output *PutObjectOutput) {
    op := &request.Operation{
        Name:       opPutObject,
        HTTPMethod: "PUT",
        HTTPPath:   "/{Bucket}/{Key+}",
    }

    if input == nil {
        input = &PutObjectInput{}
    }

    output = &PutObjectOutput{}
    req = c.newRequest(op, input, output)
    return
}

github.com/aws/[email protected]/aws/request/request.go

// PresignRequest behaves just like presign, with the addition of returning a
// set of headers that were signed. The expire parameter is only used for
// presigned Amazon S3 API requests. All other AWS services will use a fixed
// expiration time of 15 minutes.
//
// It is invalid to create a presigned URL with a expire duration 0 or less. An
// error is returned if expire duration is 0 or less.
//
// Returns the URL string for the API operation with signature in the query string,
// and the HTTP headers that were included in the signature. These headers must
// be included in any HTTP request made with the presigned URL.
//
// To prevent hoisting any headers to the query string set NotHoist to true on
// this Request value prior to calling PresignRequest.
func (r *Request) PresignRequest(expire time.Duration) (string, http.Header, error) {
    r = r.copy()
    return getPresignedURL(r, expire)
}

签名解析出来，有三个返回值（下载同）

string：HTTP请求的url
http.Header：HTTP请求的Header
error：函数返回的报错

生成下载预签名

// GetObjectRequest generates a "aws/request.Request" representing the
// client's request for the GetObject operation. The "output" return
// value will be populated with the request's response once the request completes
// successfully.
//
// Use "Send" method on the returned Request to send the API call to the service.
// the "output" return value is not valid until after Send returns without error.
//
// See GetObject for more information on using the GetObject
// API call, and error handling.
//
// This method is useful when you want to inject custom logic or configuration
// into the SDK's request lifecycle. Such as custom headers, or retry logic.
//
//
//    // Example sending a request using the GetObjectRequest method.
//    req, resp := client.GetObjectRequest(params)
//
//    err := req.Send()
//    if err == nil { // resp is now filled
//        fmt.Println(resp)
//    }
//
// See also, https://docs.aws.amazon.com/goto/WebAPI/s3-2006-03-01/GetObject
func (c *S3) GetObjectRequest(input *GetObjectInput) (req *request.Request, output *GetObjectOutput) {
    op := &request.Operation{
        Name:       opGetObject,
        HTTPMethod: "GET",
        HTTPPath:   "/{Bucket}/{Key+}",
    }

    if input == nil {
        input = &GetObjectInput{}
    }

    output = &GetObjectOutput{}
    req = c.newRequest(op, input, output)
    return
}

分段获取

HTTP 分段获取资源（HTTP Partial Content）是一种机制，允许客户端在下载大型文件时，只请求文件的一部分，而不是整个文件。这对于网络传输较慢或带宽有限的情况下非常有用，因为它可以减少等待时间并降低带宽消耗。

当客户端发送一个 HTTP 请求时，可以在请求头中包含 Range 字段来指定所需的资源范围。该字段的格式为 bytes=start-end，其中 start 和 end 分别表示资源的起始字节和结束字节的索引。服务器在接收到带有 Range 请求头的请求后，会返回指定范围的资源内容，并在响应头中包含 Content-Range 字段来指示返回内容的范围，而且返回的HTTP Code是206。

以下是一个示例的 HTTP 请求和响应头的示例：

请求头

1
2
3

GET /large-file.txt HTTP/1.1
Host: example.com
Range: bytes=0-999

响应头

HTTP/1.1 206 Partial Content
Content-Type: text/plain
Content-Length: 1000
Content-Range: bytes 0-999/50000

在上述示例中，客户端请求了 large-file.txt 文件的前 1000 个字节。

Range: bytes=0-999，代表请求数据的第0个字节到第999个字节，一共1000个字节。

服务器返回了这部分内容，Status Code是206 Partial Content，并指示这个范围是文件的总范围中的一部分。

Content-Type: text/plain代表响应的内容是纯文本
Content-Length: 1000代表响应的文件大小是1000个字节
Content-Range: bytes 0-999/50000代表响应的是文件的第0个字节到第999个字节，50000代表这个文件总大小是50000字节。

通过使用 HTTP 分段获取资源，客户端可以逐步下载大型文件，按需获取所需的部分，从而提高下载效率和用户体验。这在视频流媒体、大型文件下载和断点续传等场景中非常常见。

真实HTTP请求如下

curl 'http://xxx' \
  -H 'Range: bytes=0-' \
  --compressed \
  --insecure

浏览器默认会使用分段获取，而且传入Range的截止字节大小为0，将文件大小交由服务端处理。

实现方案

一些播放器会自动在Header中携带信息，不需要手动传入分片大小。而后端服务则需要设置最大分片大小，避免出现前端传入分片过大，一次性获取所有文件，分片传输时区意义。

func ResourcePreview(w http.ResponseWriter, r *http.Request, pathParams map[string]string) {
    ranges := r.Header.Get("Range")
    token := r.URL.Query().Get("token")       // 一些播放器无法自定义header信息，需要将一些权限信息放在query中
    fileId := pathParams["file_id"]   				// 将文件id放在path中，可以实现url唯一，具备缓存条件

    download, err := logic.PreviewResource(r.Context(), token, fileId, ranges)
    if err != nil {
        w.WriteHeader(http.StatusInternalServerError)
        io.WriteString(w, err.Error())
        return
    }
    defer download.Body.Close()

  // 将请求的s3的header放入返回的header中
    for k, values := range download.Header {
        for _, v := range values {
            w.Header().Add(k, v)
        }
    }

    io.Copy(w, download.Body)

    return
}

logic层处理token，以及Range参数

initDownloadResult.Headers["Range"] = []string{"bytes=0-"}
    if ranges != "" {
        index := strings.Index(ranges, "bytes=")
        var start, end int
        if index != -1 {
            ss := strings.Split(ranges[index+6:], "-")
            if len(ss) > 0 {
                start, _ = strconv.Atoi(ss[0])
            }
            if len(ss) > 1 && ss[1] != "" {
                end, _ = strconv.Atoi(ss[1])
            }
        }
        if end == 0 || end-start > models.VideoDefaultContentLength {
            end = start + models.VideoDefaultContentLength - 1            //  使用默认分片大小
        }
        initDownloadResult.Headers["Range"] = []string{fmt.Sprintf("bytes=%d-%d", start, end)}
    }

s3上传和下载文件有两种方式，无论使用那种方式，都需要处理请求。

如果直接使用sdk的方式，上传时，需要将文件的Body io.Reader放入参数中。

下载时，将前端传入的Header放在结构体中，例如Range、一些缓存内容等

1
2
3

if ifNoneMatch, ok := ctx.Value("If-None-Match").(string); ok {
    input.IfNoneMatch = aws.String(ifNoneMatch)
}

将返回的Response放在返回的Header中，例如一些缓存内容

cacheControl := "max-age=86400,must-revalidate"  
if s3Object.CacheControl != nil {
    cacheControl = aws.StringValue(s3Object.CacheControl)
}
w.Header().Set("Cache-Control", cacheControl)                         	// 缓存控制方案
Etag := ""
if s3Object.ETag != nil {
    Etag = aws.StringValue(s3Object.ETag)
}
w.Header().Set("ETag", Etag)                                                            // Etag信息
lastModified := time.Now()
if s3Object.LastModified != nil {
    lastModified = *s3Object.LastModified
}
w.Header().Set("Last-Modified", lastModified.Format("Mon, 02 Jan 2006 15:04:05 GMT"))
expires := time.Now().Add(24 * time.Hour)
if s3Object.Expires != nil {
    expires = *s3Object.Expires
}
w.Header().Set("Expires", expires.Format("Mon, 02 Jan 2006 15:04:05 GMT"))

获取到的结构体中包含数据内容Body io.ReadCloser。

要实现分段获取，需要传入的Header参数：

Range，例如：Range: bytes=6815744-

需要将GetObjectOutput中放入Header：

Accept-Ranges，例如：Accept-Ranges: bytes
Content-Disposition，例如：Content-Disposition: inline; filename="xxx.mp4"
Content-Type，例如：Content-Type: video/mp4
Etag，例如：Etag: "40bd8bc04fb9cd7a5795121addcb11e6-35"
Content-Range，例如：bytes 6815744-11010047/535677112
Content-Length，例如：Content-Length: 4194304

实际请求

通过查看浏览器中的请求，可以看到播放视频时，播放器会发两个请求，第一个请求Header中不携带Range，用于获取视频信息，第二个请求才是获取视频具体信息，会附带Range信息

第一个请求

第二个请求

这里需要注意，如果是以HTTP请求预签名的方式，在服务端使用签名访问s3获取数据，然后返回给客户端。

一些语言中，在HTTP client端中会存在缓存，会将第一次请求的数据会缓存起来，导致第二个请求接受到的数据可能不满足Header中的大小，导致第二个请求的数据无法加载。

因此，需要将缓存中的数据读取出来，并且丢掉。（golang不需要）

1
2
3

if ranges == "" {
    response.Body.Close()
}

s3 对象存储