aws-s3-object-multipart-copy

Copy large files in S3 using the AWS S3 Multipart API.

Installation

1$ npm install aws-s3-object-multipart-copy

Usage

1const { S3 } = require('aws-sdk')
2const copy = require('aws-s3-object-multipart-copy')
3
4const s3 = new S3()
5const source = 's3://source-bucket/path'
6const destination = 's3://destination-bucket/path'
7
8// async
9copy(source, destination, { s3 })
10  .then(() => { console.log('done') })
11
12// async with emitter
13copy(source, destination, { s3 })
14  .on('progress', console.log)
15  .then(() => { console.log('done') })

Advanced

1const { Session, Source } = require('aws-s3-object-multipart-copy')
2const { SingleBar } = require('cli-progress')
3const { S3 } = require('aws-sdk')
4
5const s3 = new S3()
6const progress = new SingleBar()
7const session = new Session({ s3 })
8const source = Source.from(session, 's3://bucket/path')
9
10progress.start(100, 0
11session.add(source, 's3://destination/path')
12session.run().on('progress', (e) => {
13  progress.update(e.value.upload.progress)
14})

API

session = copy(source, destination, opts)

Copy source into destination where source or destination can be a URI or an instance of Source and Destination respectively. opts can be:

1{
2  s3: null, // an instance of `AWS.S3` <required>
3  retries: 4, // number of max retries for failed uploads [optional]
4  partSize: 5 * 1024 * 1024, // the default part size for all uploads in this session [optional]
5  concurrency: os.cpus().length * os.cpus().length // the upload concurrency [optional]
6}

partSize = computePartSize(contentLength)

Computes the part size for a given contentLength. This is useful if you want to compute the partSize ahead of time. This module will compute the correct partSize for very large files if this number is too small.

1const s3 = new S3()
2const session = new Session({ s3 }
3const source = Source.from(session, 's3://bucket/path')
4await source.ready()
5const partSize = computePartSize(source.contentLength)

class Session

The Session class is a container for a multipart copy request.

session = new Session(opts)

Create a new Session instance where opts can be:

1{
2  s3: null, // an instance of `AWS.S3` <required>
3  retries: 4, // number of max retries for failed uploads [optional]
4  partSize: 5 * 1024 * 1024, // the default part size for all uploads in this session [optional]
5  concurrency: os.cpus().length * os.cpus().length // the upload concurrency [optional]
6}

totalQueued = session.add(source, destination)

Add a source and destination pair to the session queue.

1session.add('s3://source/path', 's3://destination/path')

await session.run()

Run the session.

session.abort()

Aborts the running session blocking until the lock is released.

1if (oops) {
2  // blocks until all requests have been aborted
3  await session.abort()
4}

session.then()

then() implementation to proxy to current active session promise.

1await session

session.catch()

catch() implementation to proxy to current active session promise.

1session.catch((err) => {
2  // handle error
3})

session.on('progress', event)

Emitted when a part is uploaded. The object emitted is the same from the 'progress' event in the Batch module. The value of event.value is a Part instance containing information about the upload (ETag, part number) and a pointer to the MultipartUpload instance at event.value.upload which contains information like how many parts have been uploaded, the progress as a percentage, and how many parts are pending.

1session.on('progress', (event) => {
2  console.log(event.value.upload.id, event.value.upload.progress)
3})

session.on('error', err)

Emitted when an error occurs during the life time of a running session.

1session.run().on('error', (err) => {
2  // handle err
3})

session.on('end')

Emitted when the session has finished running successfully.

1session.run().on('end', () => {
2  // session run finished successfully
3})

class Source

The Source class is a container for a source object in a bucket.

source = Source.from(session, uriOrOpts)

Create a new Source instance where session is an instance of Session and uriOrOpts can be a S3 URI (s3://bucket/...) or an object specifying a bucket and key ({bucket: 'bucket', key: 'path/to/file'}).

1const source = Source.from(session, 's3://bucket/path')
2// or
3const source = Source.from(session, { bucket: 'bucket', key: 'path/to/file'})

source.key

The source's key path in the S3 bucket.

source.bucket

The source's bucket in S3.

source.contentLength

The size in bytes of the source in S3.

await source.ready()

Wait for the source to be ready (loaded metadata).

1await source.ready()

class Destination

The Destination class is a container for a destination object in a bucket.

destination = Destination.from(session, uriOrOpts)

Create a new Destination instance where session is an instance of Session and uriOrOpts can be a S3 URI (s3://bucket/...) or an object specifying a bucket and key ({bucket: 'bucket', key: 'path/to/file'}).

1const destination = Destination.from(session, 's3://bucket/path')
2// or
3const destination = Destination.from(session, { bucket: 'bucket', key: 'path/to/file'})

destination.key

The destination's key path in the S3 bucket.

destination.bucket

The destination's bucket in S3.

await destination.ready()

Wait for the destination to be ready (loaded metadata).

1await destination.ready()

class MultipartUpload

The MultipartUpload class is a container for a multipart upload request tracking uploaded parts, progress, etc.

upload = new MultipartUpload(session, source, destination)

Create a new MultipartUpload instance from a session where source and destination are Source and Destination instances respectively.

1const upload = new MultipartUpload(session, source, destination)

upload.id

The identifier generated for this upload by AWS (UploadID).

upload.key

The destination key of the upload.

upload.parts

The Part instances for each uploaded part in this multipart upload. Each Part contains the ETag for the upload request.

upload.total

The total number of parts to upload.

upload.bucket

The destination bucket of the upload.

upload.pending

The total number of pending parts to upload.

upload.progress

The progress as a percentage of the multipart upload.

Tests

TODO (I am sorry)

License

MIT