NestJS AWS S3 Module Integration

A powerful and flexible AWS S3 integration module for NestJS applications with support for file uploads, signed URLs, and bucket operations.

Features

• Easy integration with AWS S3
• File upload support (PDF, CSV, and other formats)
• Chunked CSV uploads for large datasets
• Signed URL generation
• Bucket operations (list, delete)
• TypeScript support
• Configurable through environment variables

Installation

1npm install @solegence/nest-js-aws-s3

Configuration

Environment Variables

Create a .env file in your project root:

1AWS_S3_REGION=your-region
2AWS_S3_BUCKET=your-bucket
3AWS_ACCESS_KEY_ID=your-access-key
4AWS_SECRET_ACCESS_KEY=your-secret-key

Module Registration

There are two ways to register the S3Module:

1. Synchronous Registration

1import { S3Module } from '@solegence/nest-js-aws-s3';
2
3@Module({
4  imports: [
5    S3Module.register({
6      awsS3Region: process.env.AWS_S3_REGION,
7      awsS3Bucket: process.env.AWS_S3_BUCKET,
8      awsS3Accesskey: process.env.AWS_ACCESS_KEY_ID,
9      awsS3SecretKey: process.env.AWS_SECRET_ACCESS_KEY,
10      isGlobal: true, // optional: make the module global
11    }),
12  ],
13})
14export class AppModule {}

2. Asynchronous Registration

1import { S3Module } from '@solegence/nest-js-aws-s3';
2import { ConfigService, ConfigModule } from '@nestjs/config';
3
4@Module({
5  imports: [
6    S3Module.registerAsync({
7      imports: [ConfigModule],
8      useFactory: (configService: ConfigService) => ({
9        awsS3Region: configService.get('AWS_S3_REGION'),
10        awsS3Bucket: configService.get('AWS_S3_BUCKET'),
11        awsS3Accesskey: configService.get('AWS_ACCESS_KEY_ID'),
12        awsS3SecretKey: configService.get('AWS_SECRET_ACCESS_KEY'),
13        isGlobal: true, // optional: make the module global
14      }),
15      inject: [ConfigService],
16    }),
17  ],
18})
19export class AppModule {}

You can also use useClass or useExisting instead of useFactory:

1// Using useClass
2S3Module.registerAsync({
3  useClass: S3ConfigService, // Your custom config service that implements S3ModuleOptionsFactory
4});
5
6// Using useExisting
7S3Module.registerAsync({
8  useExisting: ConfigService, // Existing provider that implements S3ModuleOptionsFactory
9});

Usage Examples

Basic File Upload

1async uploadSingleFile(filename: string, buffer: Buffer) {
2  return await this.s3Service.uploadFile(this.AWS_S3_BUCKET, filename, buffer);
3}

Converting and Uploading CSV Data

1// Convert array of objects to CSV and upload
2const data = [
3  { name: 'John', age: 30, city: 'New York' },
4  { name: 'Jane', age: 25, city: 'London' }
5];
6
7// With headers (default)
8await s3Service.convertAndUploadCsv('users.csv', data);
9
10// Without headers
11await s3Service.convertAndUploadCsv('users.csv', data, false);

Uploading Files

1@Injectable()
2export class YourService {
3  constructor(private readonly s3Service: S3Service) {}
4
5  async uploadPdf(filename: string, buffer: Buffer) {
6    try {
7      const result = await this.s3Service.uploadFileInPdf(filename, buffer);
8      return result;
9    } catch (error) {
10      throw new Error(`Failed to upload PDF: ${error.message}`);
11    }
12  }
13}

Generating Signed URLs

1async getFileUrl(key: string) {
2  const url = await this.s3Service.getSignedUrl(key, 3600); // Expires in 1 hour
3  return url;
4}

Chunked CSV Upload

1async uploadLargeCsv(key: string, data: any[]) {
2  await this.s3Service.uploadCsvToS3InChunks(key, data);
3}

API Reference

S3Service Methods

Method	Description	Parameters	Return Type
uploadFile	Generic file upload	bucket: string, key: string, body: Buffer | Readable	Promise
uploadFileInPdf	PDF file upload	filename: string, fileBuffer: Buffer	Promise
uploadfileInCsv	CSV file upload	filename: string, fileBuffer: string | Buffer	Promise
convertAndUploadCsv	Convert data to CSV and upload	filename: string, data: CsvData[], includeHeader?: boolean	Promise
uploadCsvToS3InChunks	Large CSV upload in chunks	key: string, data: CsvData[]	Promise
getSignedUrl	Generate presigned URL	key: string, expires?: number	Promise
deleteMultipleObjects	Delete multiple objects	keys: string[]	Promise<DeleteObjectCommandOutput[]>
getBucketObjects	List bucket objects	keyOrPrefix: string	Promise

Best Practices

File Upload Recommendations

1. Content Type Selection

   • Use appropriate methods for specific file types:
     • uploadFileInPdf for PDF files
     • uploadfileInCsv for CSV files
     • uploadFile for generic files

2. Large Dataset Handling

   • Use uploadCsvToS3InChunks for large datasets
   • Use convertAndUploadCsv for smaller datasets

3. Error Handling

1try {
2  await s3Service.uploadFileInPdf('document.pdf', buffer);
3} catch (error) {
4  // Handle specific error types
5  if (error.name === 'NoSuchBucket') {
6    // Handle bucket not found
7  } else if (error.$metadata?.httpStatusCode === 403) {
8    // Handle permission issues
9  }
10}

Type Definitions

CsvData Interface

1interface CsvData extends Record<string, any> {
2  headers?: string[];
3  rows?: any[][];
4  [key: string]: string | number | boolean | null | string[] | any[][] | undefined;
5}

Troubleshooting

Common Issues

1. Access Denied Errors

   • Verify AWS credentials are correct
   • Check IAM permissions for the provided access key

2. Upload Failures

   • Ensure bucket name is correct
   • Verify file size is within limits
   • Check network connectivity

3. Missing Credentials

   • Ensure environment variables are properly set
   • Verify module registration includes all required options

Performance Optimization

1. Large File Uploads

   • Use multipart uploads for files > 100MB
   • Implement chunked uploads for large CSV datasets
   • Consider compression for large files

2. Bucket Organization

   • Use meaningful prefixes for better organization
   • Implement lifecycle policies for cost optimization
   • Use appropriate storage classes based on access patterns

Contributing

Contributions are welcome! Please feel free to submit pull requests.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.