async-task-runner | Node.js Async Task Runner with Concurrency Control

The most powerful async task runner for Node.js - Execute thousands of async tasks with intelligent concurrency control, automatic retries, exponential backoff, timeouts, and comprehensive reporting. Battle-tested for web scraping, API processing, file operations, data migration, ETL pipelines, and any bulk async operations.

🚀 Perfect for: Web scraping bots, API batch processing, file transformations, data migrations, parallel downloads, bulk operations, and high-performance async workflows.

✨ Features

• 🚀 Concurrency Control: Limit the number of tasks running simultaneously
• 🔄 Smart Retry Logic: Configurable retries with exponential backoff
• ⏰ Timeout Support: Automatic task cancellation with AbortController
• 📊 Comprehensive Reporting: Detailed execution summaries and error analysis
• 🔧 TypeScript Ready: Full TypeScript support with detailed type definitions
• 🎯 Production Ready: Battle-tested with extensive error handling
• 🪶 Lightweight: Zero dependencies, minimal footprint

📦 Installation

Install the most popular async task runner for Node.js:

1npm install @md-anas-sabah/async-task-runner

1yarn add @md-anas-sabah/async-task-runner

1pnpm add @md-anas-sabah/async-task-runner

🚀 Quick Start

1import { runTasks } from '@md-anas-sabah/async-task-runner';
2
3// Define your async tasks
4const tasks = [
5  () => fetch('https://api.example.com/data/1').then(r => r.json()),
6  () => fetch('https://api.example.com/data/2').then(r => r.json()),
7  () => fetch('https://api.example.com/data/3').then(r => r.json()),
8];
9
10// Run with concurrency control
11const results = await runTasks(tasks, { 
12  concurrency: 2,
13  retries: 3,
14  timeout: 5000 
15});
16
17console.log(results);
18// [
19//   { success: true, result: {...}, duration: 245.67, attempts: 1 },
20//   { success: false, error: 'Request timeout', attempts: 4 },
21//   { success: true, result: {...}, duration: 189.23, attempts: 2 }
22// ]

📚 Usage Examples

Basic Usage

1import { runTasks } from '@md-anas-sabah/async-task-runner';
2
3const tasks = [
4  () => new Promise(resolve => setTimeout(() => resolve('Task 1'), 1000)),
5  () => new Promise(resolve => setTimeout(() => resolve('Task 2'), 500)),
6  () => new Promise(resolve => setTimeout(() => resolve('Task 3'), 800)),
7];
8
9const results = await runTasks(tasks, { concurrency: 2 });

With Retry Logic and Logging

1import { runTasksWithLogging } from '@md-anas-sabah/async-task-runner';
2
3const results = await runTasksWithLogging(unreliableTasks, {
4  concurrency: 3,
5  retries: 3,
6  retryDelay: 1000,
7  exponentialBackoff: true,
8  maxRetryDelay: 10000
9});

With Comprehensive Reporting

1import { runTasksWithSummary, formatSummary } from '@md-anas-sabah/async-task-runner';
2
3const summary = await runTasksWithSummary(tasks, {
4  concurrency: 5,
5  retries: 2,
6  timeout: 3000
7});
8
9console.log(formatSummary(summary));
10// 📊 Task Execution Summary
11// ✅ Successful: 8
12// ❌ Failed: 2
13// ⏰ Timed out: 1
14// 🔄 Total retries: 4
15// ...

Advanced Configuration

1import { TaskRunner, DefaultLogger } from '@md-anas-sabah/async-task-runner';
2
3// Custom logger
4const logger = new DefaultLogger(true);
5
6// Advanced configuration
7const runner = new TaskRunner({
8  concurrency: 5,
9  retries: 3,
10  retryDelay: 1000,
11  exponentialBackoff: true,
12  maxRetryDelay: 30000,
13  timeout: 10000
14}, logger);
15
16const results = await runner.run(tasks);

🌐 Real-World Examples

Web Scraping

1import { runTasksWithSummary } from '@md-anas-sabah/async-task-runner';
2import axios from 'axios';
3import * as cheerio from 'cheerio';
4
5// Scrape product data from multiple pages
6const scrapeProduct = (url) => async () => {
7  const response = await axios.get(url);
8  const $ = cheerio.load(response.data);
9  
10  return {
11    url,
12    title: $('h1').text().trim(),
13    price: $('.price').text().trim(),
14    description: $('.description').text().trim()
15  };
16};
17
18const productUrls = [
19  'https://example.com/product/1',
20  'https://example.com/product/2',
21  // ... more URLs
22];
23
24const tasks = productUrls.map(scrapeProduct);
25
26const summary = await runTasksWithSummary(tasks, {
27  concurrency: 3,        // Don't overwhelm the server
28  retries: 2,           // Retry failed requests
29  timeout: 10000,       // 10 second timeout
30  retryDelay: 2000,     // Wait 2s between retries
31  exponentialBackoff: true
32});
33
34console.log(`Scraped ${summary.successCount} products successfully`);

File Processing

1import { runTasks } from '@md-anas-sabah/async-task-runner';
2import fs from 'fs/promises';
3import path from 'path';
4
5// Process multiple files concurrently
6const processFile = (filePath) => async () => {
7  const content = await fs.readFile(filePath, 'utf-8');
8  
9  // Process the file (e.g., transform, analyze, etc.)
10  const processed = content
11    .split('\\n')
12    .filter(line => line.trim())
13    .map(line => line.toUpperCase())
14    .join('\\n');
15  
16  const outputPath = path.join('processed', path.basename(filePath));
17  await fs.writeFile(outputPath, processed);
18  
19  return { input: filePath, output: outputPath, lines: processed.split('\\n').length };
20};
21
22const files = await fs.readdir('input-directory');
23const tasks = files
24  .filter(file => file.endsWith('.txt'))
25  .map(file => processFile(path.join('input-directory', file)));
26
27const results = await runTasks(tasks, {
28  concurrency: 4,       // Process 4 files at once
29  timeout: 30000        // 30 second timeout per file
30});
31
32console.log(`Processed ${results.filter(r => r.success).length} files`);

API Batch Processing

1import { runTasksWithLogging } from '@md-anas-sabah/async-task-runner';
2
3// Process user data through multiple API endpoints
4const processUser = (user) => async () => {
5  // Enrich user data from multiple sources
6  const [profile, preferences, activity] = await Promise.all([
7    fetch(`/api/users/${user.id}/profile`).then(r => r.json()),
8    fetch(`/api/users/${user.id}/preferences`).then(r => r.json()),
9    fetch(`/api/users/${user.id}/activity`).then(r => r.json())
10  ]);
11  
12  return {
13    ...user,
14    profile,
15    preferences,
16    activityScore: activity.score
17  };
18};
19
20const users = await fetch('/api/users').then(r => r.json());
21const tasks = users.map(processUser);
22
23const results = await runTasksWithLogging(tasks, {
24  concurrency: 10,      // 10 concurrent API calls
25  retries: 3,           // Retry failed calls
26  timeout: 5000,        // 5 second timeout
27  retryDelay: 1000,
28  exponentialBackoff: true
29});
30
31console.log(`Processed ${results.filter(r => r.success).length}/${users.length} users`);

🔧 API Reference

Functions

runTasks(tasks, options?)

Executes tasks with the specified configuration.

Parameters:

• tasks: (() => Promise<any>)[] - Array of async task functions
• options?: TaskOptions - Configuration options

Returns: Promise<TaskResult[]>

runTasksWithLogging(tasks, options?)

Same as runTasks but with verbose logging enabled.

runTasksWithSummary(tasks, options?)

Executes tasks and returns a comprehensive execution summary.

Returns: Promise<TaskExecutionSummary>

formatSummary(summary)

Formats a task execution summary into a human-readable string.

Configuration Options

Option	Type	Default	Description
concurrency	number	3	Maximum number of concurrent tasks
retries	number	0	Number of retry attempts for failed tasks
retryDelay	number	1000	Base delay between retries (milliseconds)
exponentialBackoff	boolean	false	Enable exponential backoff for retries
maxRetryDelay	number	30000	Maximum retry delay (milliseconds)
timeout	number	undefined	Maximum task duration (milliseconds)

Types

TaskResult

1interface TaskResult {
2  success: boolean;
3  result?: any;
4  error?: string;
5  taskIndex: number;
6  attempts: number;
7  duration: number;
8  timedOut?: boolean;
9  retryHistory?: Array<{
10    attempt: number;
11    error: string;
12    delay: number;
13    duration: number;
14    timedOut?: boolean;
15  }>;
16}

TaskExecutionSummary

1interface TaskExecutionSummary {
2  results: TaskResult[];
3  successCount: number;
4  failureCount: number;
5  timeoutCount: number;
6  totalRetries: number;
7  performance: {
8    totalDuration: number;
9    averageDuration: number;
10    executionTime: number;
11    startTime: Date;
12    endTime: Date;
13  };
14  errorBreakdown: Array<{
15    error: string;
16    count: number;
17    tasks: number[];
18    firstOccurrence: Date;
19    lastOccurrence?: Date;
20  }>;
21  successRate: number;
22}

🎯 Use Cases

Perfect For:

• Web Scraping: Scrape multiple pages with rate limiting and retry logic
• API Processing: Batch process API calls with intelligent retry strategies
• File Operations: Process multiple files concurrently with progress tracking
• Data Migration: Migrate data between systems with error handling
• Bulk Operations: Any scenario requiring controlled parallel execution
• ETL Pipelines: Extract, transform, and load operations with monitoring

Benefits:

• Performance: Optimal resource utilization with concurrency control
• Reliability: Automatic retries with exponential backoff
• Observability: Comprehensive logging and execution summaries
• Developer Experience: TypeScript support and intuitive API
• Production Ready: Battle-tested error handling and timeout management

🛠️ Development

1# Clone the repository
2git clone https://github.com/md-anas-sabah/async-task-runner.git
3
4# Install dependencies
5npm install
6
7# Build the project
8npm run build
9
10# Run tests
11npm test
12
13# Run example tests
14npm run test-package
15npm run test-retry
16npm run test-timeout
17npm run test-summary

📝 License

MIT © Md Anas Sabah

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📊 Examples Directory

Check out the /examples directory for more detailed use cases:

• web-scraping.js - Complete web scraping example
• file-processing.js - Batch file processing example
• api-integration.js - API batch processing example
• real-world-scenarios.js - Multiple production-ready examples

🔗 Related Projects

• p-limit - Simple concurrency limiting
• p-retry - Retry pattern implementation
• p-timeout - Timeout utility

⭐ Support

If you find this package useful, please consider giving it a star on GitHub!