Skip to main content
.NETPerformance

From 2 minutes to 30 seconds: parallelizing image downloads in a .NET import pipeline

A product import pipeline was timing out in production — Nginx was cutting connections after batches took too long to process. The bottleneck wasn't the database or the file system. It was sequential HTTP requests downloading product images one by one. Switching to parallel execution with Task.WhenAll cut batch processing time from ~2 minutes to ~30 seconds.

November 20, 20245 min read

A product import pipeline was timing out in production — Nginx was cutting connections after batches took too long to process. The bottleneck wasn't the database or the file system. It was sequential HTTP requests downloading product images one by one. Switching to parallel execution cut batch processing time from ~2 minutes to ~30 seconds.

The context

The platform supports bulk product imports via Excel files — following the same template WooCommerce uses for exports, so vendors can migrate their existing catalog without reformatting their data. Each product can have up to 6 image URLs pointing to external servers. Part of the import process is fetching those images and saving them locally so we serve them ourselves rather than depending on external URLs staying alive.

The problem

The import was split into batches of 20 products to keep memory usage manageable. But even with batching, the process was regularly hitting Nginx's request timeout and returning a 503 before finishing.

The culprit was the image download loop. For each product, we iterated over its images sequentially — downloading one, saving it, then moving to the next:

// Sequential — one image at a time
foreach (var product in batch)
{
    foreach (var imageUrl in product.ImageUrls)
    {
        var imageData = await httpClient.GetByteArrayAsync(imageUrl);
        await SaveImageToServerAsync(imageData, imageUrl);
    }
    await SaveProductToDatabase(product);
}

For a product with 6 images, each HTTP request had to complete before the next one started. Across 20 products in a batch, that's up to 120 sequential HTTP round-trips — each one waiting on a remote server to respond before we could move on.

The total time was driven by the slowest chain of responses, multiplied across every product. With variable network latency to external image servers, batches were routinely taking around 2 minutes to complete — long enough for Nginx to give up and close the connection.

The wrong fix would have been raising the Nginx timeout. That would hide the symptom while leaving the underlying inefficiency in place. A request that takes 2 minutes to process is a problem regardless of whether the client waits for it.

The fix: Task.WhenAll for parallel image downloads

The images within a single product have no dependency on each other — there's no reason to download them sequentially. We can fire all requests at once and wait for all of them to complete before moving on.

// Parallel — all images for a product downloaded simultaneously
foreach (var product in batch)
{
    var imageTasks = product.ImageUrls
        .Select(async imageUrl =>
        {
            var imageData = await httpClient.GetByteArrayAsync(imageUrl);
            await SaveImageToServerAsync(imageData, imageUrl);
        });

    await Task.WhenAll(imageTasks);
    await SaveProductToDatabase(product);
}

Task.WhenAll takes a collection of tasks and waits for all of them to complete. Instead of waiting for image 1, then image 2, then image 3 — we start all of them at once and wait for the slowest one. The total time for a product's images is now determined by the single slowest download, not the sum of all of them.

Products still process one at a time — SaveProductToDatabase still runs sequentially — but the image download bottleneck inside each product is eliminated.

The result

BeforeAfter
Batch processing time~2 minutes~30 seconds
Nginx 503 timeoutsFrequentEliminated
Images downloadedSequentialParallel per product

A batch of 20 products that regularly caused timeouts now completes in roughly 30 seconds — well within any reasonable request timeout threshold. No Nginx configuration changes were needed.

Why not parallelize across products too?

We kept product processing sequential deliberately. Parallelizing the image downloads within each product is safe — the images are independent. Parallelizing across products would mean multiple concurrent database writes, which introduces ordering and consistency concerns we didn't want to deal with for this pipeline.

Parallel where it's safe. Sequential where correctness matters.

Key takeaways

Sequential HTTP requests inside a loop are a common bottleneck. Any time you have a loop with await inside it and the iterations are independent, Task.WhenAll is worth considering.

Don't raise timeouts to fix slow processing. A long-running request is a symptom. Raising the timeout delays the failure without addressing the cause — and leaves users waiting longer than they should.

Parallelize at the right level. Not everything should run in parallel. Identify the independent work — in this case, images within a product — and parallelize only that, while keeping dependent operations sequential.

Measure before and after. The before/after numbers here (2 minutes vs 30 seconds) came from observing real batches. Without that baseline, it's hard to know whether a change actually helped or just felt faster.