Spring Batch Skip and Retry (Fault-Tolerant Batch Processing Guide)

Spring Batch Skip and Retry — Fault-Tolerant Batch Processing (Complete Guide)

In real-world enterprise applications, batch jobs often operate on hundreds of thousands or even millions of records at a time. During these long-running processes, failures are not just a possibility; they are an absolute certainty. Network connections drop, integrations with cloud providers like AWS experience transient latency, and legacy external APIs time out unexpectedly. Additionally, input data extracted from third-party vendors is rarely clean, often containing malformed strings or missing required fields.

A production-ready Spring Batch job must handle these inevitable failures gracefully. If a job processing 50,000 records fails on record 49,999 because of a single bad email address, stopping the entire job and forcing a manual restart is a massive waste of computing resources and developer time.

This guide explores exactly how to implement Skip and Retry in Spring Batch using a real, runnable example that simulates API timeouts and bad requests. We will go deep into how the framework evaluates these rules at runtime, ensuring your background processes become robust and self-healing.

🎥 Video Walkthrough:
Spring Batch Skip & Retry — Real-Time Execution Explained

TL;DR
✔ Retry: Use for transient failures (timeouts, temporary network issues, database deadlocks).
✔ Skip: Use for permanent failures (bad input formatting, strict validation errors, 400 Bad Request).
✔ Combine: Use retry alongside backoff policies and strict skip limits to ensure safe, continuous batch execution.

Understanding the High-Level Execution Flow

Before diving into code, it is crucial to understand the order of operations within a Spring Batch chunk-oriented processing step. When an item is read and passed to the processor, Spring Batch wraps the execution in a transaction. If an exception occurs, the framework does not immediately skip the item.

Spring Batch always attempts retry first, provided the exception matches your retry configuration. Only when the maximum number of retry attempts is completely exhausted does the framework evaluate your skip rules. 

Reader → Processor (Exception Thrown)
               ↓
          Retry Evaluated (Loop until limit reached)
               ↓
          Skip Evaluated (After retry exhausted or if explicitly non-retryable)
               ↓
             Writer

Sample Input Data (persons.csv)

To demonstrate this, we will use a simple CSV file containing user registrations. Notice that we have specifically injected records designed to trigger different types of failures based on their email addresses. 

id,name,email
1,John Doe,john@yopmail.com
2,Alice,retry@yopmail.com
3,BadRequest User,badrequest1@yopmail.com
4,Test,test@yopmail.com
5,BadRequest User,badrequest2@yopmail.com

Defining Custom Exceptions: Transient vs. Permanent

The most important architectural design decision you will make when building a fault-tolerant batch job is deciding which failures are retryable (transient) and which are skippable (permanent).

A transient error is something that might succeed if you simply try again a few seconds later. Examples include a 503 Service Unavailable HTTP response or a database connection timeout. For this, we create an ApiTimeoutException. 

public class ApiTimeoutException extends RuntimeException {
  public ApiTimeoutException(String message) {
    super(message);
  }
}

A permanent error will never succeed, no matter how many times you retry it. If a user submits a malformed email address, the server will consistently return a 400 Bad Request. Retrying this is a waste of CPU cycles. For this scenario, we create a BadRequestException which we will instruct Spring Batch to skip immediately. 

public class BadRequestException extends RuntimeException {
  public BadRequestException(String message) {
    super(message);
  }
}

The ItemProcessor: Where Retry Logic Shines

In most enterprise applications, data enrichment or external API calls happen inside the ItemProcessor. Therefore, this is exactly where we want our retry logic to intercept failures. 

@Component
public class PersonProcessor implements ItemProcessor {

  @Autowired
  private PersonRegistrationProcessor registrationProcessor;

  @Override
  public Person process(Person person) {
    // Attempting to register the user via an external service
    registrationProcessor.registerPerson(person.getEmail());
    return person;
  }
}

Simulating Real API Failures

In a real application, the PersonRegistrationProcessor would likely use a RestTemplate or WebClient to make an HTTP network call. To make this code runnable locally without needing a separate mock server, we are simulating the API behavior using a ConcurrentHashMap to track attempts in memory. 

@Component
public class PersonRegistrationProcessor {

  private final Map attempts = new ConcurrentHashMap<>();

  public void registerPerson(String email) {

    // Simulating a transient network glitch that resolves on the 3rd try
    if ("retry@yopmail.com".equalsIgnoreCase(email)) {
      int count = attempts.merge(email, 1, Integer::sum);
      System.out.println("Attempt " + count + " for email: " + email);
      if (count < 3) {
        throw new ApiTimeoutException("API timeout connecting to registration server");
      }
    }

    // Simulating a permanent validation error from the external API
    if (email.contains("badrequest")) {
      throw new BadRequestException("400 Bad Request: Invalid email format");
    }
  }
}

Step Configuration: Wiring Fault Tolerance

Now we tie it all together in our Step configuration. We explicitly tell Spring Batch which exceptions belong to which fault-tolerance mechanism. We also introduce a FixedBackOffPolicy. Without a backoff policy, Spring Batch would execute all 3 retries within milliseconds of each other, which defeats the purpose of waiting for a struggling downstream API to recover. 

@Bean
public Step processingStep(JobRepository jobRepository,
                           PlatformTransactionManager txManager,
                           ItemReader reader,
                           RegistrationWriter writer,
                           PersonProcessor processor) {

  FixedBackOffPolicy backOff = new FixedBackOffPolicy();
  backOff.setBackOffPeriod(2000); // Wait 2 seconds between retries

  return new StepBuilder("learn-skip-and-retry", jobRepository)
    .chunk(1, txManager)
    .reader(reader)
    .processor(processor)
    .writer(writer)
    .faultTolerant()
      .retry(ApiTimeoutException.class)
      .retryLimit(3)
      .backOffPolicy(backOff)
      .skip(BadRequestException.class)
      .skipLimit(4)
    .build();
}
Why set the chunk size to 1?
When integrating with fragile external APIs, keeping the chunk size small (or setting it to 1) is often safer. If an exception is thrown in a chunk size of 100, the entire transaction for those 100 items rolls back. Spring Batch then has to do extra work scanning the chunk to isolate the specific failing item. Processing one-by-one ensures retry and skip affect only that single isolated record.

Console Output: Watching it Work

If you run this application against our sample CSV, you will see Spring Batch orchestrating the recovery perfectly. When it encounters the `retry@yopmail.com` address, it pauses, tries again, and eventually succeeds without failing the job. 

[main] INFO  - Processing John Doe
[main] INFO  - Attempt 1 for email: retry@yopmail.com
[main] WARN  - ApiTimeoutException thrown. Waiting 2000ms...
[main] INFO  - Attempt 2 for email: retry@yopmail.com
[main] WARN  - ApiTimeoutException thrown. Waiting 2000ms...
[main] INFO  - Attempt 3 for email: retry@yopmail.com
[main] INFO  - Successfully processed Alice
[main] ERROR - Skipping badrequest1@yopmail.com due to 400 Bad Request

What Happens When the Skip Limit is Exceeded?

We configured our step with `.skipLimit(4)`. But what happens internally when the 5th bad request is encountered?

When the limit is breached, Spring Batch immediately halts the step execution. It updates the internal BATCH_STEP_EXECUTION and BATCH_JOB_EXECUTION metadata tables in your database, officially marking the job status as FAILED. Any remaining records in your CSV file are completely ignored and will not be processed. This is a crucial safety mechanism; it prevents your system from silently ignoring a massive data quality issue if, for example, an entire vendor file is corrupted.

Common Spring Batch Anti-Patterns

1. Retrying Validation Errors

One of the most common mistakes developers make is applying a blanket Exception.class to their retry logic. If a record fails because a required field is missing, retrying it 5 times will simply result in 5 identical failures. This wastes processing time, unnecessarily blocks threads, and extends the execution time of your batch window. Always strictly define which exceptions warrant a retry.

2. Using Large Chunk Sizes for Unreliable APIs

While a chunk size of 1000 is great for writing to a local database, it is terrible for calling third-party REST APIs. If the 999th API call in a chunk fails, the transaction rolls back, and Spring Batch has to manage complex state to figure out which items need to be re-processed. Keep chunk sizes small when network calls are involved.

3. Forgetting the Backoff Policy

If an external service throws a Rate Limit Exceeded (HTTP 429) error, immediately slamming the service with another request a millisecond later will only guarantee another failure. You must configure a FixedBackOffPolicy or an ExponentialBackOffPolicy to give the struggling service time to breathe and recover before attempting again.

Conclusion

Implementing Spring Batch Skip and Retry allows you to build resilient, enterprise-grade batch pipelines that can tolerate the chaos of real-world infrastructure failures. By carefully categorizing your exceptions into skippable and retryable buckets, and implementing strategic backoff policies, you can transform fragile, manual jobs into robust, self-healing systems.

🎥 Watch the complete execution walkthrough:
Spring Batch Skip & Retry — Full Video Tutorial

🧱 Spring Batch Core Components

Understand how ItemReader, ItemProcessor, and ItemWriter work together when exporting data to CSV files.

🔄 Spring Batch ItemProcessor Example

Apply transformation and formatting logic before writing records into CSV output files.

🔁 CSV to Database with Spring Batch

Compare inbound (CSV → DB) and outbound (DB → CSV) batch processing patterns.

🚫 Skip Policy & Error Handling

Handle write failures and formatting errors gracefully while exporting large datasets.

🔀 Conditional Flow in Spring Batch Jobs

Control job execution paths based on CSV generation success or failure.

🧵 Multithreaded Step in Spring Batch

Improve export performance by parallelizing data processing and CSV writing steps.