Spring Batch: How to Import Multiple CSV Files into a Database

In real-world enterprise environments, data rarely arrives neatly packaged in a single file. Third-party vendors, legacy mainframes, and external APIs often generate daily data drops consisting of dozens or even hundreds of split CSV files placed into a single directory. Processing these files sequentially using manual loops is inefficient, highly error-prone, and nearly impossible to restart if a failure occurs halfway through.

This is where Spring Batch truly shines. Instead of writing custom parsing logic, Spring Batch provides the MultiResourceItemReader—a highly robust component designed specifically to iterate over a directory of files, process them sequentially as a single continuous stream of records, and safely persist them to a database like Oracle.

📺 Watch the Complete Implementation Live!
If you prefer visual learning, check out the full step-by-step video tutorial where we build this Spring Batch multi-file architecture from scratch, run the job, and verify the database imports in real-time.

▶ Watch the Full Spring Batch Video Tutorial

1. Project Setup (pom.xml)

To get started, we need to declare our dependencies. We are using Java 21 alongside Spring Boot 3.x. The core components required are spring-boot-starter-batch for the batch processing framework, spring-boot-starter-data-jpa for our database mapping, and the ojdbc11 driver to connect natively to our Oracle Database. We also include Lombok to eliminate boilerplate code in our entity classes.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.2.0</version> <!-- Ensure compatibility with Java 21 -->
    <relativePath/> 
  </parent>
  
  <groupId>com.springjavalab</groupId>
  <artifactId>spring-batch-multi-file-import</artifactId>
  <version>0.0.1-SNAPSHOT</version>

  <properties>
    <java.version>21</java.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-batch</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
      <groupId>com.oracle.database.jdbc</groupId>
      <artifactId>ojdbc11</artifactId>
      <scope>runtime</scope>
    </dependency>
    <dependency>
      <groupId>org.projectlombok</groupId>
      <artifactId>lombok</artifactId>
      <optional>true</optional>
    </dependency>
  </dependencies>
</project>

2. Application Properties and Database Configuration

Next, we define our application properties. By setting spring.batch.job.enabled=true, we instruct Spring Boot to automatically locate our Batch Job and execute it the moment the application finishes starting up. We also configure our Oracle database connection credentials.

spring.application.name=spring-batch-multi-file-import

# JPA and Database Configuration
spring.jpa.hibernate.ddl-auto=update
spring.jpa.show-sql=true
spring.datasource.url=jdbc:oracle:thin:@//192.168.56.1:1521/freepdb1
spring.datasource.username=demo
spring.datasource.password=demo
spring.datasource.driver-class-name=oracle.jdbc.OracleDriver

# Auto-execute the batch job on startup
spring.batch.job.enabled=true

3. Sample Input Data

Create a folder named data inside your src/main/resources directory. Inside this folder, create two CSV files. Notice that both files contain a header row. Our batch configuration will need to account for this so it does not attempt to parse the header text as an integer ID.

// File 1: resources/data/employees1.csv
id,firstName,lastName
1,Rahul,Sharma
2,Priya,Patel

// File 2: resources/data/employees2.csv
id,firstName,lastName
3,Amit,Singh
4,Neha,Gupta

4. The Domain Model and Repository

Before we can write data, we must define the destination. We create a simple JPA Entity called Employee. Because we are importing the IDs directly from the CSV files, we do not use the @GeneratedValue annotation here. We follow this up with a standard Spring Data JPA Repository.

package com.springjavalab.entity;

import jakarta.persistence.Entity;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Entity
@Table(name = "employee_data")
@Data
@NoArgsConstructor
@AllArgsConstructor
public class Employee {

    @Id
    private Long id;
    private String firstName;
    private String lastName;
}

package com.springjavalab.repository;

import com.springjavalab.entity.Employee;
import org.springframework.data.jpa.repository.JpaRepository;

public interface EmployeeRepository extends JpaRepository<Employee, Long> {
}

5. The Core Architecture: Batch Configuration

This is where the magic happens. The configuration below dictates the entire flow of our application. Let's break down the architectural choices:

The Delegate Reader (FlatFileItemReader): This component knows how to open a single file, skip the header row (linesToSkip(1)), and map the comma-delimited string values directly into our Employee Java object using the BeanWrapperFieldSetMapper.
The Multi-Resource Reader: This is the orchestrator. By injecting Resource[] using the wildcard path classpath:data/*.csv, it finds every matching file in the directory. It then feeds these files, one by one, into the Delegate Reader. From the perspective of the broader batch job, it looks like one massive, continuous stream of data.
The Repository Item Writer: Rather than writing manual JDBC inserts, we leverage the RepositoryItemWriter. It takes our populated Employee objects and automatically calls the save method on our injected Spring Data JPA Repository.

package com.springjavalab.config;

import com.springjavalab.entity.Employee;
import com.springjavalab.repository.EmployeeRepository;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.job.builder.JobBuilder;
import org.springframework.batch.core.repository.JobRepository;
import org.springframework.batch.core.step.Step;
import org.springframework.batch.core.step.builder.StepBuilder;
import org.springframework.batch.infrastructure.item.data.RepositoryItemWriter;
import org.springframework.batch.infrastructure.item.file.FlatFileItemReader;
import org.springframework.batch.infrastructure.item.file.MultiResourceItemReader;
import org.springframework.batch.infrastructure.item.file.builder.FlatFileItemReaderBuilder;
import org.springframework.batch.infrastructure.item.file.mapping.BeanWrapperFieldSetMapper;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.Resource;
import org.springframework.transaction.PlatformTransactionManager;

@Configuration
public class BatchConfig {

    private final EmployeeRepository repository;

    public BatchConfig(EmployeeRepository repository) {
        this.repository = repository;
    }

    // The Delegate: Handles the actual parsing of a single file
    @Bean
    public FlatFileItemReader<Employee> delegateReader() {
        return new FlatFileItemReaderBuilder<Employee>()
                .name("employeeItemReader")
                .linesToSkip(1)
                .delimited()
                .names("id", "firstName", "lastName")
                .fieldSetMapper(new BeanWrapperFieldSetMapper<>() {{
                    setTargetType(Employee.class);
                }}).build();
    }

    // The Orchestrator: Feeds multiple files into the delegate
    @Bean
    public MultiResourceItemReader<Employee> multiResourceItemReader(
            @Value("classpath:data/*.csv") Resource[] resources) {
            
        MultiResourceItemReader<Employee> reader = new MultiResourceItemReader<>();
        reader.setDelegate(delegateReader());
        reader.setResources(resources);
        return reader;
    }

    // The Writer: Persists the data via Spring Data JPA
    @Bean
    public RepositoryItemWriter<Employee> writer() {
        RepositoryItemWriter<Employee> writer = new RepositoryItemWriter<>();
        writer.setRepository(repository);
        writer.setMethodName("save");
        return writer;
    }

    // The Step: Dictates the transactional chunk size
    @Bean
    public Step importMultiFileStep(JobRepository jobRepository, 
                                    PlatformTransactionManager transactionManager,
                                    MultiResourceItemReader<Employee> reader) {
        return new StepBuilder("importMultiFileStep", jobRepository)
                .<Employee, Employee>chunk(10, transactionManager)
                .reader(reader)
                .writer(writer())
                .build();
    }

    // The Job: The overarching execution container
    @Bean
    public Job importMultiFileJob(JobRepository jobRepository, Step step) {
        return new JobBuilder("importMultiFileJob", jobRepository)
                .start(step)
                .build();
    }
}

Understanding Chunk Processing: In our Step configuration, we define chunk(10). This is incredibly important for performance. It tells Spring Batch to read 10 records from the CSV files, hold them in memory, and then perform a single bulk insert transaction to the Oracle database. If an error occurs on record 9, only that chunk of 10 rolls back, protecting the integrity of your data.

6. Executing the Application

Finally, run your main application class. Because we set the job to trigger on startup, you will immediately see Spring Batch initialize the metadata tables, locate the wildcard CSV files, parse the data, and execute the JPA insert statements in your console logs.

package com.springjavalab;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class SpringBatchMultiFileImportApplication {
    public static void main(String[] args) {
        SpringApplication.run(SpringBatchMultiFileImportApplication.class, args);
    }
}

Conclusion

By utilizing the MultiResourceItemReader, you have successfully decoupled the complexity of file system management from your core parsing logic. This architecture is highly scalable. Whether your directory contains two files or two thousand files, Spring Batch will seamlessly stream the records into your Oracle database while strictly maintaining chunk-oriented transactional boundaries.

🔄 Deepen Your Spring Batch Knowledge

Mastering file imports is just the beginning. Explore how to add fault tolerance, validation, and multi-threading to your enterprise batch pipelines.

Spring Java Lab

📂 Explore Topics

🔥 Featured Posts

Spring Batch: How to Import Multiple CSV Files into a Database

1. Project Setup (pom.xml)

2. Application Properties and Database Configuration

3. Sample Input Data

4. The Domain Model and Repository

5. The Core Architecture: Batch Configuration

6. Executing the Application

Conclusion

🔄 Deepen Your Spring Batch Knowledge

🧱 Spring Batch Core Components

🔄 Spring Batch ItemProcessor Example

🚫 Skip Policy & Error Handling

🔁 Retry Mechanism Configuration

🧵 Multithreaded Processing

👂 JobExecutionListener Implementation

Subscribe on YouTube

Search This Blog