Hi this one is really puzzling me. I am trying to run a separate instance of a spring batch job for every file, up to 20 at a time.
I've created a stripped down job of how im using it to reproduce the issue. This is how I invoke the jobs via main class:
public class Main implements CommandLineRunner {
private final JobLauncher jobLauncher;
private final Job jobConfig;
private int parallelism = 20;
public static void main(String[] args) {SpringApplication.run(Main.class, args);}
@Override
public void run(String... args) throws Exception {
File[] files = getFiles();
ExecutorService pool = Executors.newFixedThreadPool(parallelism);
List> futures = new ArrayList<>(files.length);
for (int i = 0; i < files.length; i++) {
final File file = files[i];
futures.add(pool.submit(() -> {
try {
JobParameters params = new JobParametersBuilder()
.addString("fileName", file.getCanonicalPath(), true)
.addString("guid", UUID.randomUUID().toString(), true)
.toJobParameters();
JobExecution exec = jobLauncher.run(jobConfig, params);
return exec.getExitStatus();
} catch (Exception e) {
throw new RuntimeException(e);
}
}));
}
for (Future f : futures) {
f.get();
}
pool.shutdown();
}
public static File[] getFiles() {
File currentDir = new File("C:\\Code\\spring-batch\\input");
File[] files = currentDir.listFiles((dir, name) -> name.endsWith(".csv"));
return files;
}
And here is the job. All it's doing is creating an output file based upon the inputFile name.
@Configuration
@RequiredArgsConstructor
public class JobConfig {
@Bean
public PlatformTransactionManager transactionManager() {
return new ResourcelessTransactionManager();
}
@Bean
public JobLauncher jobLauncher(JobRepository jobRepository) throws Exception {
TaskExecutorJobLauncher launcher = new TaskExecutorJobLauncher();
launcher.setJobRepository(jobRepository);
launcher.setTaskExecutor(new org.springframework.core.task.SyncTaskExecutor());
launcher.afterPropertiesSet();
return launcher;
}
@Bean
public Job simpleJob(JobRepository jobRepository, Step simpleStep) {
return new JobBuilder("simpleJob", jobRepository)
.incrementer(new RunIdIncrementer())
.start(simpleStep)
.build();
}
@Bean
public Step simpleStep(JobRepository jobRepository,
PlatformTransactionManager transactionManager,
ItemReader reader,
ItemProcessor processor,
FlatFileItemWriter writer) {
return new StepBuilder("simpleStep", jobRepository)
.chunk(100, transactionManager)
.reader(reader)
.processor(processor)
.writer(writer)
.stream(writer)
.build();
}
@Bean
@StepScope
public FlatFileItemReader reader(
@Value("#{jobParameters['fileName']}") String fileName,
@Value("#{jobParameters['guid']}") String guid) {
return new FlatFileItemReaderBuilder()
.name("reader-" + guid)
.resource(new FileSystemResource(fileName))
.lineMapper((line, lineNumber) -> line)
.build();
}
@Bean
public ItemProcessor processor() {
return item -> item.toUpperCase();
}
@Bean
@StepScope
public FlatFileItemWriter writer(
@Value("#{jobParameters['fileName']}") String fileName,
@Value("#{jobParameters['guid']}") String guid) {
String outputName = "OUTPUT_" + getFileNameWithoutExtension(fileName) + "_" + ".out";
return new FlatFileItemWriterBuilder()
.name("writer-" + guid)
.resource(new FileSystemResource(outputName))
.lineAggregator(new PassThroughLineAggregator<>())
.build();
}
public static String getFileNameWithoutExtension(String fileName) {
String filename = java.nio.file.Paths.get(fileName).getFileName().toString();
int dotIndex = filename.lastIndexOf('.');
if (dotIndex > 0) {
return filename.substring(0, dotIndex);
}
return fileName;
}
}
I can confirm that this returns one of each CSV file, there are no duplicates and the File[] array is unique. The problem is, somehow the joblauncher is launching instances with the exact same file name and guid. Here's my output, you can see there are several filenames launched multiple times with the same UUID (ex_13.csv, ex_19.csv)
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [ool-2-thread-20] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_18.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=78696daa-4df3-473f-87cf-42b9a7dc1169, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [pool-2-thread-4] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_12.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=94a99043-91e6-4466-87e1-b70683e0ab5b, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [pool-2-thread-5] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_13.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=f9c2ecae-f68a-4c79-a5ef-4af7e0f3930c, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [ool-2-thread-15] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_13.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=f9c2ecae-f68a-4c79-a5ef-4af7e0f3930c, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [pool-2-thread-1] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_1.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=1c21c4a5-4956-4fcf-8cce-868ab1bf78cb, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [pool-2-thread-8] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_16.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=27042f9d-f963-4d96-ad6e-b5dabe18ad2e, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [pool-2-thread-7] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_15.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=ef576a4e-d225-4eb2-91a4-b3fb1d8b562d, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [pool-2-thread-3] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_11.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=1b596d93-4311-44e7-9251-84f8eaa9ba21, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [ool-2-thread-13] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_20.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=18bde259-67ea-4c0d-a20d-3c82b654019b, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [ool-2-thread-18] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_7.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=e8885719-be1e-4190-996b-011d77cc015b, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [ool-2-thread-10] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_18.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=78696daa-4df3-473f-87cf-42b9a7dc1169, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [pool-2-thread-9] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_19.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=f818a9c9-65c2-4fae-b4d4-0f39c0b79a3f, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [ool-2-thread-17] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_10.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=c47e6895-3ab0-483e-bc2a-adbd334bd33e, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [ool-2-thread-11] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_19.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=f818a9c9-65c2-4fae-b4d4-0f39c0b79a3f, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [pool-2-thread-2] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_10.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=c47e6895-3ab0-483e-bc2a-adbd334bd33e, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [ool-2-thread-19] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_8.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=1bc84fd7-4cf3-4163-a406-2edbcd297045, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [ool-2-thread-16] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_5.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=5166ebc1-5e03-4d25-a332-9a04cbc2530b, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [ool-2-thread-12] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_10.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=c47e6895-3ab0-483e-bc2a-adbd334bd33e, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [pool-2-thread-6] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_8.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=1bc84fd7-4cf3-4163-a406-2edbcd297045, type=class java.lang.String, identifying=true}}]
2026-02-28T23:15:53.606-05:00 INFO 16316 --- [ool-2-thread-14] o.s.b.c.l.s.TaskExecutorJobLauncher : Job: [SimpleJob: [name=simpleJob]] launched with the following parameters: [{JobParameter{name='fileName', value=C:\Code\spring-batch\input\ex_3.csv, type=class java.lang.String, identifying=true},JobParameter{name='guid', value=1bb5fea1-333f-4bba-b6c7-ece83149bd18, type=class java.lang.String, identifying=true}}]