Spring Batch: Running Batch Jobs Asynchronously

• Feb 11, 2023
• Batch Processing

Spring Batch: Limit How Frequent a Batch Job Runs

Often in designing batch jobs you want to limit the frequency that a job can run. For example, if you want a batch job to run once a day, you can use @Scheduled annotation to run the batch job once a day. This is enough if you only run one instance of the batch program and there are no other ways to launch the job.

But if you have manual ways to launch the same job, or you are you using kubernetes to run multiple instances of your batch program, the batch job may be attempted to run multiple times in one day, even if you intend to only run once a day.

• Jan 11, 2023
• Batch Processing

Spring Batch: Sharing Batch Step Configurations

When defining a Spring batch step, often common configurations are added to almost every step. In order to not violate DRY (Don’t Repeat Yourself), a StepBuilder can be customized upstream with all the shared configurations.

Configuring the StepBuilder

@Configuration
public class SharedStepBuilderFactory extends StepBuilderFactory {

    @Autowired
    public JobRepository jobRepository;

    @Autowired
    public PlatformTransactionManager transactionManager;

    // Inject shared values here and use them as well
    @Value
    private String logDirectory;

    public SharedStepBuilderFactory(JobRepository jobRepository, PlatformTransactionManager transactionManager) {
        super(jobRepository, transactionManager);
    }

    @Override
    public StepBuilder get(final String name) {
        // Get the default step builder
        final StepBuilder builder = super.get(name);

        // Add listeners you want for EVERY step
        stepBuilder.listener(new StepLoggingListener());
        stepBuilder.listener(getCustomLoggingListener(logDirectory);
        return stepBuilder;
        }
}

Implementation

@Configuration
public class StepConfig {

    @Bean
    public Step getStepOne(final SharedStepBuilderFactory sharedStepBuilderFactory) {
        return sharedStepBuilderFactory
        .get("stepOne") 
        .reader(stepOneReader())
        .writer(stepOneWriter())
        .build(); // The listeners are already configured for this step from the SharedStepBuilderFactory
    }
}

• Nov 23, 2022
• Java

Java Tip: Method Reference

Instead of

for(String name: names) {
	System.out.println(name);
}

Do

// This is called a method reference
names.foreach(System.out::println);

• Sep 11, 2022
• Database

Postgres: Query Results From Today Only

When you want to query only on results from today in a Postgres database use the >= now()::date syntax.

Example

select *
from orders 
where create_time >= now()::date

• Jul 14, 2020
• Programming

Rules of Thumb for Software Developers

Fetching Data

• Remember Performance issues go unnoticed during development because the data set is too small - Vlad Mihalcea
• A transaction should fetch only as much data as required by the current executing business logic - Vlad Mihalcea

Optimization

• Premature optimization is the root of all evil - Donald E. Knuth
• Measure performance before and after each attempted optimization. J Bloch
• Programs spend 90 percent of their time in 10 percent of the code.

Design

• The components of design that are most difficult to change after the fact are those specifying interactions between components and with the outside world. Cheif among these design components are APIs, wire-level protocols, and persistent data formats. Bloch
• Strive to write good programs rather than fast ones. J Block

• Jun 20, 2020
• Command Line

Find a File From the Command Line

To find a file from the command line:

find . -type f -iname "*pattern_search_for*"

• Jul 18, 2019
• DevOps

Install Heroku CLI and Setup Heroku Autocomplete

On a Mac command line

brew tap heroku/brew && brew install heroku

Enable heroku autocomplete

heroku autocomplete
printf "$(heroku autocomplete:script bash)" >> ~/.bashrc; source ~/.bashrc

To use autocomplete type: Heroku and Tab,Tab

• Apr 20, 2019
• Git

Git: Pretty Git Graphs

In order to view git history from the command line:

git log

However if you want to view the history in a cleaner format I recommend a custom git log.

Add the following to .gitconfig (typically at ~.gitconfig)

[alias]
    lg = lg1
    lg1 = lg1-specific --all
    lg2 = lg2-specific --all

    lg1-specific = log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%s%C(reset) %C(dim white)- %an%C(reset)%C(auto)%d%C(reset)'
    lg2-specific = log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset)%C(auto)%d%C(reset)%n''          %C(white)%s%C(reset) %C(dim white)- %an%C(reset)'

Now from the command line run:

• Jan 01, 0001
• Java

Java Tip: ComputeIfAbsent

Often we have to loop through a list of items and convert it into a map, with the key being some sort of category and the value being a set of the associated items ie Map<Key, Set<Object>>. The old way of doing this looped through the list and before adding checked if the key was present. Using computeIfAbsent we can clean up this syntax considerably.

Instead of

// Map of item types to items
Map<String, Set<Items>> map = new HashMap<>();

for (Item item: items) {
    if (map.containsKey(item.getType())) {
        map.get(item.getType().add(item));
    } else {
        Set newSet = new HashSet<>();
        newSet.add(item);
        map.put(item.getType(), newSet));
    }
}

Do

// Map of item types to items
Map<String, Set<Items>> map = new HashMap<>();

for (Item item: items) {
    map.computeIfAbsent(item.getType(), v -> new HashSet<>()).
    add(item);
}

• Jan 01, 0001
• Batch Processing

Spring Batch: Query All the Steps of a Batch Job

In Spring Batch, in order to get the job_execution_id of the last batch job instance for a given batch job name use this query:

select bje.job_execution_id from batch_job_instance bji
join batch_job_execution bje
on bji.job_instance_id = bje.job_instance_id
where bji.job_name = 'jobname'
order by bje.start_time desc
limit 1;

In order to get all the steps for the latest batch job instance for a given batch job name use this query:

select *
from batch_job_execution bje
join batch_job_instance bji
on bje.job_instance_id = bji.job_instance_id
join batch_job_step_execution bse
on bse.job_execution_id = bje.job_execution_id
and bje.job_execution_id = 
(
    select bje.job_execution_id from batch_job_instance bji
    join batch_job_execution bje
    on bji.job_instance_id = bje.job_instance_id
    where bji.job_name = 'jobname'
    order by bje.start_time desc
    limit 1
)
order by bse.start_time;