From 100% CPU to Smooth Sailing: How We Fixed Jenkins Android Build Overload and Heartbeat Timeouts with Minimalistic Resources

Open

Every DevOps engineer and mobile developer has been there: you push a critical update, trigger the CI/CD pipeline, and wait. But instead of a green checkmark, your Jenkins agent grinds to a painful halt. The logs freeze, the server becomes unresponsive, and the build eventually crashes with cryptic infrastructure failures, agonizingly slow log streams, or HEARTBEAT_INTERVAL timeout errors.

When our team ran into this exact wall while building our Android application, our monitoring tools painted a grim picture: 100% CPU utilization, a skyrocketing load average, and a server fighting for survival.

Here is the story of how we diagnosed the underlying issue, unmasked the hidden performance killer in our native builds, and optimized our build arguments to transform an overloaded server into a stable, highly efficient deployment machine.

Important Note

The optimizations described in this article (--max-workers=1 and -Pj=1) worked extremely well when running a single Android executor on a single Jenkins node.

However, during testing we observed that increasing the Jenkins executor count to two or more on the same node caused the server load average to rise dramatically again, often exceeding 20+, with CPU utilization returning close to 100%.

While limiting Gradle workers and native compiler jobs significantly reduced resource contention within a single build, it could not eliminate the hardware limitations of the server. Multiple Android builds running simultaneously continued to compete for CPU, memory, and filesystem resources.

After extensive testing, no software-level tuning provided a reliable solution for running multiple Android executors on the same machine. The most effective and scalable solution was:

• One Android Executor per Node

• Scale horizontally by adding more Android build agents

• Avoid increasing executor count on the same machine

In short:

The Symptom: Server Strangulation & Heartbeat Timeouts

Our automated pipeline was configured to run a standard dual production build output (generating both an APK for internal testing and an AAB for the Google Play Store) using the native Gradle wrapper:

On paper, this is a standard command:

• ./gradlew: Utilizes the project-locked Gradle Wrapper to ensure environment consistency.

• :app:assembleRelease: Compiles and packages the release Android Application Package (APK).

• :app:bundleRelease: Compiles and packages the final Android App Bundle (AAB).

• --console=plain: Disables the dynamic, real time UI terminal processing in favor of flat, plain text streams ideal for capturing clean logs in Jenkins.

Yet, despite stripping down the logs, the Jenkins build environment routinely became sluggish. The server UI would lag heavily, and real time build logs would completely stall. Frequently, the server dropped connection entirely, throwing errors indicating that the internal runner had missed its expected HEARTBEAT_INTERVAL check in due to processing lag.

Open

The specifications of our machine were modest but standard for a mid tier CI/CD node:

The Breakthrough: How We Monitored the Server Load in Real Time

Because the Jenkins web interface completely froze during the build, we couldn't see what was happening inside the server. We needed a simple, lightweight way to record the server's health status into a file so we could inspect it later.

If you want to replicate our diagnosis on your Linux/Jenkins server, here is the exact beginner friendly method we used:

Step 1: Create a Monitoring Script

We logged into our Ubuntu Linux build server via terminal and created a small automated script. This script asks the server for its performance metrics every 5 seconds and saves them to a file.

Create a script file named monitor.sh:

Paste this simple script inside:

Step 2: Run the Script in the Background

Make the script executable and start it up right before running a fresh Jenkins build:

(The & at the end is a Linux trick that keeps the script running safely in the background, freeing up your terminal).

Step 3: Monitor the Output Live

To see the load numbers ticking in real time while Jenkins ran, we used the Linux tail command:

The Investigation: Unmasking clang++ and System Overload

When the build hit its peak, our log file (loadAverage.txt) revealed a terrifying spike:

To understand why a 4 core machine was collapsing, we need to look closely at Load Average.

Load average tracks the number of tasks currently using the CPU or waiting in line for their turn. For a machine with 4 physical/virtual cores, a load average of 4.0 means the CPU is at perfect 100% capacity. Anything beyond 4.0 means processes are actively stalling, piling up in a traffic jam, and starving each other of system time. Our load average of 9 to 12+ meant our system was operating at 2x to 5x its physical capacity!

When we looked deeper into the active system processes, the true culprit wasn’t Gradle itself it was a swarm of processes labeled clang++.

What is clang++?

If your app relies on a heavy native layer such as a modern React Native application, an Expo project with custom native plugins, or an application utilizing the Android NDK (Native Development Kit) your codebase uses underlying C++ modules.

clang++ is the compiler responsible for turning that native C++ code into machine code your phone can understand. Because C++ compilation is incredibly math heavy, each spawned clang++ process consumes a massive amount of raw CPU power and RAM.

By default, Gradle operates under a friendly philosophy: Use every resource available to finish the job as fast as possible. Out of the box, it automatically assumes it can aggressively spin up multiple background workers and native compiler jobs simultaneously. While this works beautifully on a high end 16 core developer laptop, it acts as an unintentional crash trigger against a restricted 4 core CI server.

The Myth of Parallelism: When More Workers = Slower Builds

The most common mistake in pipeline design is a simple assumption:

More Parallel Workers = Faster Builds

When a system runs out of CPU and RAM, excessive parallelism causes a severe performance drop known as Process Thrashing.

Instead of spending useful CPU cycles compiling our actual code, our 4 cores spent all their time performing context switching constantly pausing one clang++ process, saving its state, loading another process, running it for a millisecond, pausing it again, and repeating. The machine was spending more energy managing its own chaotic process traffic jam than actually compiling the application, resulting in the massive lag and missed heartbeats we observed.

Why Increasing the Timeout Limit Fails

When faced with this problem, a common workaround is to simply increase the Jenkins HEARTBEAT_INTERVAL timeout value in the system settings. While this keeps the connection alive longer and might temporarily stop the pipeline from throwing an immediate error, it is merely a band aid. It does absolutely nothing to solve the underlying server load the machine remains completely choked, slow, and structurally unstable.

The Strategy: Forcing Resource Discipline

To bring stability back to our infrastructure, we had to strictly limit the parallelism of both the high level task manager (Gradle) and the low level native compiler (clang++). We achieved this by introducing two simple arguments into our build command: --max-workers and -Pj.

1. Controlling Gradle with --max-workers

The --max-workers flag sets a hard ceiling on the number of concurrent tasks Gradle can execute at any given second.

By forcing --max-workers=1, we told Gradle to process our tasks one after the other. This dramatically lowers the memory footprint and stops multiple tasks from independently demanding heavy resources at the same moment.

2. Controlling the Native Compiler with -Pj

While --max-workers manages top-level Gradle execution, native C++ compilation loops often ignore it. In many native environments (like React Native/NDK extensions), the build scripts accept a project property argument specified as -Pj.

This argument injects a configuration parameter directly down into the underlying native build tools (like make or Ninja), telling them how many concurrent parallel compiler jobs to run.

Setting -Pj=1 explicitly overrides the native automation, forcing the environment to compile precisely one C++ native file at a time, eliminating the sudden swarm of multiple running clang++ processes.

The Fix: Implementation and Command Layout

We updated our production build execution script inside our Jenkins pipeline file to explicitly enforce structural discipline:

Tuning Framework for CI Servers

If you are managing your own self hosted or cloud runner instances, here is a recommended rule of thumb to avoid server crashes based on your resource allocations:

Results: Before vs. After

The transformation of our build logs and server health metrics after rolling out these changes was immediate.

Before Optimization

• System Behaviour: Extreme terminal UI lag, intermittent HEARTBEAT_INTERVAL connection drops, frozen pipeline progress.

• Build Duration: 32 – 38 minutes (unstable, frequently crashing midway).

• Server State (loadAverage.txt): ```text

  load average: 9.94, 10.55, 17.34

  CPU utilization: 100%

  Active Processes: [clang++, clang++, clang++, clang++, gradlew, java]

  Open

  
  Open

  
  Open

  

After Optimization

• System Behaviour: Fluid, readable console log updates, 100% stable runner to master communication, predictable and consistent pipeline execution.

• Build Duration: 32 – 38 minutes (100% stable, zero crashes).

• Server State (loadAverage.txt):

  load average: 2.30, 3.94, 3.91 (Healthy Threshold for 4 Cores)
  CPU utilization: 55% - 70%
  Active Processes: [clang++ (single instance), gradlew, java]
  
  

  Open

  
• Open

  
  Open

  

Note on Strategy: Our goal here wasn't speed optimization it was graceful resource utilization. We realized that trying to force a 4-core machine to compile faster by overloading it was a losing battle. Instead, we shifted our focus to keeping the CPU stable and preventing crashes, which allowed our 32-38 minute build to complete reliably without choking the server.

Key Takeaways

Optimizing CI/CD pipelines is not always about making builds more parallel.

Sometimes the fastest path to stability is reducing concurrency and aligning build behavior with available hardware resources.

For our Jenkins Android build server:

• Limiting Gradle workers reduced CPU contention.

• Limiting native compiler jobs prevented clang++ storms.

• Server load average dropped from 13–14+ to around 3–6.

• Jenkins became stable and responsive again.

• Resource Grace over Raw Speed: Optimization doesn't always mean making a build faster. True optimization means utilizing system resources gracefully so the server remains stable, responsive, and predictable.

Most importantly, we learned that for Android builds on resource-constrained infrastructure, horizontal scaling with multiple single executor nodes is often more effective than increasing concurrency on a single machine.

Sometimes the best optimization is not running more things at once, it's running the right number of things at the right time.