Anomaly when benchmarking C binaries

06:39 17 Mar 2026

I'm in an internship in a lab and I'm doing some benchmark on micro architecture. I use https://github.com/FoRTE-Research/UlSWaP-Bench/tree/master and I have modified the mainmain.c. The goal of the bench is to mesure the maximum number of execution of a program in 5 seconds, like we run the maximum number of a specific binary in 5 seconds, and we do that 30 times. I save everything in some txt files and I draw charts of the metrics in python but that part is not important. I just find out that sometimes there's some anomaly when running the program. I run 5 warmup run, to avoid cold caches, then I run the 30 real run. I just saw on my chart that the standard deviation of the average number of execution is like abnormal. Here's my metrics:

That's the warmup runs

And that's the real run, for the lorawan_up program:

I don't know why I have a value so low, on some program I don't have that issue, on some bench, it never happen, maybe it's a linux problem ? But I don't think so, and maybe my C code smell, here's my mainmain.c that has been modify:

#include "common.h"
#include "misc.h"
#include "warmup.h"
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

int benchmark_main(void);
extern void hexstring(uint32_t num);

int main(int argc, char *argv[]) {
  char *profile = argv[1];
  char *bin = argv[2];
  char *machine = argv[3];

  printf("Running bench for machine: %s, using profile: %s\n", machine,
         profile);
  printf("Running %d warmup benchmarks.\n", WARMUP_RUN);
  printf("Running maximum benchmarks in %fs.\n", SECONDS_TO_RUN);
  printf("Running %d time the maximum benchmarks.\n", RUNS);

#ifdef CUSTOM_ARCH_STARTUP
  run_arch_startup();
#endif // CUSTOM_ARCH_STARTUP
#ifdef WARMUP_RUN
  printf("Running warmup benchmarks...\n");
  warmup_run(machine, bin, profile);
  printf("Warmup benchmarks finished.\n");
#endif
  printf("Running benchmarks...\n");

  int result_arr[RUNS];

  for (int i = 0; i < RUNS; i++) {

    // Define start and end timer variable
    time_t start, end;
    // Define the number of run than the program has make in 10 seconds
    int nb_r = 0;
    // Number of seconds elapsed
    double elapsed;
    // Define if the program should terminate or not
    int terminate = 1;
    // Start timer
    start = time(NULL);
    while (terminate) {
      // Get the time and compare with start
      end = time(NULL);
      elapsed = difftime(end, start);
      // End the program if the program run for more or equal to 10 seconds
      if (elapsed >= SECONDS_TO_RUN /* seconds */) {
        terminate = 0;
      } else {
        // Run the given benchmark
        benchmark_main();
        // Increment the number of benchmark run
        nb_r++;
      }
    }

    // Store the number of benchmark run in result_arr
    result_arr[i] = nb_r;
  }

  char result_file[60];
  // Don't handle result for now
  snprintf(result_file, "../logs/%s/%s/%s.txt", machine, profile, bin);
  // Open file, create if doesn't exist
  FILE *fp = fopen(result_file, "w");
  if (fp == NULL) {
    printf("Error opening profile result file. File coulnd't be opened.\n");
    exit(1);
  }
  for (int i = 0; i < RUNS; i++) {
    int nb = result_arr[i];
    fprintf(fp, "%d\n", nb);
  }
  fclose(fp);

#if CHECKSUM_TEST
  uint32_t checksum = get_benchmark_checksum();
  hexstring(checksum);
#endif // CHECKSUM_TEST

#ifdef CUSTOM_ARCH_FINISH
  run_arch_finish();
#endif // CUSTOM_ARCH_FINISH

  return 0;
}

The warmup_run function is like the same logic as when I run the benchmark_main and I save the metrics etc. I use a config.h to define some constant:


// Number of time we run the `SECONDS_TO_RUN`
#define RUNS 30
// Number of seconds we run benchmarks until exit
#define SECONDS_TO_RUN 5.0
// Warm up run
#ifndef WARMUP_RUN
#define WARMUP_RUN 5
#endif // WARMUP_RUN

The anomaly always happened on the first line of the real run. Sometimes on the first and second lines.

c compilation benchmarking

Your Answer

Privacy & Cookie Consent