Arrow Release Benchmark Report

Benchmark Run Summary

Run Type1	Commit SHA	Time of Commit	Hardware	Languages	Benchmark Type	Number of Benchmarks
baseline	b7d2f7f	2023-08-17 08:01:24	Intel(R) Core(TM) i9-9960X CPU @ 3.10GHz	Python, R	macrobenchmarks	218
contender	2dcee3f	2023-10-19 09:12:19	Intel(R) Core(TM) i9-9960X CPU @ 3.10GHz	Python, R	macrobenchmarks	194
baseline	b7d2f7f	2023-08-17 08:01:24	AMD Ryzen 5 PRO 4650GE with Radeon Graphics	C++, Java	microbenchmarks	3384
contender	2dcee3f	2023-10-19 09:12:19	AMD Ryzen 5 PRO 4650GE with Radeon Graphics	C++, Java	microbenchmarks	3368
baseline	b7d2f7f	2023-08-17 08:01:24	Intel(R) Core(TM) i9-9960X CPU @ 3.10GHz	JavaScript	microbenchmarks	88
1 When we compare benchmark results, we always have a contender (the new code that we are considering) and a baseline (the old code that were are comparing to). The historic distribution will be drawn from all benchmark results on commits in the baseline commit's git ancestry, up to and including all runs on the baseline commit itself. In this context, a baseline is typically the last Arrow release and the contender is the current release candidate.

Macrobenchmarks

Live Conbench UI views for the macrobenchmarks are available at this url. Conbench is an additional method to explore the results of the benchmarks particularly if you want to see results from more of the history or see more metadata.

Benchmark Percent Changes

• Benchmarks are plotted using the percent change from baseline to contender.
• Additional information on each benchmark is available by hovering over the relevant bar.

Python

dataframe-to-table

dataset-filter

dataset-read

dataset-select

dataset-selectivity

file-read

file-write

recursive-get-file-info

wide-dataframe

R

dataframe-to-table

file-read

file-write

partitioned-dataset-filter

tpch

Microbenchmarks

There are currently 3301 microbenchmarks in the Arrow benchmarks. The following comparisons are also available to be viewed in the Conbench UI.

Language	z-score threshold	Number of microbenchmarks
		Stable	Improvements	Regressions	Total
C++	5	2666	338	262	3266
Java	5	33	1	1	35

Because of the large number of benchmarks, the top 20 benchmark results that deviate most from the baseline in both the positive and negative directions are presented below. All microbenchmark results for this comparison can be explored interactively in the microbenchmark explorer.

Largest 20 regressions between baseline and contender

	Benchmark	Params	Analysis		Results
			z-score	Percent Change	Baseline result	Contender result	unit
arrow-bit-block-counter-benchmark
C++	BitBlockCounterFourWordsSumWithOffset	65536	−25.65	−297%	4,720,000,000	4,580,000,000	i/s1
C++	BitBlockCounterSumWithOffset	32768	−35.11	−594%	4,702,000,000	4,423,000,000	i/s1
arrow-small-vector-benchmark
C++	ConstructFromEmptyStdVector	<STATIC_VECTOR(int)>	−271.40	−1,434%	546,200,000	467,900,000	i/s1
parquet-bloom-filter-benchmark
C++	BM_BatchComputeHash	<DoubleType>	−40.18	−4,276%	441,400,000	252,600,000	i/s1
C++	BM_BatchComputeHash	<FloatType>	−952.50	−4,233%	518,400,000	299,000,000	i/s1
C++	BM_BatchComputeHash	<Int32Type>	−857.50	−4,714%	518,400,000	274,000,000	i/s1
C++	BM_BatchComputeHash	<Int64Type>	−44.16	−4,679%	441,400,000	234,900,000	i/s1
C++	BM_BatchComputeHash	<Int96Type>	−744.50	−3,799%	346,700,000	215,000,000	i/s1
C++	BM_BatchInsertHash	source=cpp-micro, suite=parquet-bloom-filter-benchmark	−69.44	−1,003%	151,400,000	136,200,000	i/s1
C++	BM_ComputeHash	<FloatType>	−46.65	−4,116%	329,000,000	193,600,000	i/s1
C++	BM_ComputeHash	<Int64Type>	−518.10	−4,661%	322,000,000	171,900,000	i/s1
C++	BM_ComputeHash	<Int96Type>	−467.00	−2,414%	258,600,000	196,200,000	i/s1
parquet-encoding-benchmark
C++	BM_ArrowBinaryDict	DecodeArrowNonNull_Dense/32768	−26.47	−1,277%	299	261	MB/s1
C++	BM_ArrowBinaryDict	DecodeArrow_Dense/32768	−28.26	−1,257%	299	261	MB/s1
C++	BM_DeltaBitPackingDecode_Int64_Narrow	1024	−37.98	−2,832%	6,719	4,816	MB/s1
C++	BM_DeltaBitPackingDecode_Int64_Wide	1024	−30.78	−2,686%	6,239	4,563	MB/s1
C++	BM_PlainEncodingSpacedBoolean	32768/1	−64.18	−4,447%	336	186	MB/s1
C++	BM_PlainEncodingSpacedBoolean	32768/100	−69.69	−4,480%	328	181	MB/s1
C++	BM_PlainEncodingSpacedBoolean	32768/1000	−54.68	−4,099%	273	161	MB/s1
C++	BM_PlainEncodingSpacedBoolean	32768/5000	−27.64	−2,390%	232	176	MB/s1
1 MB/s = megabytes per second; ns = nanoseconds; i/s = iterations per second

Largest 20 improvements between baseline and contender

	Benchmark	Params	Analysis		Results
			z-score	Percent Change	Baseline result	Contender result	unit
arrow-acero-aggregate-benchmark
C++	SumDoublesGroupedByMediumStringSet	1048576/0	157.20	8,678%	25,800,000	48,180,000	i/s1
C++	SumDoublesGroupedByMediumStringSet	1048576/1	166.30	9,103%	26,270,000	50,180,000	i/s1
C++	SumDoublesGroupedByMediumStringSet	1048576/10	151.10	8,360%	25,140,000	46,150,000	i/s1
C++	SumDoublesGroupedByMediumStringSet	1048576/100	159.70	8,616%	25,560,000	47,580,000	i/s1
C++	SumDoublesGroupedByMediumStringSet	1048576/10000	167.20	8,668%	25,730,000	48,040,000	i/s1
C++	SumDoublesGroupedByMediumStringSet	1048576/2	177.80	7,357%	23,390,000	40,600,000	i/s1
C++	SumDoublesGroupedBySmallIntStringPairSet	1048576/0	111.30	8,094%	25,380,000	45,930,000	i/s1
C++	SumDoublesGroupedBySmallIntStringPairSet	1048576/1	110.50	8,219%	25,510,000	46,480,000	i/s1
C++	SumDoublesGroupedBySmallIntStringPairSet	1048576/10000	111.40	8,077%	25,340,000	45,810,000	i/s1
C++	SumDoublesGroupedBySmallStringSet	1048576/0	170.40	9,960%	26,590,000	53,070,000	i/s1
C++	SumDoublesGroupedBySmallStringSet	1048576/1	177.10	10,100%	26,740,000	53,730,000	i/s1
C++	SumDoublesGroupedBySmallStringSet	1048576/10	164.70	9,384%	25,760,000	49,940,000	i/s1
C++	SumDoublesGroupedBySmallStringSet	1048576/100	170.40	9,586%	26,320,000	51,550,000	i/s1
C++	SumDoublesGroupedBySmallStringSet	1048576/10000	185.80	10,160%	26,190,000	52,800,000	i/s1
C++	SumDoublesGroupedBySmallStringSet	1048576/2	177.40	8,055%	23,740,000	42,870,000	i/s1
arrow-compute-scalar-round-benchmark
C++	RoundBinaryArrayBenchmark	<RoundBinary, UInt32Type, RoundMode::HALF_TOWARDS_ZERO>/size:524288/inverse_null_proportion:0	186.20	5,926%	345	550	MB/s1
C++	RoundBinaryArrayBenchmark	<RoundBinary, UInt32Type, RoundMode::HALF_TO_ODD>/size:524288/inverse_null_proportion:0	172.40	5,353%	348	535	MB/s1
parquet-encoding-benchmark
C++	BM_DeltaBitPackingEncode_Int64_Narrow	1024	473.00	9,765%	669	1,323	MB/s1
C++	BM_DeltaBitPackingEncode_Int64_Narrow	4096	812.40	10,600%	684	1,408	MB/s1
C++	BM_PlainEncodingBoolean	4096	159.50	1,590%	348	404	MB/s1
1 MB/s = megabytes per second; ns = nanoseconds; i/s = iterations per second

z-score distribution

Plotting the distribution of zscores for all microbenchmark results will help identify any systematic differences between the baseline and contender. The shape of the distribution of z-scores provides a sense of the overall performance of the contender relative to the baseline. Narrow distirbutions centered around 0 indicate that the contender is performing similarly to the baseline. Wider distributions indicate that the contender is performing differently than the baseline with left skewing indicating regressions and right skewing indicating improvements.

Plot.plot({
  y: {grid: true},
  x: {
    label: "z-score"
  },
  color: {legend: false},
  width: 1000,
  height: 400,
  marks: [
    Plot.rectY(microBmProced, Plot.binX({y: "count"}, {x: "analysis_lookback_z_score_z_score", fill: "grey", tip: true})),
    Plot.ruleY([0])
  ]
})

Plot = await import("https://esm.sh/@observablehq/plot");
import { aq, op } from '@uwdata/arquero';
boxWidth = 900
microBmProced = aq.from(transpose(ojs_micro_bm_proced))

Microbenchmark explorer

This microbenchmarks explorer allows you to filter the microbenchmark results by language, suite, and benchmark name and toggle regressions and improvements based on a threshold level of 5 z-scores. Languages, suite and benchmark name default to showing all results for that category. Additional benchmark parameters are displayed on the vertical axis resulting in each bar representing a case permutation. If a becnhmark does not have additional parameters, the full case permutation string is displayted. The display can be further filtered by selecting a specific language, suite, or benchmark name. Each bar can be clicked to open the Conbench UI page for that benchmark providing additional history and metadata for that case permutation.

viewof changes = Inputs.checkbox(["Regressions", "Improvements"], {
  label: md`**Benchmark Status**`,
  value: ["Regressions"]
  })

// Choose the state of the benchmark
microBmProcedChanges = {
  let microBmProcedParams;
  let hasRegressions = changes.includes("Regressions");
  let hasImprovements = changes.includes("Improvements");
  microBmProcedParams = microBmProced
      .params({hr: hasRegressions, hi: hasImprovements})
  if (hasRegressions && hasImprovements) {
    microBmProcedParams = microBmProced
      .filter((d, $) => d.analysis_pairwise_regression_indicated==$.hr || d.analysis_pairwise_improvement_indicated==$.hi);
  } else if (hasImprovements) {
    microBmProcedParams = microBmProced
      .filter((d, $) => d.analysis_pairwise_improvement_indicated==$.hi)
  } else if (hasRegressions) {
    microBmProcedParams = microBmProced
      .filter((d, $) => d.analysis_pairwise_regression_indicated==$.hr);
  } else {
    microBmProcedParams = microBmProced;
  }
  return microBmProcedParams;
}

// Choose the language
allLanguageValues = ["All languages"].concat(microBmProcedChanges.dedupe('language').array('language'))

viewof languageSelected = Inputs.select(allLanguageValues, {
    label: md`**Language**`,
    value: [allLanguageValues[0]],
    width: boxWidth
})

languages = {
  return (languageSelected === "All languages")
  ? microBmProcedChanges // If languageSelected is "All languages", no filtering is applied
  : microBmProcedChanges.filter(aq.escape(d => op.includes(d.language, languageSelected)));
}


allSuiteValues = ["All suites"].concat(languages.dedupe('suite').array('suite'))

// Choose the suite
viewof suiteSelected = Inputs.select(allSuiteValues, {
    label: md`**Suite**`,
    value: [allSuiteValues[0]],
    width: boxWidth
})


suites = {
  return (suiteSelected === "All suites")
  ? languages 
  : languages.filter(aq.escape(d => op.includes(d.suite, suiteSelected)));
}

allNameValues = ["All benchmarks"].concat(suites.dedupe('name').array('name'))

// Choose the benchmark
viewof nameSelected = Inputs.select(allNameValues, {
    label: md`**Benchmark Name**`,
    value: [allNameValues[0]],
    width: boxWidth
})

microBmProcedChangesFiltered = {
  return (nameSelected === "All benchmarks")
  ? suites 
  : suites.filter(aq.escape(d => op.includes(d.name, nameSelected)));
}

margins = {
  let hasRegressions = changes.includes("Regressions");
  let hasImprovements = changes.includes("Improvements");
  let margin = [300, 300];
  if (hasRegressions && hasImprovements) {
    margin = margin;
  } else if (hasImprovements) {
    margin = [0, 600];
  } else if (hasRegressions) {
    margin = [600, 0];
  } 
  return margin;
}

Plot.plot({
  width: 1200,
  height: (microBmProcedChangesFiltered.numRows()*30)+100, //adjust height of plot based on number of rows
  marginRight: margins[0],
  marginLeft: margins[1],
  label: null,
  x: {
    axis: "top",
    label: "% change",
    labelAnchor: "center",
    labelOffset: 30
  },
  style: {
    fontSize: "14px",
    fontFamily: "Roboto Mono"
    },
  color: {
    range: ojs_change_cols,
    domain: ojs_pn_lab,
    type: "categorical",
    legend: true
  },
  marks: [
    Plot.barX(microBmProcedChangesFiltered, {
      y: "params", 
      x: "change", 
      color: "black",
      fill: "pn_lab", 
      fillOpacity: 0.75,
      sort: {y: "x"},
      channels: {difference: "difference", params: "params"}, 
      href: "cb_url",
      tip: true
      }),
    Plot.gridX({stroke: "white", strokeOpacity: 0.5}),
    Plot.ruleX([0]),
    d3
      .groups(microBmProcedChangesFiltered, (d) => d.change > 0)
      .map(([posneg, dat]) => [
        Plot.axisY({
          x: 0,
          ticks: dat.map((d) => d.params),
          tickSize: 0,
          anchor: posneg ? "left" : "right"
        }),
      ])
  ]
})