The Post-mortem Processing & Visualization Layer

TOPSAIL post-mortem visualization relies on the MatrixBenchmarking.

MatrixBenchmarking consists of multiple components:

  • the benchmark component is in charge of running various test configurations. MatrixBenchmarking/benchmark is configured with a set of settings, with one or multiple values. The execution engine will go through each of the possible configurations and execute it to capture its performance.

  • the visualize component is in charge of the generation of plots and reports, based on the Dash and Plotly packages. MatrixBenchmarking/visualize is launched either against a single result directory, or against a directory with multiple results. The result directories can have been generated by TOPSAIL, which directly writes the relevant files (often the case there’s only one test executed, or when the test list is a simple iteration over a list of configurations), or via MatrixBenchmarking/benchmark (when the test list has to iterate over various, dynamically defined settings). This component is further described below.

  • the download component is in charge of downloading artifacts from S3, OpenShift CI or the Middleware Jenkins. Using this component instead of a simple scrapper allows downloading only the files important for the post-processing, or even only the cache file. This component is used when “re-plotting”, that is, when regenerating the visualization in the CI without re-running the tests.

  • the upload_lts component is used to upload the LTS (long term storage) payload and KPIs (key performance indicators) to OpenSearch. It is triggered at the end of a gating test.

  • the download_lts component is used to download the historical LTS payloads and KPIs from OpenSearch. It is used in gating test before running the regression analyze.

  • the analyze_lts component is used to check the results of a test against “similar” historical results. “similar” here means that the test results should have been executed with the same settings, except the so-called “comparison settings” (eg, the RHOAI version, the OCP version, etc). The regression analyze is done with the help of the datastax-labs/hunter package.

    In this document, we’ll focus on the visualize component, which is a key part of TOPSAIL test pipelines. (So are analyze_lts, download_lts and upload_lts for continuous performance testing, but they don’t require much per-project customization.)

TOPSAIL/MatrixBenchmarking visualization modules are split into two main components: the parsers (in store module) and plotters (in plotting module). In addition to that, the continuous performance testing (CPT) requires two extra components: the models (in the models module) and the regression analyze preps (in the analyze module).