Skip to main content
This document describes the JSON schema for a Trace. A Trace describes a benchmark result of a Solution on a Definition with a specific Workload. The collection of all Trace files forms the database of benchmark results. In a Trace object, solution and evaluation are optional. If they are not provided, it describes a workload entry in the dataset. In a Trace object, the definition and solution are externally linked through their names; workload and evaluation are embedded in the Trace object. This is because definition and solution are relatively large objects and will be used repeatedly in the dataset.

​
JSON Schema Description

​
Top-Level Object Structure

FieldTypeRequiredDescription
definitionstringYesThe name of the Definition used in this run.
workloadobjectYesAn object describing the specific input configuration for this run. See Workload.
solutionstringNoThe name of the Solution tested in this run.
evaluationobjectNoAn object containing the detailed results of this run.

​
evaluation : Benchmark Statistics Summary

This object represents a single, complete benchmark result.
FieldTypeRequiredDescription
statusstringYesThe final status of the evaluation run. Has to be one of the following:
"PASSED", "INCORRECT_SHAPE", "INCORRECT_NUMERICAL", "INCORRECT_DTYPE", "RUNTIME_ERROR", "COMPILE_ERROR".
logstringYesThe embedded record of the stdout and stderr of the evaluation run.
correctnessobjectYesThe summarized correctness results across all entries in the dataset.
performanceobjectYesThe summarized performance metrics across all entries in the dataset.
environmentobjectYesA snapshot of the hardware and software execution environment.
timestampstringYesThe ISO 8601 timestamp of when this summary was generated.

​
correctness : Correctness Summary

FieldTypeRequiredDescription
max_relative_errorfloatYesThe maximum relative difference found.
max_absolute_errorfloatYesThe maximum absolute difference found.

​
performance : Performance Summary

FieldTypeRequiredDescription
latency_msfloatYesThe mean latency in milliseconds per execution for this implementation.
reference_latency_msfloatYesThe mean latency of the Definition’s reference code on the same data/hardware.
speedup_factorfloatYesThe calculated speedup (reference_latency_ms / latency_ms).
Note that it’s normal for the speedup factor to be very large since the references are torch only, unoptimized implementations.

​
environment: Environment Definition Object

The environment object specifies the exact execution environment for this benchmark run.
FieldTypeRequiredDescription
hardwarestringYesThe name of the hardware, e.g., "NVIDIA_H100".
libsobjectYesA snapshot of the relevant software libraries and their versions. Keys are library names, and values are version strings.

​
The correctness and performance Nullable Table

The correctness and performance fields are set to be nullable depending on the status.
statuscorrectnessperformance
PASSEDRequiredRequired
INCORRECT_NUMERICALRequiredNone
INCORRECT_SHAPE/DTYPENoneNone
RUNTIME_ERRORNoneNone
COMPILE_ERRORNoneNone

​
Example: RMSNorm Trace

{
  "definition": "rmsnorm",
  "solution": "rmsnorm_triton_v1",
  "workload": {
    "uuid": "6120f144-b973-4bd9-b884-77ecb132914e",
    "axes": {
      "batch_size": 32
    },
    "inputs": {
      "input": {
        "type": "safetensors",
        "path": "/data/rmsnorm_evals/b32_input.safetensors",
        "tensor_key": "input"
      },
      "weight": {
        "type": "safetensors",
        "path": "/data/rmsnorm_evals/rmsnorm_weight.safetensors",
        "tensor_key": "weight"
      }
    }
  },
  "evaluation": {
    "status": "PASSED",
    "log": "...",
    "correctness": {
      "max_relative_error": 1.15e-05,
      "max_absolute_error": 0.89e-05
    },
    "performance": {
      "latency_ms": 0.008,
      "reference_latency_ms": 0.019,
      "speedup_factor": 2.375
    },
    "environment": {
      "hardware": "NVIDIA_H100",
      "libs": {
        "cuda": "12.6",
        "torch": "2.6.0",
        "triton": "2.4.0"
      }
    },
    "timestamp": "2025-06-27T12:45:00Z"
  }
}