Transaction Processing Performance Council (TPC) Launches an Artificial Intelligence Benchmark (TPCx-AI)

The first industry-standard benchmark for measuring real-world, end-to-end AI and ML scenarios – as well as data science use 

The Transaction Processing Performance Council (TPC) announced the immediate availability of TPCx-AI, the first industry-standard, vendor-neutral benchmark for measuring real-world, end-to-end AI and ML scenarios and data science use cases. TPCx-AI uses a diverse dataset and was specifically designed to be adaptable across a wide range of scale factors.

“The TPCx-AI benchmark is the result of collaboration between talented engineers and researchers at some of today’s leading AI organizations,” said Hamesh Patel, chair of the TPCx-AI committee and principal engineer at Intel Corporation. “It is designed to emulate real-world examples of organizations that use a variety of production ready data science pipelines – including both AI and ML approaches – and is now widely available to anyone who would like to download and run it. We look forward to feedback as industry experts, academics and others interested in benchmarking system performance begin to use it.”

TPCx-AI provides a means to evaluate performance for the System Under Test (SUT) as a general-purpose data science system that:

• Generates and processes large volumes of data
• Trains preprocessed data to produce realistic machine learning models
• Conducts accurate insights for real-world customer scenarios based on the generated models
• Can scale to large scale distributed configurations
• Allows for flexibility in configuration changes to meet the demands of the dynamic AI landscape

Additionally, the benchmark measures end-to-end time to provide insights for individual use cases, as well as throughput metrics to simulate multiuser environments for a given hardware, operating system, and data processing system configuration under a controlled, complex, multi-user AI or machine learning data science workload.

And as an “Express” class benchmark, TPCx-AI is an executable kit that can be rapidly deployed and measured. It is designed to provide relevant, objective performance data to industry users and is available for download via the TPC’s Web site via the following URL: http://tpc.org/tpcx-ai/default5.asp 

Organizations interested in contributing to the TPC’s benchmark development process are also encouraged to become members, and additional information is available via the following URL: http://tpc.org/information/about/join5.asp 

TPC members that contributed to the development of TPCx-AI include: Cisco, Dell, HPE, IBM, Intel, Microsoft, Red Hat, TTA, and VMware.

Source: The Transaction Processing Performance Council media announcement