Achieving High Performance With TCP Over 40 GbE on NUMA Architectures for CMS Data Acquisition

作者：Bawej Tomasz^*; Behrens Ulf; Branson James; Chaze Olivier; Cittolin Sergio; Darlea Georgiana Lavinia; Deldicque Christian; Dobson Marc; Dupont Aymeric; Erhan Samim; Forrest Andrew; Gigi Dominique; Glege Frank; Gomez Ceballos Guillelmo; Gomez Reino Robert; Hegeman Jeroen; Holzner Andre; Masetti Lorenzo; Meijers Frans; Meschi Emilio; Mommsen Remigius K; Morovic Srecko; Nunez Barranco Fernandez Carlos; O'Dell Vivian; Orsini Luciano; Paus Christoph; Petrucci Andrea

来源：IEEE Transactions on Nuclear Science, 2015, 62(3): 1091-1098.

DOI：10.1109/TNS.2015.2409898

• Data acquisition systems
• data communication
• distributed computing
• fast networks
• high energy physics computing
• software performance

TCP and the socket abstraction have barely changed over the last two decades, but at the network layer there has been a giant leap from a few megabits to 100 gigabits in bandwidth. At the same time, CPU architectures have evolved into the multi-core era and applications are expected to make full use of all available resources. Applications in the data acquisition domain based on the standard socket library running in a Non-Uniform Memory Access (NUMA) architecture are unable to reach full efficiency and scalability without the software being adequately aware about the IRQ (Interrupt Request), CPU and memory affinities. During the first long shutdown of LHC, the CMS DAQ system is going to be upgraded for operation from 2015 onwards and a new software component has been designed and developed in the CMS online framework for transferring data with sockets. This software attempts to wrap the low-level socket library to ease higher-level programming with an API based on an asynchronous event driven model similar to the DAT uDAPL API. It is an event-based application with NUMA optimizations, that allows for a high throughput of data across a large distributed system. This paper describes the architecture, the technologies involved and the performance measurements of the software in the context of the CMS distributed event building.

• 出版日期2015-6
• 单位MIT