An Efficient Task-Parallel Pipeline Programming Framework


The pipeline is a fundamental pattern to parallelize a series of stage tasks over a sequence of data in loops. Mainstream pipeline programming frameworks count on data abstractions to perform pipeline scheduling. Although this design is convenient for data-centric parallel applications, it is not efficient for algorithms that only exploit task parallelism in the pipeline. To address the limitation, we introduce a new task-parallel pipeline programming framework called Pipeflow. Pipeflow separates data abstractions and task scheduling, enabling a more efficient implementation of task-parallel pipeline algorithms than existing frameworks. We have evaluated Pipeflow on both micro-benchmarks and real-world applications. For example, in a timing analysis workload that explores pipeline parallelism to speed up the runtime performance, the Pipeflow’s implementation outperforms the oneTBB’s implementation up to 110.33% faster.

2024 International Conference on High Performance Computing in Asia-Pacific Region (HPCAsia)
Zizheng Guo
Zizheng Guo
Ph.D. Student

I am a Ph.D. candidate at Peking University. My research interests include data structures, algorithm design and GPU acceleration for combinatorial optimization problems.