FireWorks: a dynamic workflow system designed for high‐throughput applications
Lawrence Berkeley National Laboratory · University of California San Diego · +2 more institutions
Abstract
Summary This paper introduces FireWorks, a workflow software for running high‐throughput calculation workflows at supercomputing centers. FireWorks has been used to complete over 50 million CPU‐hours worth of computational chemistry and materials science calculations at the National Energy Research Supercomputing Center. It has been designed to serve the demanding high‐throughput computing needs of these applications, with extensive support for (i) concurrent execution through job packing, (ii) failure detection and correction, (iii) provenance and reporting for long‐running projects, (iv) automated duplicate detection, and (v) dynamic workflows (i.e., modifying the workflow graph during runtime). We have…
Citation impact
- FWCI
- 83.16
- Percentile
- 100%
- References
- 41
Authors
12Topics & keywords
- Workflow
- Computer science
- Supercomputer
- Python (programming language)
- Throughput
- Software
- Fireworks
- Operating system
Funding
- UDU.S. Department of EnergyAwards: -AC02-05CH11231, 05CH11231, AC02-05CH11231, DE-AC02, DE-AC02-05CH11231, DE-AC02-
- ECEuropean CommissionAwards: 321988, DE-AC02-05CH11231
- OOOffice of ScienceAwards: AC02-05CH11231, -AC02-05CH11231, DE-AC02
- OOOffice of Energy Efficiency and Renewable EnergyAward: DE‐AC02‐05CH11231
- BEBasic Energy SciencesAwards: DE-AC02, AC02-05CH11231, DE-AC02-05CH11231, -AC02-05CH11231
- BABiological and Environmental ResearchAward: DE‐AC02‐05CH11231
- LBLawrence Berkeley National LaboratoryAwards: DE-AC02-05CH11231, 05CH11231, AC02-05CH11231