DryadLINQ: a system for general-purpose distributed data-parallel computing using a high-level language

DryadLINQ is a system and a set of language extensions that enable a new programming model for large scale dis-tributed computing. It generalizes previous execution en-vironments such as SQL, MapReduce, and Dryad in two ways: by adopting an expressive data model of strongly typed.NET objects; and by supporting general-purpose imperative and declarative operations on datasets within a traditional high-level programming language. A DryadLINQ program is a sequential program com-posed of LINQ expressions performing arbitrary side-effect-free transformations on datasets, and can be writ-ten and debugged using standard.NET development tools. The DryadLINQ system automatically and trans-parently translates the…