Apache hadoop YARN: Yet another resource negotiator

Vinod Kumar Vavilapalli, Arun C. Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans, Thomas Graves, Jason Lowe, Hitesh Shah, Siddharth Seth, Bikas Saha, Carlo Curino, Owen O'Malley, Sanjay Radia, Benjamin Reed, Eric Baldeschwieler

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1359 Citations (Scopus)

Abstract

The initial design of Apache Hadoop [1] was tightly focused on running massive, MapReduce jobs to process a web crawl. For increasingly diverse companies, Hadoop has become the data and computational agorá - the de facto place where data and computational resources are shared and accessed. This broad adoption and ubiquitous usage has stretched the initial design well beyond its intended target, exposing two key shortcomings: 1) tight coupling of a specific programming model with the resource management infrastructure, forcing developers to abuse the MapReduce programming model, and 2) centralized handling of jobs' control flow, which resulted in endless scalability concerns for the scheduler. In this paper, we summarize the design, development, and current state of deployment of the next generation of Hadoop's compute platform: YARN. The new architecture we introduced decouples the programming model from the resource management infrastructure, and delegates many scheduling functions (e.g., task fault-tolerance) to per-application components. We provide experimental evidence demonstrating the improvements we made, confirm improved efficiency by reporting the experience of running YARN on production environments (including 100% of Yahoo! grids), and confirm the flexibility claims by discussing the porting of several programming frameworks onto YARN viz. Dryad, Giraph, Hoya, Hadoop MapReduce, REEF, Spark, Storm, Tez.

Original languageEnglish
Title of host publicationProceedings of the 4th Annual Symposium on Cloud Computing, SoCC 2013
PublisherAssociation for Computing Machinery (ACM)
ISBN (Print)9781450324281
DOIs
Publication statusPublished - 01-01-2013
Event4th Annual Symposium on Cloud Computing, SoCC 2013 - Santa Clara, CA, United States
Duration: 01-10-201303-10-2013

Conference

Conference4th Annual Symposium on Cloud Computing, SoCC 2013
Country/TerritoryUnited States
CitySanta Clara, CA
Period01-10-1303-10-13

All Science Journal Classification (ASJC) codes

  • Software

Fingerprint

Dive into the research topics of 'Apache hadoop YARN: Yet another resource negotiator'. Together they form a unique fingerprint.

Cite this