2012 Mar 19 at 10:30
DC 1304
Swapnil V. Patil, PhD candidate, Comp. Sci., Dept. Carnegie Mellon University
The growing ecosystem of data-intensive applications is creating new, challenging workloads for existing data storage systems. One sThe growing ecosystem of data-intensive applications is creating new, challenging workloads for existing data storage systems. One such workload is dominated by small objects, i.e. applications running on hundreds of thousands of CPU cores, and concurrently creating and accessing large numbers of small objects at very high speeds. Examples of such applications include supercomputing, Internet services, and business analytics. Unfortunately, most modern cluster file systems deliver scalable performance for large files, but not for large numbers of files. This led to the adoption of plethora of special-purpose data stores, popularly called ``NoSQL'' stores, with custom designs and semantics. This variety has its own perils: choosing the right store for your needs is hard, debugging application performance is complex, and understanding system behavior is complex. I present a two-pronged approach driven by the questions: (1) how do we extend cluster file systems with scalable small-file access? and (2) what are the implications of design decisions made by different ``NoSQL'' stores on application performance?
First, I describe a new directory architecture, called GIGA+, that enables each directory in a cluster file system to store millions to billions of files and sustain massively concurrent file creations every second. GIGA+ uses a distributed and load-balanced directory partitioning scheme that enables highly concurrent growth through asynchrony and eventual consistency. GIGA+ is implemented as a (POSIX-compliant) user-level file system that can be layered on existing, unmodified file system deployments; this prototype layered on local Linux file systems delivers a highly scalable performance -- surpassing the most demanding high-performance computing requirements. I will also discuss programmability and usability lessons learnt from running GIGA+ at scale.
Second, I present a tool, called YCSB++, that helps in understanding the effect of design choices of different ``NoSQL'' data stores on application performance. YCSB++ enables developers to construct large, parallel tests to study advanced features of table stores and to correlate observed performance with the behavior of different components in the system. YCSB++ helps in analyzing advanced features including weak consistency, high-speed ingest optimizations, server-side filtering, and access control.
Bio ==== Swapnil Patil is a Ph.D. student in the Computer Science Department of the Carnegie Mellon University. He is easily distracted by end-to-end issues in computer systems; his current focus is on large-scale systems infrastructure and software systems for emerging hardware technologies. His dissertation research on GIGA+ scalable file system directories has won the ACM Student Research Competition Award (graduate category) in 2011. He is a (die-hard) Steelers fan, an avid squash player and long-distance runner, and an espresso aficionado.