CAREER: Robust LSM-Based Data Stores

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).

Human activity generates data at an unprecedented pace. At the same time, these ever-increasing data sets are used to harness information to make critical and everyday decisions. In that setting, data systems support every aspect of human activity by offering efficient ingestion of incoming data and quick access to perform analysis tasks.

To ensure efficient data ingestion and analysis, data systems developers and administrators tune these systems based on the expectation (or knowledge) of the workload; that is, the operations to be submitted. Further, data systems aim to offer both fast and predictable performance; however, their deployment faces several challenges. On the one hand, many applications are volatile because of factors such as variations in access patterns or that the frequency of operations submitted varies wildly throughout the day.

Thus, it is hard to know what workload to expect and, as a result, to tune them. On the other hand, data systems are increasingly deployed in shared infrastructure like public and private clouds, which brings a new set of challenges: the new, more diverse hardware and infrastructure makes the execution environment even more unpredictable, since compute, memory, and storage resource availability may also vary.

Further, storing data in shared infrastructure must comply with new regulations, e.g., about privacy and data stewardship. The project will develop data systems that offer high and predictable performance and make regulatory requirements a first-class citizen. Ultimately, the project will make it easier for non-expert users to deploy data systems in the wild.

The researchers will develop a new breed of robust log-structured merge (LSM) based data systems that can offer near-optimal performance despite potential uncertainty in the execution setting and increase performance predictability. The project focuses on LSM-based data systems, a key technology used as the backbone of several data system designs today, including SQL, NoSQL, key-value stores, and time-series management.

In order to address the execution setting volatility and provide an ideal LSM data system design, the project takes a new radical approach, one that incorporates uncertainty in tuning input (e.g., workload, resource availability) and addresses both the unpredictability of key LSM operations (like compactions) and the inaccuracy of cost-models. The proposal investigates techniques that combine state-of-the-art tuning methods with robust optimization while building on recent algorithmic and hardware advancements.

This work will produce systems that depend less on the specific execution instance and exhibit stable performance despite workload and resource variability. As a result, robust LSM-based data systems will require less human intervention, and they will be more appealing to both users and administrators, allowing organizations to widely deploy reliable systems to accelerate and support data-intensive scientific discovery and applications.

Such ease of deployment is timely as data management is increasingly becoming an automatic service, where human experts cannot always be in the loop to hand-tune and optimize data systems.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.