Academia.eduAcademia.edu

Outline

Title
Abstract
Key Takeaways
Introduction
Background and Motivation
Design Challenges
Design Criteria
Related Work
Conclusion
4 Performance Study with Microbenchmark
References
All Topics
Computer Science
Distributed Systems

Distributed Lock Management with RDMA

Profile image of Dong Young YoonDong Young Yoon

2018, Proceedings of the 2018 International Conference on Management of Data

https://doi.org/10.1145/3183713.3196890
visibility

…

description

16 pages

link

1 file

Abstract

Lock managers are a crucial component of modern distributed systems. However, with the increasing availability of fast RDMAenabled networks, traditional lock managers can no longer keep up with the latency and throughput requirements of modern systems. Centralized lock managers can ensure fairness and prevent starvation using global knowledge of the system, but are themselves single points of contention and failure. Consequently, they fall short in leveraging the full potential of RDMA networks. On the other hand, decentralized (RDMA-based) lock managers either completely sacrifice global knowledge to achieve higher throughput at the risk of starvation and higher tail latencies, or they resort to costly communications in order to maintain global knowledge, which can result in significantly lower throughput. In this paper, we show that it is possible for a lock manager to be fully decentralized and yet exchange the partial knowledge necessary for preventing starvation and thereby reducing tail latencies. Our main observation is that we can design a lock manager primarily using RDMA's fetch-and-add (FA) operations, which always succeed, rather than compare-and-swap (CAS) operations, which only succeed if a given condition is satisfied. While this requires us to rethink the locking mechanism from the ground up, it enables us to sidestep the performance drawbacks of the previous CAS-based proposals that relied solely on blind retries upon lock conflicts. Specifically, we present DSLR (Decentralized and Starvation-free Lock management with RDMA), a decentralized lock manager that targets distributed systems running on RDMA-enabled networks. We demonstrate that, despite being fully decentralized, DSLR prevents starvation and blind retries by guaranteeing first-comefirst-serve (FCFS) scheduling without maintaining explicit queues. We adapt Lamport's bakery algorithm [36] to an RDMA-enabled environment with multiple bakers, utilizing only one-sided READ and atomic FA operations. Our experiments show that, on average, DSLR delivers 1.8× (and up to 2.8×) higher throughput than all existing RDMA-based lock managers, while reducing their mean and 99.9% latencies by 2.0× and 18.3× (and up to 2.5× and 47×), respectively.

Key takeaways
sparkles

AI

  1. DSLR achieves 1.8× to 2.8× higher throughput than existing RDMA-based lock managers under contention.
  2. The proposed decentralized lock manager prevents starvation and ensures fairness using fetch-and-add operations.
  3. DSLR adapts Lamport's bakery algorithm for RDMA environments, enhancing performance with atomic operations.
  4. The system can handle transaction failures using leases, enabling fault tolerance in decentralized settings.
  5. Experiments show DSLR reduces mean latencies by 2.0× and 99.9% latencies by 18.3× compared to prior methods.

Related papers

RDMA is Turing complete, we just did not know it yet!
Waleed Reda

arXiv (Cornell University), 2021

It is becoming increasingly popular for distributed systems to exploit offload to reduce load on the CPU. Remote Direct Memory Access (RDMA) offload, in particular, has become popular. However, RDMA still requires CPU intervention for complex offloads that go beyond simple remote memory access. As such, the offload potential is limited and RDMAbased systems usually have to work around such limitations. We present RedN, a principled, practical approach to implementing complex RDMA offloads, without requiring any hardware modifications. Using self-modifying RDMA chains, we lift the existing RDMA verbs interface to a Turing complete set of programming abstractions. We explore what is possible in terms of offload complexity and performance with a commodity RDMA NIC. We show how to integrate these RDMA chains into applications, such as the Memcached keyvalue store, allowing us to offload complex tasks such as key lookups. RedN can reduce the latency of key-value get operations by up to 2.6× compared to state-of-the-art KV designs that use one-sided RDMA primitives (e.g., FaRM-KV), as well as traditional RPC-over-RDMA approaches. Moreover, compared to these baselines, RedN provides performance isolation and, in the presence of contention, can reduce latency by up to 35× while providing applications with failure resiliency to OS and process crashes.

View PDFchevron_right
Towards fair, scalable, locking
Osman Unsal, Sutirtha Sanyal, Adrian Cristal

2008

Without care, Hardware Transactional Memory presents several performance pathologies that can degrade its performance. Among them, writers of commonly read variables can suffer from starvation. Though different solutions have been proposed for HTM systems, hybrid systems can still suffer from this performance problem, given that software transactions don't interact with the mechanisms used by hardware to avoid starvation.

View PDFchevron_right
Congestion Control for Large-Scale RDMA Deployments
Chuanxiong Guo

Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, 2015

Modern datacenter applications demand high throughput (40Gbps) and ultra-low latency (< 10 µs per hop) from the network, with low CPU overhead. Standard TCP/IP stacks cannot meet these requirements, but Remote Direct Memory Access (RDMA) can. On IP-routed datacenter networks, RDMA is deployed using RoCEv2 protocol, which relies on Priority-based Flow Control (PFC) to enable a drop-free network. However, PFC can lead to poor application performance due to problems like head-of-line blocking and unfairness. To alleviates these problems, we introduce DC-QCN, an end-to-end congestion control scheme for RoCEv2. To optimize DCQCN performance, we build a fluid model, and provide guidelines for tuning switch buffer thresholds, and other protocol parameters. Using a 3-tier Clos network testbed, we show that DCQCN dramatically improves throughput and fairness of RoCEv2 RDMA traffic. DCQCN is implemented in Mellanox NICs, and is being deployed in Microsoft's datacenters.

View PDFchevron_right
A log (n) multi-mode locking protocol for distributed systems
Nirmit Desai

2003

Trends in high-end cluster are to deploy an ever larger number of processors to push the envelope on parallelizing scientific applications. A similar trend exists in server computing where more and more servers are composed into a farm. One of the main challenges in such environments is to achieve scalability of synchronization, specifically for locking. Our work focuses on scalable locking in distributed systems. At its essence is a peer-to-peer protocol for multi-mode hierarchical locking, which is applicable to transaction-style processing and distributed agreement. We demonstrate high scalability combined with low response times in high-performance cluster environments. Our experiments on an IBM SP show that the number of messages approaches an asymptote at 15 nodes, from which point on the message overhead is in the order of 3-9 messages per request, depending on system parameters. At the same time, response times increase linearly with a proportional increase in requests and, consequently, higher concurrency levels. We observed response times under 2 and 10 msec for up to 25 and 80 nodes for critical sections that are one 25th the size of non-critical code. Our approach benefits in large from high levels of concurrency upon resolving requests and dynamic path compression for request propagation. Besides its technical strengths, our approach is intriguing due to its simplicity and its wide applicability, ranging from largescale clusters to server-style computing.

View PDFchevron_right
A fast read/write process to reduce RDMA communication latency
Jizhong Han

Networking, Architecture, and Storages, …, 2006

We use a mathematic model to calculate the overall latency of FRRWP. Compared to traditional RDMA operations, our results show FRRWP reduces the total communication latency dramatically in the critical data path.

View PDFchevron_right
Architecture Support for Concurrency Control in Datacenters
Dmitrii Ustiugov

2015

Modern large-scale applications are highly concurrent and require efficient concurrency control mechanisms to achieve high performance, while preserving consistency of the data that is shared between a large number of servers. Traditional software techniques, such as atomic operations, used in concurrency control mechanisms introduce considerable overheads, whereas mechanisms leveraging architecture support for transactions do not scale beyond a single coherence domain due to high design complexity. Using scale-out NUMA [1] platform, as an example of an extensible protocol and architecture, we show the insights for building an efficient software/hardware co-designed concurrency control on top of an extended set of hardware-based operations that can be leveraged by distributed transactions in a rack-scale system.

View PDFchevron_right
Effects of Processor-Native Memory Transactions in Optimizing RDMA Transfers in Distributed Shared Memory Systems
Kyriakos Paraskevas

2020

View PDFchevron_right
MP-LOCKs: replacing H/W synchronization primitives with message passing
john carter

Proceedings Fifth International Symposium on High-Performance Computer Architecture, 1999

Shared memory programs guarantee the correctness of concurrent accesses to shared data using interprocessor synchronization operations. The most common synchronization operators are locks, which are traditionally implemented via a mix of shared memory accesses and hardware synchronization primitives like test-and-set. In this paper, we argue that synchronization operations implemented using fast message passing and kernel-embedded lock managers are an attractive alternative to dedicated synchronization hardware. We propose three message passing lock (MP-LOCK) algorithms (centralized, distributed, and reactive) and provide implementation guidelines. MP-LOCKs reduce the design complexity and runtime occupancy of DSM controllers and can exploit software's inherent flexibility to adapt to differing applications lock access patterns. We compared the performance of MP-LOCKs with two common shared memory lock algorithms: test-and-test-and-set and MCS locks and found that MP-LOCKs scale better. For machines with 16 to 32 nodes, applications using MP-LOCKs ran up to 186% faster than the same applications with shared memory locks. For small systems (up to 8 nodes), three applications with MP-LOCKs slow down by no more than 18%, while the other two slowed by no more than 180% due to higher software overhead. We conclude that locks based on message passing should be considered as a replacement for hardware locks in future scalable multiprocessors that support efficient message passing mechanisms.

View PDFchevron_right
General and efficient locking without blocking
Michal Young

2008

Standard concurrency control mechanisms offer a trade-off: Transactional memory approaches maximize concurrency, but suffer high overheads and cost for retrying in the case of actual contention. Locking offers lower overheads, but typically reduces concurrency due to the difficulty of associating locks with the exact data that need to be accessed. Moreover, locking allows irreversible operations, is ubiquitous in legacy software, and seems unlikely to ever be completely supplanted.

View PDFchevron_right
A scalable lock manager for multicores
Heonyoung Yeom

Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, 2013

Modern implementations of DBMS software are intended to take advantage of high core counts that are becoming common in high-end servers. However, we have observed that several database platforms, including MySQL, Shore-MT, and a commercial system, exhibit throughput collapse as load increases, even for a workload with little or no logical contention for locks. Our analysis of MySQL identifies latch contention within the lock manager as the bottleneck responsible for this collapse. We design a lock manager with reduced latching, implement it in MySQL, and show that it avoids the collapse and generally improves performance. Our efficient implementation of a lock manager is enabled by a staged allocation and de-allocation of locks. Locks are pre-allocated in bulk, so that the lock manager only has to perform simple list-manipulation operations during the acquire and release phases of a transaction. De-allocation of the lock datastructures is also performed in bulk, which enables the use of fast implementations of lock acquisition and release, as well as concurrent deadlock checking.

View PDFchevron_right

References (64)

  1. 2016. RDMA over Converged Ethernet. http://www.roceinitiative.org/. (2016).
  2. 2017. APT. https://www.aptlab.net/. (2017).
  3. 2017. Druid | Interactive Analytics at Scale. http://druid.io/. (2017).
  4. 2017. InfiniBand Architecture Specification, Release 1.3. https://cw.infinibandta. org/document/dl/7859. (2017).
  5. 2017. Open MPI: Open Source High Performance Computing. https://www. open-mpi.org/. (2017).
  6. 2017. Teradata: Business Analytics, Hybrid Cloud & Consulting. http://www. teradata.com/. (2017).
  7. 2018. Perftest Package | Mellanox Interconnect Community. https://community. mellanox.com/docs/DOC-2802. (2018).
  8. Peter Bailis et al. 2014. Coordination avoidance in database systems. PVLDB.
  9. Claude Barthels, Simon Loesing, Gustavo Alonso, and Donald Kossmann. 2015. Rack-scale in-memory join processing using RDMA. In SIGMOD.
  10. Carsten Binnig, Andrew Crotty, Alex Galakatos, Tim Kraska, and Erfan Zamanian. 2016. The end of slow networks: it's time for a redesign. PVLDB.
  11. Pablo Brenner. 1997. A technical tutorial on the IEEE 802.11 protocol. BreezeCom Wireless Communications (1997).
  12. Mike Burrows. 2006. The Chubby lock service for loosely-coupled distributed systems. In USENIX OSDI.
  13. Yeounoh Chung and Erfan Zamanian. 2015. Using RDMA for Lock Management. arXiv preprint arXiv:1507.03274 (2015).
  14. Crispin Cowan, F Wagle, Calton Pu, Steve Beattie, and Jonathan Walpole. 2000. Buffer overflows: Attacks and defenses for the vulnerability of the decade. In DARPA Information Survivability Conference and Exposition, 2000. DISCEX'00. Proceedings.
  15. Peter B Danzig, Katia Obraczka, and Anant Kumar. 1992. An analysis of wide- area name server traffic: a study of the Internet Domain Name System. ACM SIGCOMM Computer Communication (1992).
  16. Ananth Devulapalli and Pete Wyckoff. 2005. Distributed queue-based locking using advanced network features. In ICPP.
  17. Aleksandar Dragojević, Dushyanth Narayanan, Miguel Castro, and Orion Hodson. 2014. FaRM: fast remote memory. In USENIX NSDI.
  18. Steven Fitzgerald et al. 1997. A directory service for configuring high- performance distributed computations. In High Performance Distributed Comput- ing, 1997. Proceedings. The Sixth IEEE International Symposium on.
  19. Kristen Gardner, Samuel Zbarsky, Sherwin Doroudi, Mor Harchol-Balter, and Esa Hyytia. 2015. Reducing latency via redundant requests: Exact analysis. ACM SIGMETRICS Performance Evaluation Review (2015).
  20. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. 2003. The Google file system. In ACM SIGOPS operating systems review.
  21. Kishore Gopalakrishna et al. 2012. Untangling cluster management with Helix. In Proceedings of the Third ACM Symposium on Cloud Computing.
  22. Cary Gray and David Cheriton. 1989. Leases: An efficient fault-tolerant mechanism for distributed file cache consistency.
  23. Andrew B Hastings. 1990. Distributed lock management in a transaction pro- cessing environment. In Reliable Distributed Systems, 1990. Proceedings., Ninth Symposium on.
  24. Jiamin Huang, Barzan Mozafari, Grant Schoenebeck, and Thomas Wenisch. 2017. A Top-Down Approach to Achieving Performance Predictability in Database Systems. In SIGMOD.
  25. Jiamin Huang, Barzan Mozafari, and Thomas Wenisch. 2017. Statistical Analysis of Latency Through Semantic Profiling. In EuroSys.
  26. Patrick Hunt, Mahadev Konar, Flavio Paiva Junqueira, and Benjamin Reed. 2010. ZooKeeper: Wait-free Coordination for Internet-scale Systems.. In USENIX ATC.
  27. Prasad Jayanti, King Tan, Gregory Friedland, and Amir Katz. 2001. Bounding LamportâĂŹs bakery algorithm. In International Conference on Current Trends in Theory and Practice of Computer Science.
  28. Horatiu Jula et al. 2008. Deadlock Immunity: Enabling Systems to Defend Against Deadlocks.. In OSDI.
  29. Anuj Kalia, Michael Kaminsky, and David G Andersen. 2014. Using RDMA efficiently for key-value services. In ACM SIGCOMM Computer Communication Review.
  30. Anuj Kalia, Michael Kaminsky, and David G Andersen. 2016. FaSST: Fast, Scalable and Simple Distributed Transactions with Two-Sided (RDMA) Datagram RPCs. In USENIX OSDI.
  31. Rishi Kapoor, George Porter, Malveeka Tewari, Geoffrey M Voelker, and Amin Vahdat. 2012. Chronos: Predictable low latency for data center applications. In Proceedings of the Third ACM Symposium on Cloud Computing.
  32. Nancy P Kronenberg, Henry M Levy, and William D Strecker. 1986. VAXcluster: a closely-coupled distributed system. ACM Transactions on Computer Systems.
  33. Byung-Jae Kwak, Nah-Oak Song, and Leonard E Miller. 2005. Performance analysis of exponential backoff. IEEE/ACM Transactions on Networking.
  34. Leslie Lamport. 1974. A new solution of Dijkstra's concurrent programming problem. Commun. ACM (1974).
  35. Leslie Lamport et al. 2001. Paxos made simple. ACM Sigact News (2001).
  36. Feng Li, Sudipto Das, Manoj Syamala, and Vivek R Narasayya. 2016. Accelerating relational databases by leveraging remote memory and rdma. In SIGMOD.
  37. Haoyuan Li, Ali Ghodsi, Matei Zaharia, Scott Shenker, and Ion Stoica. 2014. Tachyon: Reliable, memory speed storage for cluster computing frameworks. In Proceedings of the ACM Symposium on Cloud Computing.
  38. Jialin Li, Naveen Kr Sharma, Dan RK Ports, and Steven D Gribble. 2014. Tales of the tail: Hardware, os, and application-level sources of tail latency. In Proceedings of the ACM Symposium on Cloud Computing.
  39. Gang Luo, Jeffrey F Naughton, Curt J Ellmann, and Michael W Watzke. 2010. Transaction reordering. Data & Knowledge Engineering (2010).
  40. Christopher Mitchell, Yifeng Geng, and Jinyang Li. 2013. Using One-Sided RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store. In USENIX ATC.
  41. Barzan Mozafari, Carlo Curino, Alekh Jindal, and Samuel Madden. 2013. Perfor- mance and resource modeling in highly-concurrent OLTP workloads. In SIGMOD.
  42. Barzan Mozafari, Carlo Curino, and Samuel Madden. 2013. DBSeer: Resource and Performance Prediction for Building a Next Generation Database Cloud. In CIDR.
  43. Barzan Mozafari, Eugene Zhen Ye Goh, and Dong Young Yoon. 2015. CliffGuard: A Principled Framework for Finding Robust Database Designs. In SIGMOD.
  44. Barzan Mozafari, Jags Ramnarayan, Sudhir Menon, Yogesh Mahajan, Soubhik Chakraborty, Hemant Bhanawat, and Kishor Bachhav. 2017. SnappyData: A Unified Cluster for Streaming, Transactions, and Interactive Analytics. In CIDR.
  45. Sundeep Narravula, A Marnidala, Abhinav Vishnu, Karthikeyan Vaidyanathan, and Dhabaleswar K Panda. 2007. High performance distributed lock manage- ment services using network-based remote atomic operations. In Seventh IEEE International Symposium on Cluster Computing and the Grid.
  46. Jacob Nelson et al. 2015. Latency-tolerant software distributed shared memory. In USENIX ATC.
  47. Ravi Rajwar and James R Goodman. 2002. Transactional lock-free execution of lock-based programs. In ACM SIGOPS Operating Systems Review.
  48. Jags Ramnarayan, Barzan Mozafari, Sudhir Menon, Sumedh Wale, Neeraj Kumar, Hemant Bhanawat, Soubhik Chakraborty, Yogesh Mahajan, Rishitesh Mishra, and Kishor Bachhav. 2016. SnappyData: A hybrid transactional analytical store built on Spark. In SIGMOD.
  49. KV Rashmi, Mosharaf Chowdhury, Jack Kosaian, Ion Stoica, and Kannan Ram- chandran. 2016. EC-Cache: Load-Balanced, Low-Latency Cluster Caching with Online Erasure Coding. In USENIX OSDI.
  50. Ro Recio, P Culley, D Garcia, J Hilland, and B Metzler. 2005. An RDMA protocol specification. Technical Report. IETF Internet-draft draft-ietf-rddp-rdmap-03. txt (work in progress).
  51. Kun Ren, Alexander Thomson, and Daniel J Abadi. 2015. VLL: a lock manager redesign for main memory database systems. The VLDB Journal (2015).
  52. Wolf Rödiger, Tobias Mühlbauer, Alfons Kemper, and Thomas Neumann. 2015. High-speed query processing over high-speed networks. PVLDB.
  53. Dongin Shin et al. 2013. Dynamic Interval Polling and Pipelined Post I/O Pro- cessing for Low-Latency Storage Class Memory.. In HotStorage.
  54. Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler. 2010. The hadoop distributed file system. In 2010 IEEE 26th symposium on mass storage systems and technologies (MSST).
  55. Michael Stonebraker and Ariel Weisberg. 2013. The VoltDB Main Memory DBMS. IEEE Data Eng. Bull. (2013).
  56. Gadi Taubenfeld. 2004. The black-white bakery algorithm and related bounded- space, adaptive, local-spinning and FIFO algorithms. Distributed Computing (2004).
  57. Boyu Tian, Jiamin Huang, Barzan Mozafari, Grant Schoenebeck, and Thomas Wenisch. 2018. Contention-aware lock scheduling for transactional databases. PVLDB (2018).
  58. Ashish Vulimiri, Oliver Michel, P Godfrey, and Scott Shenker. 2012. More is less: reducing latency via redundancy. In Proceedings of the 11th ACM Workshop on Hot Topics in Networks.
  59. Carl A Waldspurger. 1995. Lottery and stride scheduling: Flexible proportional- share resource management. (1995).
  60. Xingda Wei, Jiaxin Shi, Yanzhe Chen, Rong Chen, and Haibo Chen. 2015. Fast in-memory transaction processing using RDMA and HTM. In SOSP.
  61. Cong Yan and Alvin Cheung. 2016. Leveraging lock contention to improve OLTP application performance. PVLDB.
  62. Dong Young Yoon, Ning Niu, and Barzan Mozafari. 2016. DBSherlock: A Perfor- mance Diagnostic Tool for Transactional Databases. In SIGMOD.
  63. Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: cluster computing with working sets. HotCloud (2010).
  64. Lei Zhang, Yu Chen, Yaozu Dong, and Chao Liu. 2012. Lock-Visor: An efficient transitory co-scheduling for MP guest. In ICPP.

Related papers

High Performance Distributed Lock Management Services using Network-based Remote Atomic Operations
Abhinav Vishnu

2007

Recently there has been a massive increase in computing requirements for parallel applications. These parallel applications and supporting cluster services often need to share system-wide resources. The coordination of these applications is typically managed by a distributed lock manager. The performance of the lock manager is extremely critical for application performance.

View PDFchevron_right
Acuerdo: Fast Atomic Broadcast over RDMA
Gaukas Wang

Proceedings of the 51st International Conference on Parallel Processing

Atomic broadcast protocols ensure that messages are delivered to a group of machines in some total order, even when some of these machines can fail. These protocols are key to making distributed services fault-tolerant, as their total order guarantee allows keeping multiple service replicas in sync. But, unfortunately, atomic broadcast protocols are also notoriously expensive. We present a new protocol, called Acuerdo, that improves atomic broadcast performance by using remote direct memory addressing (RDMA). Acuerdo is built from the ground up to perform communication using one-side RDMA writes, which do not use the CPU of the remote machine, and is explicitly designed to minimize waiting on the critical path. Our experimental results demonstrate that Acuerdo provides raw throughput comparable to or exceeding other RDMA atomic broadcast protocols, while improving latency by almost 2. CCS CONCEPTS • Computer systems organization → Dependable and faulttolerant systems and networks; • Networks → Network algorithms.

View PDFchevron_right
Scalable distributed concurrency services for hierarchical locking
Nirmit Desai

2003

Middleware components are becoming increasingly important as applications share computational resources in distributed environments. One of the main challenges in such environments is to achieve scalability of concurrency control. Existing concurrency protocols lack scalability. Scalability enables resource sharing and computing with distributed objects in systems with a large number of nodes. We have designed and implemented a novel, scalable and fully decentralized middleware concurrency control protocol. Our experiments on a Linux cluster indicate that an average number of three messages is required per lock request on a system with as many as 120, which is a logarithmic asymptote. At the same time, the response time for the requests scales linearly with the increase in concurrency level. A comparison to another scalable concurrency protocol shows that our protocol results in significantly superior asymptotic savings in message overhead and response time for large number of nodes. While our approach follows the specification of general CORBA concurrency services for large-scale data and object repositories, the principles are applicable to any distributed concurrency services and transaction models. The results of this work impact scalability for distributed computing facilities ranging from embedded computing with distributed objects over peer-to-peer computing environments to arbitrating accesses in very large database environments.

View PDFchevron_right
Scalable hierarchical locking for distributed systems
Nirmit Desai

Journal of Parallel and Distributed Computing, 2004

Middleware components are becoming increasingly important as applications share computational resources in distributed environments, such as high-end clusters with ever larger number of processors, computational grids and increasingly large server farms. One of the main challenges in such environments is to achieve scalability of synchronization. In general, concurrency services arbitrate resource requests in distributed systems. But concurrency protocols currently lack scalability. Adding such guarantees enables resource sharing and computing with distributed objects in systems with a large number of nodes.

View PDFchevron_right
A case for RDMA in clouds
Bernard Metzler

Proceedings of the Second Asia-Pacific Workshop on Systems - APSys '11, 2011

Modern cloud computing infrastructures are steadily pushing the performance of their network stacks. At the hardwarelevel, already some cloud providers have upgraded parts of their network to 10GbE. At the same time there is a continuous effort within the cloud community to improve the network performance inside the virtualization layers. The low-latency/high-throughput properties of those network interfaces are not only opening the cloud for HPC applications, they will also be well received by traditional large scale web applications or data processing frameworks. However, as commodity networks get faster the burden on the end hosts increases. Inefficient memory copying in socketbased networking takes up a significant fraction of the endto-end latency and also creates serious CPU load on the host machine. Years ago, the supercomputing community has developed RDMA network stacks like Infiniband that offer both low end-to-end latency as well as a low CPU footprint. While adopting RDMA to the commodity cloud environment is difficult (costly, requires special hardware) we argue in this paper that most of the benefits of RDMA can in fact be provided in software. To demonstrate our findings we have implemented and evaluated a prototype of a software-based RDMA stack. Our results, when compared to a socket/TCP approach (with TCP receive copy offload) show significant reduction in end-to-end latencies for messages greater than modest 64kB and reduction of CPU load (w/o TCP receive copy offload) for better efficiency while saturating the 10Gbit/s link.

View PDFchevron_right

Related topics

  • Computer Science
  • Distributed Computing

    • Cited by

    Storm
    Yizhou Shan

    Proceedings of the 12th ACM International Conference on Systems and Storage

    RDMA technology enables a host to access the memory of a remote host without involving the remote CPU, improving the performance of distributed in-memory storage systems. Previous studies argued that RDMA suffers from scalability issues, because the NIC's limited resources are unable to simultaneously cache the state of all the concurrent network streams. These concerns led to various software-based proposals to reduce the size of this state by trading off performance. We revisit these proposals and show that they no longer apply when using newer RDMA NICs in rack-scale environments. In particular, we find that one-sided remote memory primitives lead to better performance as compared to the previously proposed unreliable datagram and kernel-based stacks. Based on this observation, we design and implement Storm, a transactional dataplane utilizing one-sided read and writebased RPC primitives. We show that Storm outperforms eRPC, FaRM, and LITE by 3.3x, 3.6x, and 17.1x, respectively, on an InfiniBand cluster with Mellanox ConnectX-4 NICs. CCS Concepts • Networks → Network performance evaluation; • Software and its engineering → Distributed systems organizing principles;

    View PDFchevron_right
    Avoiding Register Overflow in the Bakery Algorithm
    Mohsen Sharifi

    49th International Conference on Parallel Processing - ICPP : Workshops, 2020

    Computer systems are designed to make resources available to users and users may be interested in some resources more than others, therefore, a coordination scheme is required to satisfy the users' requirements. This scheme may implement certain policies such as "never allocate more than X units of resource Z". One policy that is of particular interest is the inability of users to access a single resource at the same time, which is called the problem of mutual exclusion. Resource management concerns the coordination and collaboration of users, and it is usually based on making a decision. In the case of mutual exclusion, that decision is about granting access to a resource. Therefore, mutual exclusion is useful for supporting resource access management. The first true solution to the mutual exclusion problem is known as the Bakery algorithm that does not rely on any lower-lever mutual exclusion. We examine the problem of register overflow in realworld implementations of the Bakery algorithm and present a variant algorithm named Bakery++ that prevents overflows from ever happening. Bakery++ avoids overflows without allowing a process to write into other processes' memory and without using additional memory or complex arithmetic or redefining the operations and functions used in Bakery. Bakery++ is almost as simple as Bakery and it is straightforward to implement in real systems. With Bakery++, there is no reason to keep implementing Bakery in real computers because Bakery++ eliminates the probability of overflows and hence it is more practical than Bakery. Previous approaches to circumvent the problem of register overflow included introducing new variables or redefining the operations or functions used in the original Bakery algorithm, while Bakery++ avoids overflows by using simple conditional statements. The result is a new mutual exclusion algorithm that is guaranteed never to allow an overflow and it is simple, correct and easy to implement. Bakery++ has the same temporal and spatial complexities as the original Bakery. We have specified Bakery++ in PlusCal and we have used the TLC model checker to assert that Bakery++ maintains the mutual exclusion property and that it never allows an overflow. CCS CONCEPTS • Software and its engineering → Software organization and properties → Contextual software domains → Operating systems → Process management → Mutual exclusion •

    View PDFchevron_right
    Prism
    Sowmya Dharanipragada

    Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles CD-ROM, 2021

    Remote Direct Memory Access (RDMA) has been used to accelerate a variety of distributed systems, by providing lowlatency, CPU-bypassing access to a remote host's memory. However, most of the distributed protocols used in these systems cannot easily be expressed in terms of the simple memory READs and WRITEs provided by RDMA. As a result, designers face a choice between introducing additional protocol complexity (e.g., additional round trips) or forgoing the benefits of RDMA entirely. This paper argues that an extension to the RDMA interface can resolve this dilemma. We introduce the PRISM interface, which adds four new primitives: indirection, allocation, enhanced compare-and-swap, and operation chaining. These increase the expressivity of the RDMA interface, while still being implementable using the same underlying hardware features. We show their utility by designing three new applications using PRISM primitives, that require little to no server-side CPU involvement: (1) PRISM-KV, a key-value store; (2) PRISM-RS, a replicated block store; and (3) PRISM-TX, a distributed transaction protocol. Using a software-based implementation of the PRISM primitives, we show that these systems outperform prior RDMA-based equivalents.

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved