Transport Working Group R. Penno Internet Draft S. Raghunath Intended status: Informational Juniper Networks Expires: September 2009 J. Iyengar Franklin & Marshall College March 3, 2009 LEDBAT Practices and Recommendations draft-penno-ledbat-app-practices-recommendations-00 Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on September 3, 2009. Copyright Notice Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Penno Expires September 3, 2009 [Page 1] Internet-Draft LEDBAT Practices and Recommendations March 2009 Abstract Applications routinely open multiple TCP connections. For example, P2P applications maintain connections to a number of different peers while web browsers perform concurrent download from the same web server. Application designers pursue different goals when doing so: P2P apps need to maintain a well-connected mesh in the swarm while web browsers mainly use multiple connections to parallelize requests that involve application latency on the web server side. But this practice also has impacts to the host and the network as a whole. For example, an application can obtain a larger fraction of the bottleneck than if it had used fewer connections. Although capacity is the most commonly considered bottleneck resource, middlebox state table entries are also an important resource for an end system communication. This documents clarifies the current practices of application design and reasons behind them, and discusses the tradeoffs surrounding the use of many concurrent TCP connections to one destination and/or to different destinations. Other resource types may exist, and the guidelines are expected to comprehensively discuss them. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 Table of Contents 1. Introduction 3 2. Terminology 4 3. Multiple TCP Connections Advantages 4 3.1. Avoiding head-of-line blocking 4 3.2. Logical partitioning at application level 5 3.3. Multiple streams with different properties 5 3.4. Signaling application layer request completion 5 3.5. High bandwidth-delay links 5 3.6. Error resiliency and reliability 5 3.7. Leveraging multiple processors in a system 5 4. Multiple TCP connections Disadvantages 5 4.1. Additional connection setup overhead 6 4.2. Memory Space 6 4.3. Link Bandwidth 6 4.4. Middleboxes 7 5. Recommendations For Application Designers 7 Penno Expires September 3, 2009 [Page 2] Internet-Draft LEDBAT Practices and Recommendations March 2009 5.1. Diffserv 7 5.2. Window scale negotiation 8 5.3. Choice of number of multiple connections 8 6. Security Considerations 8 7. IANA Considerations 8 8. Conclusions 8 9. Acknowledgments 8 10. References 9 10.1. Normative References 9 10.2. Informative References 9 Author's Addresses 10 1. Introduction The use of P2P protocols by end users is widespread. These protocols are meant to exchange, replicate, stream or download files with little human intervention, trying at the same time to minimize the download time of the files requested by any single peer. This is done by opening several connections to multiple peers and downloading one or more chunks of the file from each one, selecting faster peers, amongst others. If we assume that in any file transfer the bottleneck is on the uploading peer or server side, end users that utilize P2P clients in general download the file faster and consume more bandwidth within a specific timeframe than traditional client-server applications. P2P clients can overcome the server side bottleneck by opening multiple connections to different peers. Users of P2P applications also consume bandwidth throughout the whole day since even after a file is fully downloaded it will continue to be shared with others users increasing the upstream bandwidth. We can see then that the advantages of P2P applications come from the fact that they open multiple TCP connection to different peers in order to download multiple pieces of a file in parallel, always look for faster peers. But the use of multiple TCP connections by an application is not new. Web Browsers have being it for a decade. But these are usually short- lived connections as opposed to long-lived connections. A long-lived connection in this document should be interpreted as strictly defined, i.e., a TCP connection that is simply in the established state, but not necessarily continuously transferring data. In the case of P2P protocols, e.g. Bittorrent, at any point in time a fraction of these connections is actually sending or receiving data, the others are idle or exchange occasional control information. Penno Expires September 3, 2009 [Page 3] Internet-Draft LEDBAT Practices and Recommendations March 2009 With the popularity of P2P applications, which maintain hundreds of long-lived TCP connections to multiple hosts, the issue applications making use of multiple TCP connections has been gaining attention. This documents clarifies the current practices of application design and reasons behind them, and discusses the tradeoffs surrounding the use of many concurrent TCP connections to one destination and/or to different destinations. Other resource types may exist, and the guidelines are expected to comprehensively discuss them. 2. Terminology Bandwidth: A measure of the amount of data that can be transferred per second. So, if a 1Gb file were transferred within one second the bandwidth consumption during the transfer would be 1Gb/s. If it were transferred within a day, it would be approximately 0.0002Gb/s. Volume: The total number of bytes transferred during a long time period. In both examples above the volume within a day would have been 1Gb. Capacity: The maximum bandwidth a link can sustain continuously. Long-lived connection: A TCP connection that is in the established state but not necessarily continuously transferring data. 3. Multiple TCP Connections Advantages There are good reasons for an application to use multiple TCP connections. P2P apps need to maintain a well-connected mesh in the swarm while web browsers mainly use multiple connections to parallelize requests that involve application latency on the web server side But from a P2P standpoint multiple TCP connections are at the heart of its functionality. Multiple connections allow for multiple simultaneous downloads, which improve reliability and speed. Multiple connections also allow more effective discovery of new peers, and effective peer-to-peer communication, which allows exchange of information such as which pieces of a file a client has and is available. 3.1. Avoiding head-of-line blocking Web browsers started using multiple TCP connections partly because of this reason [STEVENS]. This is especially true when the multiple TCP connections are between a pair of hosts. Penno Expires September 3, 2009 [Page 4] Internet-Draft LEDBAT Practices and Recommendations March 2009 3.2. Logical partitioning at application level Some applications such as FTP use a separate connection for control and data transfers. The advantage is that this allows a model where the data transfer is actually happening between hosts that are not local (see [RFC959], sections 2.3 & 5.2). 3.3. Multiple streams with different properties The application may need different properties on multiple streams of data (e.g., Nagle's algorithm, socket buffer sizes etc). 3.4. Signaling application layer request completion If the application assumes that connection close indicates the completion of a request, it becomes necessary to have new connections for multiple requests. This was a reason for multiple connections in HTTP 1.0. 3.5. High bandwidth-delay links In the presence of a large bandwidth-delay product, the 16-bit window size parameter in TCP header does not allow the application to fully utilize the link. In such situations, one option is to negotiate the Window Scale Option [RFC1323] In addition multiple TCP connections allow the application to achieve an effectively larger window size so that it can better utilize a link with high bandwidth-delay product (e.g. iSCSI [SCSIREF]). 3.6. Error resiliency and reliability When multiple connections are used to download a single file or webpage, for instance, there is lesser chance of a single failure on one connection having a negative impact on the whole download. Especially with P2P applications, this makes the network robust to failures and churn in participants. 3.7. Leveraging multiple processors in a system With multiple processor systems, there can be higher performance with parallelism and multiple connections spread over different processors. 4. Multiple TCP connections Disadvantages Every connected application on the Internet competes for resources. This is not specific to applications that open multiple TCP Penno Expires September 3, 2009 [Page 5] Internet-Draft LEDBAT Practices and Recommendations March 2009 connections. The use of multiple TCP connections just amplifies the issue. In the following sections we discuss these resources and how they are amplified by an application opening multiple connections. 4.1. Additional connection setup overhead TCP's mechanisms for starting up the connection and then probing the available bandwidth have to be repeated for each new connection. So there may be lesser leverage of network information. There is also the overhead of additional control traffic that may have been avoided. 4.2. Memory Space Each TCP connection needs a TCP control block (TCB) or equivalent to keep state about its connection. In operating systems where the TCP stack is part of the kernel, this would come from the kernel memory space, otherwise from userland memory. But irrespective from where the memory comes from a TCP control block requires a significant amount of memory. This is significant issue for devices that terminate TCP connections from multiple end hosts to provide functions such as Load-Balancing, Gateway and Tunneling. Some proposals have been put forward to reduce the amount of memory occupied by each TCP control block [RFC2140], but the issue remains significant and is amplified by applications that use multiple TCP connections. 4.3. Link Bandwidth The bottlenecks for these N multiple connections could be shared or separate. If separate, there's no specific bottleneck where the connections are hogging bandwidth. But from a network resource point of view, the application download still gets multiple shares. If some/all bottlenecks are shared, then two possibilities exist for shared bottleneck - bottleneck is a last-hop link (user traffic dominates link), OR - bottleneck is in-network wide-area link (background traffic dominates link) If bottleneck is the last-hop, then n transport connections compete with each other and share link bandwidth. Penno Expires September 3, 2009 [Page 6] Internet-Draft LEDBAT Practices and Recommendations March 2009 Although these connections might impact delay-sensitive traffic and increase delay, in the last hop they only affect the end end-user, which is in control of which applications run on its host. In this case the user has the option of manually choosing when to run each application, configuring the end host, amongst others. Alternatively, or in conjunction with the above, the application can be enhanced to use Diffserv and new delay sensitive congestion mechanisms. If the shared bottleneck is in-network, then the application gets unfair share of bottleneck bandwidth. This impacts flows belonging to other users in general, and most importantly delay-sensitive traffic. 4.4. Middleboxes Middleboxes are defined as any intermediary box performing functions apart from normal, standard functions of an IP router on the data path between a source host and destination host [RFC3234]. Middleboxes can be stand-alone or integrated in another device such as a router or modem. The functions that are relevant to this discussion are those that require the middlebox to keep per session state, sometimes referred as transformation services. Some of these functions are, for example, NAT, Intrusion Detection and Load-Balancing. It is easy to see that the more sessions a host initiates, the more state the middlebox will have to keep. The relationship is at least 1:1 but due asymmetric traffic, routing changes and others, this can be 1:N. Although application traffic from most broadband subscribers today go through at least one middlebox (integrated into the broadband modem), it can traverse other middleboxes that reside within the ISP's network or close the destination. These middleboxes aggregate traffic from multiple subscribers and state tables within these devices can become a premium. 5. Recommendations For Application Designers 5.1. Diffserv Applications involved in bulk data transfer with low priority in time could mark their packets according with the guidelines of RFC 3662 [RFC2662]. Penno Expires September 3, 2009 [Page 7] Internet-Draft LEDBAT Practices and Recommendations March 2009 5.2. Window scale negotiation Where appropriate, sender & receiver window should be scaled using RFC1323 based negotiation in order to make the best use of network resources. Recommendations to adjust window size are not new and have been recommended in networks where the BDP (Bandwidth Delay Product) is large [RFC3481]. 5.3. Choice of number of multiple connections In the event that an application decides to employ multiple parallel TCP connections, it is recommended empirical investigations be carried out to find a threshold beyond which additional connections yield little benefit. This kind of investigation is also worth having with existing applications that employ multiple connections. In particular with P2P systems, it would be useful to understand the impact on the health of the swarm with the number of connections. 6. Security Considerations None at this time 7. IANA Considerations None at this time 8. Conclusions TBD 9. Acknowledgments Joe Touch Penno Expires September 3, 2009 [Page 8] Internet-Draft LEDBAT Practices and Recommendations March 2009 10. References 10.1. Normative References [RFC959] J. Postel and J. Reynolds, "File Transfer Protocol (FTP)", RFC 959 (1985). [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, December 1998. 10.2. Informative References [RFC1323] V. Jacobson, B. Braden, D. Borman, TCP Extensions for High Performance, RFC 1323, May 1992 [RFC2140] J. Touch, TCP Control Block Interdependence, RFC 2140 (1997) [RFC2616] R. Fielding et al, Hypertext Transfer Protocol - HTTP/1.1, RFC 2616 (1999) [RFC3481] Inamura, H., Montenegro, G., Ludwig, R., Gurtov, A., and F. Khafizov, "TCP over Second (2.5G) and Third (3G) Generation Wireless Networks", BCP 71, RFC 3481, February 2003. [RFC3234] Carpenter, B. and S. Brim, "Middleboxes: Taxonomy and Issues", RFC 3234, February 2002. [RFC3662] Bless, R., Nichols, K., and K. Wehrle, "A Lower Effort Per-Domain Behavior (PDB) for Differentiated Services", RFC 3662, December 2003. [SCSIREF] K.Z. Meth, J. Satran, Design of the iSCSI protocol, Storage Conference (2003) [STEVENS] W. Richard Stevens et al, "Unix Network Programming, The Sockets Networking API", Volume 1, Third Edition (2003), section 10.5, page 293. Penno Expires September 3, 2009 [Page 9] Internet-Draft LEDBAT Practices and Recommendations March 2009 Author's Addresses Reinaldo Penno Juniper Networks 1194 N Mathilda Aveue Sunnyvale, CA Email: rpenno@juniper.net Satish Raghunath Juniper Networks 1194 N Mathilda Aveue Sunnyvale, CA Email: satishr@juniper.net Jana Iyengar Franklin & Marshall College Email: jiyengar@fandm.edu Penno Expires September 3, 2009 [Page 10]