George D., Here is NTAP and INTC talking about DAFS.
Direct Access File System (DAFS) protocol: What it is and why some storage industry experts think it's crucial for the future of storage
When: Thursday, November 9th at 1:30 p.m. EST (18:30 GMT)
Speakers: Mitch Shults of Intel and David Dale of Network Appliance
Moderator: Welcome to our Expert Live Q&A with Mitch Shults of Intel's Enterprise Server Group and David Dale of Network Appliance on the subject of Direct Access File System (DAFS).
lhunt903991: How does DAFS differ from conventional network file access protocols?
David_Dale: Traditional shared-file systems rely on general-purpose network protocols and interface cards to provide applications with access to files. The network protocols break data up into small packets, with all the resulting overhead of packet assembly/disassembly done in software. The resulting data is then buffered in the operating system kernel to control flow, providing general-purpose access. Applications wishing to access this data must perform context switches to the operating system to access the network buffers. DAFS, on the other hand, uses VI architecture to provide a high-performance solution to file sharing. Optimal data packet sizes are negotiated and assembled/disassembled in hardware. DAFS also bypasses kernel buffering and context switching by writing data directly into application memory space. The result is a fast, highly scalable improvement on traditional shared-file systems.
manny2674695: A quick overview of DAFS would be useful, to start.
Mitch_Shults: Good point. Technically, DAFS is intended to substantially improve performance of file-sharing applications by allowing 'direct access' across a data center network between general purpose servers and the filers that support them. It is intended as an open, interoperable specification that will allow industry participants to compete and interoperate for each of the elements of DAFS.
Moderator: We are speaking with Mitch Shults of Intel's Enterprise Server Group and David Dale of Network Appliance on the subject of Direct Access File Systems (DAFS).
terry.noonan472415: Are there any projections for performance increase .vs. NFS, CIFS
David_Dale: We expect a really significant performance increase over NFS and CIFS. In fact DAFS should provide better performance than direct-attached block-oriented devices
manny2674695: Is DAFS complementary to SoIP or iSCSI, or is it tied to VI and incompatible?
Mitch_Shults: DAFS is complementary to the various SCSI encapsulation mechanisms that exist or are being developed. SCSI encapsulation, over any transport (including VI), is a block-mode mechanism. Resource sharing is not provided. DAFS, like existing file sharing mechanisms, is inherently about resource sharing. All DAFS is doing is allowing file-sharing mechanisms to take full advantage of the greatly accelerated performance of data-center networks such as InfiniBand Architecture, Fibre Channel, Gigabit Ethernet, and proprietary interconnects, all of which support a VI-based interface. DAFS uses VI, but that isn't a source of incompatibility.
charlie464862: Do you think that DAFS will eventually obsolete protocols such as CIFS & NFS?
David_Dale: No, we see them as complementary. DAFS is designed specifically for a secure bounded network, in a data center for instance. NFS and CIFS are designed for larger scale networks and WANs
dspenard160200: How does the DAFS specification process work?
Mitch_Shults: The DAFS specification and a reference implementation will be passed on to a standards body (still to be named) for further refinement and industry approval. Additional information can be found at dafscollaborative.org.
cyndi.pearce17288: If you are using VI architecture is it safe to assume you are also using a SAN?
Mitch_Shults: Yes. VI architecture assumes a reliable interconnect. That's what provides its performance advantage, since VI-based networks don't have to use heavyweight protocols such as TCP/IP. VI-based networks typically provide hardware-based reliability protocols to ensure that everything that's sent actually arrives, exactly once, in order. VI also allows senders to place data directly in the recipient's memory, without requiring any action on the recipient's part to accept the update. Hence 'direct access'.
Dawn.Hamel7139: So in addition to the NFS and CIFS protocols used on the Network Appliance filers, you would also be implemeting this on the Filers ... and I would guess that your hopes is that the rest of the Unix world (Sun,HP, etc) and NT (cifs) would have to also implement this
David_Dale: Yes this is another protocal which will be supported on filers. To garner broad industry support, we started the DAFS Collaborative
holley_jim807388: When will DAFS run over Ethernet?
Mitch_Shults: There are VI architecture-supporting GbE adapters under development now. Expect them in 2001. The key thing needed is off-host TCP/IP capability. That's finally becoming possible. With a conformant VI interface, the DAFS client and server software stacks don't have to know or care if the interconnect is GbE, InfiniBand Architecture, or anything else.
beach4me822117: OK, what constitutes "direct" access between a processor and a disk on a filer? And how is it different from what's available today?
David_Dale: Today's filers attach to applications servers over standard TCP/IP networks. DAFS enable connection over a VI connection, which allows memory-to-memory transfers between the filer and the application
Moderator: We are speaking with Mitch Shults of Intel's Enterprise Server Group and David Dale of Network Appliance on the subject of Direct Access File Systems (DAFS).
dean.waelz810584: Who is in the DAFS collaborative, besides Intel & NA?
David_Dale: Currently around 50 companies have signed up as contributors to the DAFS effort, ranging from NIC vendors to storage vendors to server vendors, to software vendors - including ISVs, DB, etc. There is more info on the website www.DAFScollaborative.org
martimr1188685: David, what do you see as the applications most impacted by DAFS?
David_Dale: We expect big benefits to DBMS, and apps that run directly on DB engines. Also any file-oriented application, such as collaborative apps, internet apps (mail, streaming media, etc). Our focus is initially on data center and "internet data center" apps
Moderator: If you've already asked a question, please be patient with the response to it. The related question and answer may not appear in the order we receive it. Mr. Shults and Mr. Dale are working to answer these as they come in.
ehurley25910: Who will benefit from using DAFS?
Mitch_Shults: Any system that provides scalable application service can benefit from using DAFS. One example of such systems would be a set of diskless Web servers connected to one or more file servers that store Web information. Another example is a cluster of diskless servers running a highly available shared database that uses a file server to store database information. DAFS is primarily designed for clustered, shared-file network environments, where a limited number of server-class clients connect to a set of file servers via a dedicated high-speed network. The sweet spot for DAFS is where these "client" systems are centrally located and managed, and able to take advantage of VI's fast data transfer semantics.
dtart352372: What network infrastructure does DAFS use?
David_Dale: DAFS uses VI as its underlying transport mechanism. VI can be implemented on a variety of network infrastructures. Currently there are several implementations of VI that use the VI over Fibre Channel draft standard, and several implementations that use proprietary interconnection networks. In the future, the Infiniband standard will support VI. A VI over TCP/IP is also expected, which will enable the use of gigabit and 10-gigabit Ethernet networks.
mlewis27886: How will applications take advantage of DAFS?
David_Dale: DAFS will be implemented as a dynamically linkable library. The DAFS library will make use of a VI provider library by the vendor of the VI hardware. Applications can either make use of direct calls to the DAFS library using an OS-independent interface similar to normal OS I/O requests or use normal OS-dependent I/O calls through a transparent adaptation library.
daveusi: Do application programmers have to use any special API's or will normal APi's such as Windows function calls have some interface underneath?
Mitch_Shults: Application developers will have the option of using 3 distinct modes for DAFS. The simplest mode is complete transparency. The application does file operations using the standard OS API calls, which go through the standard driver stack, which includes a DAFS driver within the kernel at the lowest level. This mode is appropriate for many applications, but does not provide the best possible performance increase.
Moderator: A transcript of our entire session today with Mitch Shults and David Dale will be available on SearchStorage.com over the next few days for free download.
kums686274: Do you have an estimate of how much will there be performance increase with respect to other filesystems
David_Dale: We haven't profiled performance wrt CIFS and NFS yet.
Moderator: If you've already asked a question, please be patient with the response to it. The related question and answer may not appear in the order we receive it. Mr. Shults and Mr. Dale are working to answer these as they come in.
nik.simpson928525: Isn't the basic idea based on VIA, and if so can't the same performance improvement be realized for block oriented I/O transfers, in which case the idea that DAFS will be faster than block oriented solutions seems very dubious
Mitch_Shults: The goal with DAFS is to ensure that file-sharing approaches are fully competitive with block-mode I/O from a performance standpoint. Which will be 'faster' is a function of the application. Today, all file-sharing approaches work over conventional networking transports. They're at a profound performance disadvantage vs. block-mode I/O for this simple reason. Removing the heavyweight protocol requirement and multiple buffer-copy requirement from the transport layer provides significant acceleration for file-sharing operations.
manny2674695: Better perfromance than direct-attached block-level? Using a probabilistic IP network? What did I miss?
Mitch_Shults: Conventional TCP/IP Ethernet is a probabilistic network. Lots of folks are working on fixing that, especially with the iSCSI efforts. VI doesn't care what the underlying transport is, as long as it's reliable. Let's take the example of InfiniBand architecture.
Moderator: To hide "chatters" entering or leaving the chat, click on the "Hush" URL link found in the upper frame of your screen (you may need to use the vertical scroll bar to see this link.
bvigil823410: How does DAFS improve system performance?
Mitch_Shults: DAFS improves system performance by allowing applications to bypass operating system control, buffering, and heavy-duty network protocol operations that tend to bottleneck I/O throughput. Direct access I/O results in low-latency and low overhead data sharing, which increase system scalability and performance.
Moderator: If you've already asked a question, please be patient with the response to it. The related question and answer may not appear in the order we receive it. Mr. Shults and Mr. Dale are working to answer these as they come in.
cvanwagoner262792: How does DAFS handle file locking and permissions?
David_Dale: Again, can I point you to the spec for this ... www.DAFScollaborative.org
vijayv18472: What are the expected sources for overhead, if any?
Mitch_Shults: Using the most-optimal approach, described previously, the overhead is truly minimal. The client formats a DAFS request block in memory, including a data structure for payload information (or a receive buffer, if applicable), and invokes the user-level I/O library. That library, without a kernel transition, uses VI Architecture to communicate directly with the network transport hardware and from there to the DAFS filer's memory and I/O queue. Upon receipt at the filer, an I/O request is queued, and the filer gets to the request based on a polling cycle, so there's no ISR overhead. The only overhead in this case is setup time, VI communication time, network transport time, and filer parse/service time. The sum total of these elements will be far less than the overhead of processing a TCP/IP stack on the host, and with the right I/O hardware, like InfiniBand architecture, there are no stalls related to memory-mapped I/O and interrupt handling.
vansteenberg786577: Where can I find WhitePapers on DAFS?
David_Dale: www.DAFScollaborative.org
kkomiega124258: How does DAFS differ from conventional file systems with directly attached storage?
David_Dale: Traditional file systems that use directly attached storage are implemented inside operating system kernels and use kernel buffers to cache file system data. Application file requests must first make an operating system call and are satisfied by copying the data from the buffer cache to application buffers. In addition, sharing data with other machines requires complex proprietary software, and requires complex data management. DAFS, on the other hand, provides direct application data transfer and implicit data sharing using an open protocol, and simplifies data management.
Moderator: If you've already asked a question, please be patient with the response to it. The related question and answer may not appear in the order we receive it. Mr. Shults and Mr. Dale are working to answer these as they come in.
mhall529722: Does a disk vendor have to do anything special to support DAFS?
David_Dale: No
bsouthwell588501 : Where do you see infini band fitting into DAFS
Mitch_Shults: InfiniBand architecture fully specifies a link, a switch,a host channel adapter, and adapter form factors. InfiniBand architecture does not address higher-level transports and applications. DAFS is an example of a higher-level transport and an application, together. Therefore, it's beyond the scope of the InfiniBand Trade Association.
dspenard160200: How could DAFS improve system scalability, availability, and manageability?
Mitch_Shults: DAFS is designed to improve the efficiency and resiliency of shared file access in a clustered, System Area Network environment. Separating computing resources from storage resources allows each to be scaled and managed independently. Computing resources can be scaled by adding diskless commodity servers. Storage can be scaled either by adding more storage to existing file servers or by adding more file servers. Application servers see a common shared pool of storage. Availability improves because the architecture supports simple application server failover. File servers can be set up to failover also.
Moderator: We are no longer taking any additional questions. Mr. Shults and Mr. Dale will be in the chat room a few more minutes to answer the remaining questions that have already been submitted.
sfogarty531384: How does DAFS coordinate file sharing between systems?
David_Dale: DAFS is designed to allow high-speed, fault-tolerant, consistent views of files to a heterogeneous environment of servers that may be running different operating systems. The specification provides a consistent, cached locking mechanism that tolerates client or file server failures and failovers. DAFS also provides user authentication and cluster node access control for security.
bsheets859334: Can DAFS be used over a WAN or for general-purpose file sharing?
Mitch_Shults: In order to meet high-performance criteria, DAFS is designed for high-throughput, low-latency networks connecting a limited number of client machines that are centrally managed and have a certain level of trust between systems. DAFS may be inappropriate for some wide-area, general-purpose file-sharing environments.
Moderator: If you've already asked a question, please be patient with the response to it. The related question and answer may not appear in the order we receive it. Mr. Shults and Mr. Dale are working to answer these as they come in.
martimr1188685: Why does the DAFS collaborative elect not to address security, limiting the architecture to bounded networks?
David_Dale: We are addressing security. If you go to the website you can download the spec and get more specifics
brian721796: Is there any value in considering a DAFS/IP option? Why is DAFS so closely coupled with VI?
David_Dale: VI gives us two things - memory to meory RDMA capability, and transport independence. Initially DAFS will run on Fibre Channel and Gigabit Ethernet. Subsequently we'll see 10 Gigabit Ethernet and InfiniBand
bruce.fitch76215: At a high level, what security features will DAFS support? Native security of NFS,NT, other?
Mitch_Shults: DAFS does not bypass existing OS security mechanisms. If anything, it improves them. The DAFS approach is to use existing OS mechanisms for validating access and update rights, typically via ACL's. DAFS provides significant improvements for guaranteeing shared-file update integrity when many clients are accessing the same file, but that's independent of ACL checking.
vijayv18472: When can we expect the first implementations of DAFS, for what platforms?
David_Dale: Vendors in the DAFS Collaborative are targetting the second half of 2001 for initial products. Platforms include Unix (Solaris, Linux) and Win2k
kums686274: Direct access can bypass security mechanisms, so how could you guarantee this standard may not lead to security violations.
Mitch_Shults: Please see answer to 525.
nik.simpson928525: Would you say that it is more oriented to toward large block sequential transfers than small small random I/O?
Mitch_Shults: File-sharing generally tends to be more oriented toward large-block transfers than small-block transfers. DAFS will support both modes, of course, but there's inevitably more overhead per byte for small-block I/O than for large-block I/O in a file-sharing scheme.
lwatson609988: What type of companies have expressed interest in DAFS?
David_Dale: A really broad range of companies have expressed interest to date: hardware vendors (NICs, storage, server, networking), software vendors (apps, DBMS, admin), and service providers (SSPs, ISPs)
wskinse34783: We've been really reluctant to invest in a second robust network infrastructure (FC) to build a SAN. How will DAFS work on Gig-E? Will it require DAFS enabled Cisco boxes, new hardware, etc?
David_Dale: The VI over Ethernet NICs will work with standard Ethernet infrastructure (cables, switches, etc)
nik.simpson928525: With VI over Ethernet, would you be able to share the same adapter/network with conventional IP based ethernet traffic or would you end up with a standard ethernet network and a DAFS network alongside?
Mitch_Shults: This one is an issue for a particular IHV implementation. There's no technical reason why an IHV could not simultaneously support 'conventional mode' and 'VI-mode' communications through the same adapter. In a real-world environment, I'd probably recommend having two adapters - one for conventional IP, the other for VI architecture, since the internal paths are inevitably different, and switching between them will inevitably cause overhead. An IHV may come up with cool tricks that make that advice wrong, however.
beach4me822117: So, the filer has queued up the data from the disk and it's sitting in memory on the filer, then is served to the calling app?
David_Dale: Yes
peterlombardo812476: Is there anything currently using the VI architecture today and can you be confident in third party driver developers on supporting it?
Moderator: Our live Expert Q&A Session is being extended past 2:30p.m. (EST)so that Mr. Shults and Mr. Dale can finish answering many of the questions posed. Stay tuned for more answers.
Mitch_Shults: Oracle Parallel Server, and DB2 Extended Edition, Enterprise both ship with native VI support, now.
wskinse34783: RE: question 531. The answer to 510 suggests tat DAFS will utilize IP over Gig-E but requires fast IP processors on the smart NIC at both ends. Yes?
Mitch_Shults: That's true, if you want decent performance. It's possible to run DAFS (and iSCSI) over a conventional adapter, but host protocol-stack processing overhead is very painful performance-wise.
go4ed739544: DAFS appears to be an OS bypass protocol especially designed for storage....is this a reasonable way to view it?
David_Dale: That seems reasonable to me ...
martimr1188685: Why choose an architecture that requires re-linking at the application level, rather than an application-transparent architecture?
Mitch_Shults: The choice isn't forced. If you don't want to re-link, use the kernel-level model, with modular drivers, as described previously. Absolute best performance requires user-level I/O, which requires a minimal amount of modification to source and re-linking.
Moderator: Due to the high volume of questions not all answers will appear on line. However, Mr. Shults and Mr. Dale will answer all questions posed in our transcript which will be availabe on SearchStorage.com in a few days.
holley_jim807388: Do you have any performance expectations for DAFS that you can share with us today?
David_Dale: Nothing rigorous enough to share with you today. Performance expectations is one of the topics on the agenda at the next DAFS Collaborative Developers Conference (schedule for 11/28 at the SF airport Hyatt)
jlc68899: Is there any benefits over FC when using transparency mode?
Mitch_Shults: Not for block-mode I/O. If you're using TCP/IP over FC, which many vendors support, and you're using that link for file-sharing access, then you're suffering the performance hit of IP overhead. DAFS, using a VI-compliant FC adapter, gets rid of that overhead.
txstar969530: When will DAFS be available and who are the VI vendors.
David_Dale: Expect to see client-side and server-side DAFS products in the seconfd half of 2001. NIC vendors who have already publicly announced NIC plans include Troika Networks and Giganet
daveusi: What programming languages will the DAFS API mechanisms support?
Mitch_Shults: Under discussion. Join the collaborative and help us decide this!
balaji55699: If an application uses the "user level" DAFS API, it will not be able to use the local buffer cache. Don't you think that this will be a performance problem for reads?
Mitch_Shults: It won't be able to use the same API for the OS-provided local buffer cache, correct. But the whole idea is that the data is essentially cached on the filer, and then transmitted on-request to the user-level application I/O buffer for DAFS resources. With a data-center-class network transport, the latency for these I/O's could often be LESS than that for local I/O.
nik.simpson928525: How do you expect DAFS to do vs. conventional NAS solutions connected by the next generation of intelligent NICs that implement the IP stack in hardware on the NIC?
Mitch_Shults: We'll have to see. My bet is that DAFS will be faster, over the same NIC, than conventional sharing mechanisms, because it's specifically optimized for a data center transport, so things like timeout intervals are optimized for extreme low-latency.
nik.simpson928525: Which vendors have committed to implementing DAFS on the client side?
Mitch_Shults: Network Appliance Client Software Group (the former Orca) is working in this area. We expect the initial implementations of the DAFS client to be supported by IHV's with DAFS-capable adapters. I'm old enough to remember when that's how things worked in the early days of file serving, before PC OS's could deal with shared files natively. Similarly, we expect the OS vendors to eventually catch up to the state of the art and begin providing native DAFS support within their OS's, but DAFS' success is not dependent on any OSV action.
David_Dale: The initial client-side SDK was released at the DAFS Developers Conference held three weeks ago. This will enable us to start to engage ISV's interested in DAFS-enabling their apps. We are also actively recruiting ISV's to participate in the APIs Working Group (remember, the spec is only at rev 0.54 - there's lots of room for contribution). I don't know the exact numbers, but about 10% of the companies represented at the developers conference (on a show of hands) indicated they were working on client side implementations.
werner.glinka81587: Why is DAFS so important to the datacenter of the future?
Mitch_Shults: Customers build Internet data centers to get useful work done. That means processing transactions, serving up web pages, and generally doing value-added things. Economically, the goal is to get the most work done with the highest performance at the lowest cost. Today's data-center networking and file sharing approaches are not as efficient as they could be. That means that customers are buying more servers than they need to in order to get the required work done. And they're also not getting the best possible performance from those servers. DAFS, in combination with next-generation servers built around InfiniBand architecture, should enable a much higher level of efficiency than today's data centers can achieve.
Aziraphale: What OSI layer does DAFS live on?
Mitch_Shults: Some aspects of 2, all of 3, and some of 4 in some usage models.
balaji55699: If, as you say, DAFS is expected to run on a SAN, what are the possible applications that you see benefitting from using DAFS over NFS/CIFS?
Mitch_Shults: While there are lots of niche examples, let's focus on the biggies: web page serving and e-mail serving. Static web pages are just files. Internet data centers put those files on file servers, and configure hundreds of front-end web servers to access those files in a shared manner with file-servers. DAFS will get those files to the servers much faster and more scalably than existing approaches. In many e-mail servers, every e-mail and its associated attachments is a file or collection of files. It's efficient to partition the file I/O functions from the message-formatting and message-transfer functions. DAFS-based solutions will be able to do that faster and more scalably.
Aziraphale: How does DAFS improve performance if used in a GigE / TCP/IP environment?
Mitch_Shults: Answered previously. If the GbE adapter doesn't support VI, then DAFS won't work at all. If it does support VI, then acceleration occurs by bypassing OS mechanisms and going direct to user space (optionally). Even without user-level I/O, there are still significant benefits to VI just from a data-transfer efficiency standpoint.
laura393165: When is DAFS available and what systems architecture will be supported?
Mitch_Shults: The draft specification is available now on the DAFS collaborative website. Our goal is to finalize the spec within the next six months, and have products in 2001 to early '02. DAFS is independent of system architecture (though of course it will be supported on Intel architecture!). Nothing prevents Sun, for example, from implementing DAFS (client or server) on SPARC/Solaris. You'll have to ask them about their specific plans. My bet is that we'll get there first, but there are no guarantees where competition is concerned.
David_Dale: We expect to see Unix (Solaris and Linux) and Win2k clients first, followed by others.
laura393165: How does this protocol differ from Veritas File System (quick I/O built in capabilities)?
Mitch_Shults: Veritas File System, as I understand it, is a Distributed File System (DFS) approach. It allows block-level sharing within volumes by different OS instances. DAFS provides a completely different function. With DAFS, files are centralized on a server or servers, and the client-side DAFS stack coordinates with the filer to ensure valid I/O to shared files. With a DFS, client-side stacks communicate with one another in order to coordinate access to shared blocks. It's a fundamentally different I/O model. If Veritas has added file-sharing capability to VFS, then above discussion is wrong, but I don't believe that's the case.
john.duffy974458: Will all storage vendors (EMC, Netapp, IBM, Compaq, HDS, STK, etc) become DAFS compatible when the DAFS standard is incorporated into an operating system?
David_Dale: That really depends on where the vendor is focused. Our goal is to establish DAFS as a standard. I would expect any vendor with a NAS strategy to be seriously considering supporting DAFS. OSV support doesn't automatically mean that they will. However, storage vendors are very strongly represented in the DAFS Collaborative, so we expect to see broad adoption.
balaji55699: Do you see DAFS replacing NFS/CIFS in the future?
David_Dale: No. I believe they are complementary. NFS was originally devised to provide (Unix-based) file service to thousands of client workstations over a LAN. NFS 4.0 extends that to the wide area network. Similarly, CIFS is designed to provide (Windows-based) file service to thousands of client workstations over a standard network. DAFS, in contrast provides low latency high performance file service to applications servers (the clients in this case - which would typically be in the hundreds rather than the thousands) in a data center environment. Whereas NFS is optimized to map file system semantics onto a TCP/IP network, DAFS is optimized to map file system semantics onto a VI fabric. Both are needed.
kums686274: Does DAFS support I/O from Disks? If so, how does the application specify the location of files in disk (You've said DAFS will be implemented as a library).
David_Dale: I'm not sure I really understand your question. DAFS is designed to provide memory-to-memory file access between application servers (clients) and NAS devices (file servers) over a VI interconnect. Since it's a file service, the applications don't have to know where the blocks are on disk, but DAFS/VI optionally enables them to directly queue ops into file server memory. See also a previous answer. Hope this answers your question.
Moderator: This concludes our Live Expert Q&A Session with Mr. Shults and Mr. Dale on the subject of Direct Access File Systems (DAFS). We'd like to thank them for joining us today and thank all of our members of the audience for their questions. A transcript will be available over the next few days.
Moderator: Again, any unanswered questions will be answered in the SearchStorage.com transcript.
Moderator: Mr. Dale would be happy to answer any further questions. His e-mail address is ddale@netapp.com
searchStorage.com
c/o TechTarget.com
117 Kendrick Street
Needham, MA 02494
phone: (781) 657-1000
fax: (781) 657-1100 |
