XtreemFS

XtreemFS 1.3.1 Available Now!

2011-11-02T15:28:00.011+01:00

XtreemFS 1.3.1 is available now, which adds some minor features and fixes a wide range of bugs that were disclosed since the previous stable release.

Bug fixes mainly relate to the new client and the replication infrastructure. We included the following features:

Metadata replication. Although XtreemFS 1.3.0 already provides some preliminary support for MRC and DIR replication, the feature turned out to be fairly unstable. With XtreemFS 1.3.1, we took a big step forward in this respect. Especially the MRC replication has been thoroughly tested now and offers automatic fail-over on the client side.
Asynchronous writes. We enhanced the client with preliminary support for asynchronous writes. However, the feature is currently limited to non-replicated and failure-free scenarios, as it has not yet been integrated with the client's internal retry mechanism.
Monitoring. We added a service monitoring infrastructure. It is based on SNMP and provides information about the internal state of an XtreemFS service, such as the current memory consumption, I/O throughput, the number of stored files, etc. We also added a corresponding Ganglia plug-in.
OSD drain tool. We included a utility to remove OSDs from an XtreemFS installation. The tool relocates all files from the respective OSD to other OSDs and gracefully shuts down the OSD.
Gentoo overlay. To simplify the use of XtreemFS for our Gentoo users, we added a Gentoo overlay for installing XtreemFS on Gentoo.

For a more detailed overview of the changes, please refer to our change log. We also updated the list of known issues and moved them to the issue tracker.

We further noticed that we caused some confusion with the 1.3.0 release, as it was sometimes referred to as a release candidate. The website as well as the servers and clients themselves always stated "1.3.0 RC1" as version. However, the packaging did not allow us to release a "1.3.0-RC1" version and therefore we ended up publishing "1.3.0" packages. Blogs and news websites also referred to the released version as "1.3.0". So, we'll leave it that way: XtreemFS version 1.3.0 RC1 is regarded as 1.3.0 and now we're releasing the next version, 1.3.1.

Since we released XtreemFS 1.3.0 in August, we got quite a lot of feedback on the mailing list - thanks a lot to our user community for consistently helping us to improve XtreemFS!

To be able to quickly respond to the needs of our users, we decided to establish a separate repository with unstable packages, which we update frequently. Unstable packages are less thoroughly tested than stable releases, but they allow us to fix bugs and provide new features on short notice. A link to the unstable repository can now be found on our website at http://www.xtreemfs.org/download.php#unstable.

Public XtreemFS demo server online again

2011-08-23T16:54:00.005+02:00

Do you want to test XtreemFS without installing the servers first?

No problem! Just install the XtreemFS Client 1.3 (currently available for Linux and MacOSX, Windows will follow later) and mount our public XtreemFS demo server:

mkdir ~/xtreemfs_demo
mount.xtreemfs demo.xtreemfs.org/demo ~/xtreemfs_demo
cd ~/xtreemfs_demo

We placed a copy of the freely available short film "Big Buck Bunny" on the demo server which you can watch after mounting. (Please keep in mind that your Internet connection to our demo server has to be fast enough to watch it smoothly.)

For testing you can create any directories and files as you like. Please do not upload anything illegal or copyrighted material. For legal reasons every file create/write is logged with the IP address and timestamp. Files are automatically deleted every hour.

Upgrading from XtreemFS 1.2.x to 1.3

2011-08-11T12:49:00.001+02:00

The good news is that the on-disk formats of the DIR, MRC and OSD haven't changed which means that just need to update the packages. The bad news is that 1.2.x and 1.3.0 are protocol-incompatible, you can't mix 1.2.x servers/clients with 1.3.0 servers/clients.

To upgrade servers, shut them down. Install the 1.3 packages (or update if you have the xtreemfs repository configured) and start the servers again. For the clients, unmount all mounted volumes and install (or update) the 1.3.0.

Most important changes in the user-interface:

mount.xtreemfs has new command line options
xtfs_stat, xtfs_repl and xtfs_sp have been merged into xtfsutil
xtfsutil works on files on mounted volumes, not on URLs
you can use xtfsutil --errors /mount/point to retrieve client error messages

First Release Candidate for XtreemFS 1.3

2011-08-10T15:28:00.000+02:00

After nearly one year of development, we have finished the first release candidate for XtreemFS 1.3.
The most important new feature is the cross-site file replication with auto-failover. The new replication works with mutable files, i.e. files can be read and written.

We have created packages for Linux in our repositories on build.opensuse.org, they should become available within a few hours. For Mac OS X, we have a packaged client with installer. The sources can be downloaded at http://code.google.com/p/xtreemfs/downloads/list.

Moving towards 1.3

2011-02-17T16:09:00.000+01:00

As you might have noticed, we are currently a full year behind schedule with the 1.3 release. The good news is that we are working heavily on the new release.

We now have a new client written from scratch that implements important features such a replica failover and metadata caching. The new client already passes our complete test suite (see http://groups.google.com/group/xtreemfs-test). Currently, we are cleaning up the code, work on a libxtreemfs, porting the client to windows and a lot of manual testing for the release.

Most of the changes we have worked on since last year are invisible to the user. First of all, we have switched the internal protocol to a custom RPC format which is optimized for transfer of raw data (the file content) and uses Google protocol buffers for message encoding. For this new RPC protocol, we have re-written and optimized the client and server infrastructure. The new protocol is up to 2x faster when transferring objects. As a user, you'll notice that the URLs now start with "pbrpc://" instead of "oncrpc://".

Over the next few weeks, we'll write short posts on the stuff we have been working, e.g. the PBRPC protocol, the new client features and internals, read/write replication for files, MRC replication ...
Of course, we'll keep you updated on the 1.3 release.

For anyone who wants to throw a first glance at XtreemFS 1.3, we set up a repository with unstable packages: http://download.opensuse.org/repositories/home:/xtreemfs:/unstable/
Please be aware that these packages are experimental and may be changed or updated without prior notice.

Want to work on XtreemFS?

2010-08-02T17:58:00.004+02:00

We have positions for PhD students at the Zuse Institute Berlin (Germany) where you have the opportunity to work on XtreemFS within the CONTRAIL project. You can email (kolbeckzib.de) or call (+49-30-841-85-328) us for more information. Deadline is October 15th.

Here is the official job description:

The Zuse Institute Berlin (ZIB) is a non-university research institute under public law of the state of Berlin. In close interdisciplinary co-operation with the Berlin universities as well as national and international scientific institutions, ZIB conducts research and development in the field of information technology, applied mathematics, and computer science. To support research and development efforts in EU- and BMBF-funded projects, the department Parallel and Distributed Systems invites applications for several PhD Student or PostDoc Positions (f/m) for the duration of two years - Vgr. IIa/Ib BAT/Anwendungs-TV Land Berlin - Application code WA 22/10 As a research assistant you will explore, design, implement and evaluate scalable, fault-tolerant and distributed algorithms and systems for processing large-scale scientific data. We have developed a range of systems including: Scalaris, a structured peer-to-peer storage system; XtreemFS, a distributed and replicated file system, and BabuDB, a replicated key-value store. In co-operation with partners from science and industry we validate, extend, and optimize our solutions in production environments.

Requirements

Master's degree or Diploma in computer science
Solid fundamentals in distributed systems and algorithms
Experience with distributed file systems, databases or peer-to-peer technology
Demonstrated coding skills in C++, Java or Erlang
Familiarity with Unix/Linux
Ability to work in interdisciplinary and international teams
Fluency in English

You will work in an inspiring and pleasant environment and will receive adequate professional support. We offer challenging scientific tasks, a high degree of autonomy, and state-of-the-art technical infrastructure. You will have the opportunity to pursue a PhD or Habilitation supervised by Prof. Reinefeld. The position will be initially financed for a period of two years with the possibility of extension. The salary is based upon wage group IIa/Ib as per Berlin Collective Agreement for the Public Sector. Zuse Institute Berlin is an equal opportunity employer. We prefer to balance the number of female and male employees in our institute. Thus, we kindly encourage female candidates to apply to this job offer. Handicapped persons will be given preference to other equally qualified candidates. Please send your complete application, referring to application code WA 22/10, including cover letter, CV and relevant certificates/GPA/university transcripts until 15. October 2010 to Konrad-Zuse-Zentrum fuer Informationstechnik Berlin (ZIB), - Verwaltung -, Takustr. 7, 14195 Berlin, Germany

XtreemFS at LinuxTag 2010

2010-06-07T12:18:00.000+02:00

We'll be at LinuxTag 2010 in Berlin which starts in two days (June 9th to 12th). Visit us at the XtreemOS booth #206 in Halle 7.2a.

ISC, Summer School ...

2010-05-26T14:01:00.001+02:00

You can meet us at ISC 2010 in Hamburg from 31/05 to 02/06 at the XtreemOS booth (booth #121, next to Unicore and BSC).

The XtreemOS summer school will take place at Schloss Günzburg near Ulm from the 5th to the 9th of July. There is also a talk and a practical on XtreemFS.

Finally, the XtreemOS Challenge offers a prize of €1,000 for the best application ported to XtreemOS.

How XtreemFS uses leases for replication

2010-05-07T20:00:00.011+02:00

If you implement a distributed file systems like XtreemFS, you are dealing with many interesting problems. The most central one is probably to make files behave as if they were stored in the local file system.

The main property of this sought behavior is called strong or sequential consistency [*]. Sequential consistency requires that reads and writes (even concurrent ones) are executed in a well-defined (but random) order. Apart from sequential consistency, the file system must also ensure that reads reflect all previous writes and that concurrent reads and writes are isolated.

As XtreemFS supports fault-tolerant file replication, it has to maintain the local behavior of files while internally storing and coordinating multiple physically independent copies of the data. In technical terms, this translates to implementing sequential consistency for replicated data in a fault-tolerant way.

The simplest way to implement sequential consistency is to use a central instance that defines the order in which operations change the data (a sequencer). And indeed many distributed file systems realize sequential consistency by establishing a lock server that hands out locks to clients or storage servers. The lock holder receives all operations on the data and executes them serially. The result is a well-defined order. These locks can be made fault-tolerant by attaching a timeout to them: what you get is a lease, which can be revoked even when a client is unresponsive or dead.

A sophisticated alternative to defining a sequencer is to use a so-called replicated state machine, a distributed algorithm that is executed by all replicas of the data. If you want implement a fault-tolerant version of it, you will end up with using a Paxos derivative. The problem is that all fault-tolerant algorithms in this domain require two round-trips for both reads and writes to the data to establish sequential consistency across all replicas, which is clearly to expensive for high-rate operations like the ones on files.

So we are left with the central instance approach, fully aware that this introduces both a single point of failure and performance bottleneck. A quick back-of-the-envelope calculation reveals: assuming 50.000 open files in your cluster, with a 30 sec timeout you have 1666/sec lease renewals, which already quite some load for a lock server.

Such a high lease renewal rate is even more of a problem when you consider fault-tolerance of the lock server itself. To ensure that it is highly available, you need to replicate its state, and are again faced with a sequential consistency + replication problem. The solutions: master-slave replication with some fail-over protocol (another lease problem?) or a replicated state machine for the lease server itself. The latter has been chosen by Google for their Chubby lock service. The replication of the lock servers state solves the availability problem, but worsens the performance bottleneck. Google's "Paxos Made Live" paper cites 640 ops/sec [3]. Not enough for 50k open files (although Chubby is not used for GFS' leases).

For XtreemFS, we have chosen a different approach. Instead of using a lock service to manage leases, we let the object storage devices (OSDs) negotiate leases among themselves. For each file, all OSDs that host a replica of the file negotiate the lease. The lease-holding OSD acts as a sequencer and receives and executes all reads and writes. In turn, an OSD participates in all lease negotiations for all its file replicas that are currently open.

Negotiating leases directly by the storage servers has several advantages: it scales naturally with the number of storage servers, it saved us from implementing a lock server, and the user from the headache of provisioning and managing another service. The only problem we needed to overcome was that we first needed an algorithm that negotiates leases in a fault-tolerant, decentralized way. Such a thing didn't exist, and telling from a recent blog post from Jeff Darcy, the usage of fault-tolerant leases still seams to be its infancy [Link].

The result of our efforts are two algorithms, FatLease [1] and its successor Flease [2]. They scale to thousands of concurrent lease negotations per second - for each set of participants. For XtreemFS this means essentially that the number of open files is counted per storage server and not against the whole file system. With 1000s/sec. negotiations, this would translate to an open file count of more than 50k files per OSD.

With a fault-tolerant lease negotiation algorithm, we have solved the problem of enforcing sequential consistency and arbitrating concurrent operations. While this is the hardest part of implementing replication, the data replicas also need to be updated for every change. How this is done in XtreemFS will be a topic of a future blog post.

[*] POSIX actually mandates a strong consistency model: serializabiliy. In simple terms, it means that the file system has to take communication between its clients into account. However, this is impractical for distributed file systems, as the file system would have to control all communication channels of its clients.

References:

[1] F. Hupfeld, B. Kolbeck, J. Stender, M. Högqvist, T. Cortes, J. Malo, J. Marti. “FaTLease: Scalable Fault-Tolerant Lease Negotiation with Paxos”. In: Cluster Computing 2009.

[2] B. Kolbeck, M. Högqvist, J. Stender, F. Hupfeld. “Fault-Tolerant and Decentralized Lease Coordination in Distributed Systems”. Technical Report 10-02, Zuse Institute Berlin, 2010.

[3] Tushar Chandra, Robert Griesemer, and Joshua Redstone. "Paxos made live". PODC '07: 26th ACM Symposium on Principles of Distributed Computing.

student position in Berlin

2010-04-06T11:26:00.002+02:00

We have a student position for XtreemFS at ZIB in Berlin: http://www.zib.de/News/Jobs/index.en.html (in German)

XtreemFS user survey

2010-03-05T15:08:00.000+01:00

You can help us make XtreemFS better. Let us know what you use XtreemFS for and which features you need. Fill out our user survey at http://www.xtreemfs.org/user_survey.php

XtreemFS update 1.2.1

2010-02-04T10:31:00.002+01:00

We just released an update for XtreemFS (version 1.2.1). This version contains mainly bug fixes, e.g. for FreeBSD and Fedora 12, and enhanced replica management. The new scrubber will automatically replace failed replicas.

Source code and packages are available for download on http://www.xtreemfs.org.

There is no change in the database or OSD storage, so an update from 1.2 should work out of the box. The usual advice: backup important data before upgrading.

Luscious Lebkuchen: XtreemFS 1.2 now available

2009-12-11T17:08:00.000+01:00

XtreemFS 1.2 ("Luscious Lebkuchen") is now available at http://www.xtreemfs.org.

New features:

asynchronous MRC database backups (database dumps are created in the background)
new GRID SSL mode: SSL is used for authentication only, data is transferred via plain TCP for a better performance
Vivaldi-based replica selection and placement
MRC and OSD wait and retry for a configurable time span if they cannot reach the DIR on startup
faster file copying of files with 'cp' (by providing correct values for 'st_blksize' with 'getattr')
additional binaries with standard Unix names (mount.xtreemfs, mkfs.xtreemfs, lsfs.xtreemfs, rmfs.xtreemfs)
experimental Hadoop file system driver for XtreemFS
web-based admin console

We also fixed most of the bugs recorded in the tracker.

When upgrading your XtreemFS installation from 1.1 to 1.2, MRC and DIR databases should be converted automatically, without the need for human intervention. However, for safety reasons, we suggest to create an XML dump with the old release before updating; see "Updating" section in the XtreemFS FAQs (http://www.xtreemfs.org/faq.php).

A hadoop file system driver for XtreemFS

2009-12-09T13:15:00.000+01:00

We have started the development of a FileSystem driver for hadoop. The current version implements the basic functionality and supports parallel I/O for striped files but does not support advanced features like retrieving block locations. The driver is still in an early stage of development!

If you still want to try it, the source is in trunk/contrib/hadoop. To build it, you need the jar from the hadoop release 0.20.1 and the XtreemFS trunk (or release 1.2). Make sure to modify the path to the hadoop.jar in nbproject/project.properties. In order to use XtreemFS with hadoop, you need to load three jar files: the HadoopClient.jar, the XtreemFS.jar and the yidl.jar. You also have to add XtreemFS to your hadoop config:

<property>
  <name>fs.xtreemfs.impl</name>
  <value>org.xtreemfs.common.clients.hadoop.XtreemFSFileSystem</value>
  <description>The FileSystem for xtreemfs: uris.</description>
</property>

The URLs for XtreemFS follow this scheme: xtreemfs://volumeName@dirServiceHost:32638/path

The userID and groupID can be set in the config options xtreemfs.client.userid and xtreemfs.client.groupid

web-based admin console

2009-12-08T12:26:00.001+01:00

The next release (1.2) will include a web-based admin console. You can easily monitor the status of your installation, edit the UUID mappings, create new volumes, change policies or create MRC backups.
The console also shows the free space per volume and OSD. If you configured the "geographic_coordinates" in your service configuration files, the Google maps overlay will give you a nice overview of your entire installation.

The console .war file is in trunk/contrib/console in our svn.

Release 2.0 of XtreemOS

2009-11-17T17:50:00.001+01:00

The new release 2.0 of XtreemOS was just released here at Supercomputing'09.

XtreemFS @ SC09 (Portland, OR)

2009-11-12T12:45:00.003+01:00

You can meet us at Supercomputing in Portland, OR at the XtreemOS booth (#2388) from 11/16/09 to 11/19/09.

Cold Tamales: XtreemFS 1.1 now available

2009-09-15T15:54:00.004+02:00

XtreemFS 1.1 is now available at xtreemfs.org.

Note for users upgrading from version 1.0: the metadata database format has changed. You must use dump/restore of the database when updating from 1.0!

New features in release 1.1 include:
- POSIX advisory locks (e.g. fcntl locks)
- Global inode/dev numbers (for dovecot, tar, and other programs that rely on unique inodes in struct stat)
- On-close replication. Replicas can now be automatically created when a file is closed after having been written the first time. This causes the file to become read-only. The behavior of on-close replication can be controlled via the xtfs_repl tool.
- A new "hostname" property in the server configurations, for specifying the host name to advertise to the directory service
- gridmap-file user identification

We also fixed a number of bugs, including a problem with SSL on Windows and issues that prevented the OSD from running on Windows.

public XtreemFS demo server

2009-08-20T14:08:00.003+02:00

We have set up a public demo server for everybody to test XtreemFS. All you need is to install the client and execute

mkdir ~/xtreemfs_demo
xtfs_mount demo.xtreemfs.com/demo ~/xtreemfs_demo
cd ~/xtreemfs_demo

Please do not upload anything illegal or copyrighted material. For legal reasons every file create/write is logged with the IP address and timestamp. Files are automatically deleted after approx. 1 hour.

XtreemFS in the news

2009-08-20T13:17:00.004+02:00

heise (english) http://www.h-online.com/open/Distributed-file-system-for-the-internet--/news/114009
heise (german) http://www.heise.de/open/Verteiltes-Dateisystem-fuers-Internet--/news/meldung/143550
golem (german) http://www.golem.de/0908/69078.html
Linux Magazin (german) http://www.linux-magazin.de/NEWS/XtreemFS-1.0-ist-fertig
01net.com (french) http://pro.01net.com/editorial/505038/avec-xtreemfs-leurope-a-son-systeme-de-fichiers-en-ligne/

Jelly Donuts: Release 1.0 is here!

2009-08-13T15:17:00.006+02:00

XtreemFS 1.0 is out. XtreemFS now supports read-only replication with failover, data center replica maps, parallel reads and writes, and a native Windows client, among other new features.

Download it at xtreemfs.org.

Watch our demo of replication and failover on youtube!

Meet us at Europar in Delft, The Netherlands

2009-07-16T09:01:00.002+02:00

I will be giving a talk about the internals of striping/parallel I/O and replication at the XtreemOS summit which is held at Europar 2009 in Delft. The XtreemOS summit is a workshop and held on August 25th.

In September there is also the XtreemOS summer school were Jan or I will give a talk and hands-on session on XtreemFS. Finally, we plan an informal XtreemFS get-together in Berlin in September or October, more information will follow soon.

Join the XtreemFS team

2009-07-14T14:37:00.000+02:00

Want to join the XtreemFS team? We have a vacancy for a researcher (wiss. Mitarbeiter) at ZIB in Berlin.

XtreemFS at LinuxTag 2009

2009-06-05T10:26:00.006+02:00

We had a booth at this year's LinuxTag 2009 in Berlin. Along with XtreemFS we also presented the XtreemOS project. It was really interesting for us to see what use-cases people had in mind for XtreemFS. We will try to integrate the suggestions and ideas into our new website.

First Release Candidate for version 1.0

2009-05-14T18:11:00.003+02:00

We have put the first release candidate for version 1.0 of XtreemFS online. Most components have been rewritten, including a new client and the BabuDB-based MRC. Download source code or pre-build packages.

New MRC based on BabuDB

2009-02-26T15:55:00.002+01:00

Since the beginning of this year, we have been working on a new MRC implementation. We came to the conclusion to completely rewrite the MRC for the following reasons:

The first MRC implementation relies on an in-memory database based on Java Tree Maps, with checkpoints being created by writing the Tree Maps to disk with Java's built-in serialization support. This approach has severe limitations with respect to the size of a volume database. If volumes grow to a size of more than a couple 100k files and directories, the former MRC will run out of memory and crash when trying to create a database checkpoint.
There is quite some potential to improve the first MRC's performance. Index structures used to arrange metadata are not optimal; in the general case, several lookups are needed to retrieve metadata of a single file, and retrieving directory contents requires one lookup per directory entry. A better arrangement of indices can greatly boost the speed of metadata lookups.
There is no support for consistent metadata snapshots at runtime with the first MRC. In order to create a snapshot of the MRC database, which is e.g. done when checkpointing the database, the first implementation blocks all incoming requests until the snapshot creation process has finished, which renders the system unusable during this time. The ability to create snapshots without interrupting system operability will later become a requirement to create consistent file system snapshots and backups at runtime.

We have now completed the re-write, with a database backend based on BabuDB. The new backend exhibits a higher performance and support for larger databases than the previous backend. Besides, the MRC's architecture has been completely revised, and most of the code has been rewritten from scratch.

BabuDB - efficient key value store for java

2008-11-21T13:55:00.010+01:00

The XtreemFS MRC uses a simple key-value store for managing all metadata. We have spent a lot of time to evaluate the different storage backends: HSQLDB (and relational databases in general), BerkeleyDB for Java and simple Java TreeMaps.

Embedded SQL databases like HSQLDB are simply too slow for key-value lookups since SQL parsing is expensive. Stand-alone SQL server require IPC which also rules out that solution. We used BerkeleyDB for a while but due to the lack of documentation and stability issues we had to drop that as well. The Java TreeMaps did their job, but they have real limits in terms of size and serializing to disk is slow and interrups operations.

Finally, we decided to implement our own key-value store called BABUDB. It is based on the LSM-Tree concept and is optimized for applications that need a non-transactional key-value store. If you like to find out more, visit our google code project at http://code.google.com/p/babudb/. In the next release of XtreemFS, the MRC will be ported to the new database which will result in much better performance and a better utilization of multi-core processors.

XtreemFS 0.10.0 released: checksums and more

2008-10-27T18:51:00.006+01:00

We have just released XtreemFS 0.10.0 on http://www.xtreemfs.org/. Apart from many quality improvements, it contains several exciting new features:

checksum support for file objects, including xtfs_scrub for verification
client: caching, improved performance, new platforms (win32, ARM, OS X)
UUIDs
OSD cleanup
plug-in interface

Made popular by ZFS, checksums allow you to verify the integrity of your file data. XtreemFS OSD can now compute, maintain and verify checksums for each file object. Our new scrubbing tool xtfs_scrub allows you to verify checksums on all OSDs in parallel.

Our client, an integral part of the XtreemFS architecture, is now much faster than previous versions, and you should be network or disk bound in most cases. We also added caching for file data and metadata, which also improves performance considerably. The client is now also running on Linux on ARM devices and we have added experimental ports for Windows and OS X.

XtreemFS now identifies all its services by UUIDs. This allows you to move services and their data to different hosts and use NATed network setups where only some of the IP addresses can be used.

Further, we have now a tool xtfs_cleanup for reclaiming storage space by erasing orphaned objects from OSDs (objects can be orphaned if the client crashes while deleting a file). We extend our plug-in interface in the MRC so that you can add your own policies (written in Java) to control its operation.

Client News: Caching, Win32, OS X, ARM

2008-10-24T11:02:00.003+02:00

The client of XtreemFS contains a considerable part of the logic of XtreemFS as it has to coordinate the metadata and object storage servers (OSDs) to provide the user with a POSIX-compliant file system abstraction.

This setup is great from an architectural view point because it scales well as it takes tasks away from the metadata and storage servers. But it also makes the client a complex beast.

In the last weeks we have put a lot of work in the client and you will be able to use the features soon as part of the 0.10.0 release.

The first area of work was caching. While the operating system already caches file data in the page cache, the granularity of the accesses are mostly 4k - too small if you have to fetch each of these over the network. With client-side caching, we can now fetch data with object granularity, which is usually a few hundred k or more. We will follow-up with real measurements soon, but the performance improvements are very good.

Client-side caching is also laying the foundation for prefetching, RAID, and other things that are on our long-term roadmap.

We have also implemented a metadata cache that allows the client to retrieve the results of a readdir() together with all the stat()s in one RPC - important for long-latency networks like DSL or installations over the Internet. But it also feels more snappy on the LAN.

We have also started porting the client to other platforms than Linux/FUSE. We have been running on OS X with Mac FUSE for quite a while now (available on our download page), and have now extended our coverage of platforms to Windows and ARM Linux. The Windows client is using the fantastic Dokan library with the fuse4win adapter. It will be available on the download page soon.

FUSE performance

2008-08-20T14:03:00.000+02:00

The XtreemFS client is implemented as a FUSE driver. Therefore, the throughput of FUSE could also be a limiting factor for the overall performance of our file system. Matthias implemented a simple "emptyfs" FUSE driver which simply discards all data. I used the driver to measure bandwidth from an application through the VFS layer and FUSE to the user-level process. The machine I ran the test on has two CPUs with four cores each (Xeon E5420 @ 2.5GHz) with 16GB RAM. I used dd to transfer 2GB of data with block sizes from 4k to 64MB.

The results are plotted in two graphs. The first graph shows the throughput in MB/s as report by dd. The second graph shows the CPU usage (sy= system, us= user) and the number of context switches.

graph 1 (write bandwidth in MB/s):

graph 2 (CPU usage, context switches):

With this results (2GB/s) for 128k or larger blocks, it is easy to see that FUSE is not the limiting factor for us. But this also shows that FUSE without the direct_io options has real performance problems as all write requests are split into 4k writes. So, you have to choose between performance and the ability to execute files (mmap does not work when direct_io is enabled, see this FUSE mailing list entry).

XtreemFS 0.9.0 released

2008-08-14T11:02:00.000+02:00

We have released XtreemFS 0.9.0, a distributed file system for federated IT infrastructures. XtreemFS is an integrated part of the XtreemOS Linux operating system for the Grid, but can also be run on various other Linux distributions.

XtreemFS is a full file system that features:

full Posix compliance, incl. Posix ACLs and extended attributes
parallel access to striped files, stripe width configurable per file
scalable installations by adding more storage and metadata servers
transparent integration into XtreemOS' user and VO management
integration into various X.509 authentication infrastructures
mountable on all systems that support FUSE, incl. OS X

XtreemFS is GPL-licensed and available via http://www.xtreemfs.com as source code and packaged for various Linux distributions, incl. XtreemOS.

... Felix for the XtreemFS development team