Luscious Lebkuchen: XtreemFS 1.2 now available

XtreemFS 1.2 ("Luscious Lebkuchen") is now available at http://www.xtreemfs.org.

New features:

• asynchronous MRC database backups (database dumps are created in the background)
• new GRID SSL mode: SSL is used for authentication only, data is transferred via plain TCP for a better performance
• Vivaldi-based replica selection and placement
• MRC and OSD wait and retry for a configurable time span if they cannot reach the DIR on startup
• faster file copying of files with 'cp' (by providing correct values for 'st_blksize' with 'getattr')
• additional binaries with standard Unix names (mount.xtreemfs, mkfs.xtreemfs, lsfs.xtreemfs, rmfs.xtreemfs)
• experimental Hadoop file system driver for XtreemFS
• web-based admin console
  

We also fixed most of the bugs recorded in the tracker.

When upgrading your XtreemFS installation from 1.1 to 1.2, MRC and DIR databases should be converted automatically, without the need for human intervention. However, for safety reasons, we suggest to create an XML dump with the old release before updating; see "Updating" section in the XtreemFS FAQs (http://www.xtreemfs.org/faq.php).

A hadoop file system driver for XtreemFS

We have started the development of a FileSystem driver for hadoop. The current version implements the basic functionality and supports parallel I/O for striped files but does not support advanced features like retrieving block locations. The driver is still in an early stage of development!

If you still want to try it, the source is in trunk/contrib/hadoop. To build it, you need the jar from the hadoop release 0.20.1 and the XtreemFS trunk (or release 1.2). Make sure to modify the path to the hadoop.jar in nbproject/project.properties. In order to use XtreemFS with hadoop, you need to load three jar files: the HadoopClient.jar, the XtreemFS.jar and the yidl.jar. You also have to add XtreemFS to your hadoop config:

<property>
  <name>fs.xtreemfs.impl</name>
  <value>org.xtreemfs.common.clients.hadoop.XtreemFSFileSystem</value>
  <description>The FileSystem for xtreemfs: uris.</description>
</property>

The URLs for XtreemFS follow this scheme: xtreemfs://volumeName@dirServiceHost:32638/path

The userID and groupID can be set in the config options xtreemfs.client.userid and xtreemfs.client.groupid

web-based admin console

The next release (1.2) will include a web-based admin console. You can easily monitor the status of your installation, edit the UUID mappings, create new volumes, change policies or create MRC backups.
The console also shows the free space per volume and OSD. If you configured the "geographic_coordinates" in your service configuration files, the Google maps overlay will give you a nice overview of your entire installation.

The console .war file is in trunk/contrib/console in our svn.

Release 2.0 of XtreemOS

The new release 2.0 of XtreemOS was just released here at Supercomputing'09.

XtreemFS @ SC09 (Portland, OR)

You can meet us at Supercomputing in Portland, OR at the XtreemOS booth (#2388) from 11/16/09 to 11/19/09.

Cold Tamales: XtreemFS 1.1 now available

XtreemFS 1.1 is now available at xtreemfs.org.

Note for users upgrading from version 1.0: the metadata database format has changed. You must use dump/restore of the database when updating from 1.0!

New features in release 1.1 include:
- POSIX advisory locks (e.g. fcntl locks)
- Global inode/dev numbers (for dovecot, tar, and other programs that rely on unique inodes in struct stat)
- On-close replication. Replicas can now be automatically created when a file is closed after having been written the first time. This causes the file to become read-only. The behavior of on-close replication can be controlled via the xtfs_repl tool.
- A new "hostname" property in the server configurations, for specifying the host name to advertise to the directory service
- gridmap-file user identification

We also fixed a number of bugs, including a problem with SSL on Windows and issues that prevented the OSD from running on Windows.

public XtreemFS demo server

We have set up a public demo server for everybody to test XtreemFS. All you need is to install the client and execute

mkdir ~/xtreemfs_demo
xtfs_mount demo.xtreemfs.com/demo ~/xtreemfs_demo
cd ~/xtreemfs_demo

Please do not upload anything illegal or copyrighted material. For legal reasons every file create/write is logged with the IP address and timestamp. Files are automatically deleted after approx. 1 hour.

XtreemFS in the news

• heise (english) http://www.h-online.com/open/Distributed-file-system-for-the-internet--/news/114009
• heise (german) http://www.heise.de/open/Verteiltes-Dateisystem-fuers-Internet--/news/meldung/143550
• golem (german) http://www.golem.de/0908/69078.html
• Linux Magazin (german) http://www.linux-magazin.de/NEWS/XtreemFS-1.0-ist-fertig
• 01net.com (french) http://pro.01net.com/editorial/505038/avec-xtreemfs-leurope-a-son-systeme-de-fichiers-en-ligne/
  

Jelly Donuts: Release 1.0 is here!

XtreemFS 1.0 is out. XtreemFS now supports read-only replication with failover, data center replica maps, parallel reads and writes, and a native Windows client, among other new features.

Download it at xtreemfs.org.

Watch our demo of replication and failover on youtube!

Meet us at Europar in Delft, The Netherlands

I will be giving a talk about the internals of striping/parallel I/O and replication at the XtreemOS summit which is held at Europar 2009 in Delft. The XtreemOS summit is a workshop and held on August 25th.

In September there is also the XtreemOS summer school were Jan or I will give a talk and hands-on session on XtreemFS. Finally, we plan an informal XtreemFS get-together in Berlin in September or October, more information will follow soon.

Join the XtreemFS team

Want to join the XtreemFS team? We have a vacancy for a researcher (wiss. Mitarbeiter) at ZIB in Berlin.

XtreemFS at LinuxTag 2009

We had a booth at this year's LinuxTag 2009 in Berlin. Along with XtreemFS we also presented the XtreemOS project. It was really interesting for us to see what use-cases people had in mind for XtreemFS. We will try to integrate the suggestions and ideas into our new website.

First Release Candidate for version 1.0

We have put the first release candidate for version 1.0 of XtreemFS online. Most components have been rewritten, including a new client and the BabuDB-based MRC. Download source code or pre-build packages.

New MRC based on BabuDB

Since the beginning of this year, we have been working on a new MRC implementation. We came to the conclusion to completely rewrite the MRC for the following reasons:

• The first MRC implementation relies on an in-memory database based on Java Tree Maps, with checkpoints being created by writing the Tree Maps to disk with Java's built-in serialization support. This approach has severe limitations with respect to the size of a volume database. If volumes grow to a size of more than a couple 100k files and directories, the former MRC will run out of memory and crash when trying to create a database checkpoint.
• There is quite some potential to improve the first MRC's performance. Index structures used to arrange metadata are not optimal; in the general case, several lookups are needed to retrieve metadata of a single file, and retrieving directory contents requires one lookup per directory entry. A better arrangement of indices can greatly boost the speed of metadata lookups.
• There is no support for consistent metadata snapshots at runtime with the first MRC. In order to create a snapshot of the MRC database, which is e.g. done when checkpointing the database, the first implementation blocks all incoming requests until the snapshot creation process has finished, which renders the system unusable during this time. The ability to create snapshots without interrupting system operability will later become a requirement to create consistent file system snapshots and backups at runtime.

We have now completed the re-write, with a database backend based on BabuDB. The new backend exhibits a higher performance and support for larger databases than the previous backend. Besides, the MRC's architecture has been completely revised, and most of the code has been rewritten from scratch.