Hi everyone,
I'm finally updating this blog to give you an update.
The ZFS-FUSE wiki and mercurial repositories are now running on a GeoWirx Xen VPS, so the long downtimes will be a thing of the past (knock knock). By the way, highly recommended service - those VPSs are good and cheap - excellent experience so far.
At the moment, a new port of ZFS-FUSE to 32-bit PowerPC is being worked on, with the kind help of Thomas Riddle (Sun engineer) and Matt Sealey (Genesi manager) and his developers. A few weeks ago I've also enabled direct I/O, which slowed performance, and I will soon implement async I/O to hopefully get it back.
As you may have noticed, progress on ZFS-FUSE has been slow. These past few months I've been finishing my degree, which has required a lot of time (only one exam left.. tomorrow!) and I've been on vacations.
What's perhaps more interesting is that back in June I was invited to work for Cluster File Systems on the Lustre file system. The idea is that Lustre will be using the DMU portion of the ZFS-FUSE port to Linux as its storage backend.
This means that the talented team of engineers who have pushed Lustre into new levels of scalability will be looking to do the same with the ZFS DMU!
Of course, this doesn't necessarily mean that you will see a huge boost in performance in ZFS-FUSE, but I think many improvements are still possible. And now that I've started working for CFS I'm also learning new things by the minute :)
Oh, and expect to see the corruption issue fixed soon (kernel patch might be required, unfortunately).
What's also interesting is that just today Sun, who as you all know created ZFS, has announced that it has acquired CFS. So, if everything goes well and they still decide to keep me... next week I will be a Sun engineer!
Anyway, you may have noticed that I've removed the PayPal links (thanks to everyone who donated!), since I don't think it will make much of a difference anymore.
Well, in the midst of all this, I hope I can still dedicate some time to work on ZFS-FUSE (after all, I am also using it). And of course.. patches are always welcome!
Wednesday, September 12, 2007
Wednesday, June 20, 2007
Site down
Hi everyone,
Recently an article was posted on LinuxWorld about ZFS-FUSE. See also the Slashdot article.
In case you're wondering why the ZFS-FUSE website is down, it's because I have been without Internet access since Monday. I never thought I'd say this again but I'm using a 33Kb dial-up connection right now (I think this might be slower than this)..
As a consequence, the Mercurial repositories are also down, and I might not receive some of your emails..
On another note, there has been recent reports of corruption in Raid-Z pools, so please backup your data before trying ZFS-FUSE!
Recently an article was posted on LinuxWorld about ZFS-FUSE. See also the Slashdot article.
In case you're wondering why the ZFS-FUSE website is down, it's because I have been without Internet access since Monday. I never thought I'd say this again but I'm using a 33Kb dial-up connection right now (I think this might be slower than this)..
As a consequence, the Mercurial repositories are also down, and I might not receive some of your emails..
On another note, there has been recent reports of corruption in Raid-Z pools, so please backup your data before trying ZFS-FUSE!
Monday, April 16, 2007
ZFS in the Linux kernel?
Due to the recent surge of interest in porting ZFS to the Linux kernel (if you are in the mood to read dozens of messages, see this thread, the follow-up, plus this one and one more), I'd like to offer my view on things.
I have a feeling most Linux kernel hackers (or at least those that talk about ZFS on linux-kernel) don't really know how ZFS works or what it can do. The best example is perhaps this message from Rik van Riel.
Well, first of all, ZFS doesn't have/need fsck (what?! are you nuts??). This is because ZFS checks and repairs the filesystem online and on-the-fly, as it is being used. And when it can't repair, it will pinpoint exactly which files and which bytes in those files were corrupted. You might think this is complex or expensive, but it's really simple and beautiful actually. These slides explain this and a lot more, so please read them carefully.
The great thing is that ZFS can also repair metadata on-the-fly even on ZFS pools that don't have any inherent redundancy (in other words, this also works for single disks). This is due to a feature called ditto blocks, which basically keeps multiple copies of metadata dynamically spread through the disk. Oh and now this works for data too, so you can configure your filesystem with important files to keep 2 or even 3 copies of data on the disk (this is despite any inherent pool redundancy).
ZFS has a lot of other nice things too, like cheap and instantaneous snapshots and clones, optional compression, variable sector sizes, easy management, .. I really think interested people should read these slides and try zfs-fuse.
Now regarding a ZFS port to the Linux kernel:
1) As for technical difficulty, I don't think it is a problem. I don't know Linux VFS internals, but if I was able to port it so easily to FUSE, it certainly can be done. I don't think this is a problem at all.
2) As for the license, well.. that is a real problem. I'm a big believer in FSF's ideals, but in this case I think the GPLv2 is preventing progress. It would be a big plus to have Linux benefit from a fully open-source, useful piece of functionality with 6 years of development behind it.
Of course, as Adrian Bunk put it, I don't think it'll be possible to have 10,000 (live and dead) people to agree on a licensing change.
One option would be to reimplement ZFS (or a comparable filesystem) from scratch. I don't think this is feasible, first because it would require a huge effort and several years to reach the same level of robustness as ZFS has right now. And second because Sun has filed more than 50 patents on ZFS. Even if Sun never uses those patents against Linux, some people might see it as a risk (in the United States).
The only way I'm seeing ZFS on the Linux kernel is to convince Sun to dual-license ZFS under the GPL and the CDDL. Some people might say Sun would never do this, but Sun has been very open to the open-source community recently. And in fact, Sun's ZFS FAQ initially had an answer saying Sun was considering a ZFS port to Linux (not to FUSE, that was my idea ;).
Finally I'd like to debunk a couple of myths about zfs-fuse:
1) In terms of features, zfs-fuse will certainly be comparable to a ZFS kernel implementation (and in fact, most of it already works). The only thing that can't be done is to store swap on a ZFS pool, due to the way ZFS works. You can see the STATUS file for more details about implemented features.
2) As for performance, well.. zfs-fuse is slow right now, but it will certainly improve. I haven't even started to seriously look at performance. And FUSE-based filesystems can have comparable performance to kernel filesystems, as the bottleneck is usually the disk(s), not the CPU.
I have a feeling most Linux kernel hackers (or at least those that talk about ZFS on linux-kernel) don't really know how ZFS works or what it can do. The best example is perhaps this message from Rik van Riel.
Well, first of all, ZFS doesn't have/need fsck (what?! are you nuts??). This is because ZFS checks and repairs the filesystem online and on-the-fly, as it is being used. And when it can't repair, it will pinpoint exactly which files and which bytes in those files were corrupted. You might think this is complex or expensive, but it's really simple and beautiful actually. These slides explain this and a lot more, so please read them carefully.
The great thing is that ZFS can also repair metadata on-the-fly even on ZFS pools that don't have any inherent redundancy (in other words, this also works for single disks). This is due to a feature called ditto blocks, which basically keeps multiple copies of metadata dynamically spread through the disk. Oh and now this works for data too, so you can configure your filesystem with important files to keep 2 or even 3 copies of data on the disk (this is despite any inherent pool redundancy).
ZFS has a lot of other nice things too, like cheap and instantaneous snapshots and clones, optional compression, variable sector sizes, easy management, .. I really think interested people should read these slides and try zfs-fuse.
Now regarding a ZFS port to the Linux kernel:
1) As for technical difficulty, I don't think it is a problem. I don't know Linux VFS internals, but if I was able to port it so easily to FUSE, it certainly can be done. I don't think this is a problem at all.
2) As for the license, well.. that is a real problem. I'm a big believer in FSF's ideals, but in this case I think the GPLv2 is preventing progress. It would be a big plus to have Linux benefit from a fully open-source, useful piece of functionality with 6 years of development behind it.
Of course, as Adrian Bunk put it, I don't think it'll be possible to have 10,000 (live and dead) people to agree on a licensing change.
One option would be to reimplement ZFS (or a comparable filesystem) from scratch. I don't think this is feasible, first because it would require a huge effort and several years to reach the same level of robustness as ZFS has right now. And second because Sun has filed more than 50 patents on ZFS. Even if Sun never uses those patents against Linux, some people might see it as a risk (in the United States).
The only way I'm seeing ZFS on the Linux kernel is to convince Sun to dual-license ZFS under the GPL and the CDDL. Some people might say Sun would never do this, but Sun has been very open to the open-source community recently. And in fact, Sun's ZFS FAQ initially had an answer saying Sun was considering a ZFS port to Linux (not to FUSE, that was my idea ;).
Finally I'd like to debunk a couple of myths about zfs-fuse:
1) In terms of features, zfs-fuse will certainly be comparable to a ZFS kernel implementation (and in fact, most of it already works). The only thing that can't be done is to store swap on a ZFS pool, due to the way ZFS works. You can see the STATUS file for more details about implemented features.
2) As for performance, well.. zfs-fuse is slow right now, but it will certainly improve. I haven't even started to seriously look at performance. And FUSE-based filesystems can have comparable performance to kernel filesystems, as the bottleneck is usually the disk(s), not the CPU.
Tuesday, March 06, 2007
ZFS for Linux Beta 1
Hi everyone,
I'm happy to announce the release of ZFS on FUSE/Linux 0.4.0 beta 1.
Even though this is a beta release, it should be more stable than your typical beta filesystem. The main problems in this release are (lack of) performance and high memory usage with some load patterns.
As always, be sure to read the README and the STATUS files.
So, what's the future for zfs-fuse?
The plan is to implement all the missing features marked for 0.4.0 in the STATUS file and then release beta2. After that I intend to focus on performance, release rc1 and after extensive testing release 0.4.0 final.
I wish to thank all the users who have been testing and helping this project.
I also want to give a special thanks to Phil Worrall, Chris Samuel, David Plumpton and especially Roland (devzero) for all the patches, bug reports, tests and suggestions :)
I'm happy to announce the release of ZFS on FUSE/Linux 0.4.0 beta 1.
Even though this is a beta release, it should be more stable than your typical beta filesystem. The main problems in this release are (lack of) performance and high memory usage with some load patterns.
As always, be sure to read the README and the STATUS files.
So, what's the future for zfs-fuse?
The plan is to implement all the missing features marked for 0.4.0 in the STATUS file and then release beta2. After that I intend to focus on performance, release rc1 and after extensive testing release 0.4.0 final.
I wish to thank all the users who have been testing and helping this project.
I also want to give a special thanks to Phil Worrall, Chris Samuel, David Plumpton and especially Roland (devzero) for all the patches, bug reports, tests and suggestions :)
Thursday, January 18, 2007
News
Hi,
I haven't been able to make much progress these last couple of weeks due to school projects and exams, however there have been a few bug fixes and performance improvements. Thank you to everyone who reported bugs and sent patches (and suggestions)!
I expect to release alpha2 in the next few days, after some newer functionality is implemented.
Interestingly, Patrick Verner has been busy adding support for zfs-fuse in his Parted Magic LiveCD/USB project, which is now officially the first Linux distribution supporting ZFS!
Seems very cool, so if you want to test zfs-fuse but don't feel like compiling it yourself, give it a shot ;)
Oh, and I've created a zfs-fuse discussion group, following the suggestion of Miguel Filipe :)
By the way - if you encounter any unexpected problem using zfs-fuse, please report a bug or post a message to the zfs-fuse discussion group, otherwise I can't fix it..
I haven't been able to make much progress these last couple of weeks due to school projects and exams, however there have been a few bug fixes and performance improvements. Thank you to everyone who reported bugs and sent patches (and suggestions)!
I expect to release alpha2 in the next few days, after some newer functionality is implemented.
Interestingly, Patrick Verner has been busy adding support for zfs-fuse in his Parted Magic LiveCD/USB project, which is now officially the first Linux distribution supporting ZFS!
Seems very cool, so if you want to test zfs-fuse but don't feel like compiling it yourself, give it a shot ;)
Oh, and I've created a zfs-fuse discussion group, following the suggestion of Miguel Filipe :)
By the way - if you encounter any unexpected problem using zfs-fuse, please report a bug or post a message to the zfs-fuse discussion group, otherwise I can't fix it..
Subscribe to:
Posts (Atom)
