Hard drive thrashing? with ext4

[harmerg]

My 1st post and I'm not sure this is the correct forum but here goes...
(Moderators please move it if I've chosen wrongly)

I'm dual booting Ubuntu 9.04 and 10.04 on an ASUS P5VD2-MX SE mobo with 3 hard drives, a couple of DVD writers and a multi memory card reader (USB)
The hard drives are:-
1 250Gb SATA MAXTOR STM325082
1 80Gb PATA HITACHI IC35L090AVV207-0
1 80Gb PATA MAXTOR 6Y080P0

9.04 lives on 2 ext3 partitions of the 250G SATA
10.04 on the 2 ext4 partitions of the 80G MAXTOR
The HITATCHI has 1 ext4 partition mounted as a common data area for each O/S
The 250G drive has GRUB Legacy in the MBR with its files in a small ext3 primary partition chainloading the 2 O/S's, GRUB Legacy for 9.04 and GRUB2 for 10.04. It also contains the common SWAP partition (primary)

My problem.

When I boot into 9.04 everything works as expected with as far as I can see no unusual disk activity even when the ext4 data partition and the 2 10.04 ext4 partitions are mounted.
When 10.04 is booted however, after about 5 minutes (irrespective of whether or not the other partitions are mounted) I see the hard drive activity light come on and stay permanently on. This worries me.
The system monitor shows nothing unusual. dstat doesn't give any clues either. I've tried running iotop but it fails with a missing CONFIG_TASK_DELAY_ACCT error which I'm not sure how to deal with.

Is this a known problem with ext4, should I rebuild 10.04 onto ext3 partitions?
Are there some special ext4 mount parameters I should use? (I simply copied the fstab lines from 9.04 and just changed ext3 to ext4).
I've read that the journaling process(es?) for ext4 by default commonly access the drive(s) every 2-3 secs but that should just cause a flicker surely.

Any guidance would be greatly appreciated because I'm concerned that my hard drive life could be shortened, and although I've not done any specific timings, subjectively I'm sure the system is less responsive once the problem occurs.

Many thanks

[saikee]

(I simply copied the fstab lines from 9.04 and just changed ext3 to ext4).

Not sure what you have done. Did you format the partition to the Ext4 filing system before putting files inside?

[trilarian]

(I simply copied the fstab lines from 9.04 and just changed ext3 to ext4).

Maybe post the /etc/fstab file that you copied? I don't think there are options for EXT3 that don't work on EXT4, but maybe something will stick out. Also, like saikee mentioned: the partition was formatted as EXT4, not just changed in /etc/fstab, correct?

Here are some general tweaks you can make to EXT4 (most apply to EXT3 too) to improve performance (this assumes you are not running a mail server or something like tripwire).

Settings in /etc/fstab:
*** Listed in order of most effect to least, so you don't have to use them all if you don't want. ***

• noatime - Turns off the write to atime for every read access.
• data=writeback - This allows metadata to be written back with less priority. Improves performance, but at the cost of potential file corruption if power loss and no UPS. Sometimes it is just the metadata and the file itself is fine. Other options would be data=ordered,commit=x where x is an Integer that represents the number of seconds between syncing the journal. A bit safer than writeback, but a bit slower too. Can also do data=journal which will give instant journal sync so is the safest, but the slowest.
• barrier=0 - This is a performance boost you should only use if you are on a UPS, as power loss during a write with this turned off is almost guaranteed file corruption.
• nobh - Turns off the use of buffer headers.

So your line would look like this (assuming disc sda1 and /, change to match your setup):

/dev/sda1 / ext4 noatime,data=writeback,barrier=0,nobh,errors=remount-ro 0 1

Which options you choose are really up to you and if you are on a UPS or not. One thing you can do is run hdparm -t on the partition first, then make one change at a time, unmount and remount (or reboot if /), then run the test again. You should see just how much each option boost or doesn't effect your I/O speeds. A note on hdparm though, I'd run it a few times and average as you'll get slightly different values every time you run it.

[harmerg]

Firstly saikee, trilarian thanks for your input and yes the 10.04 drive was pre-formatted to ext4.

Now a bit of background, possibly off-topic but very possibly related, you choose

I've been running 9.04 on the 250G SATA drive for about a year now with no problems and I thought it was about time to move to 10.04 LTS. So I backed up my home partition both to a bunch of DVD's and to a new partition on that drive (ext4). I then popped in a couple of 80G IDE drives, made sure the bios order put the SATA drive 1st and booted up the 10.04 live cd and chose 'try before you buy' mode. The intention was to format the 'new' drives to ext4 one as a test area for 10.04 and the other to copy my old home to 'just-in-case'...
This is where thing started to get interesting?!
I ran gparted to do the formatting and was pretty surprised to find 10.04 had ignored the bios drive order and allocated sda and sdb to the IDE drives giving the SATA drive sdg, with the intervening letters allocated to the 4 slots in the USB memory card reader. At this point the natural caution bred from (too) many years in IT kicked in and all thoughts of just going ahead with the installation vanished, so I just went ahead with the partitioning and formatting of the new drives and escaped back to 9.04 to do some research.
This link encapsulates what I found, and believe me it's one of many. I've found some references to other distributions but it seems to be mainly Ubuntu that IGNORES the bios settings when enumerating the drives and some posters seem to think it's a bit random - not good in a multi-drive environment. Using UUID's seems to be the way to go which is why I cut and pasted the fstab lines from 9.04 once I'd set up the drives in 9.04 (after a bit of experimenting with ext3).
So I then pulled the main SATA drive and one IDE drive and went ahead with installing 10.04 on the other. Following that I put back the original drives, created a small boot partition (ext3) on the SATA installed GRUB Legacy to the MBR and set it up to chainload 9.04 on the SATA and 10.04 on the IDE and was ready to test out 10.04.
Thats when I noticed the constant drive accessing when running 10.04.

So, is it a 'feature' of ext4? Does it indicate some problem related to 10.04's communication with the bios? I just don't know, but it's certainly worrying.

I'll certainly try your suggestions trilarian but I'm not over convinced they get to the heart of the problem. I'm also going to reformat the root partition of the IDE drive to ext3 and reinstall 10.04 to see if the problem goes away, but I don't see that as a solution, merely a work-around.

Just an off-the-wall thought because I have no knowledge of the inner workings of hard drive firmware - it's not possible is it that it's something as trivial as the drive access LED getting somehow 'stuck' on is it? I only mention this because if I exercise the drives thoroughly by copying large files around, once that light is on there is absolutely NO variation in it's intensity no matter how much data is thrown around. Once on the only way to turn it off is to re-boot (no need to power down).
Re-boot to 9.04, behavior normal, re-boot to 10.04 and after about 5 min on it comes again.

I'm rather flummoxed and I'm certainly not going to inflict 10.04 on my main drive 'till I've resolved this.
Incidentally my research indicates that 10.10 is also afflicted by the drive order 'feature'.

I apologize for the length of this post but I thought it was necessary to explain the background.

[trilarian]

You'll have to forgive me, just taking a break at work so don't have the time to really go over your response (not criticizing, generally more info is better than less for troubleshooting).

The thought that came to me though, is you can monitor your disk while doing nothing to see if you see activity. iostat is the program to use, and is included in the sysstat package. Stop any services that poll the hard disk, so you will be testing if the LED is on in error or if your disk is being continuously accessed. You will want to mount with either data=writeback or set a long commit interval so that the journal doesn't give you a false positive.

Once ready, fire off:

iostat -d -x interval count

Replace interval and count with integers. 5 and 3 would output every 5 seconds, 3 times (for example). The first block should show activity from accessing iostat, but then you should see activity drop to 0.

If disk I/O is 0.00 and the LED is on, then the problem lies with the signal to the LED. If you see persistent activity then you will have to dig further (ps and top will help) to see what is active.