Syslogd doesn't check max size?

[Konan]

I am not sure if the following is a bug or just an anomaly, or is specific to Debian or Linux in total.

I was going to back up some files to a DVD so I loaded a new blank in the drive and waited for it to come ready. I decided that now was a good time to fill up my drink and left the computer for a while. When I got back about 30 minutes later, I noticed that the drive was still cycling and had not become ready. I finally got the process killed and tried another blank, which worked. Then I went on about my business.

The next day, the machine hung up on boot with Dmesg saying that it was starting the syslog daemon, but that was as far as it would go till it dropped to the maintenance shell after a few minutes. Multiple boots always did the same thing.

After considerable troubleshooting, I noticed that the syslog file was at the maximum size - about 2.5 gig I think - and was full of DVD drive errors. Deleting the file fixed the problem.

Just to test, I built a phony syslog file to the maximum size and installed it in /var/log. Bang - same problem.

So... It appears that the system logger doesn't test for max size overflow either during runtime or during boot. Interestingly, I googled up one hit for the same problem from a couple of years ago, also on Debian, but nothing in the debian.org problem lists. I guess very few users ever fill up a log file.

Konan

[Icarus]

most people either check the logs regularly or use logrotate which will take the current log and gzip it. If you don't have logrotate installed I highly recommend it

[Konan]

most people either check the logs regularly or use logrotate which will take the current log and gzip it. If you don't have logrotate installed I highly recommend it

I do have logrotate installed and it is working. In fact, I think it installs by default with Etch. Whichever, my logs rotate on schedule.

But this is the next thing that I was checking out. It appears that logrotate triggerers on a time schedule, not a size check. So it appears that if the log fills up before it rotates, the problem arises. The reason for all the "appears" is that I have not proved the above yet.

And I am trying to get an answer to another question...

If the log is full, i.e. at 2.5 gig, what happens to current log postings? Apparently they are thrown away.

Another defugalty is that on a different Etch machine (1 out of 3), the problem is completely different in that it also locks up on a full log file, but on the next reboot the logfile is rotated.

This isn't any kind of show stopper bug, although it took a couple of hours before I figured out what was happening the first time. However, if I was the sysop trying to get an important server back on line, it would have been majorly show stopping. But it is an interesting phenomenon.

Konan

[bwkaz]

And I am trying to get an answer to another question...

If the log is full, i.e. at 2.5 gig, what happens to current log postings? Apparently they are thrown away.

Well, what else are you proposing? You can't write the logs out to disk anywhere (that's why the disk is full ); what else is even possible?

(There is a limit on file size, but it's nowhere near as low as 2.5GB unless the filesystem is FAT32. ext2/3 have a limit of something like 8TB, and ext4 is even bigger. I doubt any other Linux FS has a limit any lower than ext3's. So the issue isn't that the file is 2.5GB; it's most likely that the file is taking up all your available space.)

on a different Etch machine (1 out of 3), the problem is completely different in that it also locks up on a full log file, but on the next reboot the logfile is rotated.

I believe there's an option to run logrotate as part of the bootscripts -- or at least, if it can be run from cron (which is how it normally runs), then it can also be run from the bootscripts, if your distro has decided that's a good idea.

On the locking up: That's one good reason to make /var a separate partition. That way filling it up won't screw up the rest of your machine (by filling up the root FS); it will simply reject new log lines until space is freed.

[Konan]

Sorry, not a chance. There is more than 400 gig available for /var/log. Of course the 2.5 gig limit has to be somewhere in the log code since that is far below the max file size of even a very old version of Linux. This being a very unimportant machine, I just took the easy option of everything in one partition during install. If I were running a real (i.e. money making) server, it would be a different partition setup.

But the 2.5 gig limit is real. I can recreate the problem at will on three different machines. As I said, one of those machines acts somewhat differently but it was loaded at least a year later so has much more current updates than the other two. (Yeah, I know that I should update all machines very often - but try that sometime when all that is available is a 26k dialup line.)

This weekend I plan to load a test RedHat or Slackware machine and see if the same problem happens. So far I have resisted the idea of opening the syslog source and plowing through that, since I promised myself never to inflict myself with C coding again. (Life is too short as it is, and you can't always get through on a suicide hot line.)

Again, this isn't a major problem for me, or even a problem at all, now - just an interesting gotcha that I am casually pursuing. Or not, if something else more interesting comes along.

Konan

[Satanic Atheist]

OK, I'll bite.

I'll cat 1 billion lines into /etc/syslogd and see if that makes any effect on my system. The system is coming down and moving tomorrow anyway, so I can clean the logs up and carry on as normal. I run Debian Etch (stable) but I will see if it creates the same effect. The drives it runs on (RAID-1 mirror) have 200Gb available. Shouldn't be a problem. I'll post back if it screws up.

James

Edit: Running now. You shouldn't ever fill up the log files. 2.5Gb of text is a MASSIVE amount of data. Truly staggering. There's more likely a problem with your underlying system that you need to sort first.

[Konan]

You ain't just kidding about 2.5 gig being a lot of data. If it wasn't for the tail command I wouldn't have time to load the file to look at it

Anyway, I still have my old Sarge machine loaded and just sitting there being an old backup. Loaded the too-large file on it, and it did not fail.

I remembered an having an old Toshiba laptop on the shelf with Slackware. Cranked it up, loaded the file and it didn't fail either.

So... The problem is apparently restricted to Debian Etch - probably those with early loads. The google hit that I got was dated early in Etch's release also.

I plan to haul my main machine into town to a broadband connection and do an update. I have a list of programs to apt-get so an update is in order anyway. I would rather do an incremental update so I could get an idea of what is the culprit, but that is totally impossible using my barbed wire and tin can connection at home.

I had a feeling that this was something that was not a problem across the entire Linux spectrum, since I could find so little info on it.

Konan

[happybunny]

probably not related to your post at all, but we had a problem where nfs was dumping lines into messages filling it up beyond disk capacity (4.5gigs and growing)...

since the error was irrelevant, i just added an "ignore" line to syslog-ng.conf and it stopped filling up our messages file.

If you're lucky enough to have syslog-ng instead, you could just prevent to file from getting to big in the first place

[Satanic Atheist]

OK. After cat'ting 1 billion copies of "Test" into /var/log/syslog I managed to get the file up to 2.2Gb. It never went over your 2.5Gb "limit" but the staggering amount of logging information really is not good.

Sorry, but my machine remained stable even with that much crap in the file.

James