Can the RRDtool fall behind?

[SeanFromIT]

I now have some 300 hosts with some 700 services, and the ODS directory is filled with some 2000 *.rrd files. I noticed today that all graphs in Centreon' Views had stopped at about 20:00 last night, even though service checks were happening fine. I rebooted the server and about 15 minutes later the graphs were now drawing to about 23:00 last night. But they haven't drawn anything past that yet. I'm not quite sure if I understand the process that links the rrd files and the database together, but could that large number of rrd files somehow cause it to fall behind in processing?

Another strange thing that happened is that at 09:00 this morning I measured the ODS database at 1.7GB, and after the reboot at 16:15 today it was suddenly at 1.3GB. Any idea why the database would shrink like that? The graphing problem occurred before the sudden change in database size.

[damrodd]

It turns out the above issue was due to a corrupt table in the db. Repairing the table resolved the problem.

[SeanFromIT]

The graphs stopped again around 19:00 last night. This time the database tables are okay. I did notice that the table eating the most space is ods/log at 2.2GB. This table just contains the Event Log, right? So it would be okay to clear?

[SeanFromIT]

So I cleared the log table yesterday. Graphing was working fine again yesterday up until about 19:00 last night, and then the graphing stopped again.

This morning I rebuilt the RRD files and restarted ODS and things are working fine again. Any idea why this would start happening nightly after it worked fine for like a month?

[SeanFromIT]

Again...graphing stopped around 19:00 last night...getting to be pretty regular. Any ideas? Nothing shoots off from cron around that time, and the server is dedicated, running nothing but Nagios/Centreon.