PDA

View Full Version : "Connection Failed" on Home, Monitoring, Reporting, and Graphs tabs



havok_88
16th June 2012, 00:45
Hi everyone,

First off, I must say thank you to everyone on the forums for creating such an active community - the solutions posted here have helped me a number of times in the past.Unfortunately I am unable to resolve my most recent issues with Centreon after going through many of the solutions for similar problems posted here.

I have had this happen twice now to Centreon, where an update or unplanned shutdown of the server causes Centreon to stop functioning, and upon restarting all services, I am unable to view anything on the  Home, Monitoring, Reporting, and Graphs tabs, as well as the main Centreon configuration under Configuration > Centreon. All of these pages return a "Connection failed, please contact your administrator" message.

I am able to log in correctly, edit/view host and service definitions, etc. I am also able to see the nagios statistics under Home > Nagios Statistics. I should mention that Nagios is working perfectly behind the scenes as well and sending me notifications via email.

I have checked passwords, believe my privileges are in order for mysql:
+----------------------------------------------------------------------------------------+
| Grants for centreon@localhost |
+----------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'centreon'@'localhost' IDENTIFIED BY PASSWORD '39b9344b5fdf6ccb' |
| GRANT ALL PRIVILEGES ON `centreon`.* TO 'centreon'@'localhost' WITH GRANT OPTION |
| GRANT ALL PRIVILEGES ON `centstorage`.* TO 'centreon'@'localhost' WITH GRANT OPTION |
| GRANT ALL PRIVILEGES ON `centstatus`.* TO 'centreon'@'localhost' WITH GRANT OPTION |
+----------------------------------------------------------------------------------------+



Permissions for /usr/local/centreon/bin:
-rwxr-xr-x 1 nagios nagios 25267 Apr 12 16:16 centcore
-rwxr-xr-x 1 root root 5367 Apr 12 16:16 centFillTrapDB
-rwxr-xr-x 1 root root 6614 Apr 12 16:16 centGenSnmpttConfFile
-rwxr-xr-x 1 nagios nagios 28199 Apr 12 16:15 centstorage
-rwxr-xr-x 1 root root 14456 Apr 12 16:16 centTrapHandler-2.x
-rwxr-xr-x 1 root root 13144 Apr 12 16:16 centTrapHandlerForPoller-2.x
-rwxr-xr-x 1 nagios nagios 19976 Apr 12 16:16 logAnalyser
-rwxr-xr-x 1 nagios nagios 19798 Apr 12 16:16 nagiosPerfTrace
-rwxr-xr-x 1 root root 174578 Apr 12 16:16 snmptt
-rwxr-xr-x 1 root root 30437 Apr 12 16:16 snmpttconvertmib



Permissions for /usr/local/centreon/cron:
-rwxr-xr-x 1 nagios apache 12309 Apr 12 16:14 centAcl-Func.php
-rwxr-xr-x 1 nagios apache 27026 Apr 12 16:14 centAcl.php
-rwxr-xr-x 1 nagios nagios 2403 Apr 12 16:16 centreonPurge.sh
-rwxr-xr-x 1 nagios apache 13454 Apr 12 16:14 dashboardBuilder.pl
-rwxr-xr-x 1 nagios apache 14282 Apr 12 16:14 downtimeManager.php
-rwxr-xr-x 1 nagios apache 8884 Apr 12 16:14 eventReportBuilder.pl
drwxr-xr-x 2 nagios apache 4096 Apr 12 16:14 Misc
drwxr-xr-x 2 nagios apache 4096 Apr 12 16:14 Modules
drwxr-xr-x 2 nagios apache 4096 Apr 12 16:14 perl-modules
-rwxr-xr-x 1 nagios nagios 7618 Apr 12 16:16 purgeCentstorage
-rwxr-xr-x 1 nagios apache 5876 Apr 12 16:14 purgeComments
-rwxr-xr-x 1 nagios nagios 4171 Apr 12 16:16 purgeLogs
drwxr-xr-x 2 nagios apache 4096 Apr 12 16:14 reporting

Permissions for /usr/local/nagios/bin:
-rwxrwxr-- 1 nagios nagios 585944 Apr 12 16:03 nagios
-rwxrwxr-- 1 nagios nagios 43928 Apr 12 16:03 nagiostats
-rwxrwxr-- 1 nagios nagios 227513 Apr 12 16:09 ndo2db
-rwxrwxr-- 1 nagios nagios 113348 Apr 12 16:09 ndomod.o



I do not see any ndo2db errors in nagios.log, it appears to be functioning according to the logs in fact:
[1339798581] ndomod: NDOMOD 1.4b9 (10-27-2009) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[1339798581] ndomod: Successfully connected to data sink. 0 queued items to flush.
[1339798581] Event broker module '/usr/local/nagios/bin//ndomod.o' initialized successfully.



I have backups of the 3 centreon databases as of today, as well as one from 2 weeks ago. I also have backups of the nagios /etc/ directory for the same times. I have tried restoring back to the old backups, with the same result.

I really don't know where to go from here, so any input or direction would be greatly appreciated. This has happened before and I simply rebuilt the server as I could not fix the issue. I have backups so it is not a huge deal to do so, but I would really like to solve the problem!

Thank you all in advance!


EDIT: I should note, this is Centreon 2.3.4, Nagios 3, CentOS 5.8

NDJ10
16th June 2012, 00:55
check the login informations on /etc/centreon/centreon.conf.php, /etc/centreon/conf.pm and ndo2db > database tab are correct

havok_88
16th June 2012, 01:42
I had checked this file, but here it is to look at: (mypassword is the password I verified against the DB without issue)

/etc/centreon/centreon.conf.php:
/* Database */
$conf_centreon['hostCentreon'] = "localhost";
$conf_centreon['hostCentstorage'] = "localhost";
$conf_centreon['user'] = "centreon";
$conf_centreon['password'] = "mypassword";
$conf_centreon['db'] = "centreon";
$conf_centreon['dbcstg'] = "centstorage";

/* path to classes */
$classdir='./class';
/* Centreon Path */
$centreon_path='/usr/local/centreon/';


Also checked /etc/centreon/conf.pm, here it is:

$mysql_host = "localhost";
$mysql_user = "centreon";
$mysql_passwd = 'mypassword';
$mysql_database_oreon = "centreon";
$mysql_database_ods = "centstorage";
1;


And, here is the database tab: (I have re-entered mypassword just to be sure)

1777


Keep in mind, this is a previously working system. I have not made changes to the password since it was first installed a few months ago.


Cheers.

NDJ10
16th June 2012, 11:50
Maybe you had a mysql crash? check mysql logs, and sql-error.log in centreon log folder

havok_88
16th June 2012, 18:35
Nothing interesting in mysqld.log:
120615 14:28:34 InnoDB: Starting shutdown...
120615 14:28:35 InnoDB: Shutdown completed; log sequence number 2 2051985726
120615 14:28:35 [Note] /usr/libexec/mysqld: Shutdown complete


120615 14:28:35 mysqld ended


120615 14:30:51 mysqld started
120615 14:30:52 InnoDB: Started; log sequence number 2 2051985726
120615 14:30:52 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.95' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution


However, sql-error.log shows these:
2012-06-15 14:01|0|0|DB Error: no such table QUERY : SELECT value FROM options WHERE key LIKE 'Broker'
2012-06-15 14:01|0|0|DB Error: no such table QUERY : SELECT value FROM options WHERE key LIKE 'Broker'
2012-06-15 14:01|0|0|DB Error: no such table QUERY : SELECT value FROM options WHERE key LIKE 'Broker'
2012-06-15 14:02|0|0|DB Error: no such table QUERY : SELECT \* FROM options WHERE key LIKE 'broker'
2012-06-15 14:02|0|0|DB Error: no such table QUERY : SELECT value FROM options WHERE key LIKE 'Broker'
2012-06-15 14:02|0|0|DB Error: no such table QUERY : SELECT value FROM options WHERE key LIKE 'Broker'
2012-06-15 14:02|0|0|DB Error: no such table QUERY : SELECT value FROM options WHERE key LIKE 'Broker'
2012-06-15 14:02|0|0|DB Error: no such table QUERY : SELECT value FROM options WHERE key LIKE 'Broker'
2012-06-15 14:03|0|0|DB Error: no such table QUERY : SELECT \* FROM options WHERE key LIKE 'broker'
2012-06-15 14:03|0|0|DB Error: no such table QUERY : SELECT \* FROM options


I can query the record its looking for manually, using the centreon user, like so:
mysql> use centreon;
mysql> select options.key,options.value from options where options.key like 'broker';
+--------+-------+
| key | value |
+--------+-------+
| broker | ndo |
+--------+-------+


Ideas?

NDJ10
17th June 2012, 00:05
It looks like your centreon data base is corrupted after the shutdown.
Try to launch a repair table on your centreon database

havok_88
18th June 2012, 17:40
Alright, I ran a repair on all databases. Many centreon tables stated they do not support the function, but the ones that did all ran (the options table for example, from above).


I'm not getting any new errors in sql-error.log. The only log constantly indicating an issue is centAcl.log and downtimeManager.log with the following over and over:

<br/><b>Connection failed, please contact your administrator</b><img src='./img/centreon.gif'><br/><b>Connection failed, please contact your administrator</b><img src='./img/centreon.gif'><br/><b>Connection failed, please contact your administrator</b><img src='./img/centreon.gif'><br/><b>Connection failed, please contact your administrator</b><img src='./img/centreon.gif'><br/><b>Connection failed, please contact your administrator</b><img src='./img/centreon.gif'><br/><b>Connection failed, please contact your administrator</b>


I also noticed this in eventReportBuilding.log:

[1339750802] [INFO] Processing period: Thu Jun 14 00:00:00 2012 => Fri Jun 15 00:00:00 2012
[1339750805] [INFO] Exiting program...(pid=30070)
[1339837201] [FATAL] MySQL error : cannot connect to database centstatus
[1339837201] [FATAL] Program terminated with errors
[1339923602] [FATAL] MySQL error : cannot connect to database centstatus
[1339923602] [FATAL] Program terminated with errors
[1340010002] [FATAL] MySQL error : cannot connect to database centstatus
[1340010002] [FATAL] Program terminated with errors

havok_88
18th June 2012, 18:15
Other things I've noticed:

ndo2db isn't seen as running by the system, yet Nagios says it has connected just fine... Restarting yields the same result.

# /etc/init.d/ndo2db status
ndo2db is not running but subsystem locked

BUT:

[1340035822] ndomod: Successfully connected to data sink. 0 queued items to flush.
[1340035822] Event broker module '/usr/local/nagios/bin//ndomod.o' initialized successfully.

AND:

# netstat -apn | grep 5668
tcp 0 0 0.0.0.0:5668 0.0.0.0:* LISTEN 8153/ndo2db
tcp 0 0 127.0.0.1:5668 127.0.0.1:41844 ESTABLISHED 8515/ndo2db
tcp 0 0 127.0.0.1:41844 127.0.0.1:5668 ESTABLISHED 8516/nagios


I don't know if that means anything, but there it is. The lock file *is* present in /var/lock/subsys/.