User Tools

Site Tools


server_monitoring

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
Next revisionBoth sides next revision
server_monitoring [2009/11/17 11:30] 172.26.0.166server_monitoring [2009/11/27 08:38] alan
Line 16: Line 16:
 Interesting documentation: http://www.ibm.com/developerworks/wikis/display/WikiPtype/ganglia Interesting documentation: http://www.ibm.com/developerworks/wikis/display/WikiPtype/ganglia
  
-==== Notes ====+==== Troubleshooting ==== 
 +When Ganglia has problems displaying nodes it may need to be restarted.  Restart the daemons in this order: 
 +  * Stop data collection daemon on HPC: ''service gmetad stop'' 
 +  * Stop monitoring daemon on HPC: ''service gmond stop'' 
 +  * Stop monitoring daemon on compute nodes: ''rocks run host compute %%'%%service gmond stop%%'%%'' 
 +  * Start data collection daemon on HPC: ''service gmetad start'' 
 +  * Star monitoring daemon on HPC: ''service gmond start'' 
 +  * Start monitoring daemon on compute nodes: ''rocks run host compute %%'%%service gmond start%%'%%''
  
-It appears as if other CGIAR clusters have been configured to query our ''gmond'' daemon.  I'm not sure why, as their respective ganglia pages do not show ILRI HPC stats.  In any case, the relevant configuration settings are in ''/etc/gmond.conf'' on the head node, note especially the ''trusted_hosts'' entries: +Now go check the Ganglia web interface and see if the nodes have returned.
-<file># +
-# Gmond config file for Cluster Cluster. +
-# Generated by ganglia.xml node without aid from the database. +
-+
-name "ILRI" +
-owner "Cgiar"  +
-url "http://hpc.ilri.cgiar.org/" +
-latlong "N32.87 W117.22" +
-mcast_channel "237.170.26.97" +
- +
- +
-+
-# Increase size of gmond user (gmetric) hash table. +
-+
-num_custom_metrics 2048 +
- +
-# Uncomment the next line for monitoring by the Rocks Cluster Network. +
-trusted_hosts 220.227.242.214 # hpc.icrisat.cgiar.org +
-trusted_hosts 202.123.56.187  # hpc.irri.cgiar.org +
-trusted_hosts 216.244.151.133 # ? +
-trusted_hosts 200.62.229.37   # hpc.cip.cgiar.org +
- +
-# Listen only on the private cluster interface+
-mcast_if eth0</file> +
- +
-==== Connections refused ==== +
-I kept seeing this error in ''/var/log/messages'' on the head node: +
-<code>Aug 28 12:44:24 hpc-ilri /usr/sbin/gmond[3453]: server_thread() Host 200.62.229.37 tried to connect and was refused</code> +
-Apparently that is the Potato Center's (CIP) ganglia trying to talk to our ganglia.  Adding ''trusted_hosts 200.62.229.37'' to ''/etc/gmond.conf'' on the head node and restarting gmond fixed it.+
  
 ===== Monit ===== ===== Monit =====
Line 70: Line 48:
 Nagios is a powerful monitoring system that enables organizations to identify and resolve IT infrastructure problems before they affect critical business processes. http://www.nagios.org/about Nagios is a powerful monitoring system that enables organizations to identify and resolve IT infrastructure problems before they affect critical business processes. http://www.nagios.org/about
  
-====== Installation ======+=== Installation === 
 + 
 +---- 
 Download the latest version of nagios while hot, from http://www.nagios.org/download Download the latest version of nagios while hot, from http://www.nagios.org/download
-<code>wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.0.tar.gz </code>+<file>wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.0.tar.gz  
 +$ cd nagios-3.2.0 
 +$ ./configure 
 +$ make all  
 +$ useradd nagios 
 +$ make install 
 +$ make install-init 
 +$ make install-commandmode 
 +$ make install-config 
 +$ make install-webconf 
 +</file> 
 +=== Configuration === 
 + 
 +---- 
 +Running the following command will create a new file called htpasswd.users in the /usr/local/nagios/etc directory. It will also create an username/password entry for nagiosadmin. You will be asked to provide a password that will be used when nagiosadmin authenticates to the web server. 
 +<code>htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin </code> 
 + 
 +Download and install plugins  
 +<file> 
 +$ wget http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.14.tar.gz 
 +$ tar xfz nagios-plugins-1.4.14.tar.gz 
 +$ cd nagios-plugins-1.4.14 
 +$ ./configure && make && make install  
 +</file> 
 +Edit the configuration files to add host and services to be monitored: 
 +<code>vim /usr/local/nagios/etc/objects/localhost.cfg </code> 
 + 
 +Check remote services http://wiki.nagios.org/index.php/Howtos:checkbyssh_RedHat 
 +=== Accessing Nagios === 
 + 
 +---- 
 +http://172.26.0.205:4020/nagios/ 
 + 
 +with  username = "nagiosadmin" and password = "nagios" 
 +