Announcement

Collapse
No announcement yet.

Log robocopy result with syslog-ng

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Log robocopy result with syslog-ng

    Hi guys,

    Here we are, I enjoyed this week to play a bit when I had some free time to set up a log of the robocopys result in the syslog module. That's not so easy to do when we don't know how syslog is working.

    What I want to do :
    I want to backup the result of my night robocopys to be able to see quickly which robocopy had an error and which one.

    Why I want to do that? :
    Because it's faster to have all the backup result at the same place than open all the log files one by one et chech at the end of the file every morning. And mostly because the last audit we had asked to have a log of the backup log... To be able to say if a file have been correctly copied 2 years before...yahoo..

    How I want to do that? :
    I want to modify my robocopy scripts with a bit of perl to send the result returned by robocopy to a syslog server with gonna send the syslog request a mysql database. Thanks to the syslog-ng module for centreon, at the end I will be able to display the result and be able to sort them by server and script name or more and select a time period I want.

    Beginning :
    I used this link : http://syslog.modules.centreon.com/wiki/Install_fr to install my syslog-ng. ( Thanks AkHeNaToN for the documentation )

    At the end of the installation you have to be able to fill up your database and to display thousand of syslog message. For me I just want to display my robocopy result, so I removed all the sources and destination of the file /etc/syslog-ng/syslog-ng.conf and I modified the source gaved by AkHeNaToN.

    Code:
    source s_everything { internal(); pipe("/proc/kmsg"); unix-stream("/dev/log"); udp(); };
    
    destination d_mysql {
          pipe("/var/log/mysql.pipe"
                  template("INSERT INTO logs
                  (host, facility, priority, level, tag, datetime, program, msg)
                  VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC',
                  '$PROGRAM', '$MSG' );\n") template-escape(no));
    };
    
    log {source(s_everything); destination(d_mysql);};
    by
    Code:
    source s_everything { udp(ip(#IPADDRESS#) port(514)); };
    
    
    destination d_mysql {
          pipe("/var/log/mysql.pipe"
                  template("INSERT INTO logs
                  (host, facility, priority, level, tag, datetime, program, msg)
                  VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC',
                  '$PROGRAM', '$MSG' );\n") template-escape(no));
    };
    
    log {source(s_everything); destination(d_mysql);};
    #IPADDRESS# Have to be replace by the ip address which gonna receive the syslog message. ( The ip of the syslog server )


    At this time we have a server ready to receive syslog message, and to display it in the syslog module. That's mean it's time to test if it's working! I'have installed perl 5.10 on my computer. http://www.activestate.com/downloads/ and look for how to send a syslog request with a perl command. AkHeNaToN told me Net::Syslog can help me! Ok I go to the CPAN (Community of Perl fANatic Oo) and look for Net::Syslog. OK! That's easy! Install the Net::Syslog module for perl. That's mean download it and copy the files in C:\Perl\lib\Net.

    Open a text file, copy this code, past, save as file.pl. Open a dos command and execute the script.

    Code:
    use feature ':5.10';
    use Net::Syslog;
      my $s=new Net::Syslog(Facility=>'local4',Priority=>'debug');
      $s->send('see this in syslog',Priority=>'info',SyslogHost=>'10.229.254.39');
    say (' Message envoyé');
    I think I will like perl, looks pretty easy! If every think is allright, you have to be able to see in the syslog module the message sent by the script.


    So that's sound pretty good for the moment. I've stop at this stage for today.

    So I still have to work on the robocopy script, if someone know better than me perl ( I don't know perl ) and want to help on it I will me very happy!

    I'm looking for change the display time to have 24h of view by default. I'v found in the php file how to change the time but not the day. So AkHeNaToN if you know how can I change it

    I will begin to work on the script on monday and post it when I will find how to have something well.

    Enjoy your week end!

  • #2
    ### Français ###

    Tu veux le fonctionnement suivant (si je ne me trompe pas):

    Robocopy => envoi message syslog (resultat robocopy) vers serveur (collecteur) syslog => serveur syslog (dns ton cas syslog-ng) => insertion en base de données "syslog" => affichage dans Centreon via le module Syslog.

    1. Il faut donc que ton script perl utilise net:yslog pour envoyer le résultat.
    2. Qu'un démon syslog, en occurrence syslog-ng tourne sur le collecteur
    3. Que ce démon soit paramétré pour ne recevoir que les message ne provenance de l'IP X.X.X.X sur le protocole UPD.
    4. Que si les message traverse le filtre précédant (3.) alors ils sont insérés dans une base de données
    5. Le module Syslog de Centreon affiche le contenue de la base de données.

    Pour la partie Perl, je n'ai pas développé de script qui permettent de faire ca mais je verrais en début de semaine si j'ai le temps de tester deux trois lignes de code.

    Pour la partie syslog-ng je peut t'aider. Il n'était pas nécessaire de supprimer le fichier de configuration de syslog-ng.

    syslog-ng fonctionne de la manière suivante:
    • on définit des sources d'entrées: pipe, unix-stream, internal, upd, tpc
    • on défini des filtres (optionnel) qui permettent de "filtres" les messages provenant des sources décrite.
    • on définit des destination possible: fichiers plat, pipe, ...
    • on "log", action qui met en relation une (des) source(s), des filtres (optionnel) et une (des) destination(s).


    Dans ton cas voici un exemple:
    Code:
    source s_robocopy { udp( ip(IP_MACHINE_ROBOCOPY) port(514)); };
    destination d_mysql {
          pipe("/var/log/mysql.pipe"
                  template("INSERT INTO logs
                  (host, facility, priority, level, tag, datetime, program, msg)
                  VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC',
                  '$PROGRAM', '$MSG' );\n") template-escape(no));
    };
    
    
    log {
    source(s_robocopy); 
    destination(d_mysql);
    };
    Ainsi, tous les message syslog en provenance de IP_MACHINE_ROBOCOPY envoyé par le protocole UDP sur le port 514 seront envoyés vers le pipe "/var/log/mysql.pipe".

    Il faut cependant que ce pipe existe (cf. http://syslog.modules.centreon.com/wiki/Install_fr).

    #### English (sorry but I don't speak english very well) ###

    If understood well you want operation following:

    Robocopy => send message syslog (robocopy result) to syslog server (collector) syslog => syslog server (for you syslog-ng) => insert message in "syslog" database => print result in Centreon using Syslog module.

    1. You must develop perl script using Net:yslog to eend robocopy result.
    2. to have a syslog deamon running on your collector.
    3. To configur your deamon to receive syslog message from your server with IP address X.X.X.X on UDP protocol
    4. That if the message crosses the preceding filter (3.) they are insert in a database.
    5. Syslog module print data from your database.

    For your Perl script, I never developed script script which makes it possible to do that. I will try on monday to test some sources.

    For syslog-part I can help you but it was not necessary to remove the file of configuration of syslog-ng.

    syslog-ng functions in the following way
    • you must define sources like pipe, unix-stream, internal, upd, tpc
    • you can define filters on syslog level message, description of your message (with match command), ...
    • you must define destinations: log files, pipe, ...
    • and use "log" command to link sources, filters and destinations.


    In your case, this is an example:
    Code:
    source s_robocopy { udp( ip(IP_MACHINE_ROBOCOPY) port(514)); };
    destination d_mysql {
          pipe("/var/log/mysql.pipe"
                  template("INSERT INTO logs
                  (host, facility, priority, level, tag, datetime, program, msg)
                  VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC',
                  '$PROGRAM', '$MSG' );\n") template-escape(no));
    };
    
    
    log {
    source(s_robocopy); 
    destination(d_mysql);
    };
    All syslog message from IP_MACHINE_ROBOCOPY using UPD protocol on 514 port will be sent on pipe "/var/log/mysql.pipe".

    It is necessary however that this pipe exists (http://syslog.modules.centreon.com/wiki/Install_fr) sorry but english Wiki are not available today).
    Centreon Syslog Module Manager/Developper
    Centreon E2S Module Manager/Developper
    Centreon Enterprise Server (2.x / 3.x) : Centreon Engine 1.3.x / 1.4.x, Centreon Broker 2.6.x / 2.8.x , Centreon 2.x, Centreon-Syslog 1.5.x, Centreon E2S 2.0
    Nagios 3.x et NDOutil 1.x

    Comment


    • #3
      Originally posted by AkHeNaToN View Post
      ### Français ###


      In your case, this is an example:
      Code:
      source s_robocopy { udp( ip(IP_MACHINE_ROBOCOPY) port(514)); };
      destination d_mysql {
            pipe("/var/log/mysql.pipe"
                    template("INSERT INTO logs
                    (host, facility, priority, level, tag, datetime, program, msg)
                    VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC',
                    '$PROGRAM', '$MSG' );\n") template-escape(no));
      };
      
      
      log {
      source(s_robocopy); 
      destination(d_mysql);
      };
      All syslog message from IP_MACHINE_ROBOCOPY using UPD protocol on 514 port will be sent on pipe "/var/log/mysql.pipe".

      It is necessary however that this pipe exists (http://syslog.modules.centreon.com/wiki/Install_fr) sorry but english Wiki are not available today).
      ## French below ##
      Actualy if I put the IP of the server it will work for all message I will receive. The robocopys will run from different server, so that's enough and that's working for the moment.

      I've removed the other configuration of the file because I don't need them and it easier to explain to my colleague who don't know a shit about linux.

      If you want I think I can translate quickly your tutorial.

      For the perl I think I will try to find some sources with "windows cmd" interaction. I don't know how to get the result but I think if I start the robocopy command from the perl script I will have a way to get the result at the end, or I have to send the result in a text file and get it after, I will think about that on monday =)

      Anyway, do you have any idea how can I get one full day of the logs overview in the module interface?

      ## FR ##

      En faite si j'utilise l'adress ip du server, cela me permet de capter les messages de tous mes servers sur lesquels tourne robocopy.

      J'ai supprimé le reste de la configuration dans syslog-ng parceque je n;en ai pas l'utilitée et que ca me permet d'expliquer plus facilement le fonctionnent a mon collegue qui ne connait rien a linux.

      Si tu as besoin de peux te traduire ton tutoriel. Ca devrait aller assez vite.

      Pour le script perl, je pense qu'il va falloir que je trouve quelques sources qui utilise l'interaction des command dos. Soit lancer robocopy a travers le script perl, ou le récupérer a la fin du script d'un autre maniere comme un fichier texte par exemple. Mais on verra lundi!

      Sinon tu n'a pas une petite idée sur comment je peux afficher un jour entier de log dans l'interface syslog?

      Mon premier poste est fonctionnel et c'est plus un how to bordelique (je sais tres mal expliquer et je suis tres mauvais prof et je bidouille de partout pour faire marcher les choses et comprendre) Si qqn veux l'améliorer ou faire des modifications, feel free.

      Merci encore pour l'aide que tu m'apporte AkHeNaToN ^_^

      Comment


      • #4
        Ok I have something working for the script ! There is one batch script and one perl script.


        ###script.bat###
        Code:
        @echo 
        FOR /f "tokens=1-3 delims=./- " %%F IN ("%DATE%") DO SET TD=%%F-%%G-%%H
        robocopy \\computer\folder \\computer2\folder /e /r:2 /w:2 /np /B /PURGE >C:\robocopy\backup_%TD%.txt
        if errorlevel 16 script.pl 16
        if errorlevel 8 script.pl 8
        if errorlevel 4 script.pl 4
        if errorlevel 2 script.pl 2
        if errorlevel 1 script.pl 1
        if errorlevel 0 script.pl 0
        #script.pl###
        Code:
        use feature ':5.10';
        
        my $param = shift;
        
        if ($param == 16)
        {
        use Net::Syslog;
          my $s=new Net::Syslog(Facility=>'Robocopy',Priority=>'critical');
          $s->send('***FATAL ERROR*** during the copy',Priority=>'critical',SyslogHost=>'123.123.123.123');
        }
        
        if ($param == 8)
        {
        use Net::Syslog;
          my $s=new Net::Syslog(Facility=>'Robocopy',Priority=>'high');
          $s->send('**FAILED COPIES**',Priority=>'high',SyslogHost=>'123.123.123.123');	
        }
        
        if ($param == 4)
        {
        use Net::Syslog;
          my $s=new Net::Syslog(Facility=>'Robocopy',Priority=>'debug');
          $s->send('*MISMATCHES* during the copy',Priority=>'warning',SyslogHost=>'123.123.123.123');
        }
        
        if ($param == 2)
        {
        use Net::Syslog;
          my $s=new Net::Syslog(Facility=>'Robocopy',Priority=>'debug');
          $s->send('EXTRA FILES',Priority=>'info',SyslogHost=>'123.123.123.123');
        }
        
        if ($param == 1)
        {
        use Net::Syslog;
          my $s=new Net::Syslog(Facility=>'Robocopy',Priority=>'debug');
          $s->send('Copy successful',Priority=>'info',SyslogHost=>'123.123.123.123');
        }
        
        if ($param == 0)
        {
        use Net::Syslog;
          my $s=new Net::Syslog(Facility=>'Robocopy',Priority=>'debug');
          $s->send('No change During the copy',Priority=>'info',SyslogHost=>'123.123.123.123');
        }
        Enjoy

        Edit : I've forgot, some new version of robocopy return always the errorlevel 0. Use an older version.
        Last edited by abys; 9 March 2009, 16:10. Reason: Info missing

        Comment


        • #5
          Tu arrive à avoir ton affichage dans le module Syslog dans Centreon ?
          Centreon Syslog Module Manager/Developper
          Centreon E2S Module Manager/Developper
          Centreon Enterprise Server (2.x / 3.x) : Centreon Engine 1.3.x / 1.4.x, Centreon Broker 2.6.x / 2.8.x , Centreon 2.x, Centreon-Syslog 1.5.x, Centreon E2S 2.0
          Nagios 3.x et NDOutil 1.x

          Comment


          • #6
            Yep, ca marche impec! =)

            Comment


            • #7
              Update of the scripts :

              script.bat
              Code:
              @echo 
              FOR /f "tokens=1-3 delims=./- " %%F IN ("%DATE%") DO SET TD=%%F-%%G-%%H
              robocopy \\computer\folder \\computer2\folder /e /r:2 /w:2 /np /B /PURGE >C:\robocopy\backup_%TD%.txt
              script.pl %errorlevel%
              Script.pl
              Code:
              use feature ':5.10';
              
              my $param = shift;
              
              if ($param == 16)
              {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'crit');
                $s->send('***FATAL ERROR*** during the copy',Priority=>'crit',SyslogHost=>'123.123.123.123');
              } elsif ($param == 15) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'crit');
                $s->send('**FAIL MISM XTRA COPY** during the copy',Priority=>'crit',SyslogHost=>'123.123.123.123');
              } elsif ($param == 14) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'crit');
                $s->send('**FAIL MISM XTRA** during the copy',Priority=>'crit',SyslogHost=>'123.123.123.123');
              } elsif ($param == 13) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'crit');
                $s->send('**FAIL MISM COPY** during the copy',Priority=>'crit',SyslogHost=>'123.123.123.123');
              } elsif ($param == 12) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'crit');
                $s->send('**FAIL MISM** during the copy',Priority=>'crit',SyslogHost=>'123.123.123.123');
              } elsif ($param == 11) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'crit');
                $s->send('**FAIL XTRA COPY** during the copy',Priority=>'crit',SyslogHost=>'123.123.123.123');
              } elsif ($param == 10) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'crit');
                $s->send('**FAIL XTRA** during the copy',Priority=>'crit',SyslogHost=>'123.123.123.123');
              } elsif ($param == 9) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'crit');
                $s->send('**FAIL COPY** during the copy',Priority=>'crit',SyslogHost=>'123.123.123.123');
              } elsif ($param == 8) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'err');
                $s->send('***FAIL COPIES***',Priority=>'err',SyslogHost=>'123.123.123.123');	
              } elsif ($param == 7) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'err');
                $s->send('**MISM XTRA COPY**',Priority=>'err',SyslogHost=>'123.123.123.123');	
              } elsif ($param == 6) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'err');
                $s->send('**MISM XTRA**',Priority=>'err',SyslogHost=>'123.123.123.123');	
              } elsif ($param == 5) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'err');
                $s->send('**MISM COPY**',Priority=>'err',SyslogHost=>'123.123.123.123');	
              } elsif ($param == 4) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'warning');
                $s->send('*MISMATCHES* during the copy',Priority=>'warning',SyslogHost=>'123.123.123.123');
              } elsif ($param == 3) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'warning');
                $s->send('*XTRA COPY* during the copy',Priority=>'warning',SyslogHost=>'123.123.123.123');
              } elsif ($param == 2) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'info');
                $s->send('EXTRA FILES',Priority=>'info',SyslogHost=>'123.123.123.123');
              } elsif ($param == 1) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'info');
                $s->send('Copy successful',Priority=>'info',SyslogHost=>'123.123.123.123');
              } elsif ($param == 0) {
              use Net::Syslog;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'info');
                $s->send('No change During the copy',Priority=>'info',SyslogHost=>'123.123.123.123');
              }  else {
              use Net::Syslog;
                my $result='Unknown error during the copy. Error number = '. + $param;
                my $s=new Net::Syslog(Facility=>'syslog',Priority=>'debug');
                $s->send($result,Priority=>'debug',SyslogHost=>'123.123.123.123');
              }

              For the moment it's working very good with my test machine on a little robocopy script. On a server I have some difficulties to have something in the result, still working on it. Maybe more than 16 errorlvl.

              See ya
              Last edited by abys; 12 March 2009, 14:01. Reason: code update

              Comment


              • #8
                Ok! I totally let it down my gas factory for the new system of the syslog master! => http://forum.centreon.com/showthread.php?t=7637

                To log my robocopy result I directly use the windows event log! Here is my robocopy script :

                Code:
                FOR /f "tokens=1-3 delims=./- " %%F IN ("%DATE%") DO SET TD=%%F-%%G-%%H
                
                c:\robocopy.exe \\serverA\e$ \\serverB\e$ arguments >C:\robocopy\Backup_log_%TD%.txt
                if errorlevel 16 eventcreate /L application /T ERROR/SO nameofthesourceyouwant /ID 1 /D "***FATAL ERROR***" & goto end
                if errorlevel 15 eventcreate /L application /T ERROR /SO nameofthesourceyouwant /ID 1 /D "FAIL MISM XTRA COPY" & goto end
                if errorlevel 14 eventcreate /L application /T ERROR /SO nameofthesourceyouwant /ID 1 /D "FAIL MISM XTRA" & goto end
                if errorlevel 13 eventcreate /L application /T ERROR /SO nameofthesourceyouwant /ID 1 /D "FAIL MISM COPY" & goto end
                if errorlevel 12 eventcreate /L application /T ERROR /SO nameofthesourceyouwant /ID 1 /D "FAIL MISM" & goto end
                if errorlevel 11 eventcreate /L application /T ERROR /SO nameofthesourceyouwant /ID 1 /D "FAIL XTRA COPY" & goto end
                if errorlevel 10 eventcreate /L application /T ERROR /SO nameofthesourceyouwant /ID 1 /D "FAIL XTRA" & goto end
                if errorlevel 9 eventcreate /L application /T ERROR /SO nameofthesourceyouwant /ID 1 /D "FAIL COPY" & goto end
                if errorlevel 8 eventcreate /L application /T ERROR /SO nameofthesourceyouwant /ID 1 /D "FAIL" & goto end
                if errorlevel 7 eventcreate /L application /T WARNING /SO nameofthesourceyouwant /ID 1 /D "MISM XTRA COPY" & goto end
                if errorlevel 6 eventcreate /L application /T WARNING /SO nameofthesourceyouwant /ID 1 /D "MISM XTRA" & goto end
                if errorlevel 5 eventcreate /L application /T WARNING /SO nameofthesourceyouwant /ID 1 /D "MISM COPY" & goto end
                if errorlevel 4 eventcreate /L application /T WARNING /SO nameofthesourceyouwant /ID 1 /D "MISM" & goto end
                if errorlevel 3 eventcreate /L application /T INFORMATION /SO nameofthesourceyouwant /ID 1 /D "XTRA COPY" & goto end
                if errorlevel 2 eventcreate /L application /T INFORMATION /SO nameofthesourceyouwant /ID 1 /D "XTRA" & goto end
                if errorlevel 1 eventcreate /L application /T INFORMATION /SO nameofthesourceyouwant /ID 1 /D "COPY" & goto end
                if errorlevel 0 eventcreate /L application /T INFORMATION /SO nameofthesourceyouwant /ID 1 /D "#no change#" & goto end
                :end
                That's working much better than my previous system! Also for peoples using Brightstor arcserv backup tape you can configure an alert in the event log to send it on the syslog server.

                Comment


                • #9
                  More about........Robocopy

                  Comment

                  Working...
                  X