x
login Signup

Output very large log files into a text file

I am trying to capture very large log files from findobjects api call. They are so large that I get "Out of Memory" when I run a findobjects api call on 50,000 maxIds. I need to be able to just output to a text file for optimization.

 use strict;          
  use warnings;     
  use ElectricCommander;
  $| = 1;
  
  my $ec = new ElectricCommander();
  
  my $ec_jobSteps = $ec->findObjects("jobStep", {
     filter => [{propertyName => "lastModifiedBy",
                 operator     => "notEqual",
                 operand1 => "admin"},
                 {propertyName => "resourceName",
                 operator => "contains",
                 operand1 => "Synergy"},
                 {propertyName => "start",
                 operator => "between",
                 operand1 => "2017-01-30T23:59:59.999Z",
                 operand2 => "2017-02-28T23:59:59.999Z"},
                 {propertyName => "runTime",
                 operator => "greaterThan",
                 operand1 => "0"
                 }],
     maxIds => 50000,
     sort => [{propertyName => "start",
               order => "ascending"
               }]
     });
     
     #print $ec_jobSteps->findnodes_as_string("/");
     
     my @jobSteps = $ec_jobSteps->findnodes("//object/jobStep");
     foreach my $p (@jobSteps)
     {
         my $o = $p->findvalue("owner")->value;
         my $pn = $p->findvalue("procedureName")->value;
         my $sn = $p->findvalue("stepName")->value;
         my $rt = $p->findvalue("runTime")->value;
         my $rn = $p->findvalue("resourceName")->value;
         my $s = $p->findvalue("start")->value;
         print "$o\t$pn\t$sn\t$rt\t$rn\t$s\n";
     }

How do I add the ability to print this output to a text file?

avatar image By luckyirishlife 59 asked Jun 13, 2017 at 09:53 PM
more ▼
(comments are locked)
10|750 characters needed characters left

4 answers: sort voted first

The original problem you reported about running out of memory is because the findObjects() call returns a lot of data for the job steps, so the GetObjects() calls allow you to return just a few of them at a time.

The other problems are user errors. The Perl warning about allObjectList is a syntax issue. The problems the program had with opening the output file are probably because there isn't a data/sfs/ECMigration/ directory in the EC workspace for it to put the report in.

Here is another try. The program asks the findObjects() call not to return any job step information, besides the list. Then it calls getObjects() for each 10 job steps in the list. You might be able to ask for more job steps at a time, depending on how big they are, but that is pretty easy to figure out. (There are probably better ways of controlling the loop, but this works.)

 use strict;
 use warnings;
 use ElectricCommander;
 $| = 1;
 
 my $ec = new ElectricCommander();
 #my $migrationdir = 'data/sfs/ECMigration/report.txt';
 my $migrationdir = './report.txt';
 open my $fh, '>', $migrationdir or die "Could not open file '$migrationdir' $!";
 
 # Get a list of the job steps, but no info on them yet
 my $ec_jobSteps = $ec->findObjects("jobStep", {
      filter => [{propertyName => "lastModifiedBy",
                  operator     => "notEqual",
                  operand1 => "admin"},
                  {propertyName => "resourceName",
                  operator => "contains",
                  operand1 => "Synergy"},
                  {propertyName => "start",
                  operator => "between",
                  operand1 => "2017-01-30T23:59:59.999Z",
                  operand2 => "2017-02-28T23:59:59.999Z"},
                  {propertyName => "runTime",
                  operator => "greaterThan",
                  operand1 => "0"
                  }],
      maxIds => 50000,
      numObjects => 0, # do not return any objects here
      sort => [{propertyName => "start",
                order => "ascending"
                }]
      });
 
 #print $ec_jobSteps->findnodes_as_string("/"); ###
 
 # Gather the job steps to be returned
 my @allObjectsList;
 my $nodeset = $ec_jobSteps->find('//response/objectId');
 foreach my $node ($nodeset->get_nodelist)
 {
     my $objectId = $node->string_value();
     push (@allObjectsList, $objectId);
 }
 my $num = $#allObjectsList;
 #print $num+1, "\n"; ###
 
 # Retrieve the job step information, a few at a time, and write the report
 my $first = 0;
 my $last;
 while (1)
 {
     # get the objects 10 at a time
     $last = $first + 9;
     if($last > $num) { $last = $num; }
 #print "$first, $last\n"; ###
     my @objectList = @allObjectsList[$first...$last];
     my $ec_objects = $ec->getObjects({objectId => \@objectList});
 
 #print $ec_objects->findnodes_as_string("/"); ###
 
     my @jobSteps = $ec_objects->findnodes('//object/jobStep');
     foreach my $js (@jobSteps)
     {
         my $o = $js->findvalue("owner")->value;
         my $pn = $js->findvalue("procedureName")->value;
         my $sn = $js->findvalue("stepName")->value;
         my $rt = $js->findvalue("runTime")->value;
         my $rn = $js->findvalue("resourceName")->value;
         my $s = $js->findvalue("start")->value;
         print $fh "$o\t$pn\t$sn\t$rt\t$rn\t$s\n";
     }
     if($last == $num) { last; }
     $first = $last + 1;
 }
 
 close $fh;
 print "Finished outputting report to log file.\n";

avatar image By sbarry 1.5k answered Jun 15, 2017 at 05:18 PM
more ▼
(comments are locked)
10|750 characters needed characters left

Rather than saving your output to a text file, see the EC help on the maxIds and numObjects parameters to the findObjects() call, and using that with the getObjects() call. There should be a nice example of using that in the help. That way, you can get and process the interesting data a little bit at a time, and avoid running out of memory.

avatar image By sbarry 1.5k answered Jun 13, 2017 at 11:43 PM
more ▼
(comments are locked)
avatar image luckyirishlife Jun 14, 2017 at 04:58 PM
avatar image lrochette Jun 14, 2017 at 06:24 PM

Yes this is the one. Or Check Electric Flow API documentation if you have a more recent version than 5.3: http://docs.electric-cloud.com/eflow_doc/7_3/API/HTML/APIflowHTML.htm#apimisc.htm#getObjects

10|750 characters needed characters left

I think @sbarry is the best way to do it but in order to print to a file it's just pure Perl:

 open(my $fh, '>', 'report.txt');
 Loop
   print $fh "$o\t$pn\t$sn\t$rt\t$rn\t$s\n";
 End Loop
 close $fh;

This will create a file in your current working directory (by default in the workspace job directory) You can even link it to to your job detail page for easy consulting.

avatar image By lrochette 5.9k answered Jun 14, 2017 at 04:18 AM
more ▼
(comments are locked)
10|750 characters needed characters left

Ok, here is my revised code. But its not clear to me from the example how to loop through and output the parameters I want for each of the 10 objects I have created. Maybe something like this?

 use strict;          
 use warnings;     
 use ElectricCommander;
 $| = 1;
  
 my $ec = new ElectricCommander();
 my $migrationdir = 'data/sfs/ECMigration/report.txt';
 open my $fh, '>', $migrationdir or die "Could not open file '$migrationdir' $!";
  
 my $ec_jobSteps = $ec->findObjects("jobStep", {
     filter => [{propertyName => "lastModifiedBy",
                 operator     => "notEqual",
                 operand1 => "admin"},
                 {propertyName => "resourceName",
                 operator => "contains",
                 operand1 => "Synergy"},
                 {propertyName => "start",
                 operator => "between",
                 operand1 => "2017-01-30T23:59:59.999Z",
                 operand2 => "2017-02-28T23:59:59.999Z"},
                 {propertyName => "runTime",
                 operator => "greaterThan",
                 operand1 => "0"
                 }],
     maxIds => 50000,
     numObjects => 10,
     sort => [{propertyName => "start",
               order => "ascending"
               }]
     });
     
 my @allObjectsList;
 my $ns = $ec_jobSteps->find('//response/objectId');
 
 foreach my $node ($ns->get_nodelist)
 {
     my $objectId = $node-> string_value();
     push (@allObjectsList, $objectId);
 }
     
 #print $ec_jobSteps->findnodes_as_string("/");
 for (my $i=0; $i <= 10; $i++)
 {
     my @objectList = @allObjectsList[$i];
         
     $ec_jobSteps = $ec->getObjects({objectId=>\@objectList});
     my @jobSteps = $ec_jobSteps->findnodes("//object/jobStep");
     foreach my $p (@jobSteps)
     {
         my $o = $p->findvalue("owner")->value;
         my $pn = $p->findvalue("procedureName")->value;
         my $sn = $p->findvalue("stepName")->value;
         my $rt = $p->findvalue("runTime")->value;
         my $rn = $p->findvalue("resourceName")->value;
         my $s = $p->findvalue("start")->value;
         print $fh "$o\t$pn\t$sn\t$rt\t$rn\t$s\n";
     }
 }
         
 close $fh;
 print "Finished outputting report to log file.\n";

I get the following errors:

Scalar value @allObjectsList[$i] better written as $allObjectsList[$i] at /tmp/ecmdrAgent/agent.TRKWVC8BSNCY83XR.run-1870429-12901318.cmd line 44. Could not open file 'data/sfs/ECMigration/report.txt' No such file or directory at /tmp/ecmdrAgent/agent.TRKWVC8BSNCY83XR.run-1870429-12901318.cmd line 8.

avatar image By luckyirishlife 59 answered Jun 14, 2017 at 06:39 PM
more ▼
(comments are locked)
10|750 characters needed characters left
Your answer
toggle preview:

Up to 8 attachments (including images) can be used with a maximum of 524.3 kB each and 1.0 MB total.

Follow this question


Topics:

x1004
x23
x13
x12

asked: Jun 13, 2017 at 09:53 PM

Seen: 98 times

Last Updated: Jun 15, 2017 at 05:18 PM

Related Questions