Perl One-Liner Multithreaded Ping Scanner

Just another one-liner today. This launches a thread for each address you want to scan in the range you provide.

perl -e 'use threads; ($n,@r) = @ARGV; map { $t{$_} = new threads sub { $i = shift; print "$_ up\n" if `ping -c 1 $n$i` =~ /1 received/; }, $_;} $r[0]..$r[1]; map $t{$_}->join, keys %t;' 192.168.50. 1 254

Get a List of IPs that have Been Refused

Just a one-liner today.

grep refused /var/log/secure | perl -ne '@l = $_; map { /from ::ffff:((?:\d{1,3}\.){3}\d{1,3}\b)/; $h{$1}++; } @l; END {map { print "$_ = $h{$_}\n" if $h{$_} > 10;} keys %h; }'

This will print a list of all IPs that have received a 'refused connection' message more than 10 times and print how many times each has been refused.

Forking in Perl

Yesterday I posted how to thread in perl. Today I figured I'd explain forking. There are a few reasons you might choose to fork instead of thread. One of them is that threading is a compile time option for perl that isn't enabled in every installation. Another is that not every module is thread safe. Most of the time you can just unload any modules that are not before you start threading, but if you need the module inside your thread, you'll need to switch to forking. Lastly forking is more effecient than threading in some cases. I'll actually explain this more in detail in a later post.

Here I've written a sample script that is similar to my threading function.


######################################
# http://greg-techblog.blogspot.com
# Fork child processes for a list of hosts running a function
# you pass to it. Passes host to function as argument.
# args:
# needs numbers of child to fork at a time
# ref to sub to run
# ref array of hosts or instances (will be passed to sub as arg)
# returns:
# a hash of returns from sub keyed by host or instance name
######################################
use strict;
use warnings;
my $debug;

sub fork_run {
use POSIX qw( WNOHANG );
my ($children,$func_to_fork, $hosts) = @_;
my %stdout;
my $n = 0;

# This makes sure we don't start more than $threads threads at a time
my $m = $children-1 > $#{$hosts} ? $#{$hosts} : $children-1;

while (1) {
last if $n >= $#{$hosts}+1;
foreach my $host (@{$hosts}[$n..$m]) {
print "Forking for $host $n-$m\n" if $debug;
local *FROM_CHILD;
pipe(FROM_CHILD,TO_PARENT);
my $pid = fork();
if ( not defined $pid) {
# Something is wrong
print "resources not available.\n";
}
elsif ($pid == 0 ) {
# This is the child
# Turn on autoflush
$| = 1;
close(FROM_CHILD);
select(TO_PARENT);
$func_to_fork->($host);
close(TO_PARENT);
exit;
}
else {
# This is the parent
close(TO_PARENT);
$stdout{$host} = ;
close FROM_CHILD;
}
}

# wait for some children to finish
my $kid;
do {
$kid = waitpid(-1, WNOHANG);
} while $kid > 0;

# work out the next range of instances to work on
$n = $m == 0 ? 1 : $m+1;
$m = $n+$children-1 < $#{$hosts} ? $n+$children-1 : $#{$hosts};
print "$n-$m\n\n" if $debug;
}
return %stdout;
}


To use this, simply do something like this;

use Data::Dumper;
my @hosts = qw{ host1 host2 host3 host4 host5 };
sub myFunc { my $hostname = shift; print TO_PARENT `ping -c 1 $hostname` ; return 1; }

# Launch two threads at a time that ping hosts
my %stdout = fork_run( 2, \&myFunc, \@hosts);
map { print $stdout{$_}; } keys %stdout;

Threading in Perl

Threading in Perl may seem daunting at first, but it's actually easier than it looks. In some ways, threading is more straight forward than forking. Communication between the threads and a parent process is much more straight forward in my opinion than communication between child and parent processes.

Here I've written a little sample function that will take a number of threads to launch at a time, a reference to a function and an array of hostnames. Then it will launch one thread for each hostname passing the hostname to the function, until the limit has been reached. It then waits for each thread to become joinable (finishes) and then launches more threads till the limit is reached again. When all threads are finished, it returns the output form all threads as a hash keyed by the hostnames.


######################################
# http://greg-techblog.blogspot.com
# Spawns threads for a list of hosts running a function
# you pass to it. Passes host to function as argument.
# args:
# needs numbers of threads to launch at a time
# ref to sub to run
# ref array of hosts or instances (will be passed to sub as arg)
# returns:
# a hash of returns from sub keyed by host or instance name
######################################
use threads;
my $debug;
sub run_threaded {
my ($threads,$func_to_thread, $hosts) = @_;
my (%stdout, %threads);
my $n = 0;

# This makes sure we don't start more than $threads threads at a time
my $m = $threads-1 > $#{$hosts} ? $#{$hosts} : $threads-1;
while (1) {
last if $n >= $#{$hosts}+1;
foreach my $host (@{$hosts}[$n..$m]) {
print "Launching thread for $host $n-$m\n" if $debug;
unless ( $threads{$host} = new threads $func_to_thread, $host) {
print "Error on $host\n";
}
}
map { $stdout{$_} = $threads{$_}->join if $threads{$_}; } @{$hosts}[$n..$m];
# work out the next range of instances to work on
$n = $m == 0 ? 1 : $m+1;
$m = $n+$threads-1 < $#{$hosts} ? $n+$threads-1 : $#{$hosts};
print "$n-$m\n\n" if $debug;
}
return %stdout;
}


To use this, simply do something like this;

use Data::Dumper;
my @hosts = qw{ host1 host2 host3 host4 };
sub myFunc { my $hostname = shift; return `ping $hostname` ; }

# Launch two threads at a time that ping hosts
my %stdout = run_threaded( 2, \&myFunc, \@hosts);
print Dumper \%stdout;

Kill a Ton of Queries in MySQL

Every so often I'll have a misbehaving query that read locks a table that is essential for every other query in our application. Obviously this is a bug in our application that should have never made it past the Q/A process, but lets get back to the real world where programmers make these mistakes and Q/A isn't perfect. When this happens, a ton of queries backup behind the query that read locked the table. At this point, simply killing the primary query isn't going to fix the problem, because the load generated by all the queries waiting for the read lock to end will kill the server. There is only two solutions at this point, restart MySQL or kill all the stuck queries. Killing all the queries is really the best solution for our application, but the number is typically daunting. Most of the time I've got as many queries as whatever my connection limit is set to. So I came up with this little one-liner to get a list of all the processes for a user on the server, and kill them.

mysql -u username -ppassword -e "show processlist" | sed '1d' | grep mysql_username | awk '{print "kill ", $1, ";"}' | xargs -i mysql -u username -ppassword -e "{}"

I'm sure someone has a better way to do this, but this works pretty awesomely.

Inventory Your Machines with a One-liner

The command dmidecode dumps the DMI table from your machines BIOS in human readable text. Depending on your machine, this should contain some really useful info. Basically a total hardware profile on most machines, including info like service tags and model numbers. I recently needed to quickly inventory a bunch of Dell servers and came up with this one-liner that will production a CSV line of text for the machine it's run on. Combine these lines of text into one file and import into Excel, inventory done.

Format is:
'Hostname','Distro-Release Version','UUID','Service Tag','Model','Processor Version','Memory'

echo `hostname` , `cat /etc/*-release` , `dmidecode |egrep UUID |sed "s/\s\+UUID: //"` , `dmidecode -s system-serial-number` , `dmidecode -s system-product-name` , `dmidecode -s processor-version` , `dmesg |egrep Mem |awk '{ print $2; }'`

Now of course you'll need to modify this command for your own hardware, this won't even work for every Dell. But, the output of dmidecode is readable enough to just scan through it and figure out what you need. Also look in /proc/ for other info you might want to add to your inventory.

Add Up a Column of Data with Awk

When you pipe a line of text to awk, every "word" (string of non-white space) shows up in $n, where n is equal to the order for which it appears in the line of text. For example;

>echo this is a test | awk '{print $2};'
prints
is

If you have a column of numbers, say from the output of a command or a log file, you can do math with these numbers like this

ps aux | awk '{ t =+ $6}; END { print t};'

This will give you the total for the sixth column of the output from the command ps aux. In another example, lets say I wanted to see the total number of clients with active and inactive connections on my LVS server. The command to see this info is


[root@lvs~]#ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP www:https sed
-> webapp06:https Route 110 54 86
-> webapp01:https Route 70 36 46
-> webapp04:https Route 300 148 207
-> webapp03:https Route 300 146 307
-> webapp02:https Route 120 58 77
-> webapp05:https Route 300 142 183

That's nice, but you'll notice I don't get totals. The command doesn't have any syntax to give me totals either. So we'll let awk take care of it. First we need to grep just the lines that we want to add up. In this case they all have the text 'webapp0' in them, and that string of text doesn't appear on any lines we don't want to add up.

[root@lvs~]#ipvsadm -L |grep webapp0
-> webapp06:https Route 110 72 80
-> webapp01:https Route 70 45 53
-> webapp04:https Route 300 198 267
-> webapp03:https Route 300 196 268
-> webapp02:https Route 120 79 105
-> webapp05:https Route 300 199 240

Now the two columns we'd like to add up are the 5th and the 6th.

[root@lvs~]#ipvsadm -L |grep webapp0|awk '{s += $5; i += $6} END { print s,i; };'
846 1066

Now lets say instead of wanting a total, we wanted an average? 'NR' is equal to the number of rows we sent awk.

[root@lvs~]#ipvsadm -L |grep webapp0|awk '{s += $5; i += $6} END { print s/NR,i/NR; };'
68.5833 88.4167

MaxClients in Apache with Preforking MPM

The advice for figuring out what your MaxClients setting should be in Apache with the Preforking MPM is pretty straight forward. First figure out the average amount of RAM each Apache process uses. Then divide the amount of RAM your system has, minus a little for other processes, by the average amount of RAM the average Apache process uses. In CentOS the following command should give you that number.

ps -ylC httpd |awk "{ s += \$8 } END { m = `cat /proc/meminfo|grep MemTotal|awk '{print $2}'`; printf \"MaxClients: %i\\n\", (m-(m*.15))/(s/NR) }"