Microservices, Monitoring, Infrastructure as Code & Test Automation

An interesting option I just came accross while trying to FTP a complete site down to my linux clinet from the command line

Instead of using the ftp client or mget etc

you can use

wget -r ftp://username:password@www.yourdomain.com/

But I think this may only recurse directories down to a level of 5

So using the switch -l 0

This solves that problem

wget -r  -l 0 ftp://username:password@www.yourdomain.com/

Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.

1. Install Java SDK if not already
java -version
In my case java version "1.6.0_22" was available
So
whereis javac

If you don't have a recent java sdk installed download jdk and install
wget http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-i586-rpm.bin

Make sure to note the location where you install for use later in setting JAVA_HOME in my case JAVA_HOME was set to /usr

2. Do some basic tasks to make life easy

mkdir /hadoop
useradd hadoop
passwd hadoop
groupadd hadoop

3. Get stable release of hadoop, install in to /hadoop/hadoop
cd /hadoop
wget http://ftp.heanet.ie/mirrors/www.apache.org/dist/hadoop/common/stable/hadoop-1.0.3.tar.gz

tar -xvf  hadoop-1.0.3.tar.gz

mv hadoop-1.0.3 hadoop
export HADOOP_INSTALL=/hadoop/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
Set these up in your login profile to ensure they are set each time you login.

4.Se…

Some useful grep / sed / cut commands to help you monitor your log files and detect possible attacks to your system.

Mainly based on CentOS 5.4 but can be tailored for other distros

1. Search through /var/log/secure and determine what IPs have the highest failed attempts to your FTP daemon in this instance vsftpd

grep "authentication failure" secure.* | grep vsftpd   | sed s/::ffff://g  | cut -d' ' -f14 |sort|uniq -c|sort -nr|more


2. Search through /var/log/secure and determine what IPs have the highest failed attempts to your SSHD daemon
grep "refused connect from" secure.*  | grep "sshd" | sed s/::ffff://g  | cut -d' ' -f9 |sort|uniq -c|sort -nr|more