10 Most Recently Updated Pages

use Python for http POST requests

Mon, 04 May 2009 16:40:46 +0200

A little script that shows how to use Python for HTTP Requests

import string,urllib
def format(items):
     list = []
     for k, v in items:
        list.append("%s=%s" % (k, urllib.quote_plus(v)))
     return string.join(list, '&')
data = { 'hidd':'en', 'email':'you at yourmail.com',
          'POST':'post.gif' }
d = format(data.items())
try:
     sock = urlopen(YourURL, d)
     for l in sock.readlines():
         print l
     sock.close()
     finally:
         urlcleanup()

Server Configuration

Mon, 04 May 2009 17:10:23 +0200

Usefull stuff & server configurations

Keep SSH connection open

Mon, 04 May 2009 17:10:23 +0200

Many home and office routers kill "idle" connections after a certain length of time, forcing you to log in again. From an ISP's point of view this is understandable, but imho it's just annoying to loose a connection when you just went for a coffee and forget to be back in time.
A one-line addition to the end of the client's SSH configuration file (found at /etc/ssh/ssh_config in many systems) can fix this:

ServerAliveInterval 180

That should send a little ping out every three minutes to ensure the connection is kept alive. This tip should work on most OpenSSH servers that allows access to its sshd_config file. Just make sure you close your connection if you really don't use it for some time.

LXer WebDevelopment

Mon, 04 May 2009 17:07:39 +0200

This website is currently under construction and will be ready soon.

For more information, email us at: info@lxer.nl

ICT-security

Mon, 04 May 2009 17:08:38 +0200

Tech security articles

Fail2ban - Prevent Bruteforce Attacks

Mon, 04 May 2009 17:08:38 +0200

Howto use Fail2Ban to lock out evil Haxxors after they fail trying to log in to your systems.

(ssh, apache, vsftp, proftp, wuftp, postfix, couriersmtp, sasl)

general info:

install (Debian):
apt-get install fail.ban

configuration files:
/etc/fail2ban/fail2ban.conf
/etc/fail2ban/jail.conf

fail2ban.conf
Default seetings seem to be ok.
jail.conf
In this file you can set
#maxretry change this to what you think is neccesary
#bantime #seconds for someone to be banned (default = 600)
-enable the deamons you want to monitor
-set the email address

If you want to do some extra configuration
(like set different logfiles or change the ban-action, then this can be done in this file (jail.conf). For standard use, these settings seem to be ok.

filter.d
jail.conf contains references to patterns in the folder filter.d
New configurations can be added here.

example conf.:

# "ignoreip" can be an IP address, a CIDR mask or a DNS host
ignoreip = 127.0.0.1 192.168.0.99
bantime  = 600
maxretry = 3

# "backend" specifies the backend used to get files modification. Available
# options are "gamin", "polling" and "auto".
# yoh: For some reason Debian shipped python-gamin didn't work as expected
#      This issue left ToDo, so polling is default backend for now
backend = polling

#
# Destination email address used solely for the interpolations in
# jail.{conf,local} configuration files.
destemail = root@localhost

# Default action to take: ban only
action = iptables[name=%(__name__)s, port=%(port)s]

[ssh]

enabled = true
port    = ssh
filter  = sshd
logpath  = /var/log/auth.log
maxretry = 5

[apache]

enabled = true
port    = http
filter  = apache-auth
logpath = /var/log/apache*/*error.log
maxretry = 5

[apache-noscript]

enabled = false
port    = http
filter  = apache-noscript
logpath = /var/log/apache*/*error.log
maxretry = 5

# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.
[apache-badbots]

enabled  = false
filter   = apache-badbots
action   = iptables-multiport[name=BadBots, port="http,https"]
sendmail-buffered[name=BadBots, lines=5, dest=you@mail.com]
logpath  = /var/www/*/logs/access_log
bantime  = 172800
maxretry = 1

[apache-tcpwrapper]

enabled  = false
filter   = apache-auth
action   = hostsdeny
logpath  = /var/log/apache*/*access.log
/home/www/myhomepage/access.log
maxretry = 6

# Use shorewall instead of iptables.

[apache-shorewall]

enabled  = false
filter   = apache-noscript
action   = shorewall
sendmail[name=Postfix, dest=you@mail.com]
logpath  = /var/log/apache2/error_log

# This jail uses ipfw, the standard firewall on FreeBSD. The "ignoreip"
# option is overridden in this jail. Moreover, the action "mail-whois" defines
# the variable "name" which contains a comma using "". The characters '' are
# valid too.
[ssh-ipfw]

enabled  = false
filter   = sshd
action   = ipfw[localhost=192.168.0.1]
sendmail-whois[name="SSH,IPFW", dest=you@mail.com]
logpath  = /var/log/auth.log
ignoreip = 168.192.0.1

# These jails block attacks against named (bind9). By default, logging is off
# with bind9 installation. You will need something like this:
#
# logging {
#     channel lame-servers_file {
#         file "/var/log/named/lame-servers.log" versions 3 size 30m;
#         severity dynamic;
#         print-time yes;
#     };
#     category lame-servers {
#         lame-servers_file;
#     };
# }
#
# in your named.conf to provide proper logging.
# This jail blocks UDP traffic for DNS requests.

[named-refused-udp]

enabled  = false
filter   = named-refused
action   = iptables-multiport[name=Named, port="domain,953", protocol=udp]
sendmail-whois[name=Named, dest=you@mail.com]
logpath  = /var/log/named/lame-servers.log
ignoreip = 168.192.0.1

[vsftpd]

enabled  = false
port     = ftp
filter   = vsftpd
logpath  = /var/log/auth.log
maxretry = 5

[proftpd]

enabled  = true
port     = ftp
filter   = proftpd
logpath  = /var/log/auth.log
failregex = proftpd: \(pam_unix\) authentication failure; .* rhost=<HOST>
maxretry = 5

[wuftpd]

enabled  = false
port     = ftp
filter   = wuftpd
logpath  = /var/log/auth.log
maxretry = 5

[postfix]

enabled  = false
port     = smtp
filter   = postfix
logpath  = /var/log/mail.log
maxretry = 5

[courierpop3]

enabled  = true
port     = pop3
filter   = courierlogin
failregex = courierpop3login: LOGIN FAILED.*ip=\[.*:<HOST>\]
logpath  = /var/log/mail.log
maxretry = 5

[courierimap]

enabled  = true
port     = imap2
filter   = courierlogin
failregex = imapd: LOGIN FAILED.*ip=\[.*:<HOST>\]
logpath  = /var/log/mail.log
maxretry = 5

[sasl]

enabled  = true
port     = smtp
filter   = sasl
failregex = warning: [-._\w]+\[<HOST>\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed
logpath  = /var/log/mail.log
maxretry = 5

Apache Analyser

Mon, 04 May 2009 17:01:46 +0200

Getting usefull statistics from your Apache logs

If you're running Apache or a similar web server that uses the Common Log Format, there's quite a bit of quick statistical analysis that can be done with a shell script. The standard configuration for a server has an access_log and error_log written for the site; even ISPs make these raw data files available to customers, but if you've got your own server, you should definitely have and be archiving this valuable information.

This bash script returns valuable statistics about your website:

#!/bin/sh

# webaccess - analyze an Apache-format access_log file, extracting
# useful and interesting statistics

bytes_in_gb=1048576
scriptbc="$HOME/bin/scriptbc"
nicenumber="$HOME/bin/nicenumber"
host="intuitive.com"

if [ $# -eq 0 -o ! -f "$1" ] ; then
echo "Usage: $(basename $0) logfile" >&2
exit 1
fi

firstdate="$(head -1 "$1" | awk '{print $4}' | sed 's/\[//')"
lastdate="$(tail -1 "$1" | awk '{print $4}' | sed 's/\[//')"

echo "Results of analyzing log file $1"
echo ""
echo " Start date: $(echo $firstdate|sed 's/:/ at /')"
echo " End date: $(echo $lastdate|sed 's/:/ at /')"

hits="$(wc -l < "$1" | sed 's/[^[:digit:]]//g')"

echo " Hits: $($nicenumber $hits) (total accesses)"

pages="$(grep -ivE '(.txt|.gif|.jpg|.png)' "$1" | wc -l | sed 's/[^[:digit:]]//g')"

echo " Pageviews: $($nicenumber $pages) (hits minus graphics)"

totalbytes="$(awk '{sum+=$10} END {print sum}' "$1")"

echo -n " Transferred: $($nicenumber $totalbytes) bytes "

if [ $totalbytes -gt $bytes_in_gb ] ; then
echo "($($scriptbc $totalbytes / $bytes_in_gb) GB)"
elif [ $totalbytes -gt 1024 ] ; then
echo "($($scriptbc $totalbytes / 1024) MB)"
else
echo ""
fi

# now let's scrape the log file for some useful data:

echo ""
echo "The ten most popular pages were:"

awk '{print $7}' "$1" | grep -ivE '(.gif|.jpg|.png)' | \
sed 's/\/$//g' | sort | \
uniq -c | sort -rn | head -10

echo ""

echo "The ten most common referrer URLs were:"

awk '{print $11}' "$1" | \
grep -vE "(^"-"$|/www.$host|/$host)" | \
sort | uniq -c | sort -rn | head -10

echo ""
exit 0

Retrieve links and emailaddresses from some webpage

Mon, 04 May 2009 17:00:42 +0200

This script started as a Bash coding exercise but turned out to be a very useful way of getting data from websites, and it contains many useful little coding tricks.

#!/bin/sh

##############################################
NAME_="webextract"
PURPOSE_="extract links and/or email-addresses from a webpage"
SYNOPSIS_="$NAME_ [-e  ] [-l  ] "
OPTIONS_="
   -e     extract all emailaddresses from a url
   -l     extract all links from url
   -h          show help (this)"
REQUIRES_="Curl"
VERSION_="0.9"
PUROSE_=""
#Created by Lx
#licence : GPL3
#################################################

usage () {
echo >&2 "
\033[1m$NAME_ $VERSION_ \033[0m - $PURPOSE_
Usage: $SYNOPSIS_
Options: $OPTIONS_
"
    exit 1
}

# tmp file set up
tmp_1=/tmp/tmp.${RANDOM}$$

# signal trapping and tmp file removal
trap 'rm -f $tmp_1 >/dev/null 2>&1' 0
trap "exit 1" 1 2 3 15

extract () {
 while getopts ":e:l:h:" optname
   do
      case "$optname" in
         "e")
         curl -s -S $OPTARG |{
         tr ',;<>()"\47 ' '[\n*]' | sed -n -e 's/mailto://gI' -e '/@/p'  > $tmp_1
         cat $tmp_1
         }
            ;;
         "l")
         curl -s -S $OPTARG |{
         tr '<>"\47 ' '[\n*]' | sed -n -e 's/href=//gI' -e 's/src=//gI' -e '/http:/Ip' > $tmp_1
         cat $tmp_1
         }
            ;;
         "h")
            usage
            ;;
         "?")
            echo " Unknown option $OPTARG"
            usage
            ;;
         ":")
            usage
            ;;
         *)
         # should not occur
            echo "unknown error"
            ;;
         esac
      done
   return $OPTIND
}

showargs () {
  for p in "$@"
    do
      echo "[$p]"
    done
}

###
if [ $# != 0 ]; then
   extract  $@
else
   usage
fi

Word Definitions (commandline Application)

Mon, 04 May 2009 16:57:34 +0200

Get definitions for any word using the commandline.

This script is almost the same as the Synonyms script.
(look for more info there)

lxer@server:~$ ./definition.py python
Overview of noun python
The noun python has 3 senses? (first 1 from tagged texts)
1. (3) python (large Old World boas)

2. python (a soothsaying spirit or a person who is possessed by such a spirit)

3. Python ((Greek mythology) dragon killed by Apollo at Delphi)

#!/usr/bin/python

# get synonyms for any word from synonym.com
# uses python + beautifulsoup

import re, urllib, sys
from BeautifulSoup import BeautifulSoup

if (len(sys.argv) > 1):
   word = sys.argv[1]
else:
   print " usage:  synon word"
   exit(0)

url = "http://www.synonym.com/definition/"+word
f = urllib.urlopen(url)
doc = f.read()

soup = BeautifulSoup(''.join(doc))

def strip_tags(value):
   return re.sub(r'<[^>]*?>', '', value)

divs = soup.findAll('div')
result =  str(divs[9])
result = result.replace("<br />", "\n")
result = strip_tags(result)
#remove empty lines
a = result.split('\n')
for b in a:
   if b:
      print b

Gmail use command-line to check for new messages

Tue, 14 Apr 2009 15:31:06 +0200

bash script for checking your gmail accounts

(This is probably one of the scripts that I use the most)
Having multiple gmail accounts can be usefull, however opening your browser, logging in and out (multiple times) is annoying, especially when there wasn't really a new email.

This bash script checks your gmail accounts, and reports how many new messages are in your inbox. Also, it activates your ScrLock led on your keyboard (really, when did you use that button anyway...?).

To get the led function working, you might have to do this first:

lxer@local:~$  xmodmap -e 'add mod3 = Scroll_Lock'

or add it to modmap: nano ~/.Xmodmap , add mod3 = Scroll_Lock

#! /bin/bash
# checking for new emails @ gmail

# created by Lxer 2008
#

# check your modifier keysettings:  xmodmap
# to get scroll lock led working: xmodmap -e 'add mod3 = Scroll_Lock'
#  or add it to modmap: gedit ~/.Xmodmap ,  add mod3 = Scroll_Lock

# to have this script running in the background and checking
# it  repeatedly, use:
#   sudo watch --interval=300 gmail &> /dev/null &
# or set as cronjob
#
#####################
# replace these with your account settings :
username=( username1 username2 username3 )
passwd=( pass1 pass2 pass3 )
tempfile="/var/tmp/gm/"

#########
a=0
for (( i = 0 ; i < ${#username[@]} ; i++ )) ; do
   wget -q -P $tempfile https://${username[$i]}:${passwd[$i]}@mail.google.com/mail/feed/atom
   if [ -f $tempfile"atom" ]; then
      email=`sed -n -e 's/<\/*fullcount>//pg'  $tempfile"atom"`
   if [ "$email" != "0" ]; then
      echo -e "New emails: \033[1m $email \033[0m  account: \033[1m ${username[$i]}  \033[0m"
      sed -n -e 's/< \/*summary>/\r/pg' $tempfile"atom"
      a=1
   fi
   rm $tempfile"atom"
else
   echo $name3 : Login failed.
fi
done

# print url for easy access
if [ $a -eq 1 ]; then
   echo "http://www.gmail.com"
fi

#blinkenlight
if [[ $EUID -eq 0 ]]; then # only if root
   if [ $a -eq 1 ]; then
      ledcontrol set s5 on
   else
      ledcontrol set s5 off
   fi
fi

Save this file as, for example, 'gmail' and put it in your /usr/bin folder (and make it executable). A cronjob can be used to run this script like every 5 minutes or so.