the day I had to hack wget

Thanks to Richard who insists on documenting stuff on mediawiki, I had to figure out how to make a static mirror of the html so that I a) have offline access and b) am 100% sure anything I put there is backed up properly and there is no chance of loosing any information.

I thought it was simple enough, just httrack or wget it, but it turned out to be much trickier :(

Problem #1: how to get httrack to authenticate

Mediawiki uses a form login and uses a login token that it stores in a cookie before opening the form :( which is good for security etc. but made it real tricky for me. In the end I could not get it working with httrack and moved on to wget, but not before figuring out how to intercept a cookie from my browser which was really helpful with debugging later. Some helpful links if you wanna try this:
http://www.dzone.com/snippets/using-wget-download-content
http://httrack.kauler.com/help/CatchURL_tutorial
http://www.httrack.com/html/fcguide.html

Problem #2: how to get wget to authenticate

It took some time to figure out the complicated login process mentioned above. This post from amadeus helped me get on the right track.

Problem #3: how to not 'click' on the logout link?!

Simple right just add --reject '* = *' to the configuration..

It turns out that wget (1.14 on ubuntu) for some reason first downloads all 'rejected' links and then deletes them?! !@#$!@#$

Thanks Tomàs Reverter for finally pointing me into the right direction with easy instructions for hacking wget.

His patch didn't do the trick for me so I hacked it a bit: 

Download the wget source code.

in src/recur.c on line 406 I injected an if statement so that we don't even enqueue 'rejected' urls

if (acceptable (child->url->url)) {
    url_enqueue (queue, ci, xstrdup (child->url->url),
    xstrdup (referer_url), depth + 1,
    child->link_expect_html,
    child->link_expect_css);
}

Then ./configure && make && sudo make install 
This did unfortunately not solve my problem yet, digging a bit further I noticed that the acceptable() method actually only checks regex rejects, and I thought that the glob version of reject gets converted or something.. NO! I had to specify
--reject-regex='.*=.*,.*UserLogout.*' before it finally started to work.
(The man page doesn't mention --reject-regex but I realized wget --help does )

Here is my script that downloads our internal wiki, it is very dirty and probably does loads of unnecessary stuff but it works, and it is 01:15 and I'm sick and I want to go to bed, but want to get this out while I have all the browser tabs still open:

WIKI_USERNAME='someone'
WIKI_PASSWORD='setupusthebomb'
TARGETHOST="http://intranet.example.com/mediawiki"
MW_DOMAIN="intranet.example.com"
MW_URL="$TARGETHOST"
INDEX_URL="$TARGETHOST/index.php"
MW_PAGE="Main_Page"
TARGET_DIR="/tmp/wget/"

# Mediawiki uses a login token, and we must have it for this to work.
WP_LOGIN_TOKEN=$(wget -q --no-check-certificate -O - --save-cookies cookies.txt --keep-session-cookies "$MW_URL/index.php/Special:UserLogin" | grep wpLoginToken | grep -o '[a-z0-9]\{32\}')

sleep 1

wget -q --no-check-certificate --load-cookies cookies.txt --save-cookies cookies.txt --keep-session-cookies --post-data "wpName=${WIKI_USERNAME}&wpPassword=${WIKI_PASSWORD}&wpDomain=${MW_DOMAIN}&wpRemember=1&wpLoginattempt=Log%20in&wpLoginToken=${WP_LOGIN_TOKEN}" "$INDEX_URL?title=Special:UserLogin&action=submitlogin&type=login" -O ${TARGET_DIR}/tmp

sleep 1

#not sure why this is needed, but it is (probably because mediawiki doesn't like it if you hit an 404 right after logginj in)
wget -q --no-check-certificate --load-cookies cookies.txt --keep-session-cookies --save-cookies cookies.txt "$MW_URL/index.php/${MW_PAGE}" -O ${TARGET_DIR}/tmp.html

sleep 1

/usr/local/bin/wget -t0 -T900 --limit-rate=30k --random-wait -e robots=off --no-check-certificate --load-cookies cookies.txt --keep-session-cookies --mirror --convert-links -backup-converted --html-extension --page-requisites --reject='*=*,*UserLogout*' --exclude-directories='*=*,*UserLogout*' --reject-regex='.*=.*,.*UserLogout.*'   --no-parent -P ${TARGET_DIR}  -o ${TARGET_DIR}/wget-log "$MW_URL/"

# Problem #4: the following doesn't actually work, 
# because wget decides out of its own cleverness to go into 
# background mode and I can't see a way to disable that, I smell another hack:(
#remove my username from all the pages
cd $TARGET_DIR
grep -ilr ${WIKI_USERNAME} . | xargs sed -i s/${WIKI_USERNAME}/User/g

# delete pages that include the user name
find -iname "*${WIKI_USERNAME}*" -delete

Problem #4: don't know how to force wget to not enter backrgound mode :(

Please help me in the comments..

Split local/international network traffic between different adsl accounts on kubuntu.

I was able to setup my internet on my laptop so that my local internet traffic gets charged at local rates (1/4 of the price for me).
This page helped me but it's a bit old and a little different to what I needed, so I thought I'll write down my steps (mostly copied and adjusted from said page).

My setup:
os = kubuntu 9.04 jaunty
isp = webafrica and I'm very happy with them
router = Mega 105 WR

• Some tutorials say you need to change your router mode to be bridged, but I found out that mine is automatically in semi-bridge mode: I setup my main account on the router which can be used by all the computers in the house, but on my laptop I can setup an additional connection which gets bridged through the router.
• sudo apt-get install ppp pppoeconf
  (ppp includes rp-pppoe.so, which is installed manually by the original instructions, but I didn't have to since its in ppp)
  
• Run pppoeconf
  
  • Accept the defaults provided by pppoeconf and say yes to the start at boot option.
  • I just answered yes to the "'noauth' and 'defaultroute'" question, since I changed it later in any case.
  • It asks you for a username/password, this is your local adsl account authentication parameters as provided by your ISP.
  • Now test the connection with sudo pon dsl-provider
  • If you list the interfaces with ifconfig you should see ppp0 listed (and it should get an ip address).
  • Disconnect ppp0 before continuing. sudo poff dsl-provider
• Edit /etc/ppp/peers/dsl-provider by hand.
  Only the parameters that you need to check are listed.
  A hash before the line means the line is commented out.
  sudo kate /etc/ppp/peers/dsl-provider

  noipdefault
  # defaultroute
  # replacedefaultroute
  # usepeerdns
  

  So be sure that the above 3 parameters are commented out.
  (We only want to use the DNS servers provided from the ISP from the international traffic.)
• sudo kate /etc/network/interfaces
  At the bottom of it you should have something like this:

  
  manual ppp0
  iface ppp0 inet ppp
  provider dsl-provider
  pre-up /sbin/ifconfig wlan0 up # line maintained by pppoeconf
  
  #auto wlan0
  #iface wlan0 inet manual
  

  
  
  • I make ppp0 manual since I need to wait for my wireless lan to connect before I can connect to the local internet. With this I can have full control, which is good since I don't want to leave it on too long because you are directly on the internet now and you're router's firewall isn't protecting you from local traffic.
    
  
  
  • I comment out the wlan0 section so that the network manager won't ignore it. You can remove it completely.
  
  
  • Test the connections to make sure there are no problems.
    sudo ifup ppp0
    ifconfig should list ppp0
    sudo ifdown ppp0
    
  
  
  
  
• Static routing
  Now the fun part where we actually get the traffic flowing.
  We need to tell the system to route all the South African traffic through the ppp0 interface instead of the wlan0 or eth0 interface.
  We'll use static routing to do this.
  
  Download Armin's list of local routes from http://alm.za.net/ip/localroutes4.txt
  The list is updated every 24 hours and we'll use it instead of duplicating his work.
  It contains all the network addresses that are specific to South Africa.
  Save the file in /tmp and then move it to /etc/ppp/ :
  sudo mv /tmp/localroutes4.txt /etc/ppp
  
  We need to create a small script that tells the system to add the routes to the interface when if comes up.
  Create a file in /etc/ppp/ip-up.d/ called "zanet".
  sudo kate /etc/ppp/ip-up.d/zanet
  
  Paste the following in the file :
  

  
  #!/bin/sh -e
  # Called when a new interface comes up
  
  # add custom routing for zanet (local South Africa) on ppp0 device
  if [ "$PPP_IFACE" = "ppp0" ]
  then
  cat /etc/ppp/localroutes4.txt | sed s/'\$LOCAL'/$PPP_IFACE/ | while read localroute
  do
  # Change net to host for /32 addresses
  # This is a bug workaround - Armin must fix his list script.
  if [[ $localroute =~ "/32" ]]
  then
  localroute=`echo $localroute | sed s/net/host/`
  $localroute
  else
  $localroute
  fi
  done
  fi
  
  # We need to route DNS lookups via ppp0 (SAIX) instead of ppp1 (IS)
  # The static routing will cause the DNS lookups to be made through the local IS
  # connection and the SAIX network will block the lookups because they don't originate
  # from their network.
  
  #uncomment the following if you are using different isps:
  
  #if [ "$PPP_IFACE" = "wlan0" ]
  #then
  #cat /etc/resolv.conf | sed s/nameserver// | sed s/' '// | while read nameserver
  #do
  #route add -host $nameserver $PPP_IFACE
  #done
  #fi
  

  
  Make sure you can execute it:

  
  sudo chmod a+x /etc/ppp/ip-up.d/zanet

  
  
  Static routing should now work unless you made a mistake or I forgot something.
  Test it by starting ppp0. (sudo ifup ppp0)
  If you run route -n you should see huge list of static routes for the ppp0 interface and only a few when it is not started.
  
  


• Automatic localroute updates
  You could get wget to retrieve a new list of local routes every so often manually or with a cronjob. You can stick the following script in a executable file in /etc/cron.monthly/ eg. /etc/cron.monthly/update-local-routes:
  

  
  #!/bin/bash
  
   cd /etc/ppp
   wget http://alm.za.net/ip/localroutes4.txt
  
   # If there is already a localroutes file replace it with the new one.
   if [ -f /etc/ppp/localroutes4.txt.1 ]
   then
   # Make sure that we didn't get an empty or nearly empty list
   # More than 1000 characters will qualify as a valid amount of local routes
   # Rather keep the old list if the new one is too small
   if [ `cat /etc/ppp/localroutes4.txt | wc -c` -gt 1000 ]
   then
   mv localroutes4.txt.1 localroutes4.txt
   fi
   fi
  

  
  Make sure its executable:
  

  sudo chmod a+x /etc/cron.monthly/update-local-routes

  
  


• Lastly I use iftop -i ppp0 to monitor and make 100% sure where my traffic is being routed.
  

Jaunty Jackalope (Ubuntu 9.04) is coming soon

It is expected to be released on the 23rd of April 2009.

java.net.SocketException: Too many open files

We hit this again on sles today, so I thought I should leave a note here for posterity.
java.io.FileNotFoundException: /home/tcmas/tomcat/5.0.28/work/Catalina/localhost/balancer/SESSIONS.ser (Too many open files)

1. It might be caused by the small amount of memory available! so increasing that, may fix it.
2. It may also be the the number of open files limit on linux:
   

Increase file descriptor limit under Linux to prevent java.net.SocketException: Too many open files

Get current limit:

ulimit -n

cat /proc/sys/fs/file-nr

The default limit is 1024.

Get current number of open file descriptors:

lsof [-p pid] | wc -l

Increase the limit:

Edit /etc/security/limits.conf:

username hard nofile 32768

http://hausheer.osola.com/docs/11

How I got php with oci8 working

1. Install Oracle instant client (the basic package and the SDK!). (You obviously need the 64bit version on a 64bit architecture). I tried doing this using my locally installed oracle express edition (XE) but that whas a whole lot of pain:
   

   OCIEnvNlsCreate() failed. There is something wrong with your system - please check that ORACLE_HOME is set and points to the right directory

   I ended up using oracle_instantclient_10_2 which worked!!
   
2. sudo -i
3. apt-get install apache2 php5 php5-dev php-pear libaio1
4. Make sure your $ORACLE_HOME and $LD_LIBRARY_PATH is set. Since I have 2 different versions of 'oracle' I set these two variables in /usr/sbin/apache2ctl, at the top of the 'CONFIGURATION SETION':

   export ORACLE_HOME=/stuph/install/oracle_instantclient_10_2
   export LD_LIBRARY_PATH=$ORACLE_HOME

5. echo $ORACLE_HOME >> /etc/ld.so.conf

6. ldconfig

7. Check if the symlinks named libclntsh.so and libocci.so exist in $ORACLE_HOME, which we will need later. In my case these symlinks were not created by ldconfig, so I created them manually:

   ln -s libclntsh.so.11.1 libclntsh.so; ln -s libocci.so.11.1 libocci.so

8. pecl install oci8

9. To enable the oci8 module, add "extension=oci8.so" to /etc/php5/apache2/php.ini and /etc/php5/cli/php.ini). (put this line after the examples starting with ; in the "Dynamic Extensions" section).

10. ldconfig

11. apache2ctl restart

    (or

    /etc/init.d/apache2 force-reload

    to make sure)
12. You should see the oci8 module in the output of phpinfo(). (put info.php in ~www-data or ~apache with the following content and browse to it: "<? phpinfo() ?>")
    

(I'm sorry if this assumes you know how to do certain stuff, but this is a howto for myself, so that I can remember how to do this in a mounth's time)

my bash setup

Here are some things I put in ~/.bashrc or /etc/bash.bashrc to make my life easier:

# When using bash, I use ctrl+r a lot to find commands I executed recently.
# I found that I remember my commands for much longer than bash does,
# so I increased my bash's memory:
export HISTFILESIZE=3000 # the bash history should save 3000 commands
export HISTCONTROL=ignoredups # don't put duplicate lines in the history.
export HISTCONTROL=ignoreboth # ignore same sucessive entries.
# (from http://www.novell.com/coolsolutions/tools/17142.html)

# I like to have a colour prompt with my user, the machine I'm on and the full
# path. Its a bit long so I put it on 2 lines.
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;33m\]\w\[\033[00m\]>\n\$ '

# enable bzr command auto completion
. /opt/bzr/bzr.current/contrib/bash/bzr.simple

# bzr aliasses
alias bt='./bzr --no-plugins selftest'
alias bd='bzr diff'
alias bl='bzr log'
alias bp='bzr pull -v |less'
alias bm='bzr missing |less'
alias bs='bzr status'

# bzr functions, which cant be done by alias or bzr's alias
bh(){
#the $@ inserts all the parameters you pass in
bzr help "$@" |less
}

bcd(){
bzr cdiff "$@" | less -R
}

Super cool startup script for linux

I recently had to get a startup script going which works on Ubuntu and Sles for loggerhead. I came up with these install instructions and following simple but super cool init script. Note that this should work for basically any linux daemon, you just need to rename the script (loggerheadd) and change the script a little.

1. Copy loggerheadd to /etc/init.d
2. Edit the file to configure it.
3. Register the service:

   cd /etc/init.d

• on upstart based systems like Ubuntu run:

  update-rc.d loggerheadd defaults

• on Sysvinit based systems like Centos or SuSE run:

  chkconfig --add loggerheadd

Here follows the script (the indentation is a bit messed up by blogspot)

#!/bin/sh
### BEGIN INIT INFO
# Required-Start: $local_fs $remote_fs $network
# Default-Start: 3 5
# Default-Stop: 0 1 2 6
# Short-Description: Loggerhead
# Description: Manage Loggerhead (a web viewer for projects in bazaar)
### END INIT INFO

# Configure this please: #
LOGGERHEAD_PATH=/opt/loggerhead
LOG_FOLDER=/var/log/loggerhead
LOG_FILE=$LOG_FOLDER/loggerheadd.log
PREFIX=/loggerhead
PORT=8080

# You can add additional options to serve-branches here:
START_CMD="$LOGGERHEAD_PATH/serve-branches --prefix=$PREFIX --log-folder=$LOG_FOLDER --port=$PORT /var/lib/gforge/bzrroot/"

#
# main part
#
case "$1" in
start)
python $START_CMD > $LOG_FILE 2>&1 &
echo "Started loggerhead. (See $LOG_FOLDER for details.)"
;;
stop)
pkill -f "$START_CMD"
;;
status)
proccess=`pgrep -fl "$START_CMD"`
echo "$proccess"
netstat -anp |grep -e ":$PORT"
if [ -z "$proccess" ]; then
echo "Loggerhead is not running."
else
echo "Loggerhead is running."
fi
;;

*)
echo "Usage: loggerheadd { start | stop | status }"
exit 1
esac

Set partition labels on linux.

My colleague Elardus complained that my flash drive didn't have volume labels set and I also wanted automounting to make a nice consistant mountpoint eg. /media/my_flash_drive.
I didn't think think it would be that hard, I really thought that a gui like gparted or qtparted would be able to do that, but was disappointed. So I had to do it from the commandline.

To set the label for ext2 or ext3 partitions:

sudo e2label /dev/sdb2 new_label

where /dev/sdb2 is the partition you want to label. (To get a list of mounted partitions you can do `df -ahT`)


For FAT partitions you first need to configure the drive:
you have to edit /etc/mtools.conf, make f: drive be /dev/sdb1

sudo mlabel f:new_label

Initially I got the following error.

Total number of sectors (256976) not a multiple of sectors per track (63)!
Add mtools_skip_check=1 to your .mtoolsrc file to skip this test

I just did what it asked and then it worked:

echo "mtools_skip_check=1" > ~/.mtoolsrc

I found my info on this page:
http://fedoraforum.org/forum/archive/index.php/t-86378.html