Hacking websites

This is something I've started doing a lot these days out of need. First what I mean by hacking is figuring out how stuff works i.e. how the web client (the web browser) and web server (Apache, IIS, Tomcat) interact. There are a few tools freely available which can go a long in elucidating how stuff works. A lot of them are simple plugins for Firefox(LiveHTTP headers, Tamper Data, Firebug( or other web browsers. Others are standalone applications like web proxies (Scarab) or network interface sniffers (Wireshark, tcpdump).

The most obvious of hacking a website is simulating a web browser such as Firefox. In my case the language of choice to do that is Python. At the place I'work we've got a browser class written in Python which uses the built-in httplib, urllib and urlparse modules.
It also handles session cookies by storing them in a dict. The code is not fully RFC compliant but does the job for most websites.
By using the browser module we can try to simulate all post, get requests that go from a web browser like Firefox towards a web server. Obviously any client side script that usually runs in a compliant web browser needs to be simulated in Python code if it is found to be necessary in the client-server communication.
Simulating a web browser in Python is fairly interesting in itself because it does open your eyes to many vulnerabilities in server-client side scripts.
The really interesting stuff though is when web site designers try to stop us from accessing their web site programatically. Ways of accomplishing this usually involve the test which are trivial for humans but hard for machines (e.g. captcha codes).

...