Recently I was doing an online course "CS 101:Building A Search Engine" at www.udacity.com. In this course I learnt about basics of python programming, crawlers and search engine. I thought of building a small utility using crawlers to locate broken links in a web site if any. This utility crawls every page of your web site and checks if all the links are accessible. In case of any error it returns the url along with their http error code. This utility is suitable for that kind of web sites where you have multiple pages connected through links(using "href") and does not have too many form submission and AJAX calls.
then it will crawl root url domain and domain b, c and other domains given in above statement.
How to use:
My knowledge of Python programming is of intermediate level. So, probably there may be some issues with this utility. Please use it at your own risk :)
Please let me know the issues you faced while using this utility.