Crawling a Website

These past two weeks my main task was to crawl various websites and download the pages. I then had to get data from google analytics and cross reference the pages which are getting traffic and suggest which one’s should be kept, redirected or removed. However this task was only meant for the pages which had news articles.

The tool I used for this task was Screaming Frog. Its a powerful, yet simple and straightforward tool.

All you need to do is either enter a URL or upload and then start crawling. You can then add filters and customize the data as you want or need.

This task was quite long, messy and not to mention manual. But being a trainee, I was more than happy to get my hands dirty.

Here’s what I learnt.

• User experience: Having dead pages on your website ruins the user experience for a person. Not only does it take longer to load which increases bounce rate but it may also mean that the user has to navigate through unnecessary pages on your website.
• Analysis: The fact that I was given the responsibility to suggest which pages to keep, redirect or remove mean that I got to sharpen my data analytical skills.
• I got to experience in a new tool and got more comfortable with using google analytics.

The best part was that I got praise from the person who assigned me the task.

Now I want to hear about a task that you’ve been assigned. How did you find it and what did you learn?