I’ve been curious about building a web scraper for some time now and I decided to look into building one for alerting me on new jobs for companies I am interested in.
For the uninitiated (Including myself in that category.) here’s a brief explanation of a web scraper:
Per the good people of Techopedia, webscraping is defined as “Web scraping is a term for various methods used to collect information from across the Internet. Generally, this is done with software that simulates human Web surfing to collect specified bits of information from different websites. Those who use web scraping programs may be looking to collect certain data to sell to other users, or to to use for promotional purposes on a website.” – https://www.techopedia.com/definition/5212/web-scraping.
Here’s the guide I am going off of for this – https://realpython.com/beautiful-soup-web-scraper-python/
I’m planning on building a scraper that will look for two particular items for me. A Hobie kayak and a stand up paddle board. I don’t know the specific models at the moment. Once I’ve got the models dialed in, I’ll use the scraper to target sites like Craig’s List, REI, Ebay, Amazon, etc to look for the specific models.
This is my current project. I’ll post more on it as time permits.