Sheets Genie Project Templates are useful for quickly getting up and running with web scraping and automations, but what if you want to edit those templates and recipes to make them more useful and relevant to you?
Or what if there isn’t a template yet for the website, tool, or automation you want?
This is when the valuable skill of DIY (Do It Yourself) enters the pictures.
With just a little bit of knowledge you can do a lot
Even if you have no experience whatsoever with scraping websites or building automations, you can do this.
With just a little bit of knowledge (the knowledge below) you’ll have a lot more power and capabilities. So let’s get started.
How to scrape websites and build online automations: 3 steps
Scraping websites and automating online activities boils down to 3 steps:
- Select elements on a website page
- Do stuff with those selected elements (like click on them, extract text or links from them, input data, etc.)
- Have a way to access any data collected as a result of your activities
How to select website page elements for Sheets Genie
Sheets Genie uses CSS Selectors and JQuery Selectors (CSS Selectors on Steroids) to select page elements. To identify these selectors you can take one of two basic approaches:
- Inspect the website page html to identify selectors
- Use a tool that visually selects elements on the page for you and gives you the CSS selector
My flow is the following:
- Start with a tool that helps select elements / returns CSS selectors (WebScraper.io chrome extension)
- Then do manual cleanup (inspecting a page’s HTML) for selectors that could be improved or the tool couldn’t identify
Use the WebScraper.io chrome extension to select elements on the page
This is what I always use at the beginning of building every Sheets Genie scrape or automation, because:
- It does a good enough job the majority of the time (no tool will be 100% perfect, just FYI)
- It saves a lot of time selecting elements initially
- It gives you visual feedback on your progress (what selected elements look like, what the predicted data returned will be)
- It already helps you with the “doing stuff to elements” phase (ex: extract text or a link from the selected element, click on it, etc.)
- You can export your WebScraper.io setup and put it in a Sheets Genie spreadsheet, and the majority of the time Sheets Genie will do a decent job translating your WebScraper.io setup to a Sheets Genie setup (again, no tool is 100% perfect, not even Sheets Genie!)
How I use the WebScraper.io extension to jumpstart Sheets Genie scrapes and automations
I always categorize scrapes and automations in one of 2 ways that is reflected in my tasks setup with Sheets Genie:
- I’m targeting multiple elements of the same kind (like reviews, tweets, products, etc.), or…
- I’m targeting various unique elements on a page of my choice (in other words, not targeting multiple elements of the same kind)
So I set up my scrapes with WebScraper.io in one of those 2 ways, and then I do the following:
- I import my WebScraper.io scrapes into my Sheets Genie spreadsheet
- I convert them to Sheets Genie tasks
- I run those tasks and see how well the tasks converted to a Sheets Genie setup
- If it doesn’t turn out to my liking, I make modifications and go from there
My advice to you if you are new to all of this:
- You don’t have to be an expert to achieve the majority of what you want to achieve
- Start with learning how to use the WebScraper.io tool to jumpstart your automations
- By importing WebScraper.io scrapes into your Sheets Genie spreadsheet template, you will automatically learn how to create your own Sheets Genie task setup (for example, where CSS selectors go in the template and how to format them)
- Develop the skill of modifications / advanced CSS selection last (after you get the basic concept)
Videos of how I use WebScraper.io with Sheets Genie
The best way to learn how to do this is to see how I actually do it, so here you go.
A few quick notes:
- The videos don’t have sound for some reason (but they still work well enough for our purposes)
- I’ll probably update the videos at some point, but this is enough to get you started…
- Shows how I do both categories of scrapes (multiple elements vs. unique elements)
- Also shows you how to export from WebScraper.io and import into Sheets Genie
WebScraper.io: scrape multiple elements of same kind for Sheets Genie
WebScraper.io: scrape unique page elements for Sheets Genie
How I fix broken automations with WebScraper.io
I have a separate post on how to fix broken automations, but I’m including the video for that here, as well, because it’s also another relevant WebScraper.io tutorial for you.
Also, once you start building your own automations, you will inevitably find that they stop working at some point, so it’s good to be prepared for that and know what to do if / when that happens.
Some good news: this video actually has sound.
WebScraper.io + Sheets Genie is the top DIY Sheets Genie skill
Learning how to set up WebScraper.io scrapes and import them into your Sheets Genie template is by far the most powerful DIY Skill that you can develop for jumpstarting your Sheets Genie automations and scrapes.
This might even be enough for you, depending on what you want to achieve. (And it takes a relatively small amount of time to develop this skill compared to the massive payoff and power it provides you, so it’s definitely worth learning.)
My advice to you: definitely learn to do this before you move on to the more advanced stuff below. It will already help you achieve 50% to 100% of what you want to achieve, depending on your goals.
WebScraper.io + Sheets Genie = 3 steps completed for scrapes and automations
At the beginning of this tutorial, you learned that web scraping and automations boil down to 3 steps.
Don’t remember them? Look them up at the top. And realize that this WebScraper.io + Sheets Genie skill already completes these 3 steps, and that everything else you’ll learn from here is a more advanced skill that builds upon this basic, foundational skill.
After you know how to use WebScraper.io with Sheets Genie…
You might want to know the following:
- How to select CSS elements in other ways (to better target elements on the page)
- How to use Sheets Genie to scale up your automations
- This information is coming soon (again, starting with WebScraper.io + Sheets Genie is already huge at this point)