Python for SEO: a beginner’s guide to automating SEO tasks

Python for SEO means using short Python scripts to automate repetitive SEO tasks, such as checking status codes, pulling title tags from hundreds of pages or combining keyword exports. You don’t need to be a developer: a few lines of code can replace hours of copying and pasting in spreadsheets.

Most SEO work involves the same jobs on a large scale. Check these 2,000 URLs. Merge these three exports. Find every page missing a meta description. Spreadsheets struggle once the data gets big, and tools don’t always do exactly what you need.

Python fills that gap. It’s free, easy to read and has libraries built for fetching web pages and handling data. This guide covers how to start and gives three small scripts you can run today.

Why SEOs use Python

SEOs use Python because it handles large datasets quickly and repeats the same task without mistakes. A script that takes ten minutes to write can save hours every month, and it runs the same way each time.

It’s most useful for:

• Checking status codes and redirects across a list of URLs
• Extracting titles, meta descriptions and headings from pages
• Merging and cleaning exports from Search Console, analytics and keyword tools
• Grouping keywords by shared words or intent, a starting point for keyword clustering
• Parsing server logs to see how Googlebot crawls a site, the basis of log file analysis
• Pulling data from APIs, such as the Search Console API, without hitting export limits in the interface

How to get started

To start, install Python, add three libraries and run your first script in a notebook. The setup takes about 15 minutes.

1. Install Python. Download it from python.org, or use a free online notebook such as Google Colab so there’s nothing to install.
2. Install the libraries. Requests fetches web pages, Beautiful Soup reads HTML and pandas works with tables of data.
3. Use a notebook. Jupyter or Colab lets you run code a few lines at a time and see the result straight away.
4. Start with a real task. Pick something you already do by hand each month. Learning is much faster when the script solves a problem you have.

Install the libraries with one command:

pip install requests beautifulsoup4 pandas

Example 1: check status codes for a list of URLs

This script checks whether each URL returns a 200, a redirect or an error. It’s useful after a migration or when cleaning up a sitemap.

import requests

urls = [
    "https://www.example.com/",
    "https://www.example.com/about/",
    "https://www.example.com/old-page/",
]

for url in urls:
    response = requests.get(url, allow_redirects=False, timeout=10)
    print(url, response.status_code, response.headers.get("Location", ""))

Setting allow_redirects=False shows you the first response, so you can see a 301 and where it points rather than only the final page.

Example 2: pull title tags and meta descriptions

This one fetches each page and reads its title and meta description, so you can spot missing or duplicate ones. Our guide to title tags explains what good ones look like.

import requests
from bs4 import BeautifulSoup

urls = ["https://www.example.com/", "https://www.example.com/about/"]

for url in urls:
    html = requests.get(url, timeout=10).text
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.string.strip() if soup.title and soup.title.string else "MISSING"
    meta = soup.find("meta", attrs={"name": "description"})
    description = meta["content"].strip() if meta and meta.get("content") else "MISSING"
    print(url, "|", title, "|", description)

Example 3: find quick wins in a Search Console export

Export the Queries report from Search Console as a CSV, then use pandas to list queries with plenty of impressions but a low click-through rate. Those are often pages where a better title could win more clicks.

import pandas as pd

df = pd.read_csv("Queries.csv")
df["CTR"] = df["CTR"].str.rstrip("%").astype(float)

quick_wins = df[(df["Impressions"] > 500) & (df["CTR"] < 2)]
print(quick_wins.sort_values("Impressions", ascending=False).head(20))

Column names can vary slightly between exports, so check the header row of your file first.

Using Python safely and responsibly

Only run scripts against sites you own or have permission to check, and keep the request rate low. Add a short pause between requests on larger lists so you don’t slow the server down. Never paste API keys or passwords straight into a notebook you share.

Python also doesn’t replace a proper crawler for full audits. Tools built for site crawls handle JavaScript rendering, robots.txt rules and scale far better. Python is best for the gaps around them: custom checks, data merging and repeat reports.

If you’d rather spend your time on the findings than the scripts, our technical SEO team automates this kind of checking for clients. Book a free strategy session to talk it through.

Related reading: our guides to technical SEO and JavaScript SEO.

Common questions