DEV Community

Zahar Baranenko
Zahar Baranenko

Posted on Edited on

How to pull every open job from Workday, Greenhouse, Lever, Ashby and 3 more ATS with public APIs (and monitor changes)

#webscraping #api #automation #jobs

Job postings are one of the most underrated public data sources on the internet. Recruiters use them to spot placement opportunities, B2B teams read them as buying signals (a new Head of Data means data-tooling budget), and job seekers want to apply on day one — not when a posting finally reaches the aggregators.

The usual instinct is to scrape career pages. Don't. Most companies host their careers page on one of a handful of Applicant Tracking Systems (ATS), and the big ones — Workday, Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee and Personio — all expose public JSON (or XML) endpoints. No auth. No proxies. No brittle HTML selectors. The career page itself loads the same data you're about to fetch.

In this tutorial we'll build a single-file Python tool that:

  1. fetches every open job for a company from any of the seven ATS,
  2. auto-detects which ATS a company uses,
  3. normalizes everything into one clean schema,
  4. monitors changes — run it on a schedule and get only new / removed / changed postings.

The seven endpoints

ATS Endpoint
Greenhouse GET https://boards-api.greenhouse.io/v1/boards/{slug}/jobs?content=true
Lever GET https://api.lever.co/v0/postings/{slug}?mode=json
Ashby GET https://api.ashbyhq.com/posting-api/job-board/{slug}
SmartRecruiters GET https://api.smartrecruiters.com/v1/companies/{slug}/postings (paginated)
Recruitee GET https://{slug}.recruitee.com/api/offers/
Personio GET https://{slug}.jobs.personio.de/xml (XML feed)
Workday POST https://{tenant}.{wdN}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs (paginated)

The first six are documented by the vendors. The Workday one is the CXS endpoint every Workday career site loads its own listings from — public and unauthenticated, just not in a docs portal.

The {slug} is the company identifier you see in career-page URLs: boards.greenhouse.io/stripe → stripe, jobs.lever.co/spotify → spotify, jobs.ashbyhq.com/linear → linear, careers.smartrecruiters.com/Visa → Visa.

Try one right now — no API key needed:

curl -s "https://api.ashbyhq.com/posting-api/job-board/linear" | head -c 400
Enter fullscreen mode Exit fullscreen mode

Step 1 — fetchers, one per ATS

Each API returns a different shape, so we normalize as we fetch. Here are the first four (Python 3, only requests):

import requests

UA = {"User-Agent": "ats-jobs-tutorial/1.0"}

def get_json(url, params=None):
    r = requests.get(url, params=params, headers=UA, timeout=30)
    r.raise_for_status()
    return r.json()

def fetch_greenhouse(slug):
    d = get_json(f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs",
                 {"content": "true"})
    return [{
        "job_id": str(j["id"]),
        "title": j.get("title"),
        "location": (j.get("location") or {}).get("name"),
        "url": j.get("absolute_url"),
        "published_at": j.get("first_published"),
    } for j in d.get("jobs", [])]

def fetch_lever(slug):
    d = get_json(f"https://api.lever.co/v0/postings/{slug}", {"mode": "json"})
    return [{
        "job_id": str(j["id"]),
        "title": j.get("text"),
        "location": (j.get("categories") or {}).get("location"),
        "url": j.get("hostedUrl"),
        "published_at": j.get("createdAt"),  # milliseconds since epoch!
    } for j in d]

def fetch_ashby(slug):
    d = get_json(f"https://api.ashbyhq.com/posting-api/job-board/{slug}")
    return [{
        "job_id": str(j["id"]),
        "title": j.get("title"),
        "location": j.get("location"),
        "url": j.get("jobUrl") or j.get("applyUrl"),
        "published_at": j.get("publishedAt"),
    } for j in d.get("jobs", []) if j.get("isListed") is not False]

def fetch_smartrecruiters(slug):
    out, offset = [], 0
    while True:
        d = get_json(f"https://api.smartrecruiters.com/v1/companies/{slug}/postings",
                     {"limit": 100, "offset": offset})
        items = d.get("content", [])
        for j in items:
            loc = j.get("location") or {}
            out.append({
                "job_id": str(j["id"]),
                "title": j.get("name"),
                "location": ", ".join(filter(None, [loc.get("city"), loc.get("country")])) or None,
                "url": f"https://jobs.smartrecruiters.com/{slug}/{j['id']}",
                "published_at": j.get("releasedDate"),
            })
        offset += len(items)
        if not items or offset >= d.get("totalFound", 0):
            break
    return out
Enter fullscreen mode Exit fullscreen mode

Three real-world quirks worth knowing (each cost me a debugging session):

  • Lever returns timestamps in milliseconds (createdAt: 1721900000000), not ISO strings. Divide by 1000 before datetime.fromtimestamp.
  • Ashby includes unlisted postings — filter out isListed: false or you'll "discover" jobs the company never published.
  • SmartRecruiters answers 200 OK with an empty list for any slug, even one that doesn't exist. An empty SmartRecruiters response is not proof the company uses SmartRecruiters.

Step 1.5 — Recruitee, Personio and Workday

Recruitee is another plain JSON GET:

def fetch_recruitee(slug):
    d = get_json(f"https://{slug}.recruitee.com/api/offers/")
    return [{
        "job_id": str(j["id"]),
        "title": j.get("title"),
        "location": j.get("location"),
        "url": j.get("careers_url"),
        "published_at": j.get("published_at"),
    } for j in d.get("offers", [])]
Enter fullscreen mode Exit fullscreen mode

Personio publishes an XML feed (stdlib handles it fine):

import xml.etree.ElementTree as ET

def fetch_personio(slug):
    r = requests.get(f"https://{slug}.jobs.personio.de/xml", headers=UA, timeout=30)
    r.raise_for_status()
    out = []
    for p in ET.fromstring(r.content).iter("position"):
        jid = p.findtext("id")
        out.append({
            "job_id": jid,
            "title": p.findtext("name"),
            "location": p.findtext("office"),
            "url": f"https://{slug}.jobs.personio.de/job/{jid}",
            "published_at": p.findtext("createdAt"),
        })
    return out
Enter fullscreen mode Exit fullscreen mode

Workday is the interesting one — it's the ATS of most large enterprises, and it can't be probed from a bare slug. A Workday career URL carries three parts: the tenant, the instance (wd1, wd5, …) and the site name — e.g. https://adobe.wd5.myworkdayjobs.com/external_experienced. All three go into the CXS endpoint, which is a POST with pagination:

import re

def fetch_workday(career_url):
    m = re.search(r"([\w-]+)\.(wd\d+)\.myworkdayjobs\.com(/[^?#]*)?", career_url)
    tenant, wd, path = m.group(1), m.group(2), m.group(3) or ""
    site = next((s for s in path.strip("/").split("/")
                 if s and not re.match(r"^[a-z]{2}([-_]\w{2,4})?$", s)), tenant)
    base = f"https://{tenant}.{wd}.myworkdayjobs.com"
    out, offset, total = [], 0, None
    while True:
        r = requests.post(f"{base}/wday/cxs/{tenant}/{site}/jobs",
                          json={"limit": 20, "offset": offset,
                                "searchText": "", "appliedFacets": {}},
                          headers=UA, timeout=30)
        r.raise_for_status()
        d = r.json()
        if total is None:
            total = d.get("total", 0)   # only present on the first page!
        posts = d.get("jobPostings", [])
        for j in posts:
            out.append({
                "job_id": (j.get("bulletFields") or [j.get("externalPath")])[0],
                "title": j.get("title"),
                "location": j.get("locationsText"),
                "url": base + (j.get("externalPath") or ""),
                "published_at": None,  # the list API only gives "Posted N Days Ago"
            })
        offset += len(posts)
        if not posts or offset >= total:
            break
    return out
Enter fullscreen mode Exit fullscreen mode

Three more quirks, all Workday:

  • limit is capped at 20. Ask for 100 and you get 400 Bad Request. A 2,000-job board is 100 requests — budget for it.
  • total is only present on the first page — later pages report total: 0. Cache it from page one or your loop stops after 40 jobs (mine did).
  • The site name can't be guessed. nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite — nobody types that from memory. Always start from the career-page URL. 
FETCHERS = {
    "greenhouse": fetch_greenhouse,
    "lever": fetch_lever,
    "ashby": fetch_ashby,
    "smartrecruiters": fetch_smartrecruiters,
    "recruitee": fetch_recruitee,
    "personio": fetch_personio,
}
# workday is separate: it needs a URL, not a slug
Enter fullscreen mode Exit fullscreen mode

Step 2 — auto-detect the ATS

You usually don't know (or care) which ATS a company uses. Probe them in order and keep the first that answers with jobs:

def detect_and_fetch(slug):
    for ats, fetch in FETCHERS.items():
        try:
            jobs = fetch(slug)
        except Exception:
            continue
        # SmartRecruiters gives 200 + [] for any slug — an empty board
        # during detection means "not found", not "no openings".
        if ats == "smartrecruiters" and not jobs:
            continue
        return ats, jobs
    return None, []
Enter fullscreen mode Exit fullscreen mode 
>>> detect_and_fetch("linear")
('ashby', [{'job_id': '...', 'title': 'Senior / Staff Fullstack Engineer', ...}])
Enter fullscreen mode Exit fullscreen mode

In production you'd cache the detected ATS per company so you don't re-probe on every run — four HTTP calls when one is enough.

Step 3 — monitor changes

Pulling all jobs is a one-liner now. The genuinely useful part is knowing what changed: which postings appeared today, which quietly disappeared, which got edited. That's a diff against the previous state.

Fingerprint each job, store {job_id: fingerprint} between runs, compare:

import hashlib, json, pathlib

STATE = pathlib.Path("state.json")

def fingerprint(job):
    src = "|".join(str(job.get(f, "")) for f in ("title", "location", "url"))
    return hashlib.sha1(src.encode()).hexdigest()[:16]

def diff(prev, jobs):
    cur = {j["job_id"]: j for j in jobs}
    changes = []
    for jid, job in cur.items():
        if jid not in prev:
            changes.append({"change": "new", **job})
        elif prev[jid]["fp"] != fingerprint(job):
            changes.append({"change": "changed", **job})
    for jid, snap in prev.items():
        if jid not in cur:
            changes.append({"change": "removed", "job_id": jid, "title": snap["title"]})
    return changes

def monitor(companies):
    state = json.loads(STATE.read_text()) if STATE.exists() else {}
    for slug in companies:
        ats, jobs = detect_and_fetch(slug)
        if not ats:
            print(f"{slug}: no supported board found")
            continue
        prev = state.get(slug)
        if prev is None:
            print(f"{slug} [{ats}]: baseline saved, {len(jobs)} jobs")
        else:
            for c in diff(prev, jobs):
                print(f"{slug}: {c['change'].upper()} — {c.get('title')}")
        state[slug] = {j["job_id"]: {"fp": fingerprint(j), "title": j.get("title")}
                       for j in jobs}
    STATE.write_text(json.dumps(state))

monitor(["stripe", "linear", "spotify"])
Enter fullscreen mode Exit fullscreen mode

First run saves a baseline. Every later run prints only the delta:

stripe: NEW — Backend Engineer, Payments
linear: REMOVED — Account Executive, Growth
Enter fullscreen mode Exit fullscreen mode

Put it on cron (or GitHub Actions on a schedule) and pipe the output into Slack, a spreadsheet, or an n8n/Make webhook — you'll know a company is hiring the day the posting goes live.

What it takes to run this seriously

The 100-line version above works. Running it reliably for a real watchlist grows the usual operational tail: state storage that survives machines, retry/backoff when an API hiccups, distinguishing "board is gone" from "request failed" (so you don't fire 200 false REMOVED alerts), department/salary/remote fields where each ATS hides them differently, caching ATS detection, and a scheduler that doesn't silently die.

All of that is maintenance, not insight. If you'd rather not own it, I packaged this exact pipeline — all seven ATS, Workday included — as an Apify Actor: ATS Jobs Scraper & Change Monitor:

  • paste slugs or career-page URLs, get the normalized dataset (department, salary where exposed, remote flags included),
  • monitor mode with hosted state and per-change pricing ($0.001 per job) — watching 100 companies daily costs a few dollars a month,
  • native @apify scheduling, dataset exports (CSV/JSON), webhooks, and it's callable by AI agents via Apify MCP.

The DIY script above gets you 80% of the way for $0 — start there (full runnable version on GitHub). When babysitting it stops being fun, the Actor is the same logic with the ops solved.

Questions about a specific ATS or an edge case? Drop a comment — I've probably hit it.

Top comments (0)