# Groupon deals scraper

`groupon_scraper.php` pulls public deal listings off `groupon.com` for a given
city + category and returns them as structured JSON. Public pages only - no
login, no account access.

## How it works

Earlier attempts (see git history) drove a real headless Chrome browser via
Symfony Panther, waiting for the deal grid to render client-side and then
scraping text out of CSS-selected DOM nodes. That approach turned out to be
the wrong tool for two reasons, discovered by actually running it and
inspecting a screenshot of what the browser saw:

1. **Groupon's bot-detection blocks headless Chrome outright** - every
   headless-Chrome request got a hard `403 Access Restricted` page (a
   Cloudflare-style challenge page), regardless of which CSS selector the
   script was waiting for or how long it waited.
2. **A plain `curl` request with a normal desktop-browser User-Agent is
   served the real page, no challenge at all** - Groupon's deal grid is
   server-rendered (it's a Next.js app), so the HTML `curl` gets back
   already contains the full first page of deal cards. No JavaScript
   execution or browser is needed.
3. **Each deal card carries a `data-bhd` attribute holding a JSON blob** with
   the card's title, subtitle/location, full pricing breakdown
   (list price, sell price, discount %, promo code, currency), and rating -
   already structured, and far more robust than parsing rendered text out of
   Groupon's classnames (which are partly hashed/dynamic and have already
   changed at least once, per the earlier scraper variants in git history).

So the script just does: `curl` the category page with a real UA -> find
every `[data-item-type="card"]` element with `DOMDocument`/`DOMXPath` -> read
and JSON-decode that element's `<a data-bhd="...">` attribute -> also grab
the `<a href>` (deal URL) and first `<img src>` (thumbnail).

## Usage

CLI:
```
php groupon_scraper.php [city-slug] [category-slug]
php groupon_scraper.php los-angeles things-to-do
php groupon_scraper.php new-york things-to-do
```
Defaults to `los-angeles` / `things-to-do` if no args given. Prints the
scraped deals as JSON to stdout and writes the same JSON to
`results/groupon-<city>-<category>-<timestamp>.json` next to the script (the
`results/` folder is created on first run - it's not committed to git, add a
`.gitignore` entry if you don't want these piling up on disk).

Web (e.g. for manually testing from a browser or letting another one of your
own apps call it over HTTP):
```
https://findyournet.com/template00/tools/groupon_scraper.php?city=los-angeles&category=things-to-do
```
Returns a JSON response (`{city, category, count, deals}`) and also writes
the same results file server-side.

## Output shape

Each entry in `deals`:
```json
{
    "title": "Aquarium of the Pacific Deals – Save Up to 33% on Admission",
    "location": "100 Aquarium Way, Long Beach",
    "list_price": 53.29,
    "sell_price": 35.95,
    "discount_pct": 33,
    "promocode": "SUMMER",
    "currency": "USD",
    "rating": 4.6,
    "rating_count": 1887,
    "url": "https://www.groupon.com/deals/gl-aquarium-of-the-pacific?redemptionLocationId=...",
    "image": "https://img.grouponcdn.com/deal/.../t2001x1212.webp"
}
```
Prices are plain dollars (the source data is in cents). Any field can come
back `null` if that particular card didn't have it (e.g. some deals have no
rating yet, some have no separate list/discount and only `sell_price`).

## City/category slugs

The URL pattern is `groupon.com/local/<city-slug>/<category-slug>`. Valid
slugs have to match Groupon's actual taxonomy - browse groupon.com yourself
to find the slug for a city/category combo you want (e.g. `things-to-do`,
`restaurants`, `beauty-spas`...). An invalid category slug doesn't error, it
just 301-redirects to the generic city landing page, which has no
`[data-item-type="card"]` elements, so you'll silently get zero deals back -
if a run returns `0 deals`, double-check the slug in a browser first before
assuming the scraper itself broke.

## Known limitations

- **Only the first page of results** (9 deals as of this writing) is
  returned. The full category page has hundreds/thousands of deals
  (`totalCount` in the embedded page data can be 900+) loaded via an
  internal "Load More" API call that fires client-side and wasn't
  reverse-engineered here - appending `?offset=9` to the URL does **not**
  work (Next.js SSR ignores it; only the client-side JS re-fetch respects an
  offset). Getting more than the first page would mean finding and calling
  that internal API directly.
- **Fragile to Groupon changing their card markup.** The `data-bhd` JSON
  shape (`body.sectionN.type`/`.content`) is read defensively (keyed off
  each section's own `type` field rather than assuming `section1` is always
  the title), so minor reordering won't break it, but a bigger redesign of
  their card component could remove or rename this attribute entirely. If
  the script starts returning 0 deals on a URL you've confirmed is correct
  in a browser, that's the first thing to check - fetch the URL yourself
  and grep for `data-item-type="card"` and `data-bhd` to see if the shape
  changed.
- Not scheduled anywhere (no cron entry) - run it manually, or wire it into
  a cron job / your own import pipeline if you want it running on a
  schedule.