Using AI to Extract Structure from Messy Websites

Websites are databases that forgot to be databases. The information is there: pricing, amenities, hours, staff qualifications, service descriptions. But it is trapped in marketing copy, buried in paragraph text, scattered across pages, and formatted differently on every site.

For any project that needs structured data from the web, this is the fundamental problem. The information exists. It just was not meant to be read by machines.

Why traditional scraping is not enough

A web scraper can reliably extract text from a page. With enough effort, you can write selectors for specific elements, handle pagination, and deal with JavaScript rendering. For sites with consistent structure: job boards, e-commerce platforms, government databases. This works well enough.

The difficulty arises when every site is different. We ran into this building a senior care search platform. Facilities have websites with rich information about their amenities, dining programs, staffing, activities, pricing, and specialties. But there is no standard format. One facility lists amenities in bullet points. Another embeds them in a paragraph about "our campus." A third puts everything in a PDF.

No amount of CSS selectors can extract "has a swimming pool" from a paragraph that reads "residents enjoy our heated aquatic center." That requires understanding language, not parsing HTML.

The extraction pipeline

We built a pipeline that turns unstructured website content into validated, structured data:

Scrape. Fetch the facility's website pages: homepage, amenities, dining, services. Standard HTTP requests with HTML parsing.

Clean. Strip everything that is not content. Cookie consent banners, phone number CTAs, navigation menus, footer boilerplate, social media widgets. This noise confuses the model and wastes context window space.

Truncate and organize. Trimmed to roughly 8,000 characters, organized by section. The amenities page matters more than the careers page.

Extract. The prepared content goes to Gemma 3 27B via Ollama, with temperature set to 0.1. The prompt asks the model to extract structured data across nine categories. JSON mode is enforced so the output is always valid JSON.

Validate. Extracted JSON gets validated against Zod schemas, one per category, defining expected fields, types, and enum values.

Store. Validated data goes into the database.

Lenient validation as a feature

Schema validation is where most extraction pipelines get too strict and break.

A rigid validator says: the amenities field must contain only values from this enum. If the model returns "swimming_pool" but the enum expects "pool," the validator rejects the response. The entire extraction fails because the model used a synonym.

We took the opposite approach. The validator filters out values that do not match the enum but keeps everything that does. If the model returns five amenities and two do not match, we keep the three that do. The record is incomplete rather than empty.

This reflects a simple reality about LLM outputs: they are approximately correct most of the time. A pipeline that demands perfection will reject most of what it gets. A pipeline that accepts partial results will accumulate useful data from every run.

Prompt discipline: extract, do not infer

The prompt explicitly instructs the model to extract only information that is directly stated on the website. If the site does not mention memory care, the extraction should not include it, even if the facility type makes it statistically likely.

This matters because LLMs are trained to be helpful, and "helpful" often means filling in gaps with plausible information. But inference is not extraction. We need the data to reflect what the facility actually says about itself.

The prompt reinforcement is specific:

• Extract only explicitly stated information
• Do not infer capabilities from context
• Return null where the website provides no relevant content
• Distinguish between "not mentioned" and "not available"

This discipline costs some recall. Facilities that offer services they do not mention on their website will have incomplete records. But the data we do capture is trustworthy, which matters more when families are making care decisions based on it.

Temperature and determinism

We run extraction at temperature 0.1, nearly deterministic. The same content through the pipeline twice should produce substantially the same output.

Higher temperature introduces variation that is useful for creative tasks but harmful for extraction. If the model sometimes returns "swimming pool" and sometimes "aquatic center" for the same facility, downstream systems cannot reliably deduplicate or compare.

Low temperature also makes debugging easier. When an extraction produces unexpected results, we can re-run it and get the same output, which means the problem is in the content or the prompt, not in random variation.

Caching expensive operations

Each extraction takes several seconds of model inference time. Multiply by hundreds of facilities and the pipeline takes hours.

The caching strategy is tiered:

• Successful extraction: cached for 30 days
• Partial extraction: cached for 14 days, worth retrying sooner
• Failed extraction: cached for 7 days, something went wrong, retry relatively soon

When a facility's website changes, the cache expires naturally and the next pipeline run captures the new information.

Respecting human edits

Some facility providers log into the platform and manually update their information. The pipeline never overwrites these manual edits. Human-provided data takes precedence over extracted data.

This is a small implementation detail with large trust implications. A provider who corrects their listing and sees the correction overwritten by a bot will not correct it again.

Where this pattern applies

The specific domain is incidental. The pattern works anywhere that useful information exists on the web in unstructured form:

• Real estate listings where property features are described in prose
• Restaurant menus that exist as images or styled HTML
• Product specifications scattered across manufacturer sites
• Job postings on company career pages with no standard schema
• Local business directories with hours and services in free text

The pipeline stages are the same: scrape, clean, extract, validate leniently, cache, and respect human overrides. The schemas and prompts change. The architecture does not.

The LLM is not doing anything magical. It is doing what a human data entry person would do, reading a page, understanding what it says, and filling in a form. It just does it at scale, consistently, and without getting bored. The discipline is in treating it like a data entry process: define the schema precisely, validate the output rigorously, accept partial results gracefully, and never confuse extraction with inference.