An AI-powered compliance checker that reads a lease clause by clause against 73 real legal requirements, citing its source for every finding.
Motivation & what it does
This one is personal, not hypothetical: I've been a landlord in Pittsburgh myself, and I've felt firsthand how landlords here piece together lease compliance from scattered blog posts, outdated templates, and guesswork, and still come up short. Pittsburgh landlords answer to four overlapping layers of law at once: federal Fair Housing and lead-disclosure rules, Pennsylvania's Landlord and Tenant Act, Allegheny County's health code, and City of Pittsburgh ordinances. That's 70+ real requirements, and no single tool checked a lease against all four. Having gone through it myself, I wanted to solve it properly, using LLMs.
Pittsburgh landlords answer to four overlapping legal layers, federal, Pennsylvania state, county, and city, with more than 70 real requirements and no single tool that checks a lease against all of them. Pitt-Lords does: upload a lease, get it checked clause by clause against 73 hand-sourced requirements, every finding backed by a clickable citation to the primary government source.
- Checks an uploaded lease clause by clause against 73 hand-sourced requirements.
- Covers all four overlapping jurisdictions: federal, Pennsylvania, county, and city.
- Backs every finding with a clickable citation to the primary government source.
- Rejects any finding whose citation wasn't actually part of what retrieval returned.
System architecture
Storage
PostgreSQL with the pgvector extension, deliberately avoiding a separate vector database so the relational and semantic data live in one place.
Four-agent pipeline
A parser agent matches lease clauses to requirements, a diff agent renders the compliance verdict, a verifier agent double-checks that reasoning using a second, independent model (GPT-4o-mini via LiteLLM, not Claude again) specifically to catch a single model's own blind spots, and a report builder assembles the final output with zero additional LLM calls.
No LangChain
The pipeline is deliberately hand-rolled rather than framework-driven, which kept the agent handoffs and failure modes fully visible while debugging the retrieval regression below.
Guardrails
Built-in checks reject any finding that cites a source the retrieval step didn't actually surface, closing off the most common way these tools hallucinate legal authority.
Ambiguity handling
Background law a landlord should simply be aware of is surfaced as its own category, separate from verdicts, never forced into a false compliant or non-compliant call it can't actually support.
Deployment
Next.js on Vercel for the frontend, FastAPI and PostgreSQL on Railway for the backend, with an async background-task pattern and live status polling for long-running lease analyses.
Decisions & tradeoffs
The eval harness that caught a real regression
I built a synthetic test suite with ground-truth answers, weighted so a missed violation scores far worse than a false flag, since a missed real violation and an unnecessary flag aren't equally bad. Partway through the build, the harness surfaced a real gap: short, sparse leases sometimes starved certain requirements of any retrieval signal, so real clauses got missed. The obvious fix was to add a second retrieval direction, letting each requirement also search for its own best-matching text. It backfired: with no relevance floor, it started attaching a closest-available chunk to every requirement regardless of actual relevance, diluting results that were already correct. I tried scoping it down, then tuning a similarity threshold. Two more rounds, two more partial fixes, each trading one failure mode for another, and the eval suite kept catching it every time. I reverted to the simpler, proven retrieval approach and documented why in the commit history, rather than keep patching under pressure. An eval suite that only confirms your instincts isn't doing its job.
Why four narrow agents instead of one large one
Splitting parsing, verdict rendering, verification, and report assembly into separate steps meant each agent's failure mode was isolated and debuggable on its own, and the final report builder needed zero extra LLM calls, which kept both cost and hallucination surface area down at the one step where the user actually reads the output.
Why citations are a hard guardrail, not a prompt instruction
Telling a model 'always cite your source' in the prompt is not the same as guaranteeing it. The verifier step structurally rejects any output whose citation wasn't part of what retrieval actually returned, so an invented citation is a pipeline failure, not a policy the model can quietly ignore under pressure.
Concepts, skills & tools
AI & ML infrastructure
Frameworks & tools
Concepts & patterns
Screenshots
Upload a lease and see how the review works before you start
Live status while a second AI model double-checks every finding
The final compliance report, with a citation for every finding