Legal and ethical basics of collecting web data for research
Three things determine most of the risk in collecting web data: whether the site's terms of service prohibit automated access, whether you respect its robots.txt signals, and whether any of the data you collect counts as personal data under rules like the GDPR. This is general information, not legal advice — treat it as a starting checklist, not a substitute for counsel on a specific project.
The short answer
Before collecting data from a website for research, check the terms of service for language prohibiting automated access or scraping, check and respect robots.txt as a baseline courtesy even though it is not legally binding by itself, avoid collecting personal data you do not need, and keep in mind that "publicly accessible" is not the same as "free of all legal risk." This is general information, not legal advice.
Terms of service: the most common source of risk
Most consequential web-scraping disputes in the US have turned on breach of contract or breach of a website's terms of service, rather than on a criminal anti-hacking statute. That distinction matters: even where scraping publicly available pages does not violate a computer-crime law, a site's terms of service can still create civil liability if you agreed to them (for example by creating an account) and then violated a clause against automated collection. Before collecting data at any scale, read the target site's terms of service specifically for language about "automated access," "scraping," "bots" or "data mining," and note whether those terms apply only to registered users or to anyone visiting the site.
robots.txt: a convention, not a law
robots.txt is a plain-text file that site operators place at the root of their domain to tell automated crawlers which parts of the site they would prefer not be crawled. Google's own documentation on how it interprets the robots.txt specification describes it as a set of rules — user-agent, disallow, allow — that well-behaved crawlers are expected to follow voluntarily. Nothing about the mechanism itself is legally binding; it is a technical courtesy that responsible crawlers, including major search engines, choose to respect. For research collection, treat robots.txt as a floor, not a ceiling: respecting it is standard good practice and reduces the chance that ignoring an explicit signal gets treated as evidence of bad faith in a later dispute, even though the file alone does not create or remove legal rights.
What hiQ v. LinkedIn actually decided
The hiQ Labs v. LinkedIn litigation is the case most often cited in scraping-legality discussions, and it is frequently oversimplified. In April 2022 the Ninth Circuit held, for a second time, that scraping data from publicly accessible web pages that do not require an account to view does not by itself violate the US Computer Fraud and Abuse Act's prohibition on accessing a computer "without authorization." That part of the ruling is genuinely significant. But the case did not end there: in November 2022, the parties settled, with hiQ agreeing to a permanent injunction to stop scraping LinkedIn and to delete the data, code and algorithms it had built from the scraped data, after a separate ruling found hiQ had breached LinkedIn's user agreement. The takeaway is narrower than "scraping is legal": a specific anti-hacking statute may not apply to public pages, but a contract you agreed to can still bind you regardless.
Personal data: apply extra care by default
If any of the data you plan to collect could identify a real person — names, usernames, email addresses, photos, or combinations of otherwise anonymous fields — treat it as personal data and apply the stricter standard even if you are not certain a specific regulation applies. The EU's General Data Protection Regulation, in force since 2018 and available in full at EUR-Lex, defines personal data broadly as any information relating to an identified or identifiable natural person, and applies its protections to processing that data, not only to its initial collection. For research, that means minimizing what you collect to what the project actually needs, avoiding retention of identifying fields you will not use, and being able to explain the legal basis and purpose for any personal data you do keep.
Pre-collection checklist
| Check | Why it matters |
|---|---|
| Read the site's terms of service for automated-access clauses | Most scraping disputes turn on contract, not a computer-crime statute |
| Check and respect robots.txt | Voluntary convention; not legally binding by itself, but standard good practice |
| Confirm whether pages require login | Authorization analysis differs sharply between public and account-gated pages |
| Screen fields for personal data | Names, usernames, emails and combined identifiers may trigger GDPR-style obligations |
| Minimize and document your purpose | Keep only what the research needs; be able to explain why you kept it |
FAQ
Is web scraping for research legal?
Collecting publicly accessible web data for research is generally lower-risk than doing so for a commercial product, and the 2022 hiQ v. LinkedIn ruling found that scraping publicly accessible pages does not by itself violate the US Computer Fraud and Abuse Act. But the case also ended in a settlement in which hiQ agreed to stop scraping LinkedIn and pay damages for breaching LinkedIn's user agreement, so a site's terms of service can still create contractual liability even when a specific anti-hacking statute does not apply. This is general information, not legal advice.
Does robots.txt have any legal force?
By itself, robots.txt is a voluntary technical convention that tells automated crawlers which paths a site would prefer they not access; it is not a law and does not by itself create a binding legal obligation. That said, ignoring it can be used as evidence of intent in a broader legal dispute, and reputable research crawlers respect it as a baseline of good practice regardless of its legal status.
What counts as personal data when I am scraping web pages for research?
Under the EU's General Data Protection Regulation, personal data is broadly defined as any information relating to an identified or identifiable natural person, which can include names, usernames, email addresses, and even indirect identifiers when combined. If your collection touches EU residents' data, or you are otherwise subject to comparable rules, treat any field that could identify a real person as personal data by default and apply minimization and a lawful basis for processing it.
This is general information, not legal advice. If a data collection plan carries real legal exposure, confirm the specific approach with qualified counsel before you start.
Need a dataset without the collection risk?
DeepSData can run a real availability search and verify sources for you — finding datasets that already exist through vetted, licensed channels, so you are not building your own scraper and taking on that risk yourself.