When businesses need web scraping services vs official APIs and data feeds
Before building any custom data extractor, engineering teams must verify whether an official data channel already exists. Many platforms offer public APIs, partner feeds in XML, CSV, or YML formats, or structured data exports. When a business collects catalog details or marketplace pricing, an official integration is always more reliable and cheaper to maintain than parsing raw HTML.
Using an official API eliminates the risk of breakage caused by frontend redesigns, minimizes network traffic, and provides an explicit data schema. For instance, when interacting with major e-commerce platforms, the primary approach is a direct Wildberries and Ozon API integration, where catalog structures, stock levels, and price tiers are documented and supported. Similar rules apply to enterprise data exchange using integration services.
Custom web scraping services for business become justifiable only when the target resource provides no programmatic interface, partner channels are unavailable, and market intelligence is essential for commercial operations. In such scenarios, the scraper should be engineered as an external data adapter feeding standardized events into internal systems, following the principles of system integration with APIs and webhooks.
- Use an official API: when public documentation, authentication tokens, and service level agreements (SLAs) are available.
- Request partner feeds: suppliers frequently export structured catalogs via scheduled CSV, JSON, or XML feeds over HTTPS or SFTP.
- Build a custom scraper: when no public API exists, partner data is inaccessible, and public web pages contain unstructured market data.
- Decline data extraction: when site terms explicitly prohibit automated access, content is copyrighted, or pages contain personal data.
Competitor price monitoring: product matching and history storage
Developing an automated competitor price tracking scraper requires solving two engineering challenges: accurate catalog matching and time-series price storage. Collecting raw numbers from a web page is useless if the system cannot link the scraped product to your internal stock keeping unit (SKU). A robust scraper must extract stable identifiers: manufacturer part numbers, EAN-13 barcodes, brand names, and technical specifications.
In e-commerce, identical items often have divergent titles across stores. Competitor listings frequently omit barcodes, include promotional slogans, or use alternative measurement units. To match items reliably, engineers implement a cascading strategy: exact matching by barcode and part number, string normalization (lowercasing, symbol stripping, unit standardization), and attribute-based comparison for difficult edge cases.
Price history must be persisted using an append-only time-series model rather than updating records in place. Historical trend analysis requires recording timestamps, stock availability indicators, and multiple pricing tiers. Modern online retailers display base retail prices, promotional discounts, loyalty program prices, and location-dependent shipping fees. The database schema must isolate these attributes into dedicated columns.
- Price tier separation: isolating base retail prices, promotional discounts, coupon codes, and loyalty membership rates.
- Stock status tracking: capturing in-stock flags, warehouse locations, and estimated delivery dates alongside prices.
- Cascading matching: resolving SKUs sequentially via barcodes, manufacturer part numbers, normalized titles, and technical attributes.
- Tiered crawling intervals: scheduling frequent crawls for high-priority benchmark items (KVI) and infrequent checks for long-tail products.
Legal and ethical guardrails: terms of service, robots.txt, and privacy
The technical ability to fetch public web pages does not confer unrestricted legal rights to harvest commercial data. Any automated extraction pipeline must account for compliance risks before writing code. The foundational engineering principle is straightforward: extract only publicly visible data, never bypass authentication gates, and never attempt to access private account areas.
The first compliance baseline is the target site's robots.txt file and Terms of Service (ToS). Directives such as Disallow and Crawl-delay communicate server administrator rules for automated crawlers. Ignoring these directives risks server strain and IP bans. Furthermore, if a website's terms explicitly prohibit automated extraction for commercial reuse, operating scrapers may expose your business to breach-of-contract claims.
Collecting and storing personal data without explicit user consent is strictly prohibited under data privacy laws such as the European Union's General Data Protection Regulation (GDPR) and regional equivalents. Scrapers must never harvest personal names, telephone numbers, email addresses, or user profiles. In addition, proprietary catalog databases and creative descriptions may be protected by intellectual property laws.
The Anaxita development team does not provide legal advice. Before launching an automated data collection pipeline against external websites, company leadership must submit the planned data fields and extraction architecture to legal counsel to assess unfair competition and copyright risks.
- Review robots.txt: honoring disallow paths and respecting recommended crawl delays between consecutive requests.
- Analyze Terms of Service: reviewing terms to identify explicit restrictions on commercial data extraction.
- Exclude personal data: actively filtering out names, emails, phone numbers, and profile data from parsing logic.
- Prevent server strain: capping request rates to prevent performance degradation on target infrastructure.
- Seek legal counsel: validating the extraction model with qualified legal advisors prior to production deployment.
Scraper resilience: handling layout shifts, schema validation, and alerts
The fundamental vulnerability of any web scraper is the fragility of HTML markup. Unlike versioned APIs, web layouts evolve constantly: e-commerce teams run A/B experiments, adopt new CSS libraries, rename class names, and restructure DOM hierarchies. A scraper built on brittle XPath queries or deeply nested CSS selectors will fail as soon as frontend code updates.
To build durable scrapers, extraction logic must be decoupled from network fetching and database storage. Extractors should target semantic markup rather than autogenerated CSS classes like .product-card_3xZ1a. Schema.org microdata, Open Graph tags, itemprop attributes, and JSON-LD script blocks provide significantly greater stability because e-commerce platforms maintain them for search engine indexing.
Because semantic metadata can also change, production scrapers require strict schema validation and anomaly detection. If a crawler extracts zero items from a category that previously held hundreds, or if prices across the majority of cards return null, the pipeline must treat this as an extraction failure rather than a market reality.
Upon detecting schema anomalies, the pipeline must halt updates for that specific source, alert an on-call engineer via Telegram or Slack with a raw HTML dump, and preserve the original response for rapid troubleshooting.
Dynamic JavaScript pages and headless browsers: why headless is a last resort
Many modern web applications are built as Single Page Applications (SPAs) using React, Vue, or Angular. Initial HTTP requests receive an empty HTML shell, while content renders client-side via JavaScript. Inexperienced developers often default to headless browsers like Chromium via Playwright or Puppeteer to render the entire page. In commercial backend development, this is an inefficient last resort.
Full headless browsers consume massive infrastructure resources: a Chromium instance demands hundreds of megabytes of RAM and heavy CPU utilization. Data throughput drops dramatically: instead of sub-second HTTP responses, browser pages take several seconds to load scripts, web fonts, stylesheets, and third-party trackers. Maintaining browser process pools in worker services frequently leads to memory leaks and hanging instances.
In most cases, inspecting network traffic in browser developer tools reveals that the frontend application fetches clean JSON from internal REST or GraphQL endpoints. Replicating those underlying HTTP requests directly in Go using lightweight clients and standard headers is exponentially faster, cheaper, and more stable than rendering visual DOM trees.
Headless browsers should only be deployed when data extraction is strictly coupled to complex client-side calculations or encrypted tokens. When browser rendering is unavoidable, configure the instance in a stripped-down mode: completely disable images, stylesheets, custom fonts, analytics trackers, and video streams.
- Inspect network traffic: examine internal XHR and fetch requests to find raw JSON endpoints before launching headless tools.
- Optimize compute resources: a Go HTTP client processes hundreds of requests per second per core, whereas headless browsers handle single digits.
- Isolate browser processes: configure automatic worker restarts after a set number of pages to prevent browser memory leaks.
- Disable non-essential assets: blocking images, CSS stylesheets, and tracking scripts accelerates headless page rendering substantially.
Queues, scheduling, and request rate limiting in Go
Reliable data extraction cannot be executed as a monolithic, synchronous script. A production scraping architecture is structured across three decoupled layers: a job scheduler, a persistent task queue, and a pool of stateless workers. The scheduler generates crawl targets and pushes jobs into the queue, while workers pull jobs, fetch pages, parse payloads, and commit structured records.
This decoupled design ensures fault tolerance during network timeouts or process restarts. Every crawl task maintains a unique identifier and execution state. Mechanisms for building robust task pipelines with PostgreSQL and worker pools are covered in our guide on background jobs and queues in Go. When a worker encounters an error or timeout, the task is returned to the queue with an exponential retry delay.
A critical architectural component is polite rate limiting. Uncontrolled request spikes overwhelm target web servers, trigger IP rate limits, and degrade user experience for legitimate visitors. The scraping pipeline must enforce strict limits on concurrent connections and inter-request delays per target domain.
When a target server responds with HTTP 429 (Too Many Requests) or HTTP 503 (Service Unavailable), workers must immediately pause and apply backoff algorithms. A thorough breakdown of retry headers and backoff mathematics can be found in our guide to API rate limits and exponential backoff.
Data cleaning, normalization, and relational storage pipelines
Raw scraped text is invariably noisy: strings contain non-breaking spaces, currency symbols, unescaped HTML entities, line breaks, and mixed date formats. The data normalization phase transforms this unstructured input into strictly typed records before writing to analytical databases.
Prices must be stripped of currency signs and converted into integer minor units (such as cents or kopecks) to eliminate floating-point calculation errors. Measurement units (grams, liters, meters) are converted to standard SI metrics for reliable price-per-unit comparisons. Normalization pipelines should also store raw HTML or JSON payloads in JSONB columns for auditability and schema re-parsing.
The database write layer must enforce idempotent operations, as detailed in our guide on API idempotency and retries. If a product page is parsed multiple times during an interrupted crawl, the system must not create duplicate entries. This is achieved using upsert operations (INSERT ... ON CONFLICT DO UPDATE) backed by composite unique indexes. Designing clean contracts and schemas is central to custom Go backend development.
- Strict data typing: converting price fields to integers in minor currency units with explicit ISO currency codes (USD, EUR, RUB).
- Unit standardization: normalizing dimensions, weights, and volumes to universal measurement standards for direct cross-catalog comparison.
- Raw payload archiving: preserving original HTML snippets or JSON payloads in JSONB columns for debugging and historic auditing.
- Idempotent persistence: eliminating duplicate records using composite unique keys and upsert statements in relational databases.
Polite crawling, anti-bot realities, and sustainable data collection
A persistent myth in data extraction is that commercial scraping problems are solved by purchasing proxy pools and automated CAPTCHA solvers. From an engineering and financial perspective, attempting to defeat anti-bot perimeters (Cloudflare, Akamai, or custom WAFs) is an unsustainable strategy. Bypassing active protections leads to an expensive technological arms race: anti-bot vendors continually update detection heuristics, scrapers break without notice, and maintenance overhead destroys project ROI.
A sustainable data extraction approach relies on polite crawling and transparent behavior. Well-behaved scrapers provide clear User-Agent headers with contact information, respect crawl rate limits, leverage HTTP caching headers (If-Modified-Since and If-None-Match) to avoid downloading unchanged pages, and schedule intensive crawl operations during off-peak hours.
When a web platform implements aggressive bot challenges or blocks automated traffic, mature businesses view this as a clear signal to reassess their data acquisition strategy. In almost every scenario, establishing direct commercial partnerships, purchasing official API subscriptions, or licensing data feeds provides greater stability and zero operational downtime. Long-term business operations must not depend on fragile evasion techniques.