Web scraping and data integration

A parser is a program that collects data where people otherwise search by hand every day: tender and auction platforms, registries, competitor and supplier sites, PDF documents. The data is then split into the fields you need and delivered where it is convenient to work with. RandomStar’s own service LotRadar collects tens of thousands of Prozorro.Sale lots this way — so your task gets proven solutions.

What can be automated

• Tenders and auctions. New procurements, lots and their changes matching your filter — at once, not the next morning.
• Competitor prices and stock. A daily table of who charges what and what has gone off sale.
• Registries and open data. Government and industry databases, listings, catalogues — with a change history.
• Supplier catalogues. Prices, stock and product descriptions from supplier files or sites — straight into your online shop.
• Documents. Invoices, waybills and acts in PDF: seller, buyer, line items and totals reach the database with no manual typing.
• Exchange between systems. Your site, CRM, ERP (including 1C) and marketplaces exchange orders, products and statuses via APIs or exchange files.

Already in production

• LotRadar. Every 5 minutes the collector fetches changes from the official Prozorro.Sale API: the database holds about 36,000 lots. The service works out what changed — price, dates, cancellation, result — and delivers it to Telegram, email, Excel, an API and webhooks. Details are in the LotRadar case.
• E-Rechnung Pro. The converter reads an ordinary PDF invoice, finds the seller, buyer, line items and totals and builds an e-invoice from them. It passes 91 of 93 real invoices with different layouts. Details are in the E-Rechnung Pro case.

How it works

• API first, pages second. If a source has an open API or an export, that is used — it is more reliable. Page scraping is for when there is no other way.
• Only what is new. The parser remembers where it stopped and fetches only changes; duplicates are dropped.
• Change history. A source shows only the current state, so changes — a new price, a moved date, a removed item — are derived by comparing with the previous version.
• Gentle on the source. Pauses between requests and a sensible frequency — the parser does not load someone else’s site and does not get blocked.
• Monitoring. The admin panel shows whether each source is working, silent or failing — you learn about a failure from the system, not from customers.
• Tested on samples. For document parsing a base of real files is collected and fully re-run after every change.

Where the data goes

To Telegram and email — notifications and daily summaries; to Excel and CSV — tables; to your database, CRM or online shop — directly; to other programs — via an API and signed webhooks.

What is not taken on

Collecting personal data, bypassing passwords, CAPTCHAs or other protection, and scraping that breaks a source site’s rules or overloads it. In LotRadar, data on private individuals is removed before anything reaches the database — and the same goes for your project.

How the work goes

1. Sources and fields. Where to collect from, which data exactly, how often and where to deliver it.
2. Estimate. Stages with the cost and timing of each; you pay for an accepted result.
3. Trial run. You get a data sample and check that exactly the right data is collected.
4. Launch. The parser runs on a schedule on your server; the code and access are handed over to you.
5. Maintenance. Source sites change — the parser is adjusted to them as part of support.

How much it costs

The price depends on the number of sources, whether they have an API, how often data is collected and where it goes. That is why there is no “from $25” price list — after a request and a short call you get an estimate for your task.

Frequently asked questions

Is this legal?

Only open data and public pages are collected — under the source’s rules, without personal data and without bypassing protection; that is how all of RandomStar’s parsers work. If there is any doubt about a particular source, it is discussed before work starts.

What happens when the source site changes?

Source monitoring spots the failure at once, and the fix is part of maintenance. That is why support after launch matters especially for parsers.

How often can the data be updated?

From every few minutes to once a day — depending on the source and your task. LotRadar, for example, checks for changes every 5 minutes.

Can the data go to 1C or a CRM?

Yes: via their API or a scheduled exchange file — products, prices, stock, orders.

Discuss data collection