Data leak scanner

Context

An Italian fintech working in cybersecurity. Its clients are banks, and every bank has customers whose personal data (name, email address, phone number) can end up somewhere it should not.

The business needed a tool that searches the open web for that data: any page or file that opens without logging in, whether HTML, PDF, Excel or another office document. When a customer’s data shows up, the bank should know where.

Problem

The first version ran three services at the same time, all the time. It was hard to debug, hard to test, and it did a lot of work that never produced an alert.

The volume made every wasted step expensive: the system checked more than a million URLs a day, and every search query and every page fetch costs money, in EC2 time and at the search provider.

Architecture

A backend-only system on AWS, written in Node.js: the services run on EC2, and PostgreSQL on EBS is the single source of truth. Five services, each with one job:

1. Query builder takes the customers’ data and builds search queries from combinations of it: name with email, name with phone number, and so on.
2. Search sends the queries to a web search API and stores the URLs it returns.
3. Blacklist drops URLs that are known not to matter before anything is downloaded. Some are excluded already in the search request, the rest by a query against tens of millions of stored URLs.
4. Source scan opens each remaining page or file and looks for the customer’s data. A match counts only if the pieces sit close together: a first name on line one and a surname on line three thousand are a coincidence, not a leak. A real match becomes an alert.
5. Orchestrator runs the stages in order instead of all at once.

A separate service turns alerts into PDF reports and notifies the banks.

Distributed, and parallel inside each machine

JavaScript runs on one thread per process. Waiting for the network is cheap in Node.js, but parsing a PDF or an Excel file, or checking how close the pieces of a name sit on a page, blocks that thread, and everything else on it waits.

So the work is split twice. Across machines: several EC2 instances, all reading and writing the same PostgreSQL database. Inside each machine: every instance starts a pool of worker threads (the workerpool library), each with its own event loop, so a slow page in one worker does not hold up the others. In production that was three instances with ten workers each; more of either is a configuration change.

• Pool size per stage. Each stage has its own minimum and maximum number of workers, set in configuration. The source scan is the bottleneck and gets the most, split between workers that render HTML pages and workers that download files; search runs few, so the search API never hits its rate limit.
• No shared memory. Workers talk only through the database. A worker claims a batch of rows, processes it and writes the result; adding a machine or a worker needs no other change.
• Restart, not crash. If a pool fails, it is shut down and started again after a delay, a limited number of times.

Pages are opened with Playwright, which the EC2 environment did not support out of the box. Custom setup scripts made it run.

Decisions & trade-offs

1. Orchestrate the stages instead of running everything at once

   The original services ran in parallel all the time. An orchestrator running them in order made the system easier to maintain, debug and test.

   TRADE-OFF Each stage waits for the one before it, so a batch takes longer end to end

2. Filter before fetching

   Fetching and scanning a page is the most expensive step, so the cheapest page is the one never opened. The blacklist service removed about 60% of the work that never led to an alert, counted as the rows in the database marked as blacklisted.

   TRADE-OFF A URL on the list is never opened, so a wrong entry means a leak there goes unseen

3. Many machines, many workers, no duplicate work

   Parallel workers on several machines made the volume possible, and made it necessary to guarantee that two workers never process the same data at the same time. Every stage keeps its work items in a PostgreSQL table, and a worker claims a batch in a single statement with FOR UPDATE SKIP LOCKED: rows taken by one worker are invisible to the others, and nothing is lost when a machine restarts.

   TRADE-OFF More moving parts than a single process

4. Algorithmic optimisation where volume multiplies it

   First filter the input, then measure where the time goes, then remove the calculations that repeat. At tens of millions of operations, doing two operations instead of ten is a cost decision, not a style choice.

   TRADE-OFF Faster code is often harder to read than the obvious version

5. Choosing a search provider by test, then by trust

   When the search API the system used became more than ten times more expensive per query, several thousand identical queries were run against three providers and the results compared. The business chose a well-known, trusted provider even though its results were not the best in the test.

   TRADE-OFF Not the best results of the three

Results

The business received monthly reports, generated by scripts written for that purpose, showing what the system found and how it performed.

At the same company I also worked on a separate system: alerting jobs over hundreds of terabytes in Athena. How those queries went from hitting the 240-minute limit to about 30 minutes is in a post of its own.

Stack

• Node.js
• Express
• workerpool
• Playwright
• PostgreSQL
• AWS EC2
• AWS EBS
• Web search APIs