This presentation by Yaroslav Harahuts offers a technical and organizational overview of building and operating large-scale data leak databases, using his project focusing on the personal data of Russian citizens as a case study. Harahuts details the entire workflow, from sourcing data via public leaks on Telegram and hacker forums to processing and storage. The technical architecture relies on the "Follow the Money" ontology for data modeling, a custom processing pipeline, a single-server database leveraging ZFS for efficiency, and Manticore Search for full-text search capabilities.
A significant portion of the talk is dedicated to the legal and ethical dimensions of the project. The speaker candidly admits the work is "super unethical" and likely illegal in most jurisdictions, including Ukraine. He explains that the database contains highly sensitive information on both military and civilian individuals. However, he justifies these actions by the context of the ongoing war, arguing that any potential benefit to investigators and journalists outweighs the ethical concerns and individual harm. The database is accessible to approved users, such as law enforcement and journalists, through both free and paid models.
Building and Operating a Large-Scale Data Leak Database
0:42 Project Overview: The speaker, Yaroslav Harahuts, introduces his project: a large-scale database of leaked personal data of Russian citizens. It is used by journalists, military personnel, and law enforcement for investigations.
1:58 Data Sourcing: Data is collected from publicly available sources like hacker group publications and Telegram channels. A key principle is that data is never purchased.
3:14 Data Modeling: The system uses the "Follow the Money" (FtM) ontology to structure the data. This allows each piece of information (a "statement") to be stored with rich metadata, such as its source and acquisition date.
4:26 Processing Pipeline: A custom-built pipeline is used to process each dataset. It follows rules defined in a mapping file to clean, normalize, and ingest the varied data formats.
5:36 Technical Infrastructure: All data is stored in a single database on one server. Queries are performed using simple key-based indexes for speed and efficiency.
6:31 Filesystem and Search: The ZFS filesystem is used for its compression capabilities, which saves disk space and accelerates database operations. For full-text search, the system utilizes Manticore Search.
8:09 Legal Status: Harahuts states that such activities are considered illegal in Ukraine and most other jurisdictions, and he explicitly advises the audience not to replicate his work.
8:52 Ethical Considerations: The speaker acknowledges the database contains highly sensitive personal information, including medical records, of both ordinary civilians and military personnel.
9:38 Access Model: Both free and paid access to the database is provided, primarily for journalists and law enforcement.
10:06 Ethical Justification: Despite admitting the project is "super unethical," Harahuts justifies it within the context of the war. He states that as long as the work provides some benefit, he does not focus on the harm it may cause to individuals.
15:07 Deletion Policy: When asked about requests for data removal, he confirms that none have been received and that any such requests would be ignored.