Data engineering & analytics
The full data path — from tracking and ingestion pipelines through to the dashboards and reports that turn raw events into decisions, built on firsthand experience running real-time analytics at billions of events a month.
Last updated:Data engineering is the practice of building the pipelines and storage that move raw events into a structured, queryable form for analysis. Revenant Systems builds the full data path — tracking and ingestion through to warehousing, dashboards, and reporting — using Python, PostgreSQL, ClickHouse, DuckDB, and Airflow.
What does the data path include?
The data path is the route raw events take from source to decision: capture, ingestion, storage, transformation, and reporting. A complete data path makes the same numbers reproducible across dashboards and reports, so teams can trust what they measure rather than reconciling conflicting figures.
Revenant Systems builds each stage — ingestion in Python and Airflow, storage in PostgreSQL, ClickHouse, or DuckDB, and reporting on top — as one pipeline rather than disconnected scripts, drawing on the founder's firsthand experience running ClickHouse through hundreds of billions of rows.
- Event tracking and ingestion
- Warehousing and data modelling
- Dashboards and reporting
- Scheduling and orchestration (Airflow)
Which databases does Revenant use, and when?
Revenant Systems selects the data store by workload. PostgreSQL handles transactional and general-purpose data; ClickHouse handles high-volume analytical queries; DuckDB handles fast in-process analytics and local transformation. Matching the engine to the workload keeps queries fast and infrastructure simple.
| Engine | Workload | Chosen when |
|---|---|---|
| PostgreSQL | Transactional and general purpose | The default store for application data |
| ClickHouse | Large-scale analytics | Analytical volume and cardinality outgrow the transactional store |
| DuckDB | In-process analytics | Fast local analysis and transformation without a server |
| Airflow | Orchestration | Pipelines need scheduling, dependencies, and visibility |
What's included
- Tracking and ingestion pipelines
- Data warehouse design
- Transformation and data modelling
- Dashboards and reporting
Stack Python · PostgreSQL · ClickHouse · DuckDB · Data Warehousing · Airflow
Frequently asked questions
Our dashboards and reports disagree — why?
Usually because the numbers travel different paths: separate scripts, exports, and hand-maintained queries that each transform the data slightly differently. A complete data path — capture, ingestion, storage, transformation, reporting — makes the same numbers reproducible everywhere, so teams trust what they measure instead of reconciling conflicting figures.
Can you build the event tracking as well as the warehouse?
Yes — the engagement covers the full path, from event tracking and ingestion through warehousing and data modelling to the dashboards and reports on top. Building the stages as one pipeline, rather than disconnected scripts, is the point of the service.
What scale does this experience come from?
The founder's prior roles: real-time analytics platforms ingesting billions of events a month, and ClickHouse warehouses that have worked through hundreds of billions of high-cardinality rows. Smaller estates get the simplest pipeline that stays correct — the engine is matched to the workload, not to a reference architecture.
Do we need ClickHouse, or will PostgreSQL do?
The split is clean: PostgreSQL for transactional and general-purpose data, ClickHouse where analytical query volume and cardinality outgrow it, DuckDB for fast in-process analytics. Matching the engine to the workload keeps queries fast and infrastructure simple — and the audit stage settles the choice with evidence rather than fashion.
Our existing PostgreSQL is slow — can you help?
Yes — that is a fixed-scope package: the PostgreSQL performance audit examines query load, indexing, autovacuum and bloat, configuration, and storage overhead, and delivers a severity-ranked report with a remediation roadmap. The same audit exists for ClickHouse. Either is a common starting point for wider data work.
Can you replace reporting that lives in a spreadsheet?
Yes, and the safe route starts with understanding it: the Excel workbook audit maps what the workbook actually does before pipelines and dashboards replace it. Reporting logic that grew up in a spreadsheet usually contains business rules nobody has written down anywhere else.
Our reporting still runs on manual exports — is that fixable?
Yes — manual exports and copy-paste steps are usually the symptom of a missing pipeline stage, and they are where numbers quietly diverge. Replacing them with automated ingestion and transformation makes the reporting reproducible and removes the person-shaped single point of failure in the process.
Every engagement follows the same process — see how we work.