PROJECT 02
CleverTap Integration
Eight systems. Five hotels. Two parks. One reliable customer view.
- Python
- ETL
- CleverTap
- Docker
- Data Quality
01 Context
Customer data was spread across eight disconnected systems serving five hotels and two parks. Without a single view, the same guest could appear many times — or not at all — in the CRM. I unified these sources into one CleverTap customer view.
02 Challenges
- Eight sources with different schemas, date formats and missing values.
- Duplicate records across and within systems.
- Phone numbers in many local formats.
- The pipeline had to run continuously without supervision.
03 My role
Built the multi-source ETL pipeline, normalisation and deduplication logic, and its deployment on an Ubuntu VPS.
04 Decisions
Hash-ledger deduplication
Each record is fingerprinted with an MD5 hash and stored in a ledger, so unchanged records are skipped on every hourly run and duplicates aren't re-sent.
Normalise before you merge
Dates, null values and phone numbers (to E.164) are normalised first, so matching works on consistent data rather than on luck.
Hourly, not real-time
Hourly ingestion kept the system simple and robust while staying fresh enough for CRM use.
05 Architecture
- Source connectors for eight independent systems.
- Normalisation layer: dates, nulls, E.164 phone numbers.
- MD5 hash ledger for change detection and deduplication.
- Upload to CleverTap; scheduled hourly, running continuously on an Ubuntu VPS with Docker.
06 Results
- Achieved 98–100% record coverage through data validation and integration logic.
NEXT PROJECT · 03
OCR Pipeline Upgrade