mhedhbihamza11@gmail.com+216 93 377 688WhatsApp
← All work

PROJECT 02

CleverTap Integration

Eight systems. Five hotels. Two parks. One reliable customer view.

  • Python
  • ETL
  • CleverTap
  • Docker
  • Data Quality
SRC 01SRC 02SRC 03SRC 04SRC 05SRC 06SRC 07SRC 08NORMALISEMD5 LEDGERCleverTapONE VIEWHOURLY
Simplified architecture diagram.

01  Context

Customer data was spread across eight disconnected systems serving five hotels and two parks. Without a single view, the same guest could appear many times — or not at all — in the CRM. I unified these sources into one CleverTap customer view.

02  Challenges

  • Eight sources with different schemas, date formats and missing values.
  • Duplicate records across and within systems.
  • Phone numbers in many local formats.
  • The pipeline had to run continuously without supervision.

03  My role

Built the multi-source ETL pipeline, normalisation and deduplication logic, and its deployment on an Ubuntu VPS.

04  Decisions

Hash-ledger deduplication

Each record is fingerprinted with an MD5 hash and stored in a ledger, so unchanged records are skipped on every hourly run and duplicates aren't re-sent.

Normalise before you merge

Dates, null values and phone numbers (to E.164) are normalised first, so matching works on consistent data rather than on luck.

Hourly, not real-time

Hourly ingestion kept the system simple and robust while staying fresh enough for CRM use.

05  Architecture

  • Source connectors for eight independent systems.
  • Normalisation layer: dates, nulls, E.164 phone numbers.
  • MD5 hash ledger for change detection and deduplication.
  • Upload to CleverTap; scheduled hourly, running continuously on an Ubuntu VPS with Docker.

06  Results

  • Achieved 98–100% record coverage through data validation and integration logic.

NEXT PROJECT · 03

OCR Pipeline Upgrade

→