Data Engineering interview question · Most asked
How do you handle personal data (PII) in a data warehouse?
Summary
Minimise what you load, keep personal attributes in one restricted identity table, use pseudonymous keys elsewhere, tag personal columns and track lineage, apply column-level access control and masking, automate deletion requests, and set retention periods for snapshots, exports and backups.
Longer Answer
Under the GDPR, pseudonymised data is still personal data, while properly anonymised data is not. Keyed hashes (HMAC with a secret) are safer pseudonyms than plain hashes, which can be reversed by dictionary attack.
Key Points
- Minimise and separate identity.
- Pseudonymous keys (HMAC).
- Tag, mask and restrict access.
- Automated deletion and retention.
Deep dive: ETL and Data Pipelines: Personal Data, Privacy and Retention
Explanation
What interviewers look for
Design-level answers, not just 'encrypt it'.
More most-asked Data Engineering questions
- Batch or streaming: how do you decide?
- What is the difference between ETL and ELT, and when would you choose each?
- What is change data capture (CDC), and why is log-based CDC preferred over polling?
- What does it mean for a data pipeline to be idempotent, and how do you achieve it?
- How do you implement incremental extraction, and what can go wrong?
- How do you deduplicate events, and what does 'exactly-once' really mean?
- What is a star schema, and why is it used in data warehouses?
- What is the grain of a table, and what is join fan-out?