Note: The job is a remote job and is open to candidates in USA. Celebal Technologies is seeking a highly skilled and security-conscious Senior Databricks Data Engineer to design, build, and optimize their enterprise cloud data platform. The role involves leading the implementation of modern data pipelines within AWS GovCloud and migrating legacy metastores to Databricks Unity Catalog, ensuring compliance with federal security frameworks.
Responsibilities
- Design & Build: Develop robust, scalable, and highly secure ETL/ELT pipelines using PySpark, Delta Lake, and Databricks Workflows
- Implement optimal ingestion strategies utilizing Databricks Auto Loader, Structured Streaming, and Delta Live Tables (DLT)
- Optimize compute clusters, Photon engine usage, and Delta Lake layouts (Z-Order, Liquid Clustering) to balance processing efficiency with cloud expenditures
- Define and implement attribute-based and role-based access control (ABAC/RBAC) utilizing Unity Catalog schemas, catalogs, and external storage locations
- Build dynamic row-level filtering and column-level masking rules to safeguard personally identifiable information (PII) and highly sensitive federal data
- Configure and maintain end-to-end column-level data lineage and audit logging to guarantee traceability for strict compliance audits
- Architect and interact with Databricks Workspaces deployed on AWS GovCloud (US) , respecting boundaries defined by FedRAMP High and DoD IL5
- Manage data movement ensuring all backend and frontend data connections route exclusively through AWS PrivateLink, AWS VPCs, and secure endpoints
- Integrate Databricks with AWS services safely by provisioning IAM Roles, AWS Secrets Manager, and vended storage credentials through Unity Catalog
Skills
- 5+ years of dedicated Data Engineering experience, with at least 2+ years architecting enterprise production environments on Databricks
- In-depth understanding of Delta Lake architecture, Unity Catalog administration, Databricks SQL Warehouses, and Serverless compute configurations
- Hands-on experience working within AWS environments (EC2, S3, IAM, CloudWatch, Secrets Manager). GovCloud experience is highly preferred
- Expert proficiency in Python (PySpark) and advanced SQL
- Familiarity with Git integration inside Databricks (Databricks Git Folders/Repos) and infrastructure-as-code tools like Terraform for deploying workspaces
- Solid understanding of security protocols, data encryption (at rest and in transit via hardware-implemented AWS Nitro instances), and federal data handling guidelines
- Databricks Certified Data Engineer Professional
- Databricks Certified Platform Administrator
- AWS Certified Data Engineer – Associate / Professional
Company Overview