Note: The job is a remote job and is open to candidates in USA. Diverse Lynx is a company seeking an AWS Data Engineer to support their data engineering initiatives. The role involves developing and managing data pipelines, implementing event-driven architectures, and optimizing cloud infrastructure on AWS.
Responsibilities
- Develop a comprehensive plan for migrating near real-time fraud detection campaigns from on-premises systems to AWS
- Design and implement event-driven architectures to process inbound dialer data (fraud events) using services such as Amazon EventBridge, Kafka, Kinesis Data Streams, and Kinesis Firehose
- Build and manage scalable data pipelines using AWS Glue (ETL jobs, Crawlers), PySpark, and Python for data ingestion, transformation, and processing
- Configure and manage Glue Crawlers to automatically Client schemas and update the Data Catalog
- Store and optimize data using Parquet format and enable analytics through Amazon Athena for efficient querying
- Develop integrations between Customer Profiles and messaging platforms to automatically trigger profile updates and downstream processes
- Implement automation to trigger fraud-related outbound calls based on updates in customer profiles
- Design and orchestrate workflows using AWS Step Functions to manage complex processing pipelines
- Provision and manage cloud infrastructure using Terraform (Infrastructure as Code)
- Optimize system architecture for scalability, reliability, cost-efficiency, and ensure data integrity and security
- Conduct end-to-end testing of the entire framework to validate functionality, performance, and reliability
- Deploy, automate, and manage resources using CI/CD pipelines
- Continuously monitor system performance and implement optimizations post-deployment
- Maintain detailed documentation of architecture, workflows, and operational processes
Skills
- Strong expertise in AWS services including: Lambda, S3, EventBridge, Kinesis (Data Streams & Firehose), Glue (ETL + Crawlers), Step Functions, Amazon Connect, Athena, Macie
- Proficient in Python
- Proficient in PySpark
- Experience in Glue Crawlers for schema discovery and cataloging
- Experience in Parquet-based data storage
- Experience in building scalable data pipelines
- Strong understanding of event-driven architectures (Pub/Sub model)
- Hands-on experience with Terraform (Infrastructure as Code)
- Familiarity with CI/CD tools and automation pipelines
- Experience with messaging platforms like Kafka and Amazon EventBridge
- Experience in designing real-time data processing systems
- Experience in using Glue Crawlers + Athena for data lake architectures
Benefits
- Benefits
Company Overview
Company H1B Sponsorship