Incept Data Solutions, Inc | Jobs | Sr . Data Engineer | BigDataKB.com | 24-02-22

โ€”

by

Job Location: remote

Responsibilities:

  • Architecting, building, and maintaining modern, scalable data architectures in the cloud preferably AWS.
  • Clean, transform, and analyze vast amounts of raw data from various systems using Spark to provide ready-to-use data to our feature developers and business analysts.
  • Involves both ad-hoc requests as well as data pipelines that are embedded in our production environment.
  • Design develop frameworks for increasing the overall efficiency of bringing data into the data lake, processing and delivery of data; Encode best practices into reusable tools that can be shared across the team.
  • Automate the deployment of changes to production to improve data reliability quality
  • Work closely with upstream and downstream stakeholders to collect business requirements and develop data models to satisfy those requirements.
  • Design and deliver event-driven solutions using a streaming platform such as Kafka.
  • Mentor data engineers and other stakeholders on the best practices of managing a large-scale production data platform and developing a data-driven operations culture within the data engineering team.
  • Oversee the migration of data from legacy systems to new solutions.
  • Experience with Machine Learning or other leading-edge technologies.

Job requirements:

  • Exposure to at least two to three domains (such as Sales Marketing, Supply Chain, Finance, Data Science, Analytics).
  • ETL Pipeline experience Hadoop Ecosystem, Kafka, Spark
  • Workflow management oozie, Airflow
  • 5+ years of hands-on experience building large-scale data systems.
  • 5+ years of experience with architectural patterns, building APIs, microservices, event streams, and high throughput systems.
  • 5+ years of using AWS ecosystem tools preferred, including Redshift, S3, EMR, Lambda, Athena, Kinesis, ECS, Glue, and/or another cloud provider s equivalent stack.
  • Expert level proficiency with Python and SQL. Professional experience with at least one other programming language (Java, Scala, Go, Ruby, JavaScript)
  • 5+ years of experience building ETL pipelines and working with Cloud Data Warehouses (such as Redshift, Snowflake, Azure SQL Data Warehouse, Big Query).
  • Experience operating very large data warehouses or data lakes.
  • Experience designing and implementing data models for enterprise data warehouses.
  • Demonstrated Kafka, streaming experience
  • 5+ years of experience with ETL tools (Matillion, Informatica, DataStage, Talend)
  • Experience working with orchestration tools (Airflow, Luigi, Apache Nifi, Step Functions, etc.)

Apply Here

Submit CV To All Data Science Job Consultants Across India For Free

๐Ÿ” Explore All Related ITSM Jobs Below! ๐Ÿš€ โœ… Select your preferred “Job Category” in the Job Category Filter ๐ŸŽฏ ๐Ÿ”Ž Hit “Search” to find matching jobs ๐Ÿ”ฅ โž• Click the “+” icon that appears just before the company name to see the Job Detail & Apply Link ๐Ÿ“๐Ÿ’ผ

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *