Job Location: United States
The Data Engineer will manage and manipulate data and data flows for both existing and new systems. Additionally, they will provide support in the areas of data extraction, transformation, and load (ETL), data mapping, analytics, operations, databases, and maintenance of data and associated systems. As a member of the team, candidate will work in a multi-tasking, quick-paced, dynamic, process-improvement environment that requires experience with the principles of large-scale (terabytes) database environments, large-scale file manipulation, data modeling, data mapping, data testing, data quality, and documentation preparation.
Responsibilities
- Provide support in the areas of data extraction, transformation, and load (ETL), data mapping, analytics, operations, database administration, and maintenance of data and associated systems.
- Develop and manage complex data flows or makes significant enhancements to existing pipelines.
- Troubleshoot complex problems and provide customer support for the ETL process.
- Advise hardware engineers on machine characteristics that affect software systems, such as storage capacity, processing speed, and input/output requirements.
- Conduct investigations and tests of considerable complexity.
- Provide ongoing maintenance, support, and enhancements in existing systems and platforms
- Collaborate cross-functionally with software engineers, data scientists, analysts, project managers, and other engineering groups.
- Research emerging cloud-native technologies to determine impact on application execution.
- Communicate clearly and effectively with teammates, customers, and external partners.
- Prepare written and verbal communications on analyses, findings, and project progress.
- Write and update technical documentation such as user manuals, system documentation, training materials, processes, and procedures.
- Provide recommendations for continuous improvement.
Requirements
- Five (5) + years of experience in working with Big Data technology and using methods to ingest, process, clean, and analyze big data.
- Bachelor’s degree in Computer Science, Information Technology, or other related discipline, or equivalent combination of education, technical certifications, training, and work/military experience
- Experience working with large and complex data sets as well as experience analyzing large volumes of data.
- Experience working with relational databases such as Postgres, MySQL, and Microsoft SQL Server.
- Experience with data ingest tools (e.g., Sqoop, Kafka, and Spark Streaming)
- Experience in system management expertise with monitoring, disaster recovery, backup, automated testing, automated schema migration, and continuous deployment.
- Experience in Agile software methodologies.
- Security+ Required
- Secret Clearance or ability to obtain
Preferred Qualifications
- Five (5) + years of implementation experience in Hadoop technologies and have worked in multiple Hadoop distributions like AWS and Cloudera.
- Demonstrated experience with large data stores such as data lakes, data warehouses, and VLDB sharded RDBMS databases.
- Intensive skill in scripting using Bash, Python, and Shell
- Demonstrated experience in creating indexes and working with indexes.
- Demonstrated experience in Kibana and Elasticsearch.
- Demonstrated experience in ETL, data integration, and migration.
- Experience with different file formats like ORC, Parquet, AVRO, JSON.
- Experience in data cleansing scripts like Spark and MapReduce.
- Orchestrated multiple Hadoop application jobs using Oozie or Airflow
- Experience in working with various IDE’s such as Eclipse, VS Code, and PyCharm
- Exposure in infrastructure as Code languages like Chef, Puppet, Ansible, Cloudformation.
All qualified applicants will receive consideration for employment without regard to race, color, sex, sexual orientation, gender identity, religion, national origin, disability, veteran status, age, marital status, pregnancy, genetic information, or other legally protected status.
Required Skills
Required Experience
Submit CV To All Data Science Job Consultants Across United States For Free