Job Location: Bangalore/Bengaluru
What you ll do:
- Curate very large-scale data from a multitude of sources into appropriate sets for research and development for the data science, threat analysts, and developers across the company
- Design, test, and implement storage solutions for various consumers of the data
- Design and implement mechanisms to monitor data sources over time for changes using summarization, monitoring, and statistical methods
- Leverage computer science algorithms and constructs, including probabilistic data structures, to distill large data into sources of insight and enable future analytics
- Convert prototypes into production data engineering solutions through disciplined software engineering practices, Spark optimizations, and modern deployment pipelines
- Collaborate on design, implementation, and deployment of applications with the rest of software engineering
- Support data scientists and product teams in building, debugging, and deploying Spark applications that best leverage data
- Build and maintain tools for automation, deployment, monitoring and operations
- Create test plans, test cases, and run tests with automated tools
What you ll bring:
- 6+ years of experience with Python3, and 2+ years experience with Spark Scala experience is helpful
- 5+ years of experience in data engineering, data science, and related data-centric fields using large-scale data environments
- 3+ years of experience in using SQL and working with modern relational databases, including MySQL or PostgreSQL
- 3+ years of experience with developing ETL pipelines and data manipulation scripts
- Proficiency with object oriented design and SOLID principles
- Strong emphasis on unit testing and code quality
- Proficiency with AWS products EMR, S3, VPC, EC2, Athena, etc.
- MS or BS in computer science or a related field, or equivalent work experience
What success looks like:
After six months, you will
- Complete onboarding by demonstrating knowledge of the data lake and associated technologies and CI/CD processes by deploying ETL pipelines for curation and warehousing to production
- Complete rotations on support/duty where you will gain experience with the different systems, tools, and processes in the data lake by resolving reported issues
- Contribute to the teams velocity by participating in scrum and driving stories to completion
After about a year, you will
- Be an expert on the data lake target state architecture and drive engineering design and grooming sessions for new feature development
- Apply coding best practices and provide in-depth reviews on the team s pull requests
- Be a thought leader in one or more domains of the data lake, driving development and mentoring teammates in this domain
Submit CV To All Data Science Job Consultants Across India For Free
🔍 Explore All Related ITSM Jobs Below! 🚀
✅ Select your preferred "Job Category" in the Job Category Filter 🎯
🔎 Hit "Search" to find matching jobs 🔥
➕ Click the "+" icon that appears just before the company name to see the Job Detail & Apply Link 📝💼