Avispa Technology | Senior Data Engineer (H) | Albany, NY | United States | BigDataKB.com | 2023-01-15

Senior Data Engineer 14924 (H)

A leading tech company is seeking a Senior Data Engineer. The successful candidate will establish a fully functional object store compatible with S3 access APIs. The ideal candidate is an expert Senior Data Engineer with Spark, S3, and Rapid prototyping. The company offers a great work environment!

Senior Data Engineer Pay and Benefits:

• Hourly pay: $80/hr
• Worksite: Leading technology and data solutions company (Albany, NY 12201, Open to remote)
• W2 Employment, Group Medical, Dental, Vision, Life, Retirement Savings Program
• 40 hours/week, 3 Month Assignment

Senior Data Engineer Responsibilities:

• Establish a fully functional object store compatible with S3 access APIs which will:
  • Provide the ability to define fine-grain access control lists on our content platform objects.
  • Support multiple storage tiers in the view of cost optimization and allows moving objects between storage tiers.
  • Support event notifications over objects, allowing a subscription model to build data pipelines
  • Record all the object-level operations, and expose access logs for audit
  • Support multi-part uploads to move handle large objects into the store
  • Support replication of objects between different object stores
  • Provide comprehensive monitoring and observability metrics, and generate alerts on configured thresholds.
• Create appropriate documentation for the activities above Integrate Iceberg with our content platform, Spark, and Trino which will:
  • Provide a central metastore or catalog pluggable with multiple compute engines.
  • Support time travel, and automated maintenance like purging, and compression.
  • Support native reads on Iceberg, which can be leveraged by compute engines.
  • Create appropriate documentation for the activities above.

Senior Data Engineer Qualifications:

• +10 years of relevant experience.
• Senior Data Engineer / Spark / S3 / Rapid prototyping.
• Spark expertise.
• S3 storage and data management.
• Agile.
• Apache Iceberg (Nice to have) or similar.
• Can search and investigate on tools.
• PoC approach implementation experience.
• Delivery Ownership.
• Autonomous.

This is a remote position.