Avispa Technology | Senior Data Engineer (H) | Albany, NY | United States | BigDataKB.com | 2023-01-15

Before u proceed below to check the jobs/CVs, please select your favorite job categories, whose top job alerts you want in your email & Subscribe to our Email Job Alert Service For FREE


Job Location: Albany, NY

Job Detail:

Senior Data Engineer 14924 (H)

A leading tech company is seeking a Senior Data Engineer. The successful candidate will establish a fully functional object store compatible with S3 access APIs. The ideal candidate is an expert Senior Data Engineer with Spark, S3, and Rapid prototyping. The company offers a great work environment!

Senior Data Engineer
Pay and Benefits:

BigDataKB.com Jyotish
BigDataKB.com Jyotish - Career & Life Prediction
  • Hourly pay: $80/hr
  • Worksite: Leading technology and data solutions company (Albany, NY 12201, Open to remote)
  • W2 Employment, Group Medical, Dental, Vision, Life, Retirement Savings Program
  • 40 hours/week, 3 Month Assignment

Senior Data Engineer

  • Establish a fully functional object store compatible with S3 access APIs which will:

    • Provide the ability to define fine-grain access control lists on our content platform objects.
    • Support multiple storage tiers in the view of cost optimization and allows moving objects between storage tiers.
    • Support event notifications over objects, allowing a subscription model to build data pipelines
    • Record all the object-level operations, and expose access logs for audit
    • Support multi-part uploads to move handle large objects into the store
    • Support replication of objects between different object stores
    • Provide comprehensive monitoring and observability metrics, and generate alerts on configured thresholds.
  • Create appropriate documentation for the activities above Integrate Iceberg with our content platform, Spark, and Trino which will:
    • Provide a central metastore or catalog pluggable with multiple compute engines.
    • Support time travel, and automated maintenance like purging, and compression.
    • Support native reads on Iceberg, which can be leveraged by compute engines.
    • Create appropriate documentation for the activities above.

Senior Data Engineer Qualifications:

  • +10 years of relevant experience.
  • Senior Data Engineer / Spark / S3 / Rapid prototyping.
  • Spark expertise.
  • S3 storage and data management.
  • Agile.
  • Apache Iceberg (Nice to have) or similar.
  • Can search and investigate on tools.
  • PoC approach implementation experience.
  • Delivery Ownership.
  • Autonomous.

This is a remote position.

Apply Here

Submit CV To All Data Science Job Consultants Across United States For Free


Please enter your comment!
Please enter your name here