In-Demand IT Careers

Big Data Engineer

Greater Pittsburgh Area, PA 15222

Posted: 11/16/2018 Job Number: 3168

The ideal Big Data Engineer candidate that will work on collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across the company.

RESPONSIBILITIES

  • Develop custom batch-oriented and real-time streaming data pipelines, migrating flows from ELT to ETL

  • Ensure proper data governance policies are followed by implementing or validating data lineage, quality checks, classification, etc.

  • Act in a technical leadership capacity: Mentor junior engineers and new team members, and apply technical expertise to challenging programming and design problems

  • Resolve defects/bugs during QA testing, pre-production, production, and post-release patches

  • Conduct design and code reviews

  • Analyze and improve efficiency, scalability, and stability of various system resources

  • Contribute to the design and architecture of the project

  • Operate within Agile Development environment and apply the methodologies

REQUIREMENTS

  • Bachelor's degree in Computer Science or similar field of study, preferred

  • Demonstrate advanced knowledge of data architectures, data pipelines, real time processing, streaming, networking, and security

  • Experience developing ETL processing flows using MapReduce technologies like Spark and Hadoop

  • Experience developing with ingestion and clustering frameworks

  • Experience with building stream-processing systems, using solutions

  • Experience with various messaging systems, such as Kafka or RabbitMQ

  • Good knowledge of Big Data querying tools, such as Pig or Hive

  • Good understanding of Lambda Architecture, along with its advantages and drawbacks

  • Proficiency with MapReduce, HDFS

  • Experience with integration of data from multiple data sources

  • Ability to solve any ongoing issues with operating the cluster

Send an email reminder to:

Share This Job:

Related Jobs:

Login to save this search and get notified of similar positions.

"Whether you are looking to fill or looking to advance your career, I would strongly suggest speaking to Scott and his team at OpenArc. Scott and his team provided a consultative and hands on approach throughout the entire process in my recent transition. Their unique approach was unlike anything I've experienced either as a candidate or a hiring manager,"

-Mark Quinn, Service Sales Executive at Siemens Medical Solutions, USA, Inc.

"Working with OpenArc was a great experience. The position I accepted was accurately described and my questions on benefits, paid time-off, and holidays and more were promptly answered. The entire OpenArc team was a very professional and to the point. I highly recommended OpenArc to all of my peers."

- Lisa McLaughlin, Executive Director, ACAPT

Take the Next Step

Find the right job now