Greater Pittsburgh Area, PA 15222
The ideal Big Data Engineer candidate that will work on collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across the company.
Develop custom batch-oriented and real-time streaming data pipelines, migrating flows from ELT to ETL
Ensure proper data governance policies are followed by implementing or validating data lineage, quality checks, classification, etc.
Act in a technical leadership capacity: Mentor junior engineers and new team members, and apply technical expertise to challenging programming and design problems
Resolve defects/bugs during QA testing, pre-production, production, and post-release patches
Conduct design and code reviews
Analyze and improve efficiency, scalability, and stability of various system resources
Contribute to the design and architecture of the project
Operate within Agile Development environment and apply the methodologies
Bachelor's degree in Computer Science or similar field of study, preferred
Demonstrate advanced knowledge of data architectures, data pipelines, real time processing, streaming, networking, and security
Experience developing ETL processing flows using MapReduce technologies like Spark and Hadoop
Experience developing with ingestion and clustering frameworks
Experience with building stream-processing systems, using solutions
Experience with various messaging systems, such as Kafka or RabbitMQ
Good knowledge of Big Data querying tools, such as Pig or Hive
Good understanding of Lambda Architecture, along with its advantages and drawbacks
Proficiency with MapReduce, HDFS
Experience with integration of data from multiple data sources
Ability to solve any ongoing issues with operating the cluster
"Whether you are looking to fill or looking to advance your career, I would strongly suggest speaking to Scott and his team at OpenArc. Scott and his team provided a consultative and hands on approach throughout the entire process in my recent transition. Their unique approach was unlike anything I've experienced either as a candidate or a hiring manager,"-Mark Quinn, Service Sales Executive at Siemens Medical Solutions, USA, Inc.
"Working with OpenArc was a great experience. The position I accepted was accurately described and my questions on benefits, paid time-off, and holidays and more were promptly answered. The entire OpenArc team was a very professional and to the point. I highly recommended OpenArc to all of my peers."- Lisa McLaughlin, Executive Director, ACAPT