Job Details

Lead Big Data Engineer

Advertiser
Global Bridge InfoTech Inc.
Location
Wayzata, Minnesota, United States
Rate
-
Job Title Lead Big Data Engineer with Kafka, Spark, and Hadoop Location Wayzata, MN. Duration 12+Months Job Description You will spend time on a deep technical review or a complete organizational review, helping understand the potential that data brings to solve the most pressing problems You will partner with teammates to create complex data processing pipelines in order to solve the most ambitious challenges You will collaborate with Data Scientists to design scalable implementations of models You will pair to write re-usable, efficient, clean and iterative code based on TDD by following best practices Leverage various continuous delivery practices to deploy, support and operate data pipelines. Provide the team with thought leadership to promote re-use and develop consistent, scalable patterns. Advise and educate clients on how to use different distributed storage and computing technologies from the plethora of options available Develop and operate modern data architecture approaches to meet key business objectives and provide end-to-end data solutions Participate in Designing data models, data solutions, data architecture and speak to the tradeoffs of different modeling approaches On other projects, you might be acting as the architect, leading the design of technical solutions, or perhaps overseeing a program inception to build a new product Seamlessly incorporate data quality into your day-to-day work as well as into the delivery process Herersquos what wersquore looking for As a Lead Engineer, yoursquoll take the lead as youhellip Use your technology expertise to apply and maintain knowledge of current and emerging technologies within your specialized area(s) of the technology domain. Evaluate new technologies and participates in decision-making, accounting for several factors such as viability within clientrsquos technical environment, maintainability, and cost of ownership. Initiate and execute research and proof-of-concept activities for new technologies. Lead or set strategy for testing and debugging at the platform or enterprise level. In complex and unstructured situations, serve as an expert resource to create and improve standards and best practices to ensure high-performance, scalable, repeatable, and secure deliverables Lead the design, lifecycle management, and total cost of ownership of services. Participate in planning services that have enterprise impact. Provide suggestions for handling routine and moderately complex technical problems, escalating issues when appropriate. Gather information, data, and input from a wide variety of sources identify additional resources when appropriate, engage with appropriate stakeholders, and conduct in-depth analysis of information. Develop plans and schedules, estimate technical pre-requisites for project requirements, and define milestones and deliverables. Monitor workflow and risks play a leadership role in mitigating risks and removing obstacles. Lead and participate in complex construction, automation, and implementation activities, ensuring successful implementation with architectural and operational requirements met. Establish new standards and best practices to monitor, test, automate, and maintain IT components or systems. Serve as an expert resource in disaster recovery and disaster recovery planning. Stay current with clientrsquos technical capabilities, infrastructure, and technical environment. Develop fully attributed data models, including logical, physical, and canonical. Influence data standards, policies, and procedures. Install, configure, andor tune data management solutions with minimal guidance. Monitor data management solution(s) and identify optimization opportunities You are equally happy coding and leading a team to implement a solution You have a track record of innovation and expertise in Data Engineering Yoursquore passionate about craftsmanship and have applied your expertise across a range of industries and organizations You have a deep understanding of data modelling and experience with data engineering tools and platforms such as Kafka, Spark, and Hadoop You have built large-scale data pipelines and data-centric applications using any of the distributed storage platforms such as HDFS, S3, NoSQL databases (Hbase, Cassandra, etc.) and any of the distributed processing platforms like Hadoop, Spark, Hive, Oozie, and Airflow in a production setting Hands on experience in MapR, Cloudera, Hortonworks andor cloud (AWS EMR, Azure HDInsights, Qubole etc.) based Hadoop distributions You are comfortable taking data-driven approaches and applying data security strategy to solve business problems Yoursquore genuinely excited about data infrastructure and operations with a familiarity working in cloud environments Working with data excites you you have created Big data architecture, you can build and operate data pipelines, and maintain data storage, all within distributed systems

Send application

Mail this job to me so I can apply later

Apply With CV

You are not logged in. If you have an account, log in to your account. If you do not have an account, why not sign up? It only takes a minute!

latest videos

Upcoming Events