Big Data Engineer

Company: Yoh
Location: Ridgefield Park
Posted on: September 14, 2023

Job Description:

**NO Corp2Corp!!**US CITIZENS ONLYQualifications: Strong PySpark experience Hands-on experience with Python, SQL, Hadoop, Hive, Presto/PrestoDB Ability to articulate PySpark on the backend expertise Experienced with Big Data platforms (preferably GCP over AWS / Azure) Regarding cloud expertise with GCP - looking for Dataproc, GCS (Google Cloud Storage), and Cloud Composer Strong coding expertise -as an individual contributor - at least 70 to 80% of time spent for developing code Good knowledge of ETL processes and building custom ingestion / transformation pipelines on BigData systems Exposure to cloud big data systems as well (this is crucial) Description: Big Data Engineers serve as the backbone of the Strategic Analytics organization, ensuring both the reliability and applicability of the team's data products to the entire organization. They have extensive experience with ETL design, coding, and testing patterns as well as engineering software platforms and large-scale data infrastructures. Big Data Engineers have the capability to architect highly scalable end-to-end pipeline using different open source tools, including building and operationalizing high-performance algorithms. Big Data Engineers understand how to apply technologies to solve big data problems with expert knowledge in programming languages like Java, Python, Linux, PHP, Hive, Impala, and Spark. Big data engineers implement complex big data projects with a focus on collecting, parsing, managing, analyzing, and visualizing large sets of data to turn information into actionable deliverables across customer-facing platforms. They have a strong aptitude to decide on the needed hardware and software design and can guide the development of such designs through both proof of concepts and complete implementations. Responsibilities: Translate complex functional and technical requirements into detailed design. Design for now and future success Hadoop technical development and implementation. Loading from disparate data sets. by leveraging various big data technology e.g. Kafka Pre-processing using Hive, Impala, Spark, and Pig Design and implement data modeling Maintain security and data privacy in an environment secured using Kerberos and LDAP High-speed querying using in-memory technologies such as Spark. Following and contributing best engineering practice for source control, release management, deployment etc. Production support, job scheduling/monitoring, ETL data quality, data freshness reportingNote: Any pay ranges displayed are estimations. - Actual pay is determined by an applicant's experience, technical expertise, and other qualifications as listed in the job description. - All qualified applicants are welcome to apply.Yoh, a Day & Zimmermann company, is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.Visit -to contact us if you are an individual with a disability and require accommodation in the application process.

Keywords: Yoh, Hackensack , Big Data Engineer, Other , Ridgefield Park, New Jersey