Job Title: Big Data Engineer
Location: Plano, Texas
Job Description: The Big Data Engineer will be responsible for building big data pipelines using open-source tools and enterprise frameworks in response to new business requests. This individual will work closely with data scientists and SMEs to define and map data requirements which can be translated to executable data processing pipes.
Design and implementation of specific data models to ultimately help drive better business decisions through insights from a combination of external and internal AT&T data assets. This role is also accountable for developing the necessary enablers and data platform in the Big Data Computing Environment and maintaining its integrity across the data life cycle phases.
- Build high performance streaming and batch data pipelines that generate business ready datasets and
- insights to support existing use case scenarios
- Work directly with the client user community and data analysts to define and document data
- requirements for data integration and business intelligence applications.
- Determine and document data mapping rules for movement of medium to high complexity data between
- applications. Adhere to and promote the use of data administration standards.
- Support data selection, extraction, and cleansing for enterprise applications, including data warehouse
- and data marts.
- Investigate and resolve data issues across platforms and applications, including discrepancies of
- definition, format, and function.
- Employ data mining techniques to achieve data synchronization, redundancy elimination, source
- identification, data reconciliation, and problem root cause analysis. May also be responsible for quality
- Analyze existing or build new PySpark/Scala/Snow SQL code wherever necessary to evolve existing
- prototypes into modern scalable data processing pipelines using Snowflake and Databricks
- Familiarity with SCD and lambda processing pipelines in Azure Data Factory
- Build high-performance algorithms, prototypes, predictive models and proof of concepts
- Create reusable software components (e.g. specialized spark UDFs) and analytics applications Support architecture evaluation of the enterprise data platform through implementation and launch of data preparation and data science capabilities
Required Qualifications and Skills:
- 8+ years of Data Ware housing and Big Data Technology experience.
- 3+ years of experience with Hadoop and/or Plantir database.
- 3+ of strong PySpark scripting experience.
- 2+ years of Snowflake experience required
- Prior experience with deploying complex data science solutions is a strong plus
- Strong understanding and use of databases: relational as well as NoSQL datastores
- Prior experience with data exploration, prototyping and visualization tools: e.g., Zeppelin, Jupyter, Power BI, Tableau
- Experience working in telecommunications industry
Education: Bachelors or Master’s in computer science or equivalent