Big Data Engineer with PySpark

Requirements:

• 7+ years of professional experience in Big Data Engineering
• Hands-on experience with Apache Spark and PySpark
• Programming experience with Python and/or Scala
• Strong SQL skills and experience with relational databases
• Hands-on experience with Hadoop, HDFS, Hive, and MapReduce
• Experience with Apache Kafka and real-time data processing
• Experience building ETL/ELT pipelines and processing large datasets
• Knowledge of distributed computing, Big Data architecture, and data warehouse or data lake concepts
• Experience with Git and CI/CD
• Strong troubleshooting and performance-tuning skills
• Experience with one or more of these cloud and modern data technologies is highly preferred: AWS (S3, EMR, Glue, Redshift, Lambda), Azure (ADLS, Databricks, Azure Data Factory, Synapse), GCP (BigQuery, Dataproc, Cloud Storage), Databricks, or Snowflake
• Preferred: Experience developing batch and streaming pipelines, Apache Airflow or similar orchestration tools, data modeling, Docker/Kubernetes, data governance, security, data quality frameworks, and Agile/Scrum practices
• Strong communication and analytical problem-solving skills

Responsibilities:

• Design, develop, and maintain scalable Big Data pipelines and data-processing solutions
• Build batch and real-time applications using Apache Spark and PySpark
• Develop and optimize ETL/ELT pipelines for large-scale processing
• Use Hadoop, HDFS, Hive, Spark, Kafka, and other Big Data technologies
• Write complex SQL queries and optimize processing jobs for performance
• Create data ingestion and transformation processes for structured and unstructured sources
• Develop real-time streaming pipelines using Kafka and Spark Streaming
• Perform data cleansing, transformation, validation, and quality checks
• Troubleshoot pipeline failures and improve scalability and reliability
• Work with cloud-based data platforms and storage services
• Collaborate with data architects, data scientists, analysts, application developers, and engineering teams
• Apply data security, governance, and access-control best practices
• Participate in code reviews, testing, deployment, and production support
• Contribute to Agile/Scrum activities and technical documentation

Technologies:

• Airflow
• AWS
• Lambda
• Redshift
• Azure
• Big Data
• BigQuery
• CI/CD
• Cloud
• Data Warehouse
• Databricks
• Docker
• ETL
• GCP
• Git
• Hadoop
• Hive
• Support
• Kafka
• Kubernetes
• Python
• PySpark
• SQL
• Scala
• Security
• Snowflake
• Spark

More

We are hiring a Big Data Engineer for a full-time hybrid role in Irving, Texas. The position involves working closely with data scientists, analysts, application developers, and business stakeholders to deliver scalable data solutions. The interview process includes a face-to-face interview.

last updated 43 week of 2026