Big Data Engineer with PySpark
VITS US
- Location
- Irving, US
- Workplace
- On-site
- Job type
- Full-time
- Salary
- $154k–$194k
- Posted
- 2 days ago
- Source
- DevITJobs
Requirements:
- 7+ years of professional experience in Big Data Engineering
- Hands-on experience with Apache Spark and PySpark
- Programming experience with Python and/or Scala
- Strong SQL skills and experience with relational databases
- Hands-on experience with Hadoop, HDFS, Hive, and MapReduce
- Experience with Apache Kafka and real-time data processing
- Experience building ETL/ELT pipelines and processing large datasets
- Knowledge of distributed computing, Big Data architecture, and data warehouse or data lake concepts
- Experience with Git and CI/CD
- Strong troubleshooting and performance-tuning skills
- Experience with one or more of these cloud and modern data technologies is highly preferred: AWS (S3, EMR, Glue, Redshift, Lambda), Azure (ADLS, Databricks, Azure Data Factory, Synapse), GCP (BigQuery, Dataproc, Cloud Storage), Databricks, or Snowflake
- Preferred: Experience developing batch and streaming pipelines, Apache Airflow or similar orchestration tools, data modeling, Docker/Kubernetes, data governance, security, data quality frameworks, and Agile/Scrum practices
- Strong communication and analytical problem-solving skills
Responsibilities:
- Design, develop, and maintain scalable Big Data pipelines and data-processing solutions
- Build batch and real-time applications using Apache Spark and PySpark
- Develop and optimize ETL/ELT pipelines for large-scale processing
- Use Hadoop, HDFS, Hive, Spark, Kafka, and other Big Data technologies
- Write complex SQL queries and optimize processing jobs for performance
- Create data ingestion and transformation processes for structured and unstructured sources
- Develop real-time streaming pipelines using Kafka and Spark Streaming
- Perform data cleansing, transformation, validation, and quality checks
- Troubleshoot pipeline failures and improve scalability and reliability
- Work with cloud-based data platforms and storage services
- Collaborate with data architects, data scientists, analysts, application developers, and engineering teams
- Apply data security, governance, and access-control best practices
- Participate in code reviews, testing, deployment, and production support
- Contribute to Agile/Scrum activities and technical documentation
Technologies:
- Airflow
- AWS
- Lambda
- Redshift
- Azure
- Big Data
- BigQuery
- CI/CD
- Cloud
- Data Warehouse
- Databricks
- Docker
- ETL
- GCP
- Git
- Hadoop
- Hive
- Support
- Kafka
- Kubernetes
- Python
- PySpark
- SQL
- Scala
- Security
- Snowflake
- Spark
More
We are hiring a Big Data Engineer for a full-time hybrid role in Irving, Texas. The position involves working closely with data scientists, analysts, application developers, and business stakeholders to deliver scalable data solutions. The interview process includes a face-to-face interview.
last updated 43 week of 2026