Data Engineer - Databricks

Requirements:

• Bachelors or masters degree in Computer Science, Data Engineering, Statistics, or a related field
• 5+ years of data engineering experience
• 2+ years applying machine learning in production
• Strong Databricks expertise
• Experience with advanced SQL optimization and complex query design
• Experience developing Kafka streaming applications and connectors
• Experience with Databricks workflows and medallion architecture
• Knowledge of data governance, compliance, security, privacy, and performance tuning for large-scale data processing
• Experience developing Apache NiFi pipelines for invoice and purchase order processing
• Experience integrating data stores such as Druid, MongoDB, OpenSearch, and Postgres
• Experience building and maintaining machine learning pipelines for model training, deployment, and monitoring
• Ability to design data architectures for large-scale machine learning workloads
• Familiarity with LLM solutions, RAG architectures, and generative AI
• Strong collaboration, mentoring, analytical, communication, process-improvement, and data-quality skills
• Experience communicating with data platform vendors, evaluating tools, providing product roadmap feedback, and coordinating integration projects

Responsibilities:

• Modernize and scale our business intelligence capabilities
• Design, build, and integrate end-to-end data pipelines
• Deploy machine learning solutions and operationalize intelligent analytics to support organizational decision-making
• Develop Kafka streaming applications and connectors, Databricks workflows, and Apache NiFi pipelines
• Implement data governance and compliance practices, and optimize performance, security, and privacy
• Build and maintain machine learning pipelines for model training, deployment, and monitoring
• Design and optimize data architectures for large-scale machine learning workloads
• Explore and implement LLM-based solutions, RAG architectures, and generative AI for business use cases
• Collaborate with product and engineering teams and mentor junior data engineers
• Communicate data requirements to stakeholders and improve processes for efficiency, accuracy, and reliability
• Engage data platform vendors, evaluate tools, provide technical feedback on product roadmaps, and coordinate integration projects

Technologies:

• AI
• OpenSearch
• Business Intelligence
• Databricks
• Support
• Kafka
• LLM
• Machine Learning
• MLOps
• Model Training
• MongoDB
• RAG
• SQL
• Security

More

We are seeking a Data Engineer to help advance our business intelligence and analytics capabilities. The role combines data engineering, applied machine learning, MLOps, and AI. Compensation varies based on factors such as skills, experience, training, certifications, location, and organizational needs; discretionary bonuses may be available. Eligible employees may participate in medical, dental, vision, and 401(k) benefits. Vaco by Highspring is an equal opportunity employer and provides reasonable accommodations during the application and employment process. Applications may undergo preliminary AI screening for minimum job requirements, with further assessment by recruiters and hiring managers. This posting also includes notices concerning candidate representation, privacy, and applicable pay transparency.

last updated 44 week of 2026