Databricks Data Engineer/Architect
Our Take
Design and scale Spyrosoft's enterprise data platform using Databricks and Python.
What you’ll do
- Design and maintain data pipelines
- Implement data governance via Unity Catalog
- Develop relational and non-relational data models
- Improve Big Data workflow performance
- Integrate DevOps and CI/CD
- Guide team on big data architecture
What they’re looking for
- 8+ years Data Engineering
- 4+ years Databricks (Unity Catalog)
- Expert Python and SQL
- Strong Data Warehousing, ETL
- Cloud experience (Azure, AWS, GCP)
- Deep Data Lakehouse/Mesh understanding
- Agile (Scrum/Kanban) experience
- English B2/C1 proficiency
Skills & Focus Areas
- Databricks
- Python
- SQL
- Azure
- AWS
- GCP
- DevOps
- Agile
As posted by Spyrosoft
Tech stack:
Databricks (Unity Catalog, Delta Live Tables)
Python (PySpark), SQL
Azure, AWS, or GCP
Data Lakehouse, Data Mesh, Data Marts
DevOps, CI/CD Pipelines
Agile (Scrum/Kanban)
Requirements:
At least 8 years in Data Engineering, with a minimum of 2 years specifically in Big Data environments.
4+ years of hands-on experience with Databricks services, including data pipelines and Unity Catalog.
Expert-level skills in Python and SQL.
Strong background in Data Warehousing, ETL, and distributed data processing.
Deep understanding of Data Lakes, Data Warehouses, and Data Mesh concepts.
Experience with at least one public cloud (Azure, AWS, or GCP) and strong design skills for both relational and non-relational storage.
Analytical mindset capable of troubleshooting complex issues in a big data landscape.
Very good verbal and written English B2/C1
Experience working in Agile (Scrum/Kanban) environments.
Project description:
Join our data engineering team as we develop and scale our enterprise data platform. We are building a high-performance ecosystem designed to manage large-scale datasets, ranging from structured to unstructured formats. In this role, you will help modernize our data infrastructure by implementing cutting-edge storage and processing solutions. You will play a key part in designing how we ingest, process, and govern data to provide reliable insights across the organization.
Main responsibilities:
Design and maintain robust data pipelines and distributed data processing systems using Databricks.
Implement and manage data governance and security frameworks via Unity Catalog.
Develop sophisticated data models (Relational and Non-Relational) to support complex analytical requirements.
Improve the performance and reliability of Big Data workflows and ETL processes.
Work within an Agile environment, integrating DevOps and CI/CD principles into the data lifecycle.
Act as a subject matter expert, guiding the team through complex big data challenges and architectural decisions.