25.11.2025 aktualisiert
Premiumkunde
100 % verfügbarSenior Data Engineer | PySpark | Azure Data Factory | Azure Data Fabric | Databricks | Snowflake
Frankfurt am Main, Deutschland
Weltweit
Bachelor of Engineering in Computer scienceÜber mich
Microsoft & Databricks Senior Data Engineer with 15+ years of IT Consulting experience working with clients like AKKODIS, Uniper Energy, Deutsche Bank, Deutsche Boerse, Commerzbank, Aldi Sued, KARSTAT, etc. for their various Data Lakehouse, Data Lake, Data Warehouse, Data Migration projects.
Skills
SQLPythonSnowflakeDatabricksMicrosoft AzureAzure Data Factorydata engineer PySparkAzure Data fabricAzure Data EngineerETL PipelineAzure Service FabricdbtSpark
Summary:
- Senior Data Engineer with 15+ years of IT Consulting experience working with clients like AKKODIS, Uniper Energy, Deutsche Bank, Deutsche Boerse, Commerzbank, Aldi Sued, KARSTAT, etc. for their various Data Lakehouse, Data Lake, Data Warehouse, Data Migration, Data Integration projects.
- Highly proficient in Architecture, Design, Development, Implementation and Support of ETL or ELT data processing pipelines for Data warehouses or Data Lakes using ETL tools like Microsoft Azure Data factory V2, Informatica Powercenter, Talend, etc and using PySpark in Databricks SaaS applications.
- Proficient in designing & customizing data models using data modelling techniques likes dimensional modelling, data vault modelling, etc.
- Worked with various structured and semi structured data sources like SAP, Azure SQL Server Database, REST API, CSV, XML, JSON, PARQUET, etc.
- Designed various data layers e.g. stage layer, core layer, reporting layers, etc. for efficient & in time data processing with high quality.
- Worked closely with Data Architect, Business Analysts, Product Owners, Tech Leads to help in drafting design, architecture, requirement specification, etc for developing ETL/ELT data processing pipelines.
Skills
ETL Tools
PySpark, Azure Data Factory V2, Informatica PowerCenter 10.4, Informatica Integration Cloud Services ie, IICS (Data Integration & Application Integration), Talend Data Integration 7.2
Services/Application Integration Tools
Informatica Intelligent Cloud Services (IICS) Data & Application Services, Talend Open Studio for ESB, Talend Real-Time Big Data Platform 7.2
Databases
Azure SQL Database, Oracle 19c, Oracle 12c, Oracle Exadata 12c, Microsoft SQL Server 2016, IBM DB2, Hadoop HDFS, PostgreSQL 10.7, MySQL
Large Language Model Tools:
ChatGPT, Google Bard
Cloud Based Datawarehouse:
Snowflake, Azure Synapse Analytics
Data Processing Frameworks/Platforms:
Apache Spark, Databricks
Big Data Ecosystem:
Hadoop 2.0, HDFS, Sqoop, Hive
Cloud Technologies:
Azure Storage Accounts, Azure Batch Account, Azure Data Factory, Azure SQL Server, Azure Data Lake Storage, Azure Functions, Azure Logic Apps , Azure Key Vault, Azure DevOps, Amazon Web Services (AWS), AWS S3
Modeling
3-NF, Dimensional Modeling (Star & Snow Flake), Data Vault (Raw Vault & Business Vault)
Modeling Tools
PowerDesinger
Software Development Methods
Agile, SCRUM, Waterfall
Programming Languages
Python 3, PySpark, JAVA, SQL, T-SQL, PL/SQL, UNIX/Bash Shell scripting
Scheduling Tools
BMC Control-M 9.0.19, Automic UC4, Informatica Scheduler, Talend Management Console (TMC)
Database Deployment & Change Control Tools:
Liquibase 4.21.1
Version Control
Subversion, GitHub
Operating Systems
Windows 7/10, Linux, Solaris
Sprachen
DeutschGrundkenntnisseEnglischMuttersprache
Projekthistorie
Responsibilities:
Data Pipeline/ELT:
- Design, Develop and Maintain ETL/Data pipelines using Azure Data Factory and Python.
- Designed and led the implementation of end-to-end data pipelines on Azure Data Factory, ensuring efficient data movement and transformation across multiple sources. Resulted in a 30% reduction in data processing time and improved data accuracy.
- Setup all meta data tables, their configurations, store procedures, views for pipeline reusability to load using Generic Import pipelines.
- Reduced 60% to 70% development time of source to data lake and data lake to staging lay mappings by developing generic ADF pipelines.
- Setup all database objects needed for logging pipeline run information.
- Creation of ADF Linked Services, Data Sets, pipelines to read data from SAP tables using SAP Table linked service and load data into Azure Data Lake Storage Gen2.
- Creation of various types of Data Sources, Linked Services, Pipelines, Global Variables, Triggers, etc ADF objects required for pipeline development.
- Creation of Global, Linked Service, Data Source, pipeline parameters for reusability.
- Create ADF pipelines using various activities like Copy Data, Web, Lookup, foreach, store procedure, execute pipeline, etc.
- Uses various data flow transformations such as select, filter, join, derive column, exists, sequence, etc.
- Create ADF Self Hosted Runtime and read data from on premises source system like SAP, etc.
- Debugging ADF pipelines using Data Flow Debug Clusters for verifying the data or transformation results.
- Creation of Generic SCD Type 2 pipelines for loading data into historized tables.
- Creation of documentation of various processes, data models, data flow diagrams, ETL Architecture on Confluence.
- Configuration of GIT Repositories for various environments and releases.
- Creation of Azure Key Vault resource for password encryption in data pipelines.
- Creation of Azure Pipelines to execute PySpark Notebooks from Azure Data Bricks workspace.
- Creation of PySpark Notebooks in Azure Databricks to perform various transformations and loading.
- Creation of various azure resource consumption reports for budget optimization.
- Creation of Azure Logic App workflows for email notification in case of Data Pipeline failure or Fatal errors.
- Used ChatGPT to find various performance optimization, Data Testing techniques.
Database Tasks:
- Creation of various Azure SQL Server database objects such as schemas, tables, sequences, store procedures, views, etc.
- Help business analyst to identify various dimensions as per Report requirements and optimize model.
- Creating various Master Data and Meta Data tables, views & store procedures for data enrichment and job run logging information.
Documentation:
- Creation of documentation of various processes, data models, data flow diagrams, ETL Architecture, Data Pipelines, Database Objects on Confluence.
Team Activities:
- Participating in various SCRUM meetings for creating user stories, estimation, backlog grooming, retrospective, etc.
DevOps:
- Creation of code repositories in Azure DevOps and developing CI/CD release pipelines for deployment to UAT & PROD environment.
- Creation of CI/CD release pipelines to automatically deploy application code objects from Dev to UAT & PRD DevOps repositories.
- Creation of Azure Key Vault, credentials and integrating it with ADF Linked services, activities for retrieving the secrets.
Data Pipeline/ELT:
- Design, Develop and Maintain ETL/Data pipelines using Azure Data Factory and Python.
- Designed and led the implementation of end-to-end data pipelines on Azure Data Factory, ensuring efficient data movement and transformation across multiple sources. Resulted in a 30% reduction in data processing time and improved data accuracy.
- Setup all meta data tables, their configurations, store procedures, views for pipeline reusability to load using Generic Import pipelines.
- Reduced 60% to 70% development time of source to data lake and data lake to staging lay mappings by developing generic ADF pipelines.
- Setup all database objects needed for logging pipeline run information.
- Creation of ADF Linked Services, Data Sets, pipelines to read data from SAP tables using SAP Table linked service and load data into Azure Data Lake Storage Gen2.
- Creation of various types of Data Sources, Linked Services, Pipelines, Global Variables, Triggers, etc ADF objects required for pipeline development.
- Creation of Global, Linked Service, Data Source, pipeline parameters for reusability.
- Create ADF pipelines using various activities like Copy Data, Web, Lookup, foreach, store procedure, execute pipeline, etc.
- Uses various data flow transformations such as select, filter, join, derive column, exists, sequence, etc.
- Create ADF Self Hosted Runtime and read data from on premises source system like SAP, etc.
- Debugging ADF pipelines using Data Flow Debug Clusters for verifying the data or transformation results.
- Creation of Generic SCD Type 2 pipelines for loading data into historized tables.
- Creation of documentation of various processes, data models, data flow diagrams, ETL Architecture on Confluence.
- Configuration of GIT Repositories for various environments and releases.
- Creation of Azure Key Vault resource for password encryption in data pipelines.
- Creation of Azure Pipelines to execute PySpark Notebooks from Azure Data Bricks workspace.
- Creation of PySpark Notebooks in Azure Databricks to perform various transformations and loading.
- Creation of various azure resource consumption reports for budget optimization.
- Creation of Azure Logic App workflows for email notification in case of Data Pipeline failure or Fatal errors.
- Used ChatGPT to find various performance optimization, Data Testing techniques.
Database Tasks:
- Creation of various Azure SQL Server database objects such as schemas, tables, sequences, store procedures, views, etc.
- Help business analyst to identify various dimensions as per Report requirements and optimize model.
- Creating various Master Data and Meta Data tables, views & store procedures for data enrichment and job run logging information.
Documentation:
- Creation of documentation of various processes, data models, data flow diagrams, ETL Architecture, Data Pipelines, Database Objects on Confluence.
Team Activities:
- Participating in various SCRUM meetings for creating user stories, estimation, backlog grooming, retrospective, etc.
DevOps:
- Creation of code repositories in Azure DevOps and developing CI/CD release pipelines for deployment to UAT & PROD environment.
- Creation of CI/CD release pipelines to automatically deploy application code objects from Dev to UAT & PRD DevOps repositories.
- Creation of Azure Key Vault, credentials and integrating it with ADF Linked services, activities for retrieving the secrets.
Contract Type: Contract
Role: Data Engineer
Project: Energy Data Lake
Project Technology Stack
Cloud Applications: Microsoft Azure
Source System: REST API, MS SQL Server, Snowflake, CSVs, XMLs
Target System: MS SQL Server, Snowflake
ETL Tool/Programming Language: Azure Data Factory, Talend Data Integration, Python
Other programming languages: Python, SQL, SnowSQL
Scheduling Tool: Azure Batch Service, Talend Management Console
Role: Data Engineer
Project: Energy Data Lake
Project Technology Stack
Cloud Applications: Microsoft Azure
Source System: REST API, MS SQL Server, Snowflake, CSVs, XMLs
Target System: MS SQL Server, Snowflake
ETL Tool/Programming Language: Azure Data Factory, Talend Data Integration, Python
Other programming languages: Python, SQL, SnowSQL
Scheduling Tool: Azure Batch Service, Talend Management Console
- Implement data pipelines using Azure Data Factory
- Migration from the old system to the new system
- Connect, process, implement and store data sources
- Data processing with Azure MS SQL database
- Create SQL procedures that contain the data processing logic
- Migration from the old system to the new system
- Connect, process, implement and store data sources
- Data processing with Azure MS SQL database
- Create SQL procedures that contain the data processing logic