Abstraindo a lógica do Airflow para usuários finais usando templates, empoderando o time de dados e outras áreas.
O Apache Airflow se tornou uma das principais ferramentas de engenharia de dados no mercado, dominando as novas implementações de Data Lake como o padrão de orquestração de jobs. Não é por acaso: é uma ferramenta completa (resolve do scheduling à execução), escalável, fácil de instalar+configurar, compatível com praticamente tudo e usa Python (outro padrão para a área de dados).
Mesmo sendo uma ferramenta simples de se aprender, ela é mais uma ferramenta para o já lotado stack de dados. O profissional…
Apache Airflow has become one of the most widely used data engineering tool on the market. That’s no coincidence, as it is an awesome tool. Easy to install, scalable, compatible with pretty much everything in the market and very easy to use.
Even though Airflow is easy to get the hang of, it is yet another tool to learn. For teams that will not work with it on a daily basis, it is not so trivial. That could lead to an overloaded data engineering team, working overtime to automate jobs that could easily be handled automatically.
For example: a Data…
Starting a recurring topic about the boldest, sleekest, most unconventional approaches to software development.
These programming masterpieces are way ahead of their time, but we shall appreciate them as they deserve. Here are some top notch stunts I’ve ever seen developers do.
I hate long tutorials that don’t explain the absolute basics, so this quick and practical read should get you up and running in no time.
Easiest thing in the world. Let’s get straight to it. This is the most simple way to do it. First, you will need a few things from Azure:
You will need to download and install the azure BLOB library.
pip install azure-storage-blob
Now, the actual code to upload a “Hello World” file:
from azure.storage.blob import BlockBlobService, PublicAccessblock_blob_service = BlockBlobService(account_name='account_name', account_key='account_key')block_blob_service.create_blob_from_text("container_name", filename_with_path, "Hello World", encoding='utf-8')
You will now…
Analytics is something every company has been looking into. From startups to traditional companies, we are seeing a boom in data usage and analysis.
Stepping away from privacy issues and AI takeover panic, here are 5 mistakes I see companies (old and new) make when it comes to Analytics:
Data analysis has a lot of tools that do a lot of things. However, I see companies moving the wrong way: they pick the tool first, then look for problems to hit them with. …
Analytics/Data Engineering/Software Developer