Abstraindo a lógica do Airflow para usuários finais usando templates, empoderando o time de dados e outras áreas.

Apache Airflow

O Apache Airflow se tornou uma das principais ferramentas de engenharia de dados no mercado, dominando as novas implementações de Data Lake como o padrão de orquestração de jobs. Não é por acaso: é uma ferramenta completa (resolve do scheduling à execução), escalável, fácil de instalar+configurar, compatível com praticamente tudo e usa Python (outro padrão para a área de dados).

Desafio

Mesmo sendo uma ferramenta simples de se aprender, ela é mais uma ferramenta para o já lotado stack de dados. O profissional…


Apache Airflow logo

Apache Airflow has become one of the most widely used data engineering tool on the market. That’s no coincidence, as it is an awesome tool. Easy to install, scalable, compatible with pretty much everything in the market and very easy to use.

Even though Airflow is easy to get the hang of, it is yet another tool to learn. For teams that will not work with it on a daily basis, it is not so trivial. That could lead to an overloaded data engineering team, working overtime to automate jobs that could easily be handled automatically.

For example: a Data…


Starting a recurring topic about the boldest, sleekest, most unconventional approaches to software development.

I got this

These programming masterpieces are way ahead of their time, but we shall appreciate them as they deserve. Here are some top notch stunts I’ve ever seen developers do.

  • Use staging environment to push data to production environment, for “performance purposes”. And permission issues, of course.


I hate long tutorials that don’t explain the absolute basics, so this quick and practical read should get you up and running in no time.

Easiest thing in the world. Let’s get straight to it. This is the most simple way to do it. First, you will need a few things from Azure:

  1. Account Name
  2. Account Key
  3. Container Name

You will need to download and install the azure BLOB library.

pip install azure-storage-blob

Now, the actual code to upload a “Hello World” file:

from azure.storage.blob import BlockBlobService, PublicAccessblock_blob_service = BlockBlobService(account_name='account_name', account_key='account_key')block_blob_service.create_blob_from_text("container_name", filename_with_path, "Hello World", encoding='utf-8')

You will now…


Analytics is something every company has been looking into. From startups to traditional companies, we are seeing a boom in data usage and analysis.

Stepping away from privacy issues and AI takeover panic, here are 5 mistakes I see companies (old and new) make when it comes to Analytics:

Fancy new tool [ Photo by Markus Spiske on Unsplash ]

Find a nail for that fancy new hammer

Data analysis has a lot of tools that do a lot of things. However, I see companies moving the wrong way: they pick the tool first, then look for problems to hit them with. …


Brazilian data analyst with too many ideas to let them sit idly. Entrepreneur wannabe and technology aficionado.

I plan to write about many topics, focusing on Analytics technologies and (actual) good practices, software development, tech in general and entrepreneurship.

Daniel R Carletti

Analytics/Data Engineering/Software Developer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store