Modern Data Analytics - Know more with less
What he said is astonishingly simple: Success requires knowledge of the truths (for example, the company figures), on the basis of which decisions can then be made. That's neither new nor breathtakingly innovative, and yet I see companies fail on this very point every day.
Challenge
Managers need a clean basis for decision-making - this includes facts and figures on the actual situation of the company.
It is still difficult to generate these figures - in many organizations, figures are compiled manually and sent as Excel or PowerPoint. Reporting that is set up in this way has several disadvantages.
- High costs due to manual effort
- High error rate due to copy/paste errors
- No traceability of how the figures were arrived at (compliance)
- Low reliability, as the figures can be changed by anyone
Modern Data Architecture
Data Factory
With the Data Factory, data from any source (SAP, SQL, Dynamics, SalesForce, Files etc.) can be read and prepared. The result of the processing is stored in the DataLake as a file (.json, .txt, ...). PowerBI itself can access these files like a DB server.
The whole process is automated and generates hardly any costs. The data can be updated daily, hourly or minutely. Any company can gain valuable insights and make better decisions with this architecture.
Benefits
Benefits of Modern Data Architecture
- Azure Data Factory generates pay-per-use costs, i.e. costs are only incurred when data is read and processed.
- Data storage in Datalake is very attractively priced compared to SQL DWH, billed per GB.
- The process of data preparation in the Data Factory can be configured by a "Citizen Developer" - the dependencies on the specialized Data Team are eliminated.
- Deployment and operation can be fully automated.
- Full cost transparency.
Where is the SQL?
But where is the SQL server, Snowflake or my data warehouse? We have become so accustomed to these buzzwords that it is often not questioned whether they are needed at all. These tools (SQL-DWH, Snowflake etc) generate high costs and are not always necessary. Analytics and reporting are possible even without a full-blown DWH.
Conclusion
In data projects the proven rule «start small, move fast» applies. From my experience, I can say that the above architecture is highly expandable and enables rapid further development. Nothing is obstructed and the initial costs are much more attractive than in a classic DWH project.
