Snowflake Data Warehouse Solution

Over the last few years, Cloud Data Warehouses have grown rapidly in popularity. Snowflake Data Warehouse is a Cloud Data Platform that is supported on all public clouds. All major Cloud vendors (AWS, Azure, Google) offer their proprietary versions of data warehouses as well. By being cloud agnostic, Snowflake Data Warehouse is an ideal solution for companies looking for a cloud agnostic or hybrid cloud solution.

 

At Data Aces, we specialize in designing and building Snowflake Data Warehouse Solutions. These include highly performant data models to ensure responsive, scalable business analytics to meet customers’ needs. We have built ELT pipelines that integrate several dozen data sources to extract, load and transform large data sets into Snowflake. Using technologies such as Spark that are highly scalable, multi-terabytes of data can be processed quickly to meet business needs.The diagram below shows our reference architecture for Snowflake Data Warehouse Solution.

 

 

 

 

 

 

 

 

 

 

 

 

 

​

​

The following sections describe the key components of this Snowflake Data Warehouse architecture.

​

Data Ingestion

A wide variety of data needs to be ingested into the Snowflake Data Warehouse. Data from external sources arrives into object stores (e.g AWS S3). Task processors from workflow schedulers are triggered that start ingesting the data and passing it downstream to Spark jobs. In addition to watching for external data, there can also be tasks to proactively pull data via APIs from internal operational systems such as CRM, Finance, HR and other systems.

 

Data Transformation

Once the data is ingested in its raw form from the external sources, it needs to be transformed in the Snowflake Data Warehouse. This is the ‘T” part of ELT; it applies certain rules and logic to get the data in a state that is useful for analytics. There are a wide variety of ETL tools that can help with Data Transformation such as Talend, FiveTran, Matillion etc. Some tools are configuration driven while others need custom coding, sometimes in a proprietary language. Our reference architecture uses Apache Spark –a highly scalable, performance technology that can process and transform very large data sets quickly. Transformation tasks map the input data into appropriate tables to facilitate downstream consumption for analytics.

 

Data Store

The transformed data is stored at multiple levels in the Snowflake Data Warehouse. The designing of the data models and transformations are based on the business rules and analytics requirements. The goal is to ensure that business users have access to the data they need.

 

Snowflake Data Warehouse Analytics

Snowflake Data Warehouse Analytics involves creation of pre-defined dashboards and reports to serve a wide variety of business users across the organization. These can be sales, marketing, operations, compliance, etc. At Data Aces, we have built dashboards for Tableau, PowerBI, Looker, etc.

In addition to pre-built dashboards, business analysts need the ability to perform ad-hoc analysis of data in the Snowflake Data Warehouse. The analysis tables need to be designed and implemented in a way that makes sense to business users and give them the flexibility to perform their own analysis.

 

For more information on how we can help your organization build a Snowflake Data Warehouse and associated Analytics, please Contact us.

​