Redshift is uniquely architected to allow for both vertical and horizontal scaling, execution of queries utilizing massively parallel processing, prioritization of user groups and query types, along with compression efficient and aggregation friendly columnar storage. I demonstrate how to export, transform, and load intermediate files into AWS S3 from the Postgres side then, finish by showing how to load the S3 files into Redshift staging tables then utilize an upsert pattern to load into the target production tables. I conclude the article with a section on using AWS Glue for performing Extract, Transform, and Load from a Postgres compatible Aurora database loaded with the OLTP Pagila schema. Additionally, all source code for this tutorial is available on my GitHub account in a public repo. For reference I've created the following AWS Pricing Calculator estimate of the charges based off the services being used. I do demonstrate how to destroy the Terraform provisioned services as well so, definitely use them along with the Terraform docs if necessary to cleanup when you are finished to avoid unnecessary charges. However, please understand that these AWS services being provisioned do cost money at a rate of somewhere between $1 - $2 USD per hour. In order to lower the barrier for a reader to get up and running with the technologies used in this tutorial I've provided fairly complete () which can be used to provision resources in the AWS cloud and follow along. Then armed with this basic knowledge of Redshift architecture I move on to give a practical example of designing a schema optimal for Redshift based off the Pagila sample dataset. I start with a basic overview of the unique architecture Redshift uses to accomplish its scalable and robust use case as an enterprise cloud data warehouse. In this article I give a practical introductory tutorial to using Amazon Redshift as an OLAP Data Warehouse solution for the popular Pagila Movie Rental dataset.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |