{"id":1044211,"date":"2021-03-28T00:00:00","date_gmt":"2021-03-28T00:00:00","guid":{"rendered":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/"},"modified":"2024-02-20T02:50:29","modified_gmt":"2024-02-20T02:50:29","slug":"leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud","status":"publish","type":"customer-story","link":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/","title":{"rendered":"Leading digital advertising agency advances data analytics using Vertica on AWS cloud"},"content":{"rendered":"\n<p><strong>THE CHALLENGE<\/strong><\/p>\n\n\n<p>The client wanted to use their business intelligence tool to query their historical data for 90 days in under two seconds with more than 80 concurrent users. Their current system included data in Hadoop clusters and Infobright DB on-premise cluster, which was unable to handle their data analytics requirements.<\/p>\n\n\n<p><strong>THE SOLUTION<\/strong><\/p>\n\n\n<p>Beyondsoft\u2019s Big Data consulting team proposed a solution using Vertica, a columnar database on Amazon Web Services (AWS). The AWS cloud solution would provide the added scalability, elasticity, and performance that the customer wanted. The project consisted of creating Vertica clusters in a repeatable manner and a pipeline-based approach for Vertica DDL. It also included moving large amounts of data daily to the Vertica cluster.<\/p>\n\n\n<p><strong>The project consisted of three phases:<\/strong><\/p>\n\n\n<ul class=\"wp-block-list\"><li>Phase 1 involved creating a repeatable deployment process through infrastructure as code (Terraform) for Vertica cluster. The Vertica AMI was procured from AWS Marketplace. Beyondsoft engineers added the ability to launch different Vertica nodes as part of a cluster through tags to AWS Elastic Load Balancer (ELB). The Vertica cluster consist of two AZ\u2019s in active\/active nodes, with 16 nodes total and 90 days of data, which came to approximately 40TB. AWS Systems Manager Service (SSM) and CloudWatch Logs are used for administration of the cluster. This Vertica infrastructure as code also integrates with the customer&#8217;s self-servicing tool for their developers.<\/li><li>Phase 2 included a pipeline-based approach for Vertica DDL. Vertica DDL is pushed through the pipeline using Liquibase, a java framework for database change and deployment. This ensures that production Vertica clusters are not touched manually for schema changes.<\/li><li>Phase 3 involved setting up ETL from the Hadoop cluster to Vertica using a producer\/consumer pattern. The ETL code is written in python code with on-demand Fargate containers, which extract data from Hadoop and store it in zipped files in S3. From there, jobs are created to load data into Vertica from S3. The data is around 120GB\/day with around 570M rows loaded at its peak. The customer-facing java application has several dashboards which are able to procure data from Vertica in under two seconds query time with concurrent usage.<\/li><\/ul>\n\n\n<p><\/p>\n\n\n<p><strong>TECHNOLOGIES USED<\/strong><\/p>\n\n\n<p>Vertica on AWS, AWS SSM, AWS CloudWatch logs, AWS S3, AWS ELB, AWS Fargate, AWS Parameter Store, AWS ECR, Python, Jenkins, etc.<\/p>\n\n\n<p><\/p>\n\n\n<p><strong>KNOWLEDGE TRANSFER<\/strong><\/p>\n\n\n<p>Beyondsoft educated the client&#8217;s data analytics team around the newly created solution and Terraform and provided a runbook, to enable them to both manage and add to the solution in future. Beyondsoft also provided education on the various AWS services and customized training sessions on various topics.<\/p>\n\n\n<p><\/p>\n\n\n<p><strong>BENEFITS<\/strong><\/p>\n\n\n<p>Moving from an on-premise cluster to the cloud increased the scalability, agility, and performance of the whole solution. Taking a DevOps approach through data pipelines decreased go-to-market time for code changes. Infrastructure as code provided a repeatable way to create infrastructure, increasing operational consistency and reducing bugs. <\/p>\n\t\t\n\t\t\t<a href=\"http:\/\/www.beyondsoft.com\/wp-content\/uploads\/2021\/05\/Leading-digital-advertising-agency-advances-data-analytics-using-Vertica-on-AWS-cloud.pdf\" role=\"button\">\n\t\t\t\t\t\tDownload Case Study\n\t\t\t\t\t<\/a>","protected":false},"excerpt":{"rendered":"<p>Integral Ad Science (IAS) is a global market leader in digital ad verification, offering technologies that drive high-quality advertising media. IAS equips advertisers and publishers with the insight and technology to protect advertising investments from fraud and unsafe environments as well as to capture consumer attention and deliver business outcomes. Founded in 2009, IAS is headquartered in New York, with global operations in 18 offices across 13 countries.<\/p>\n","protected":false},"featured_media":1044139,"template":"","service-category":[45,44],"industry-category":[51],"external-author":[],"class_list":["post-1044211","customer-story","type-customer-story","status-publish","has-post-thumbnail","hentry","service-category-cloud-adoption-and-transformation","service-category-data-and-ai-modernization","industry-category-high-tech"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Leading digital advertising agency advances data analytics using Vertica on AWS cloud - Beyondsoft<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Leading digital advertising agency advances data analytics using Vertica on AWS cloud - Beyondsoft\" \/>\n<meta property=\"og:description\" content=\"Integral Ad Science (IAS) is a global market leader in digital ad verification, offering technologies that drive high-quality advertising media. IAS equips advertisers and publishers with the insight and technology to protect advertising investments from fraud and unsafe environments as well as to capture consumer attention and deliver business outcomes. Founded in 2009, IAS is headquartered in New York, with global operations in 18 offices across 13 countries.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/\" \/>\n<meta property=\"og:site_name\" content=\"Beyondsoft\" \/>\n<meta property=\"article:modified_time\" content=\"2024-02-20T02:50:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2023\/12\/case-study-sample.jpeg\" \/>\n\t<meta property=\"og:image:width\" content=\"724\" \/>\n\t<meta property=\"og:image:height\" content=\"483\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/\",\"url\":\"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/\",\"name\":\"Leading digital advertising agency advances data analytics using Vertica on AWS cloud - Beyondsoft\",\"isPartOf\":{\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2023\/12\/case-study-sample.jpeg\",\"datePublished\":\"2021-03-28T00:00:00+00:00\",\"dateModified\":\"2024-02-20T02:50:29+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/#primaryimage\",\"url\":\"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2023\/12\/case-study-sample.jpeg\",\"contentUrl\":\"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2023\/12\/case-study-sample.jpeg\",\"width\":724,\"height\":483},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.beyondsoft.com\/sg\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Customer Stories\",\"item\":\"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Leading digital advertising agency advances data analytics using Vertica on AWS cloud\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/#website\",\"url\":\"https:\/\/www.beyondsoft.com\/sg\/\",\"name\":\"Beyondsoft\",\"description\":\"Choose Beyondsoft for comprehensive business and IT services\",\"publisher\":{\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.beyondsoft.com\/sg\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/#organization\",\"name\":\"Beyondsoft\",\"url\":\"https:\/\/www.beyondsoft.com\/sg\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2024\/03\/cropped-bci-logo.png\",\"contentUrl\":\"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2024\/03\/cropped-bci-logo.png\",\"width\":757,\"height\":201,\"caption\":\"Beyondsoft\"},\"image\":{\"@id\":\"https:\/\/www.beyondsoft.com\/sg\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Leading digital advertising agency advances data analytics using Vertica on AWS cloud - Beyondsoft","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/","og_locale":"en_US","og_type":"article","og_title":"Leading digital advertising agency advances data analytics using Vertica on AWS cloud - Beyondsoft","og_description":"Integral Ad Science (IAS) is a global market leader in digital ad verification, offering technologies that drive high-quality advertising media. IAS equips advertisers and publishers with the insight and technology to protect advertising investments from fraud and unsafe environments as well as to capture consumer attention and deliver business outcomes. Founded in 2009, IAS is headquartered in New York, with global operations in 18 offices across 13 countries.","og_url":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/","og_site_name":"Beyondsoft","article_modified_time":"2024-02-20T02:50:29+00:00","og_image":[{"width":724,"height":483,"url":"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2023\/12\/case-study-sample.jpeg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/","url":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/","name":"Leading digital advertising agency advances data analytics using Vertica on AWS cloud - Beyondsoft","isPartOf":{"@id":"https:\/\/www.beyondsoft.com\/sg\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/#primaryimage"},"image":{"@id":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/#primaryimage"},"thumbnailUrl":"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2023\/12\/case-study-sample.jpeg","datePublished":"2021-03-28T00:00:00+00:00","dateModified":"2024-02-20T02:50:29+00:00","breadcrumb":{"@id":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/#primaryimage","url":"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2023\/12\/case-study-sample.jpeg","contentUrl":"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2023\/12\/case-study-sample.jpeg","width":724,"height":483},{"@type":"BreadcrumbList","@id":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/leading-digital-advertising-agency-advances-data-analytics-using-vertica-on-aws-cloud\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.beyondsoft.com\/sg\/"},{"@type":"ListItem","position":2,"name":"Customer Stories","item":"https:\/\/www.beyondsoft.com\/sg\/customer-stories\/"},{"@type":"ListItem","position":3,"name":"Leading digital advertising agency advances data analytics using Vertica on AWS cloud"}]},{"@type":"WebSite","@id":"https:\/\/www.beyondsoft.com\/sg\/#website","url":"https:\/\/www.beyondsoft.com\/sg\/","name":"Beyondsoft","description":"Choose Beyondsoft for comprehensive business and IT services","publisher":{"@id":"https:\/\/www.beyondsoft.com\/sg\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.beyondsoft.com\/sg\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.beyondsoft.com\/sg\/#organization","name":"Beyondsoft","url":"https:\/\/www.beyondsoft.com\/sg\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.beyondsoft.com\/sg\/#\/schema\/logo\/image\/","url":"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2024\/03\/cropped-bci-logo.png","contentUrl":"https:\/\/www.beyondsoft.com\/sg\/wp-content\/uploads\/sites\/6\/2024\/03\/cropped-bci-logo.png","width":757,"height":201,"caption":"Beyondsoft"},"image":{"@id":"https:\/\/www.beyondsoft.com\/sg\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.beyondsoft.com\/sg\/wp-json\/wp\/v2\/customer-story\/1044211","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.beyondsoft.com\/sg\/wp-json\/wp\/v2\/customer-story"}],"about":[{"href":"https:\/\/www.beyondsoft.com\/sg\/wp-json\/wp\/v2\/types\/customer-story"}],"version-history":[{"count":1,"href":"https:\/\/www.beyondsoft.com\/sg\/wp-json\/wp\/v2\/customer-story\/1044211\/revisions"}],"predecessor-version":[{"id":1044481,"href":"https:\/\/www.beyondsoft.com\/sg\/wp-json\/wp\/v2\/customer-story\/1044211\/revisions\/1044481"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.beyondsoft.com\/sg\/wp-json\/wp\/v2\/media\/1044139"}],"wp:attachment":[{"href":"https:\/\/www.beyondsoft.com\/sg\/wp-json\/wp\/v2\/media?parent=1044211"}],"wp:term":[{"taxonomy":"service-category","embeddable":true,"href":"https:\/\/www.beyondsoft.com\/sg\/wp-json\/wp\/v2\/service-category?post=1044211"},{"taxonomy":"industry-category","embeddable":true,"href":"https:\/\/www.beyondsoft.com\/sg\/wp-json\/wp\/v2\/industry-category?post=1044211"},{"taxonomy":"external-author","embeddable":true,"href":"https:\/\/www.beyondsoft.com\/sg\/wp-json\/wp\/v2\/external-author?post=1044211"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}