Post Syndicated from Siamak Ziraknejad original https://aws.amazon.com/blogs/big-data/amazon-identity-services-uses-amazon-quicksight-to-empower-partners-with-self-serve-data-discovery/
Amazon Identity Services is responsible for the way Amazon customers—buyers, sellers, developers—identify themselves on Amazon. Our team also manages customers’ core account information, such as names and delivery addresses. Our mission is to deliver the most intuitive, convenient, and secure authentication experience. We’re in charge of account security for Amazon, worldwide, on all device surfaces.
Identity systems make millions of security decisions per second. We ingest datasets at a large scale—processing 9 TB per hour—and produce analytical datasets that grow by billions of rows per hour. Our core business metrics within the Amazon Identity Services team are built on top of these datasets, which we use for leadership meetings, product launch decisions, metric movement investigations, and discovering new innovation opportunities to simplify security experiences for our customers.
In this post, we discuss how we use Amazon QuickSight to empower partners with self-serve data discovery.
Inaccessible insights block data-driven decisions
The sheer volume of our datasets made gathering insights a slow process. Not only that, but datasets weren’t accessible to a wide audience outside our team, such as partners, program managers, product managers, and so on. As a result, Business Intelligence Engineers (BIEs) spent a lot of time writing ad hoc queries, which then took a long time to run. When the insights were ready, BIEs were tasked with answering questions via manual processes that didn’t scale.
We chose QuickSight to not only speed up our processing times, but to create efficiencies with self-serve insight access via analyses (exploration and authoring) and dashboards for consumption. With partners and stakeholders having the ability to access insights without assistance, our BIEs were able to shift focus from ad hoc requests to more impactful projects that were a better use of their skills and expertise.
In the following sections, we discuss what we were looking for in BI capabilities, and how QuickSight satisfied those requirements for our team.
Removing the middle person with QuickSight
Imagine being the pilot of a commercial airline, navigating in cloudy conditions. You know your destination lies ahead, but you can’t see it; you have to rely on your dashboard of instruments to navigate so you’ll arrive safely. It’s similar when working on large-scale consumer products. Although our team gathers anecdotes and reviews feedback to form hypotheses on what our customers need, only by analyzing data at scale can we truly understand customer problems and design appropriate solutions.
The status quo that positioned our BIEs between stakeholders and the insights that were needed required a manual, error-prone process, with a time-to-insight that could take weeks. Even more problematic was that insights were limited to what the requestor envisioned. There was no flexibility to explore and visualize data with a simple drag-and-drop UI. This inability to explore and interact with available data meant stakeholders didn’t know the best questions to ask. Our team needed to make data more accessible to partner teams and non-BIE users, and we needed that access to be fast, intuitive, and to provide a single, indisputable source of truth.
With our research into BI tool options, we were looking for the following:
- Accessible insights – We needed to ensure users from all levels of technical experience would be able to access and understand the insights provided to them
- Speed – With an ingestion rate of 9 TB of data per hour, we needed our BI tool to be fast and reliable
- Security – Built-in row-level and column-level security would give us the ability to provide on-demand access to thousands of users across AWS
The first option we considered had a lot of great features, but it wouldn’t scale without a server. The next option we looked at was very capable in a lot of areas, but it wouldn’t be as accessible for non-BIE users, and it also required a team to manage a server. QuickSight was a great fit because it’s not only serverless, but also has enough visualization capabilities to make it useful for self-service data.
QuickSight also offered seamless integration with Amazon Redshift, and the ability to publish our business metrics to QuickSight SPICE (Super-fast, Parallel, In-memory Calculation Engine), its robust in-memory engine. SPICE performs rapid advanced calculations and serves data. What we love most about SPICE is that it greatly reduces time-to-insight, supports column-level security for staying in compliance, and most importantly it’s super fast for data exploration within analyses.
Self-service data discovery just a few clicks away
Publishing our metrics to QuickSight SPICE enabled us to create pre-authored dashboards, and empowered users to create their own analytical content via analyses. Our technical program managers have all been trained on how to use QuickSight to create visualizations, whereas our BIE team members are dedicating their time to creating better datasets. Our non-tech partners and product managers no longer need to depend on a BIE to get answers to their questions, because they can create analyses and query billions of records with a drag-and-drop interface to instantly visualize data.
The following screen shot shows what our year-to-date visualization looks like, with all sensitive data redacted.
The BIE time saved as a result of stakeholders getting self-service answers can now be invested in building richer and better quality datasets, creating a virtuous cycle to help accelerate our ability to adapt and improve to meet our customers’ needs.
Another important benefit of using QuickSight was that we centralized a semantic layer, unifying the language we speak across departments by publishing authoritative datasets in SPICE with proper access control. Because the data was easy to use and accessible with pre-calculated metrics, our partner teams didn’t have to re-invent the metric definitions. To ensure everyone stays on the same page, we publish all documentation to internal wikis.
More efficient business reviews with paginated reports and Amazon QuickSight Q
Our north star is to completely automate our periodic business review processes, similar to how the AWS Analytics Sales team is currently using QuickSight Q in their monthly business reviews. Because Q enables simple querying of data in real time via natural language, we can reduce the time it takes to author analytical content, eliminate redundant manual work, and simplify interactivity with data.
With QuickSight, we are automating all the dashboard and analyses generation for the business reviews. Doing so enables us to focus more on generating insights and conducting relevant investigations every month, rather than spending time and energy querying for data. Specifically, the new paginated report object type enables us to produce highly formatted content for leadership and formal reviews.
To learn more about how you can embed customized data visuals, interactive dashboards, and natural language querying into any application, visit Amazon QuickSight Embedded.
About the Authors
Siamak Ziraknejad leads the technical product management team for Amazon Identity Services. His team formulates the technical product strategy and plans for account security (authentication and authorization across all surfaces, worldwide) and the consumer identity foundations for all Amazon programs and products (entitlement management, benefit sharing, and personalization).
Abhinav Mehta is a Senior Product Manager (Technical) with the Amazon Identity Services team. He is focused on the product strategy and innovations for fast and secure authentication methods at Amazon.