All posts by Vetri Natarajan

Automate and accelerate your Amazon QuickSight asset deployments using the new APIs

Post Syndicated from Vetri Natarajan original https://aws.amazon.com/blogs/big-data/automate-and-accelerate-your-amazon-quicksight-asset-deployments-using-the-new-apis/

Business intelligence (BI) and IT operations (BIOps) teams often need to automate and accelerate the deployment of BI assets to ensure business continuity. We heard that you wanted an automated and scalable way to deploy, back up, and replicate Amazon QuickSight assets at scale so that BIOps teams within your organization can work in an agile manner.

Today, we are releasing six new QuickSight APIs to allow programmatic access to export and import QuickSight assets—dashboards, analyses, datasets including ingestion schedules, data sources, themes, and VPC configurations—across accounts and environments. These new APIs allow you to interact with a collection of assets in a lift-and-shift manner for deployment across QuickSight accounts, enable backup and restore, and support replication so you can automate workflows and achieve your desired infrastructure setup. These new capabilities bring greater agility to your BIOps teams, allowing you to automate and seamlessly integrate QuickSight assets into existing infrastructure.

Prior to this launch, you needed to have an in-depth understanding of QuickSight asset relationships and couldn’t deploy, back up, or replicate at scale in an automated manner. In this post, we cover the capabilities of the new APIs in detail and go over common use cases.

Export APIs

You can use the following APIs to initiate, track, and describe the export jobs that produce the bundle files from the source account. A bundle file is a zip file (with the .qs extension) that contains assets specified by the caller, and optionally all dependencies of the assets. The APIs are as follows:

  • StartAssetBundleExportJob – Use this asynchronous API to export an asset bundle file.
  • DescribeAssetBundleExportJob – Use this synchronous API to get the status of your export job. When successful, this API call response will have a presigned URL to fetch the asset bundle.
  • ListAssetBundleExportJobs – Use this synchronous API to list past export jobs. The list will contain both finished and running jobs from the past 15 days.

Import APIs

These APIs initiate, track, and describe the import jobs that take the bundle file as input and create or update assets in the destination account:

Supported assets

You can export and import the following assets with these APIs:

  • Analyses
  • Dashboards
  • Data sources
  • Datasets including scheduled and incremental refreshes
  • Themes
  • VPC connections

Asset bundle output format

The output of the export job is a single zip file with the .qs extension. This zip file contains a separate folder for each asset type. Each folder contains a single JSON file for each asset with the resourceId as the file name. This folder structure makes it easy to commit the contents into a version control system like Git, so you can get the benefits of a complete version history.

The Asset-bundle API can also export QuickSight assets as AWS CloudFormation templates, one of the most popular infrastructure as code (IaC) frameworks. It makes it easy to manage your QuickSight assets at scale and automate your deployments. AWS CloudFormation also has built-in transaction and rollback capabilities, ensuring that all your environments are consistent and your assets are deployed correctly every time. Finally, you can use the CloudFormation templates to recreate your QuickSight resources in case of a disaster.

Permissions required

These APIs are available to users with AWS Identity and Access Management (IAM) permissions to run these APIs. The following IAM policy allows an IAM user to get access to these APIs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [          
                "quicksight:StartAssetBundleImportJob",
                "quicksight:DescribeAssetBundleImportJob",
                "quicksight:ListAssetBundleImportJobs",
                "quicksight:StartAssetBundleExportJob",
                "quicksight:DescribeAssetBundleExportJob",
                "quicksight:ListAssetBundleExportJobs"
            ],
            "Resource": "*"
        }
    ]
}

Use case overview

Let’s consider a fictional company, AnyCompany, which owns healthcare facilities across the globe. They have set up a development QuickSight account for authors to create and update QuickSight assets and a separate production account. In some cases due, to data residency regulation, they have to maintain the same assets across multiple Regions. AnyCompany is scaling their business and they want to automate deployment within and across multiple QuickSight accounts and back up QuickSight assets on a schedule.

AnyCompany has the following key deployment and backup requirements:

  • Deployment – Deploy QuickSight assets across Regions and multiple accounts:
    • Deployment to the production account – AnyCompany wants to automate the deployment of QuickSight assets from their development to their production account.
    • Deployment to different Regions in the same account – AnyCompany’s central IT team needs to deploy dashboards and datasets across various Regions to meet data residency requirements.
    • Deployment to multiple accounts in different Regions – To meet their end customer requirements of separate QuickSight accounts, AnyCompany needs to deploy the dashboards and datasets across multiple accounts.
    • Deployment in the same account and same Region – AnyCompany consolidates all non-production environment into one QuickSight account. However, there has to be different dashboards and datasets for each non-production environment, such as development and testing.
  • Backup and restore – As AnyCompany rolls out critical dashboards for business, it needs to ensure high availability of the dashboards. As part of their strategy, AnyCompany wants to maintain a backup of assets to restore in case of disasters.
  • Deployment history – As part of the governed process, AnyCompany’s central IT team needs to have a history of deployments in each environment.

In the following sections, we discuss how to meet these requirements.

Deploy to a production account

The following figure shows the sequence of steps in the deployment process through the new asset deployment APIs.

For deployments, the import job API provides the capability to pass data source configurations to point to the respective test or production instances of data sources. In the preceding sequence flow, we use the AWS Command Line Interface (AWS CLI) to showcase the capability, but you can invoke the APIs through your automation pipeline using AWS SDKs.

For this use case, AnyCompany used Amazon Simple Storage Service (Amazon S3) to store their asset bundles..

On the development QuickSight account, complete the following steps:

  1. Use the StartAssetBundleExportJob API to export the dashboard and its dependencies.
  2. Place the output asset bundle in an S3 bucket in the production account.

On the production QuickSight account, complete the following steps:

  1. Use the StartAssetBundleImportJob API with the asset bundle in Amazon S3 as the source, overriding the data source details.
  2. Run the import job.

The following code shows their StartAssetBundleExportJob API call to export the dashboard and its dependencies:

aws quicksight start-asset-bundle-export-job 
--aws-account-id $AAI 
--asset-bundle-export-job-id job-1 
--resource-arns arn:aws:quicksight:$IR:$AAI:dashboard/<<dashboard-id>> 
--include-all-dependencies 
--export-format QUICKSIGHT_JSON

The following code is for the DescribeAssetBundleExportJob API:

aws quicksight describe-asset-bundle-export-job 
--aws-account-id $AAI 
--asset-bundle-export-job-id job-1

The output of the DescribeAssetBundleExportJob API call contains the presigned URL, which you use to download your respective assets and subsequently upload them to a dedicated S3 bucket in the target account.

The import job (StartAssetBundleImportJob) is initiated in the target account using S3Uri as one of the input parameters. You can also change the data source configuration while initiating the job. In the following example, the S3 manifest file location for the S3 data source is overridden:

aws quicksight start-asset-bundle-import-job
 --aws-account-id $AAI 
--asset-bundle-import-job-id job-1 
--asset-bundle-import-source "{\"S3Uri\": \"<<qsfile location\"}"
 --override-parameters '{"DataSources": 
 [{"DataSourceId": "<<DataSourceID>>", 
 "DataSourceParameters": 
 { "S3Parameters": {"ManifestFileLocation": 
 {"Bucket": "<<bucket name>>", "Key": "<<key for manifest file>>"}}}}]}' 
--region us-west-2

DescribeAssetBundleImportJob lets you to monitor the status of the import job:

aws quicksight describe-asset-bundle-import-job 
--aws-account-id $AAI 
--asset-bundle-import-job-id job-1 
--region us-west-2

The following screenshot shows the response.

Deploy to different Regions in the different accounts

To comply with data residency regulations, data can’t be moved outside a Region in certain countries. Therefore, the dashboards have to be deployed in each of these Regions. QuickSight provides the option to pass an asset bundle extracted from the source environment as a base64 encoded string for the import job (StartAssetBundleImportJob):

aws quicksight start-asset-bundle-import-job \
 --aws-account-id $AAI  \
--asset-bundle-import-job-id <<job-id>> \ 
--asset-bundle-import-source Body="$(base64 assetbundle-extract.qs)"

Deploy within a single account in the same Region

When using a centralized account approach for all the lower environments, AnyCompany wanted to have the same dashboards within a single Region to be able to connect with different data sources. To achieve this, they used the optional parameter resource-id-prefix in the import job (StartAssetBundleImportJob) to create a unique ID for each environment:

aws quicksight start-asset-bundle-import-job
--aws-account-id $AAI 
--asset-bundle-import-job-id job-1 
--asset-bundle-import-source "{\"S3Uri\": \"s3://qs file s3 location\"}"
--override-parameters '{"DataSources": 
[{"DataSourceId": "<<DataSource ID>>", 
"DataSourceParameters":{ "S3Parameters": {"ManifestFileLocation": 
{"Bucket": "ee-assets-prod-us-west-2", 
"Key": "modules/337d5d05acc64a6fa37bcba6b921071c/v1/SalesDataManifest.json"}}}}]}' 
--region us-west-2
--resource-id-prefix "test-"

Backup and version control

AnyCompany deploys business-critical dashboards, and it’s important for them to have proper backup and version control processes. They run scheduled export jobs at regular intervals along with asset deployments. Additionally, they use asset bundle APIs to meet their version control requirements.

The following screenshot shows the content of a sample bundle.

Deployment history

AnyCompany needs to track the deployment history of all the assets in all environments. They achieved this goal by using the ListAssetBundleExportJobs and ListAssetBundleImportJobs APIs to fetch the deployment history in a given account.

The following code is for ListAssetBundleExportJobs:

aws quicksight list-asset-bundle-export-jobs \
 --aws-account-id $AAI  \

The following code is for ListAssetBundleImportJobs:

aws quicksight list-asset-bundle-export-jobs \
 --aws-account-id $AAI  \

Conclusion

Asset bundle APIs provide methods for automation and acceleration in the deployment process across multiple environments. This post illustrated various use cases where you can apply these APIs for automation and scale. For more information, refer to Amazon QuickSight and What’s New in the Amazon QuickSight User Guide.

If you have any questions or feedback, please leave a comment. For additional discussions and help getting answers to your questions, check out the QuickSight Community.

About the authors

Vetri Natarajan is a Specialist Solutions Architect for Amazon QuickSight. Vetri has 15 years of experience implementing enterprise business intelligence (BI) solutions and greenfield data products. Vetri specializes in integration of BI solutions with business applications and enable data-driven decisions.

Zhao Pan is a software development manager for Amazon QuickSight. He is working to provide a delightful developer experience to our customers to automate and streamline their BI operations. He has 20 years of software development experience in various tech stacks. Prior to QuickSight he was a people and technical leader at ADP building a next-gen platform for human capital management. When he is not at his desk, he can usually be found in his garage building one contraption or another.

Mayank Agarwal is a product manager for Amazon QuickSight, AWS’ cloud-native, fully managed BI service. He focuses on embedded analytics and developer experience. He started his career as an embedded software engineer developing handheld devices. Prior to QuickSight he was leading engineering teams at Credence ID, developing custom mobile embedded device and web solutions using AWS services that make biometric enrollment and identification fast, intuitive, and cost-effective for Government sector, healthcare and transaction security applications.

Scale Amazon QuickSight embedded analytics with new API-based domain allow listing

Post Syndicated from Vetri Natarajan original https://aws.amazon.com/blogs/big-data/scale-amazon-quicksight-embedded-analytics-with-new-api-based-domain-allow-listing/

Amazon QuickSight is a fully-managed, cloud-native business intelligence (BI) service that makes it easy to connect to your data, create interactive dashboards, and share these with tens of thousands of users, either within QuickSight itself or embedded in apps and portals.

QuickSight Enterprise Edition recently introduced the ability to dynamically allow list the domains where QuickSight content can be embedded. This allows developers to quickly embed dashboards across multiple apps, portals, or websites, without needing to make this change on the QuickSight administrative console every time. Together with QuickSight’s existing dashboard theming and templating capabilities, this new feature allows developers to rapidly develop and deploy QuickSight dashboards and visualizations for a variety of use cases across various applications with ease. Let’s take a look at how this works.

Solution overview

To embed a QuickSight dashboard using APIs, you can use one of the following embedding APIs:

In these APIs, you can now pass the domain where you want to embed your dashboard using the new parameter AllowedDomains:

POST /accounts/AwsAccountId/embed-url/registered-user HTTP/1.1
Content-type: application/json
 
{
   "AllowedDomains": [ "string" ],
   "ExperienceConfiguration": { 
      "Dashboard": { 
         "InitialDashboardId": "string"
      },
      "QSearchBar": { 
         "InitialTopicId": "string"
      },
      "QuickSightConsole": { 
         "InitialPath": "string"
      }
   },
   "SessionLifetimeInMinutes": number,
   "UserArn": "string"
}

You can add up to three domains in a single API call as an array list. All the domains need to be SSL enabled (using HTTPS protocol). If you want to test out the embedded dashboard on your local machine, you can allow list http://localhost via the AllowedDomains parameter. For example, if you want to embed a dashboard in your SaaS application called https://myorders.simplelogistics.com, you set AllowedDomains to be https://myorders.simplelogistics.com in the API call. You can also enable sub domains by passing *, for example, https://*.myorders.simplelogistics.com.

AllowedDomains is an optional parameter. If you don’t specify any domains via this parameter, you can still use the domains allow listed via the QuickSight console. But if you specify domains via this parameter, then the embedding URL returned as part of the API call is only embeddable in these domains (even if you have a list of static domains entered on the QuickSight console).

Prior to this capability, the Content-Service-Policy in the request header listed all the domains allow listed in QuickSight console. Now when allow listing the domains using the API, the Content-Service-Policy only shows the domains that are allow listed in the API call.

With this new capability, ISVs that have different applications for different customers can allow list specific domains at runtime, enabling them to scale easily for different customers and to hundreds of thousands of end-users.

As an added security, the AWS Identity and Access Management (IAM) admin of your QuickSight account can restrict the domains that can be allow listed. This can be done when your IAM admin sets up permissions for your application or server. As part of this step, you can specify the list of domains that can be allow listed via the embedding APIs. For example, let’s assume you want your developers to only allow list the following domains:

You can set these domains in the quicksight:AllowedEmbeddingDomain of the permissions setup. The following code is a sample for the GenerateEmbedURLForAnonymousUser API:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
            "quicksight:GenerateEmbedUrlForAnonymousUser"
            ],
            "Resource": "arn:partition:quicksight:region:accountId:user/namespace/userName",
            "Condition": {
                "ForAllValues:StringEquals": {
                    "quicksight:AllowedEmbeddingDomains": [
                        "https://myorders.simplelogistics.com",
                        "https://cheapelectornics.simplelogistics.com",
                        "https://myreturns.simplelogistics.com"
                    ]
                }
            }
        }
    ]
}

Sample use case

In this example use case, Travel Analytics is a software as a service (SaaS) provider with travel-related solutions for various travel agencies. They have a SaaS application for these agencies to track different metrics on how their business is performing. Because Travel Analytics is scaling their business, they have different sites for different travel agencies. With the newly launched domain allow listing with APIs, they’re able to scale with ease. They allow list the specific domains, depending on the customer, via the API when generating the embedding URL.

The following code shows their sample GenerateEmbedURLForAnonymousUser API call with the domain added to the request:

The returned URL can only be embedded in the domain that was allow listed as part of the preceding request. The following is a screenshot of the embedded dashboard in this domain.

The CSP header has only the specific allow listed domain via the API when the dashboard is embedded.

Conclusion

Runtime domain allow listing using embedding APIs enables developers to scale their embedded offerings with QuickSight dashboards, visuals, QuickSight Q (natural language querying), or authoring experience across different domains for their different customers easily. All of this is done without any infrastructure setup or management, while scaling to millions of users. For more information, refer to Amazon QuickSight Embedded Analytics and What’s New in the Amazon QuickSight User Guide.

About the authors

Vetri Natarajan is a Specialist Solutions Architect for Amazon QuickSight. Vetri has 15 years of experience implementing enterprise Business Intelligence (BI) solutions and greenfield data products. Vetri specializes in integration of BI solutions with business applications and enable data-driven decisions.

Kareem Syed-Mohammed is a Product Manager at Amazon QuickSight. He focuses on embedded analytics, APIs, and developer experience. Prior to QuickSight he has been with AWS Marketplace and Amazon retail as a PM. Kareem started his career as a developer and then PM for call center technologies, Local Expert and Ads for Expedia. He worked as a consultant with McKinsey and Company for a short while.