Tag Archives: AWS

Send voice appointment reminders using Amazon Pinpoint custom channels and Amazon Connect

Post Syndicated from Ryan Lowe original https://aws.amazon.com/blogs/messaging-and-targeting/send-voice-appointment-reminders-using-amazon-pinpoint-custom-channels-and-amazon-connect/

Introduction

In this post, we will walk through setting up an always-on appointment reminder campaign in Amazon Pinpoint. No-show rates are a constant challenge for service providers. Industries such as hospitality estimate 20% of diners miss reservations in big cities,1 while salons average five missed appointments per week.2 Professional services such as financial institutions and sales teams have similar challenges to ensure clients do not miss meetings. To these businesses, an appointment missed represents lost revenue. As a result, the no-show rate is a key metric to improve. An outbound voice message provides another way to reach customers versus emails or SMS, and voice reminders give customers the choice of channels based on personal preferences.

Overview

Amazon Pinpoint is a multichannel communications service enabling customers to send both promotional and transactional messages across email, SMS, push notifications, voice, and custom channels. Amazon Connect is an easy to use omnichannel cloud contact center that helps companies provide superior customer service at a lower cost.

There are benefits of using these services together. Amazon Pinpoint allows you to build a segment of users which can be used within a campaign. Amazon Connect can enable customers to send outbound voice messages at scale should your user audience be large and require a high number of transactions per second (TPS).

To use these services together, you setup custom channels in Amazon Pinpoint, which can be created via an AWS Lambda function. These functions enable you to call APIs to trigger message sends as part of Amazon Pinpoint campaigns. Amazon Pinpoint has developed a new AWS Lambda function which can be used to send outbound voice messages via Amazon Connect. This configuration allows you to define the voice message to be sent, define the segment of users you would like to target, and send voice messages at scale through Amazon Connect via the Amazon Pinpoint custom channel.

The audience for this solution are technical customers who are used to working with multiple AWS services and are familiar with AWS Lambda functions. The solution built relies on the Amazon Pinpoint custom channel feature and targeting, along with the Amazon Connect outbound voice API called via a prepared AWS Lambda function. Once completed, you will be able to create an evergreen campaign which will send outbound voice messages to your patients who have an appointment the following day.

The costs associated with this solution will be:

  1. Amazon Connect outbound voice calls per minute
  2. Amazon Connect claimed phone number(s)
  3. Amazon Pinpoint Monthly Targeted Audience (MTA) costs.

The costs for a outbound voice reminder system that sends 10k messages per day, with an average length of 20 seconds per call, to an total monthly audience of 300k, in the US are as follows. Note that prices with vary for other countries. Complete Amazon Connect outbound call pricing can be found here.

Solution

Prerequisites:

For this walkthrough the article assumes:

  • An AWS account
  • Basic understanding of IAM and privileges required to create the following; IAM identity provider, roles, policies, and users
  • Basic understanding of Amazon Pinpoint and how to create a project
  • Basic understanding of Amazon Connect and experience in creating contact flows. More information on setup of Amazon Connect can be found here.

Step 1: Create an Appointment Reminder custom event

The first step in setting up this solution is to create and report a custom event to Amazon Pinpoint. There are multiple ways to report events in your application. Ffor demonstration purposes, below are two example event calls using the AWS SDK for Python (Boto3) from inside an AWS Lambda Function.

It is important to note that the Amazon Pinpoint events API can also be used to update endpoints when the event gets registered. In the below example, the first API call will update the endpoint attributes AppointmentDate and AppointmentTime with the details of the upcoming appointment. These attributes will be used in the outgoing message to the end-user

Sample Event: Appointment Coming Up

import boto3

client = boto3.client('pinpoint')
app_id = '[PINPOINT_PROJECT_ID]'
endpoint_id = '[ENDPOINT_ID]'
address = '[PHONE_NUMBER]'

def lambda_handler(event, context):

client.put_events(
ApplicationId = applicationId,
EventsRequest={
'BatchItem': {
endpoint_id: {
'Endpoint': {
'ChannelType': 'CUSTOM',
'Address': address,
'Attributes': {
'AppointmentDate': ['December 15th, 2020'],
'AppointmentTime': ['2:15pm']
}
},
'Events':{
'appointment-event': {
'Attributes':{},
'EventType': 'AppointmentReminder',
'Timestamp': datetime.datetime.fromtimestamp(time.time()).isoformat()
}
}
}
}
}
)

NOTE: The following steps assume that the AppointmentReminder event is being reported to Amazon Pinpoint. If you are unable to integrate the above API call into your application, you can manually create an AWS Lambda function using a Python runtime with the above code to trigger sample events.

Step 2: Create an Amazon Connect contact flow for outbound calls

This article assumes that you have an Amazon Connect contact center already setup and working. In this step, we will set up our Amazon Connect contact flow to dial our recipients and play read the message before hanging up.

  1. Log in to your Amazon Connect instance using your access URL (https://<alias>.awsapps.com/connect/login).
    Note: Replace alias with your instance’s alias.
  2. In the left navigation bar, pause on Routing, and then choose Contact flows.
  3. Under Contact flows, choose a template, or choose Create contact flow to design a contact flow from scratch. For more information, see Create a New Contact Flow.
  4. Download the sample JSON contact flow configuration file Outbound_calling.json.
  5. Choose the dropdown menu under Save and choose Import flow (beta).
  6. Select the Outbound_calling.json file in the Import flow (beta) dialog and choose Save.
  7. Choose Save to open the Save flow dialog. Then choose Save to close the dialog.
  8. Choose Publish to open the Publish dialog. Then choose Publish to close the dialog.
  9. In the contact flow designer, expand Show additional flow information.
  10. Under ARN, copy the Amazon Resource Name (ARN) contact flow. It looks like the following:
    arn:aws:connect:region:123456789012:instance/[ConnectInstanceId]/contact-flow/[ConnectContactFlowId]Note the ConnectInstanceId and ConnectContactFlowId from the ARN, they will be used in the next step.
  11. In the left navigation bar, pause on Routing and then choose Queues.
  12. Choose the queue you wish to use for the outbound calls.
  13. In the Edit queue screen, expand Show additional queue information.
  14. Under ARN, copy the Amazon Resource Name (ARN) for the queue. It looks like the following:
    arn:aws:connect:region:123456789012:instance/[ConnectInstanceId]/contact-flow/[ConnectQueueId]Note the ConnectQueueId from the ARN. It will be used in the next step.

Step 3: Deploy and modify the Amazon Pinpoint to the Amazon Connect custom channel with AWS Lambda function

Next, we will need to deploy an Amazon Pinpoint custom channel. Custom channels in Amazon Pinpoint allow you to send messages through any service with an API, including Amazon Connect. The AWS Serverless Application Repository contains an open-sourced AWS Lambda function that we will use for our custom channel. After deploying the AWS Lambda function, we will customize it to match our requirements.

  1. Navigate to the AWS Lambda Console, then choose Create function.
  2. Under Create function, Choose Browser serverless app repository.
  3. Under Public applications, choose the checkbox next to Show apps that create custom IAM roles or resource policies and enter amazon-pinpoint-connect-channel in the search box.
  4. Choose the amazon-pinpoint-connect-channel card from the list and review the Application details.
  5. Under Application settings enter the details for ConnectContactFlowId, ConnectInstanceId, and ConnectQueueId from the previous step.
  6. After reviewing all the details, choose the checkbox next to I acknowledge that this app creates custom IAM roles and resource policies and choose Deploy.
  7. Wait a couple minutes for the application to deploy two AWS Lambda functions and an AWS Simple Queue Service queue.
  8. Under Resources, choose the PinpointConnectQueuerFunction resource to open the AWS Lambda function configuration. This is the AWS Lambda function that Amazon Pinpoint will call when the message is crafted.
  9. Under Function code, scroll down to line 31 and replace
    message = "Hello World! -Pinpoint Connect Channel"
    with
    message = "This is a reminder of your upcoming appointment on {0} at {1}".format(endpoint_profile["Attributes"]["AppointmentDate"][0], endpoint_profile["Attributes"]["AppointmentTime"][0])
  10. Choose Deploy.

Step 4: (Optional) Modify the custom channel AWS Lambda function to meet change the rate of outgoing calls

By default, the custom channel we deployed in the previous step will place outbound calls through Amazon Connect at a rate of 1 call every 3 seconds. This allows you to configure how many active outbound calls to avoid running into service limits. Review your current service limits in Amazon Connect for more details.

  1. Navigate to the AWS Lambda Console, then choose AmazonPinpointConnectChannel-backgroundprocessor function.
  2. Under Function code, scroll down to line 73 and replace the sleep timer, currently set with 3 seconds, with your requirements.
  3. Choose Deploy.

Step 5: Create a Pinpoint custom campaign with your lambda function and segment

  1. Create a CSV file to import endpoints with the attributes of AppointmentDate and AppointmentTime.
    Example:
    Id,Address,ChannelType,Attributes.AppointmentDate,Attributes.AppointmentTime
    1,+1[PHONE_NUMBER],SMS,November 30 2020,9:00am
    2,+1[PHONE_NUMBER2],SMS,November 30 2020,10:00am
  2. Navigate to the Amazon Pinpoint console.
  3. In the All Projects list, select your project.
  4. In the navigation pane, choose Segments.
  5. Choose Create a Segment.
  6. Choose Import a segment and upload your CSV file and choose Create segment.
  7. In the navigation pane, choose Campaigns.
  8. Choose Create campaign.
  9. In the Create a campaign wizard, enter a name for campaign name.
  10. Under Channel choose Custom.
  11. Choose Next.
  12. On the Choose a segment screen, choose the segment created above, and choose Next.
  13. On the Create your message screen, do the following:
    a) For Lambda function choose AmazonPinpointConnectChannel that we deployed in Step 3 above.
    b) For endpoint Options choose SMS.
    c) Choose Next.
  14. On the Choose when to send the campaign screen, do the following:
    a) Choose When an event occurs.
    b) Under Events, choose the AppointmentReminder event.
    c) Under campaign dates, choose a Start date and time and an End date and time to be used as the campaign’s duration.
  15. Choose Next.
  16. Review the campaign details and choose Launch campaign.

Cleanup:

To remove the two AWS Lambda functions and the Amazon Simple Queue Service queue provisioned in the steps above in order not to incur further charges, please follow these steps below.

  1. Navigate to the Amazon CloudFormation Console.
  2. Choose severlessrepo-amazon-pinpoint-connect-channel and choose Delete.
  3. Choose Delete stack in the delete confirmation window.

 

Next Steps:

You can continue to iterate on this experience using Amazon Pinpoint and Amazon Connect to create a custom user experience.

To learn more about these services, please visit the Amazon Pinpoint or Amazon Connect web pages.

(1) https://www.scisolutions.com/uploads/news/Missed-Appts-Cost-HMT-Article-042617.pdf

(2) https://blog.carbonfreedining.org/the-ultimate-guide-to-restaurant-no-shows

Auto-reply to incoming emails using Amazon Simple Email Service (SES)

Post Syndicated from Ilya Pupko original https://aws.amazon.com/blogs/messaging-and-targeting/auto-reply-to-incoming-emails-using-amazon-simple-email-service-ses/

Both Amazon Pinpoint and Amazon Simple Email Service (SES) are known for their ability to send out transactional and promotional emails at scale and with ease. However, both are often not set up to receive email replies. Owners often assume that the “no-reply” addresses they are using do not require much consideration. This means that if a customer does reply, they would get an unhelpful server rejection indicating that the address is invalid. They would also not be able to unsubscribe via the simple reply, which is an otherwise established common practice. Automated guidance that the address is not monitored and who and how to reach for assistance would never be provided. In summary, a very unprofessional experience.

If you do have full control over the DNS and are not already receiving emails at the subdomain used for these emails, you can follow this short guide. It walks you through all the setup needed to have automated and templated responses to any address at the domain. This includes the address you use to send emails. Follow this post to ensure that your Amazon SES and Amazon Pinpoint are set up in accordance with common configuration and best business practices to have professional auto-reply to emails sent to the configured sending email addresses.

Solution overview

The proposed solution does not rely on any additional services. It does not add any additional charges beyond the cost directly associated with receiving and sending the emails and the minimal AWS Lambda function for the automated logic. It relies on SES built-in capability to receive emails, Amazon Pinpoint native templates, and uses Lambda for basic orchestration.

lambda diagram for response

Note, in this walkthrough and related code, we are using Amazon Pinpoint templates as they can be managed and maintained directly via the console, but you can choose to use SES templates (via the CreateTemplate API) or, if it makes better sense in your scenario, even just hardcode the template into the AWS Lambda function itself.

To complete the setup, all you must do is follow these steps:

      1. Confirm (Sub-) Domain setup in SES (even if you use Amazon Pinpoint to send your emails out, the SES portion of the console should show the validated domain as well). See SES Developer Guide.
      2. Ensure that your SES domain is verified and you are out of the sandbox. If still in the sandbox, you can only send emails to the Amazon SES mailbox simulator addresses and email addresses/domains that you have pre-verified. See Moving out of the Amazon SES sandbox.
      3. Configure SES to receive incoming emails. Please note that this must be done on the whole subdomain you use, not just a single email address. See Setting up Amazon SES email receiving.
      4. Create/add a new template you want to use via Amazon Pinpoint. Simply switch the console over to Amazon Pinpoint, select Message templates, click Create, select Email, and fill out the rest of the self-explanatory field.
        1. Plaintext portion is optional – you can either skip it or fill it out and enable in the Lambda function we are deploying in the next step.
        2. Similarly, if you prefer to use the SES template, you can instead. Just use the associated line in that same code.
        3. Same with a hardcoded template, if you prefer that for some reason.
      5. Have this pre-defined CloudFormation create the required SES receive rule, and Lambda function. This processes the incoming email and sends back the response, all using the code shared in the dedicated portion of our GitHub, AWS Digital User Engagement Reference Architectures repository. Specifically:
          1. Download the YAML from SES_Auto_Reply.yaml.
          2. Go to CloudFormation in AWS Management Console. (Remember to choose the region you want it deployed on)
          3. Click Create Stack and then choose With new resources
          4. Leave the default “Template is ready“, but switch to ”Upload a template file“ and choose the file you just downloaded
          5. Follow the wizard to give the “stack” a new name and enter the name of the template you created in step 4.
          6. Optionally you can also set the default response address, the addresses and/or domains you want to limit the auto-response to, and adjust the incoming email rule-set it should be stored under (the default should be fine, unless you have manually adjusted it in the past)
      6. Once deployed, the behavior is immediately active and you can further adjust any of these elements.

 

Conclusion and what’s next?

This architecture, once deployed, sends out the templated auto-response using the SES/Pinpoint domain/email address it received the original email on.

The new rule is added to the SES email receiving rule set to allow further customization:

  1. The rule can be limited to specific email address, specific domain, or just be set to be across all domains.
  2. It can also have the default response address set or reuse the address that the original rejected email was sent to.
  3. It can be moved down on the priority with other rules taking precedence and possibly even overriding it.
  4. It can have other actions added to it, like notifying SNS for additional tracking.

The Lambda function looks up the chosen Amazon Pinpoint template and uses it to reply. Here are some of the customizations you may want to consider within this function and the template:

  1. When sending the automated reply, by default, the template’s configured subject is appended with the original incoming email subject. You can adjust this to fit your company’s brand better.
  2. By default, the function supports an optional template tag %%NAME%% and %%ID%%. If the first appears in the template, it is automatically replaced with the original email’s FROM address. And if %%ID%% appears in the template, it is replaced with the SES’s original email message id, to help with any required audits.
  3. It is assumed that no additional tracking and actions are needed on such rejected and auto-replied emails, but you can further modify the flow by moving the rule around and adding more actions (as mentioned above), and even specify a particular/different SES Configuration Set for the outgoing emails.

Are you using this flow as a baseline for a more complex business flow or have other questions about it? We want to hear back – please comment here or file an issue in the GitHub repository. If you want to file a pull request to make it even more useful for others, please do so, we do appreciate community participation.

If you liked this article, we are continually expanding our Amazon Pinpoint and SES Architecture References and publish new solutions for these and other services. For most recent SES documentation, please see official SES documentation site, and for Amazon Pinpoint, please see Amazon Pinpoint documentation site.

 

 

 

Why AWS Dominates The Cloud Services Marketplace

Post Syndicated from Laurent Harvey original https://www.anchor.com.au/blog/2020/11/why-aws-dominates-the-cloud-services-marketplace/

Year after year, AWS maintains a very significant lead in the cloud marketplace over its closest competitors, including Microsoft Azure, Google Cloud Platform, as well as a number of other smaller cloud providers.

According to recent research published by Synergy Research Group, Amazon has 33% of the cloud infrastructure market share, with Microsoft trailing behind at 18%, and Google sitting at 9%.

So why has Amazon always been the leader of the pack when it comes to the major cloud service providers? The reasons for AWS’ significant lead may be more simple than you would first think. Below, we will go into just a few of the many reasons AWS has maintained such a dominant lead since its conception.

It’s Been Around The Longest

In any race, one of the most valuable things you can have is a head start. Amazon launched Amazon Web Services (AWS) back in 2006 and began offering their initial cloud computing service offerings to, primarily, developers. Their initial service offerings included Amazon S3 cloud storage, SQS, and EC2.

Google and Microsoft had dabbled in their own Cloud offerings at the time, but didn’t put the same resources into it that Amazon did. Google launched their Cloud Platform in 2008, and Microsoft Azure later followed them and launched in 2010. However, neither Google nor Microsoft invested the same amount of resources early on. As a result, Amazon was able to establish a firm lead in the cloud services market. Other providers have been playing a never-ending battle of catch-up ever since.

Constant Innovation

Although we can attribute a lot of AWS’ success to their early foothold, they wouldn’t be able to maintain such a significant market share on that alone. Since the beginning, Amazon has continually innovated year after year.

Since 2006, Amazon have greatly increased their service offerings and created many innovative services. In fact, at the time of writing, AWS offers an astounding 175 individual products and services to consumers. Many of these services are original Amazon innovations. You would be hard-pressed to find a task you can’t accomplish with one of Amazon’s many services, and they’re only adding more and more to their catalogue each year. We expect to see a specific focus on Artificial Intelligence Services from Amazon in the next few years, as it’s one of the fastest-growing areas of cloud computing.

Price Cuts

One of the greatest reasons AWS stays not only incredibly competitive, while still leading the market, is their constant efforts to reduce consumer costs. In fact, research published in 2018 by TSO Logic found that AWS costs get lower every year. AWS has no problem maintaining their existing customer base with increasingly diminishing prices, while also attracting new customers. Plus, the larger AWS gets, the more ability they have to achieve even higher economies of scale, thus passing more price cuts onto their customers.

In Amazon’s own words, they state the following on their website:

“Benefit from massive economies of scale – By using cloud computing, you can achieve a lower variable cost than you can get on your own. Because usage from hundreds of thousands of customers is aggregated in the cloud, providers such as AWS can achieve higher economies of scale, which translates into lower pay as-you-go prices.”

Backed by Amazon

With the full long-term backing of Amazon, which comes in at #3 worldwide of all public corporations by market capitalization, AWS is quite simply a juggernaut of resources and capability. At the time of writing, Amazon has an estimated net value of $1.14 trillion. Amazon’s founder, Jeff Bezos, has an estimated worth of $190 billion.

With these kinds of numbers, Amazon are of course a formidable opponent for any newcomers to the cloud services marketplace. They also don’t look to be slowing down anytime soon in terms of their vision for the future and upcoming technological innovations.

Conclusion

AWS provides a platform which is ever-evolving and constantly becoming more financially accessible for businesses of all sizes. With new offerings, technology, features and opportunities for performance improvements every year, AWS provides a solid and proven platform for businesses who are looking to bring their business into the cloud.

If you think your business may benefit from taking advantage of AWS’ huge range of services, get in touch with us today for an expert consultation on how we can assist you in your journey to the cloud.

The post Why AWS Dominates The Cloud Services Marketplace appeared first on AWS Managed Services by Anchor.

3 Common Problems Businesses Face Which Can Be Solved By AWS Cloud.

Post Syndicated from Andy Haine original https://www.anchor.com.au/blog/2020/11/3-common-problems-businesses-face-which-can-be-solved-by-aws-cloud/

Business leaders know all too well the long list of challenges involved in taking any business to the next level. Cash flow, human resources, infrastructure, growing marketing spend, refining or expanding on processes, and company culture are just a few of the many considerations. One particularly important challenge is choosing the right software and tools early on, to allow your business to provide its products or services efficiently and cost-effectively.

One of the greatest ways to achieve reliable and harmonious business practices is to ensure the technological infrastructure that your business is built upon is carefully planned to not only cater to your immediate business needs but also to be flexible for future growth.

Cloud computing services are more popular than ever before, and even in the face of the COVID-19 pandemic, have continued to grow just as steadily. Below, we’ve outlined 5 common business problems that are solved by migration to AWS cloud. If you’ve been considering the potential advantages of AWS for your business, read on!

Common problem: Convoluted/expensive/unnecessary services within pre-packaged traditional hosting plans.

With traditional hosting services, products tend to be pre-packaged with a selection of commonly required services as well as tiered/set resources. As a business grows larger and requires more heavy-duty online infrastructure, the cost of pre-packaged services can become much more expensive than it needs to be. That is because you may not be using some of the inclusions provided with these services, or require less of one resource or more of another. Pre-packaged service pricing also generally has factored in the cost of software licences needed to deliver all of the inclusions offered. If you’re not using these services, why should you be paying for them?

How AWS cloud computing services solves this: With AWS cloud hosting, each individual service is billed separately, and charges are based on different metrics such as the number of hours or seconds a service is online, or how much data transfer takes place. This allows a business to have very granular control over where they are directing their spend, as well as offering the ability to avoid paying for service inclusions that they are simply not using.

Common problem: Cost creep over time.

Cost creep is a common problem both in traditional hosting services and cloud computing services. As your business grows and evolves, your online infrastructure may need access to more services, features or integrations, as well as more computing resources. Each of these things almost always comes with an additional cost.

How AWS cloud computing services solves this: Between traditional hosting services and cloud computing, cloud is the only one that offers a plethora of ways to prevent, monitor and even reverse cost creep over time. Cost creep is a common occurrence for many businesses, especially in the early deployment stages when traffic is the least predictable and resource requirements are difficult to allocate in advance. This is something that can be greatly improved over time as usage data becomes available, along with traffic and resource usage patterns of your website or application. With proper maintenance and the utilisation of AWS reserved instances (which can provide the same resources at a greatly lower cost), there are many opportunities to minimise, and even reverse cost creep over time with cloud services.

Common problem: Infrastructure that offers a lack of horizontal scaling.

Horizontal scaling can translate to cost efficiencies, by adding or removing computing resources, and only paying for them while you are actually using them. For example, say you were running a food delivery application where you required a lot of computing resources to handle traffic during the lunch and dinner rush. If you were to purchase a computing instance with enough power to handle the rush hour, that might become an expensive waste of resources to still be running when business is quiet at 4 am. This is where horizontal scaling can come in to maximise efficiency through the addition and reduction of additional computing power, as needed.

Traditional hosting services rarely offer horizontal scalability, meaning you will be overpaying for resources or services that you aren’t utilising a lot of the time.

How AWS cloud computing services solves this: AWS offers powerful options when it comes to horizontally scaling computing power on demand. Scaling horizontally means adding additional computing instances to support an application running on an existing instance, as needed.

One of the greatest advantages of cloud computing services such as AWS is that their range of services are billed by the amount of time you are using them. So horizontal scaling can translate to cost efficiencies, by adding or removing computing resources, and only paying for them while you are actually using them.

The post 3 Common Problems Businesses Face Which Can Be Solved By AWS Cloud. appeared first on AWS Managed Services by Anchor.

Why is it vital for businesses to outsource AWS management?

Post Syndicated from Andy Haine original https://www.anchor.com.au/blog/2020/10/why-is-it-vital-for-businesses-to-outsource-aws-management/

Some of us may have learnt during 2020 that there are simply some things that one cannot DIY without proper skills and expertise. Perhaps during the pandemic lockdown, your local hairdresser was closed, and you turned to a DIY YouTube tutorial and learnt this the hardest way of all. But, even if you survived 2020 without a fringe 2 inches too short, managing AWS services is a whole other ball game that requires years of training and dedicated skill to properly deploy, manage and keep expenses under control.

As powerful as AWS is, and as much as it can do for your business, it can be all-but impossible to do it right if you have never set foot in the AWS Management console before. AWS is complex, and requires expertise to truly get the most from it. While you may be able to perform basic provisioning tasks and perhaps get a service up and running, ensuring that that service is performing optimally and cost-efficiently is where professional AWS management can truly revolutionise your infrastructure strategy and budget.

Managed AWS services is one of the largest outsourced areas of the IT industry. According to a recent Gartner forecast, almost 19% of cloud budgets are spent on cloud management-related services, such as cloud consulting, implementation, migration and managed services – with this percentage expected to increase in the next few years (and for good reason). In this article we will delve into just a few of the reasons why you’re far better off putting your AWS management in the hands of experts.

Cost Savings

Wasted cloud spend is a very common occurrence within the cloud services industry, with many cloud users not optimising costs where they can. In a 2019 report from Flexera, they measured the actual waste of cloud spending at 35 percent.

One of the most beneficial things an AWS management service provider can offer your business is ensuring that you’re only paying for what your business needs. It may save your business significantly more money in the long run, even when factoring in management fees.

Free Up Your Time

You should focus on what you and your business do best. Sure, you could put in many hours to understand as much as possible and get up and running yourself, but many businesses find that time is much better spent on focussing on your core service offerings and leaving management to your managed service provider.

On top of the initial learning curve, there is also the time investment needed for ongoing training as new AWS cloud services are released and new management tools are developed. Best practice changes very frequently, and it can be a significant undertaking to try and keep your finger on the pulse while simultaneously trying to handle every other area of your business.

Proactive Management

Ensuring that your business leverages AWS’ ability to scale and adjust depending on your current needs is essential. An AWS partner and managed service provider can help you understand your businesses needs, and adjust course as necessary to meet each new scenario.

A good example of scaling to meet current needs is the COVID-19 pandemic. The cloud services industry has seen significant growth during 2020 due to its ability to rapidly scale and support sudden growth. For example, web traffic and bandwidth requirements skyrocketed in 2020 with more people turning to eCommerce to acquire their everyday household items as well as remotely attend school and work.

Avoiding Downtime and Increasing Stability

Any number of things can happen to your hosted services, and when they do, it’s crucial that you have an experienced team on hand to tackle whatever comes your way. There’s nothing worse than hosting your mission-critical services on AWS and not having the experience to get services up and running as soon as possible when things go wrong.

A qualified AWS management team will also put best practice measures into place to improve the resilience of your configuration, and minimise the chance of anything going wrong in the first place.

Conclusion

When deciding what is the best course of action for your business, it’s imperative to ensure that your mission-critical cloud services are in good hands. It can be shown that in many cases, having AWS experts handle your businesses cloud needs can more than repay the associated management fees, leaving you better off both in terms of support and costs.

If you’re looking for advice on AWS cloud migration best practices, don’t hesitate to get in touch with one of our expert cloud architects today.

The post Why is it vital for businesses to outsource AWS management? appeared first on AWS Managed Services by Anchor.

Anchor Joins AWS Service Delivery Program: Amazon EC2 for Windows Server

Post Syndicated from Andy Haine original https://www.anchor.com.au/blog/2020/10/anchor-joins-aws-service-delivery-program-amazon-ec2-for-windows-server/

We are excited to announce that Anchor has joined the Amazon EC2 for Windows Service Delivery Program (SPD). This new SDP classification complements an expanding portfolio of AWS certifications, substantiating our commitment to both our AWS partnership and enabling the cloud for Australian businesses.

The AWS Service Delivery Program acknowledges select partners within the AWS Partner Network (APN) who have demonstrated technical proficiency across specialised solution areas. Achieving SDP status involves a stringent validation process to certify a deep understanding of, and adherence to, AWS architectural best practices. 

Amazon EC2 for Windows Server Partners are certified for delivering Windows Server environments on Amazon Elastic Compute Cloud (Amazon EC2). They are recommended by AWS for managing secure, reliable, and high-performance environments for deploying Windows-based applications and workloads. 

As part of the validation process for this competency, Anchor was required to demonstrate proven customer success through real customer engagement to validate that we had the technical proficiency and resources to help customers migrate, manage, or deploy Microsoft Workloads to AWS.

This AWS SDP reinforces Anchors expertise in helping businesses both modernise and future-proof legacy windows applications by replatforming onto AWS Cloud. 

“Our new SDP status for Amazon EC2 for Windows provides our customers with the highest level of confidence in our technical aptitude and alignment to AWS best practices.” – Josh Chiswell, Director of Architecture and Professional Services, APAC, Anchor Systems.

The proficiency also provides exclusive access to service-specific funding programs which partners can pass on to customers. If your business is looking to modernise or replatform old windows workloads, contact our cloud consultants for a complimentary cloud assessment.

 

Why Anchor? We exist to help SMBs and emerging enterprises who need managed AWS and cloud engineering services. Anchor enables the cloud by deeply engaging with your business. We architect, deploy, run and optimise cloud workloads and advocate for cloud best practices. Anchor’s team of certified engineers can support your workloads in three different time zones with 24x7x365 coverage from Sydney.

The post Anchor Joins AWS Service Delivery Program: Amazon EC2 for Windows Server appeared first on AWS Managed Services by Anchor.

Analyze and improve email campaigns with Amazon Simple Email Service and Amazon QuickSight

Post Syndicated from Apoorv Gakhar original https://aws.amazon.com/blogs/messaging-and-targeting/analyze-and-improve-email-campaigns-with-amazon-simple-email-service-and-amazon-quicksight/

Email is a popular channel for applications, used in both marketing campaigns and other outbound customer communications. The challenge with email is that it can become increasingly complex to manage for companies that must send large quantities of messages per month. This complexity is especially true when companies need to measure detailed email engagement metrics to track campaign success.

As a marketer, you want to monitor several metrics, including open rates, click-through rates, bounce rates, and delivery rates. If you do not track your email results, you could potentially be wasting your campaign resources. Monitoring and interpreting your sending results can help you deliver the best content possible to your subscribers’ inboxes, and it can also ensure that your IP reputation stays high. Mailbox providers prioritize inbox placement for senders that deliver relevant content. As a business professional, tracking your emails can also help you stay on top of hot leads and important clients. For example, if someone has opened your email multiple times in one day, it might be a good idea to send out another follow-up email to touch base.

Building a large-scale email solution is a complex and expensive challenge for any business. You would need to build infrastructure, assemble your network, and warm up your IP addresses. Alternatively, working with some third-party email solutions require contract negotiations and upfront costs.

Fortunately, Amazon Simple Email Service (SES) has a highly scalable and reliable backend infrastructure to reduce the preceding challenges. It has improved content filtering techniques, reputation management features, and a vast array of analytics and reporting functions. These features help email senders reach their audiences and make it easier to manage email channels across applications. Amazon SES also provides API operations to monitor your sending activities through simple API calls. You can publish these events to Amazon CloudWatch, Amazon Kinesis Data Firehose, or by using Amazon Simple Notification Service (SNS).

In this post, you learn how to build and automate a serverless architecture that analyzes email events. We explore how to track important metrics such as open and click rate of the emails.

Solution overview

 

The metrics that you can measure using Amazon SES are referred to as email sending events. You can use Amazon CloudWatch to retrieve Amazon SES event data. You can also use Amazon SNS to interpret Amazon SES event data. However, in this post, we are going to use Amazon Kinesis Data Firehose to monitor our user sending activity.

Enable Amazon SES configuration sets with open and click metrics and publish email sending events to Amazon Kinesis Data Firehose as JSON records. A Lambda function is used to parse the JSON records and publish the content in the Amazon S3 bucket.

Ingested data lands in an Amazon S3 bucket that we refer to as the raw zone. To make that data available, you have to catalog its schema in the AWS Glue data catalog. You create and run the AWS Glue crawler that crawls your data sources and construct your Data Catalog. The Data Catalog uses pre-built classifiers for many popular source formats and data types, including JSON, CSV, and Parquet.

When the crawler is finished creating the table definition and schema, you analyze the data using Amazon Athena. It is an interactive query service that makes it easy to analyze data in Amazon S3 using SQL. Point to your data in Amazon S3, define the schema, and start querying using standard SQL, with most results delivered in seconds.

Now you can build visualizations, perform ad hoc analysis, and quickly get business insights from the Amazon SES event data using Amazon QuickSight. You can easily run SQL queries using Amazon Athena on data stored in Amazon S3, and build business dashboards within Amazon QuickSight.

 

Deploying the architecture:

Configuring Amazon Kinesis Data Firehose to write to Amazon S3:

  1. Navigate to the Amazon Kinesis in the AWS Management Console. Choose Kinesis Data Firehose and create a delivery stream.
  2. Enter delivery stream name as “SES_Firehose_Demo”.
  3. Under the source category, select “Direct Put or other sources”.
  4. On the next page, make sure to enable Data Transformation of source records with AWS Lambda. We use AWS Lambda to parse the notification contents that we only process the required information as per the use case.
  5. Click the “Create New” Lambda function.
  6. Click on “General Kinesis Data FirehoseProcessing” Lambda blueprint and this opens up the Lambda console. Enter following values in Lambda
    • Name: SES-Firehose-Json-Parser
    • Execution role: Create a new role with basic Lambda permissions.
  7. Click “Create Function”. Now replace the Lambda code with the following provided code and save the function.
    • 'use strict';
      console.log('Loading function');
      exports.handler = (event, context, callback) => {
         /* Process the list of records and transform them */
          const output = event.records.map((record) => {
              console.log(record.recordId);
              const payload =JSON.parse((Buffer.from(record.data, 'base64').toString()))
              console.log("payload : " + payload);
              
              if (payload.eventType == "Click") {
              const resultPayLoadClick = {
                      eventType : payload.eventType,
                      destinationEmailId : payload.mail.destination[0],
                      sourceIp : payload.click.ipAddress,
                  };
              console.log("resultPayLoad : " + resultPayLoadClick.eventType + resultPayLoadClick.destinationEmailId + resultPayLoadClick.sourceIp);
              
              //const parsed = resultPayLoad[0];
              //console.log("parsed : " + (Buffer.from(JSON.stringify(resultPayLoad))).toString('base64'));
              
              
              return{
                  recordId: record.recordId,
                  result: 'Ok',
                  data: (Buffer.from(JSON.stringify(resultPayLoadClick))).toString('base64'),
              };
              }
              else {
                  const resultPayLoadOpen = {
                      eventType : payload.eventType,
                      destinationEmailId : payload.mail.destination[0],
                      sourceIp : payload.open.ipAddress,
                  };
              console.log("resultPayLoad : " + resultPayLoadOpen.eventType + resultPayLoadOpen.destinationEmailId + resultPayLoadOpen.sourceIp);
              
              //const parsed = resultPayLoad[0];
              //console.log("parsed : " + (Buffer.from(JSON.stringify(resultPayLoad))).toString('base64'));
              
              
              return{
                  recordId: record.recordId,
                  result: 'Ok',
                  data: (Buffer.from(JSON.stringify(resultPayLoadOpen))).toString('base64'),
              };
              }
          });
          console.log("Output : " + output.data);
          console.log(`Processing completed.  Successful records ${output.length}.`);
          callback(null, { records: output });
      };

      Please note:

      For this blog, we are only filtering out three fields i.e. Eventname, destination_Email, and SourceIP. If you want to store other parameters you can modify your code accordingly. For the list of information that we receive in notifications, you may check out the following document.

      https://docs.aws.amazon.com/ses/latest/DeveloperGuide/event-publishing-retrieving-firehose-examples.html

  8. Now, navigate back to your Amazon Kinesis Data Firehose console and choose the newly created Lambda function.
  9. Keep the convert record format disabled and click “Next”.
  10. In the destination, choose Amazon S3 and select a target Amazon S3 bucket. Create a new bucket if you do not want to use the existing bucket.
  11. Enter the following values for Amazon S3 Prefix and Error Prefix. When event data is published.
    • Prefix:
      fhbase/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/
    • Error Prefix:
      fherroroutputbase/!{firehose:random-string}/!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd}/
  12. You may utilize the above values in the Amazon S3 prefix and error prefix. If you use your own prefixes make sure to accordingly update the target values in AWS Glue which you will see in further process.
  13. Keep the Amazon S3 backup option disabled and click “Next”.
  14. On the next page, under the Permissions section, select create a new role. This opens up a new tab and then click “Allow” to create the role.
  15. Navigate back to the Amazon Kinesis Data Firehose console and click “Next”.
  16. Review the changes and click on “Create delivery stream”.

Configure Amazon SES to publish event data to Kinesis Data Firehose:

  1. Navigate to Amazon SES console and select “Email Addresses” from the left side.
  2. Click on “Verify a New Email Address” on the top. Enter your email address to which you send a test email.
  3. Go to your email inbox and click on the verify link. Navigate back to the Amazon SES console and you will see verified status on the email address provided.
  4. Open the Amazon SES console and select Configuration set from the left side.
  5. Create a new configuration set. Enter “SES_Firehose_Demo”  as the configuration set name and click “Create”.
  6. Choose Kinesis Data Firehose as the destination and provide the following details.
    • Name: OpenClick
    • Event Types: Open and Click
  7. In the IAM Role field, select ‘Let SES make a new role’. This allows SES to create a new role and add sufficient permissions for this use case in that role.
  8. Click “Save”.

Sending a Test email:

  1. Navigate to Amazon SES console, click on “Email Addresses” on the left side.
  2. Select your verified email address and click on “Send a Test email”.
  3. Make sure you select the raw email format. You may use the following format to send out a test email from the console. Make sure you send out this email to a recipient inbox to which you have the access.
    • X-SES-CONFIGURATION-SET: SES_Firehose_Demo
      X-SES-MESSAGE-TAGS: Email=NULL
      From: [email protected]
      To: [email protected]
      Subject: Test email
      Content-Type: multipart/alternative;
          		boundary="----=_boundary"
      
      ------=_boundary
      Content-Type: text/html; charset=UTF-8
      Content-Transfer-Encoding: 7bit
      This is a test email.
      
      <a href="https://aws.amazon.com/">Amazon Web Services</a>
      ------=_boundary
  4. Once the email is received in the recipient’s inbox, open the email and click the link present in the same. This generates a click and open event and send the response back to SES.

Creating Glue Crawler:

  1. Navigate to the AWS Glue console, select “crawler” from the left side, and then click on “Add crawler” on the top.
  2. Enter the crawler name as “SES_Firehose_Crawler” and click “Next”.
  3. Under Crawler source type, select “Data stores” and click “Next”.
  4. Select Amazon S3 as the data source and prove the required path. Include the path until the “fhbase” folder.
  5. Select “no” under Add another data source section.
  6. In the IAM role, select the option to ‘Create an IAM role’. Enter the name as “SES_Firehose-Crawler”. This provides the necessary permissions automatically to the newly created role.
  7. In the frequency section, select run on demand and click “Next”. You may choose this value as per your use case.
  8. Click on add Database and provide the name as “ses_firehose_glue_db”. Click on create and then click “Next”.
  9. Review your Glue crawler setting and click on “Finish”.
  10. Run the above-created crawler. This crawls the data from the specified Amazon S3 bucket and create a catalog and table definition.
  11. Now navigate to “tables” on the left, and verify a “fhbase” table is created after you run the crawler.

If you want to analyze the data stored until now, you can use Amazon Athena and test the queries. If not, you can move to the Amazon Quicksight directly.

Analyzing the data using Amazon Athena:

  1. Open Athena console and select the database, which is created using AWS Glue
  2. Click on “setup a query result location in Amazon S3” as shown in the following screenshot.
  3. Navigate to the Amazon S3 bucket created in earlier steps and create a folder called “AthenaQueryResult”. We store our Athena query result in this bucket.
  4. Now navigate back to Amazon Athena and select the Amazon S3 bucket with the folder location as shown in the following screenshot and click “Save”.
  5. Run the following query to test the sample output and accordingly modify your SQL query to get the desired output.
    • Select * from “ses_firehose_glue_db”.”fhbase”

Note: If you want to track the opened emails by unique Ip addresses then you can modify your SQL query accordingly. This is because every time an email gets opened, you will receive a notification even if the same email was previously opened.

 

Visualizing the data in Amazon QuickSight dashboards:

  1. Now, let’s analyze this data using Amazon Athena via Amazon Quicksight.
  2. Log into Amazon Quicksight and choose Manage data, New dataset. Choose Amazon Athena as a new data source.
  3. Enter the data source name as “SES-Demo” and click on “Create the data source”.
  4. Select your database from the drop-down as “ses_firehose_glue_db” and table “fhbase” that you have created in AWS Glue.
  5. And add a custom SQL based on your use case and click on “Confirm query”. Refer to the example below.
  6. You can perform ad hoc analysis and modify your query according to your business needs as shown in the following image. Click “Save & Visualize”.
  7. You can now visualize your event data on Amazon Quicksight dashboard. You can use various graphs to represent your data. For this demo, the default graph is used and two fields are selected to populate on the graph, as shown below.

 

Conclusion:

This architecture shows how to track your email sending activity at a granular level. You set up Amazon SES to publish event data to Amazon Kinesis Data Firehose based on fine-grained email characteristics that you define. You can also track several types of email sending events, including sends, deliveries, bounces, complaints, rejections, rendering failures, and delivery delays. This information can be useful for operational and analytical purposes.

To get started with Amazon SES, follow this quick start guide and you can learn more about monitoring sending activity here.

About the Authors

Chirag Oswal is a solutions architect and AR/VR specialist working with the public sector India. He works with AWS customers to help them adopt the cloud operating model on a large scale.

Apoorv Gakhar is a Cloud Support Engineer and an Amazon SES Expert. He is working with AWS to help the customers integrate their applications with various AWS Services.

 

Additional Resources:

Amazon SES Dedicated IP Pools

Amazon Personalize optimizer using Amazon Pinpoint events

Template Personalization using Amazon Pinpoint

 

 

Complete CI/CD with AWS CodeCommit, AWS CodeBuild, AWS CodeDeploy, and AWS CodePipeline

Post Syndicated from Nitin Verma original https://aws.amazon.com/blogs/devops/complete-ci-cd-with-aws-codecommit-aws-codebuild-aws-codedeploy-and-aws-codepipeline/

Many organizations have been shifting to DevOps practices, which is the combination of cultural philosophies, practices, and tools that increases your organization’s ability to deliver applications and services at high velocity; for example, evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes.

DevOps-Feedback-Flow

An integral part of DevOps is adopting the culture of continuous integration and continuous delivery/deployment (CI/CD), where a commit or change to code passes through various automated stage gates, all the way from building and testing to deploying applications, from development to production environments.

This post uses the AWS suite of CI/CD services to compile, build, and install a version-controlled Java application onto a set of Amazon Elastic Compute Cloud (Amazon EC2) Linux instances via a fully automated and secure pipeline. The goal is to promote a code commit or change to pass through various automated stage gates all the way from development to production environments, across AWS accounts.

AWS services

This solution uses the following AWS services:

  • AWS CodeCommit – A fully-managed source control service that hosts secure Git-based repositories. CodeCommit makes it easy for teams to collaborate on code in a secure and highly scalable ecosystem. This solution uses CodeCommit to create a repository to store the application and deployment codes.
  • AWS CodeBuild – A fully managed continuous integration service that compiles source code, runs tests, and produces software packages that are ready to deploy, on a dynamically created build server. This solution uses CodeBuild to build and test the code, which we deploy later.
  • AWS CodeDeploy – A fully managed deployment service that automates software deployments to a variety of compute services such as Amazon EC2, AWS Fargate, AWS Lambda, and your on-premises servers. This solution uses CodeDeploy to deploy the code or application onto a set of EC2 instances running CodeDeploy agents.
  • AWS CodePipeline – A fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates. This solution uses CodePipeline to create an end-to-end pipeline that fetches the application code from CodeCommit, builds and tests using CodeBuild, and finally deploys using CodeDeploy.
  • AWS CloudWatch Events – An AWS CloudWatch Events rule is created to trigger the CodePipeline on a Git commit to the CodeCommit repository.
  • Amazon Simple Storage Service (Amazon S3) – An object storage service that offers industry-leading scalability, data availability, security, and performance. This solution uses an S3 bucket to store the build and deployment artifacts created during the pipeline run.
  • AWS Key Management Service (AWS KMS) – AWS KMS makes it easy for you to create and manage cryptographic keys and control their use across a wide range of AWS services and in your applications. This solution uses AWS KMS to make sure that the build and deployment artifacts stored on the S3 bucket are encrypted at rest.

Overview of solution

This solution uses two separate AWS accounts: a dev account (111111111111) and a prod account (222222222222) in Region us-east-1.

We use the dev account to deploy and set up the CI/CD pipeline, along with the source code repo. It also builds and tests the code locally and performs a test deploy.

The prod account is any other account where the application is required to be deployed from the pipeline in the dev account.

In summary, the solution has the following workflow:

  • A change or commit to the code in the CodeCommit application repository triggers CodePipeline with the help of a CloudWatch event.
  • The pipeline downloads the code from the CodeCommit repository, initiates the Build and Test action using CodeBuild, and securely saves the built artifact on the S3 bucket.
  • If the preceding step is successful, the pipeline triggers the Deploy in Dev action using CodeDeploy and deploys the app in dev account.
  • If successful, the pipeline triggers the Deploy in Prod action using CodeDeploy and deploys the app in the prod account.

The following diagram illustrates the workflow:

cicd-overall-flow

 

Failsafe deployments

This example of CodeDeploy uses the IN_PLACE type of deployment. However, to minimize the downtime, CodeDeploy inherently supports multiple deployment strategies. This example makes use of following features: rolling deployments and automatic rollback.

CodeDeploy provides the following three predefined deployment configurations, to minimize the impact during application upgrades:

  • CodeDeployDefault.OneAtATime – Deploys the application revision to only one instance at a time
  • CodeDeployDefault.HalfAtATime – Deploys to up to half of the instances at a time (with fractions rounded down)
  • CodeDeployDefault.AllAtOnce – Attempts to deploy an application revision to as many instances as possible at once

For OneAtATime and HalfAtATime, CodeDeploy monitors and evaluates instance health during the deployment and only proceeds to the next instance or next half if the previous deployment is healthy. For more information, see Working with deployment configurations in CodeDeploy.

You can also configure a deployment group or deployment to automatically roll back when a deployment fails or when a monitoring threshold you specify is met. In this case, the last known good version of an application revision is automatically redeployed after a failure with the new application version.

How CodePipeline in the dev account deploys apps in the prod account

In this post, the deployment pipeline using CodePipeline is set up in the dev account, but it has permissions to deploy the application in the prod account. We create a special cross-account role in the prod account, which has the following:

  • Permission to use fetch artifacts (app) rom Amazon S3 and deploy it locally in the account using CodeDeploy
  • Trust with the dev account where the pipeline runs

CodePipeline in the dev account assumes this cross-account role in the prod account to deploy the app.

Do I need multiple accounts?
If you answer “yes” to any of the following questions you should consider creating more AWS accounts:

  • Does your business require administrative isolation between workloads? Administrative isolation by account is the most straightforward way to grant independent administrative groups different levels of administrative control over AWS resources based on workload, development lifecycle, business unit (BU), or data sensitivity.
  • Does your business require limited visibility and discoverability of workloads? Accounts provide a natural boundary for visibility and discoverability. Workloads cannot be accessed or viewed unless an administrator of the account enables access to users managed in another account.
  • Does your business require isolation to minimize blast radius? Separate accounts help define boundaries and provide natural blast-radius isolation to limit the impact of a critical event such as a security breach, an unavailable AWS Region or Availability Zone, account suspensions, and so on.
  • Does your business require a particular workload to operate within AWS service limits without impacting the limits of another workload? You can use AWS account service limits to impose restrictions on a business unit, development team, or project. For example, if you create an AWS account for a project group, you can limit the number of Amazon Elastic Compute Cloud (Amazon EC2) or high performance computing (HPC) instances that can be launched by the account.
  • Does your business require strong isolation of recovery or auditing data? If regulatory requirements require you to control access and visibility to auditing data, you can isolate the data in an account separate from the one where you run your workloads (for example, by writing AWS CloudTrail logs to a different account).

Prerequisites

For this walkthrough, you should complete the following prerequisites:

  1. Have access to at least two AWS accounts. For this post, the dev and prod accounts are in us-east-1. You can search and replace the Region and account IDs in all the steps and sample AWS Identity and Access Management (IAM) policies in this post.
  2. Ensure you have EC2 Linux instances with the CodeDeploy agent installed in all the accounts or VPCs where the sample Java application is to be installed (dev and prod accounts).
    • To manually create EC2 instances with CodeDeploy agent, refer Create an Amazon EC2 instance for CodeDeploy (AWS CLI or Amazon EC2 console). Keep in mind the following:
      • CodeDeploy uses EC2 instance tags to identify instances to use to deploy the application, so it’s important to set tags appropriately. For this post, we use the tag name Application with the value MyWebApp to identify instances where the sample app is installed.
      • Make sure to use an EC2 instance profile (AWS Service Role for EC2 instance) with permissions to read the S3 bucket containing artifacts built by CodeBuild. Refer to the IAM role cicd_ec2_instance_profile in the table Roles-1 below for the set of permissions required. You must update this role later with the actual KMS key and S3 bucket name created as part of the deployment process.
    • To create EC2 Linux instances via AWS Cloudformation, download and launch the AWS CloudFormation template from the GitHub repo: cicd-ec2-instance-with-codedeploy.json
      • This deploys an EC2 instance with AWS CodeDeploy agent.
      • Inputs required:
        • AMI : Enter name of the Linux AMI in your region. (This template has been tested with latest Amazon Linux 2 AMI)
        • Ec2SshKeyPairName: Name of an existing SSH KeyPair
        • Ec2IamInstanceProfile: Name of an existing EC2 instance profile. Note: Use the permissions in the template cicd_ec2_instance_profile_policy.json to create the policy for this EC2 Instance Profile role. You must update this role later with the actual KMS key and S3 bucket name created as part of the deployment process.
        • Update the EC2 instance Tags per your need.
  3. Ensure required IAM permissions. Have an IAM user with an IAM Group or Role that has the following access levels or permissions:

    AWS Service / Components Access LevelAccountsComments
    AWS CodeCommitFull (admin)DevUse AWS managed policy AWSCodeCommitFullAccess.
    AWS CodePipelineFull (admin)DevUse AWS managed policy AWSCodePipelineFullAccess.
    AWS CodeBuildFull (admin)DevUse AWS managed policy AWSCodeBuildAdminAccess.
    AWS CodeDeployFull (admin)All

    Use AWS managed policy

    AWSCodeDeployFullAccess.

    Create S3 bucket and bucket policiesFull (admin)DevIAM policies can be restricted to specific bucket.
    Create KMS key and policiesFull (admin)DevIAM policies can be restricted to specific KMS key.
    AWS CloudFormationFull (admin)Dev

    Use AWS managed policy

    AWSCloudFormationFullAccess.

    Create and pass IAM rolesFull (admin)AllAbility to create IAM roles and policies can be restricted to specific IAM roles or actions. Also, an admin team with IAM privileges could create all the required roles. Refer to the IAM table Roles-1 below.
    AWS Management Console and AWS CLIAs per IAM User permissionsAllTo access suite of Code services.

     

  4. Create Git credentials for CodeCommit in the pipeline account (dev account). AWS allows you to either use Git credentials or associate SSH public keys with your IAM user. For this post, use Git credentials associated with your IAM user (created in the previous step). For instructions on creating a Git user, see Create Git credentials for HTTPS connections to CodeCommit. Download and save the Git credentials to use later for deploying the application.
  5. Create all AWS IAM roles as per the following tables (Roles-1). Make sure to update the following references in all the given IAM roles and policies:
    • Replace the sample dev account (111111111111) and prod account (222222222222) with actual account IDs
    • Replace the S3 bucket mywebapp-codepipeline-bucket-us-east-1-111111111111 with your preferred bucket name.
    • Replace the KMS key ID key/82215457-e360-47fc-87dc-a04681c91ce1 with your KMS key ID.

Table: Roles-1

ServiceIAM Role TypeAccountIAM Role Name (used for this post)IAM Role Policy (required for this post)IAM Role Permissions
AWS CodePipelineService roleDev (111111111111)

cicd_codepipeline_service_role

Select Another AWS Account and use this account as the account ID to create the role.

Later update the trust as follows:
“Principal”: {“Service”: “codepipeline.amazonaws.com”},

Use the permissions in the template cicd_codepipeline_service_policy.json to create the policy for this role.This CodePipeline service role has appropriate permissions to the following services in a local account:

  • Manage CodeCommit repos
  • Initiate build via CodeBuild
  • Create deployments via CodeDeploy
  • Assume cross-account CodeDeploy role in prod account to deploy the application
AWS CodePipelineIAM roleDev (111111111111)

cicd_codepipeline_trigger_cwe_role

Select Another AWS Account and use this account as the account ID to create the role.

Later update the trust as follows:
“Principal”: {“Service”: “events.amazonaws.com”},

Use the permissions in the template cicd_codepipeline_trigger_cwe_policy.json to create the policy for this role.CodePipeline uses this role to set a CloudWatch event to trigger the pipeline when there is a change or commit made to the code repository.
AWS CodePipelineIAM roleProd (222222222222)

cicd_codepipeline_cross_ac_role

Choose Another AWS Account and use the dev account as the trusted account ID to create the role.

Use the permissions in the template cicd_codepipeline_cross_ac_policy.json to create the policy for this role.This role is created in the prod account and has permissions to use CodeDeploy and fetch from Amazon S3. The role is assumed by CodePipeline from the dev account to deploy the app in the prod account. Make sure to set up trust with the dev account for this IAM role on the Trust relationships tab.
AWS CodeBuildService roleDev (111111111111)

cicd_codebuild_service_role

Choose CodeBuild as the use case to create the role.

Use the permissions in the template cicd_codebuild_service_policy.json to create the policy for this role.This CodeBuild service role has appropriate permissions to:

  • The S3 bucket to store artefacts
  • Stream logs to CloudWatch Logs
  • Pull code from CodeCommit
  • Get the SSM parameter for CodeBuild
  • Miscellaneous Amazon EC2 permissions
AWS CodeDeployService roleDev (111111111111) and Prod (222222222222)

cicd_codedeploy_service_role

Choose CodeDeploy as the use case to create the role.

Use the built-in AWS managed policy AWSCodeDeployRole for this role.This CodeDeploy service role has appropriate permissions to:

  • Miscellaneous Amazon EC2 Auto Scaling
  • Miscellaneous Amazon EC2
  • Publish Amazon SNS topic
  • AWS CloudWatch metrics
  • Elastic Load Balancing
EC2 InstanceService role for EC2 instance profileDev (111111111111) and Prod (222222222222)

cicd_ec2_instance_profile

Choose EC2 as the use case to create the role.

Use the permissions in the template cicd_ec2_instance_profile_policy.json to create the policy for this role.

This is set as the EC2 instance profile for the EC2 instances where the app is deployed. It has appropriate permissions to fetch artefacts from Amazon S3 and decrypt contents using the KMS key.

 

You must update this role later with the actual KMS key and S3 bucket name created as part of the deployment process.

 

 

Setting up the prod account

To set up the prod account, complete the following steps:

  1. Download and launch the AWS CloudFormation template from the GitHub repo: cicd-codedeploy-prod.json
    • This deploys the CodeDeploy app and deployment group.
    • Make sure that you already have a set of EC2 Linux instances with the CodeDeploy agent installed in all the accounts where the sample Java application is to be installed (dev and prod accounts). If not, refer back to the Prerequisites section.
  2. Update the existing EC2 IAM instance profile (cicd_ec2_instance_profile):
    • Replace the S3 bucket name mywebapp-codepipeline-bucket-us-east-1-111111111111 with your S3 bucket name (the one used for the CodePipelineArtifactS3Bucket variable when you launched the CloudFormation template in the dev account).
    • Replace the KMS key ARN arn:aws:kms:us-east-1:111111111111:key/82215457-e360-47fc-87dc-a04681c91ce1 with your KMS key ARN (the one created as part of the CloudFormation template launch in the dev account).

Setting up the dev account

To set up your dev account, complete the following steps:

  1. Download and launch the CloudFormation template from the GitHub repo: cicd-aws-code-suite-dev.json
    The stack deploys the following services in the dev account:

    • CodeCommit repository
    • CodePipeline
    • CodeBuild environment
    • CodeDeploy app and deployment group
    • CloudWatch event rule
    • KMS key (used to encrypt the S3 bucket)
    • S3 bucket and bucket policy
  2. Use following values as inputs to the CloudFormation template. You should have created all the existing resources and roles beforehand as part of the prerequisites.

    KeyExample ValueComments
    CodeCommitWebAppRepoMyWebAppRepoName of the new CodeCommit repository for your web app.
    CodeCommitMainBranchNamemasterMain branch name on your CodeCommit repository. Default is master (which is pushed to the prod environment).
    CodeBuildProjectNameMyCBWebAppProjectName of the new CodeBuild environment.
    CodeBuildServiceRolearn:aws:iam::111111111111:role/cicd_codebuild_service_roleARN of an existing IAM service role to be associated with CodeBuild to build web app code.
    CodeDeployAppMyCDWebAppName of the new CodeDeploy app to be created for your web app. We assume that the CodeDeploy app name is the same in all accounts where deployment needs to occur (in this case, the prod account).
    CodeDeployGroupDevMyCICD-Deployment-Group-DevName of the new CodeDeploy deployment group to be created in the dev account.
    CodeDeployGroupProdMyCICD-Deployment-Group-ProdName of the existing CodeDeploy deployment group in prod account. Created as part of the prod account setup.

    CodeDeployGroupTagKey

     

    ApplicationName of the tag key that CodeDeploy uses to identify the existing EC2 fleet for the deployment group to use.

    CodeDeployGroupTagValue

     

    MyWebAppValue of the tag that CodeDeploy uses to identify the existing EC2 fleet for the deployment group to use.
    CodeDeployConfigNameCodeDeployDefault.OneAtATime

    Desired Code Deploy config name. Valid options are:

    CodeDeployDefault.OneAtATime

    CodeDeployDefault.HalfAtATime

    CodeDeployDefault.AllAtOnce

    For more information, see Deployment configurations on an EC2/on-premises compute platform.

    CodeDeployServiceRolearn:aws:iam::111111111111:role/cicd_codedeploy_service_role

    ARN of an existing IAM service role to be associated with CodeDeploy to deploy web app.

     

    CodePipelineNameMyWebAppPipelineName of the new CodePipeline to be created for your web app.
    CodePipelineArtifactS3Bucketmywebapp-codepipeline-bucket-us-east-1-111111111111Name of the new S3 bucket to be created where artifacts for the pipeline are stored for this web app.
    CodePipelineServiceRolearn:aws:iam::111111111111:role/cicd_codepipeline_service_roleARN of an existing IAM service role to be associated with CodePipeline to deploy web app.
    CodePipelineCWEventTriggerRolearn:aws:iam::111111111111:role/cicd_codepipeline_trigger_cwe_roleARN of an existing IAM role used to trigger the pipeline you named earlier upon a code push to the CodeCommit repository.
    CodeDeployRoleXAProdarn:aws:iam::222222222222:role/cicd_codepipeline_cross_ac_roleARN of an existing IAM role in the cross-account for CodePipeline to assume to deploy the app.

    It should take 5–10 minutes for the CloudFormation stack to complete. When the stack is complete, you can see that CodePipeline has built the pipeline (MyWebAppPipeline) with the CodeCommit repository and CodeBuild environment, along with actions for CodeDeploy in local (dev) and cross-account (prod). CodePipeline should be in a failed state because your CodeCommit repository is empty initially.

  3. Update the existing Amazon EC2 IAM instance profile (cicd_ec2_instance_profile):
    • Replace the S3 bucket name mywebapp-codepipeline-bucket-us-east-1-111111111111 with your S3 bucket name (the one used for the CodePipelineArtifactS3Bucket parameter when launching the CloudFormation template in the dev account).
    • Replace the KMS key ARN arn:aws:kms:us-east-1:111111111111:key/82215457-e360-47fc-87dc-a04681c91ce1 with your KMS key ARN (the one created as part of the CloudFormation template launch in the dev account).

Deploying the application

You’re now ready to deploy the application via your desktop or PC.

  1. Assuming you have the required HTTPS Git credentials for CodeCommit as part of the prerequisites, clone the CodeCommit repo that was created earlier as part of the dev account setup. Obtain the name of the CodeCommit repo to clone, from the CodeCommit console. Enter the Git user name and password when prompted. For example:
    $ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/MyWebAppRepo my-web-app-repo
    Cloning into 'my-web-app-repo'...
    Username for 'https://git-codecommit.us-east-1.amazonaws.com/v1/repos/MyWebAppRepo': xxxx
    Password for 'https://[email protected]/v1/repos/MyWebAppRepo': xxxx

  2. Download the MyWebAppRepo.zip file containing a sample Java application, CodeBuild configuration to build the app, and CodeDeploy config file to deploy the app.
  3. Copy and unzip the file into the my-web-app-repo Git repository folder created earlier.
  4. Assuming this is the sample app to be deployed, commit these changes to the Git repo. For example:
    $ cd my-web-app-repo 
    $ git add -A 
    $ git commit -m "initial commit" 
    $ git push

For more information, see Tutorial: Create a simple pipeline (CodeCommit repository).

After you commit the code, the CodePipeline will be triggered and all the stages and your application should be built, tested, and deployed all the way to the production environment!

The following screenshot shows the entire pipeline and its latest run:

 

Troubleshooting

To troubleshoot any service-related issues, see the following:

Cleaning up

To avoid incurring future charges or to remove any unwanted resources, delete the following:

  • EC2 instance used to deploy the application
  • CloudFormation template to remove all AWS resources created through this post
  •  IAM users or roles

Conclusion

Using this solution, you can easily set up and manage an entire CI/CD pipeline in AWS accounts using the native AWS suite of CI/CD services, where a commit or change to code passes through various automated stage gates all the way from building and testing to deploying applications, from development to production environments.

FAQs

In this section, we answer some frequently asked questions:

  1. Can I expand this deployment to more than two accounts?
    • Yes. You can deploy a pipeline in a tooling account and use dev, non-prod, and prod accounts to deploy code on EC2 instances via CodeDeploy. Changes are required to the templates and policies accordingly.
  2. Can I ensure the application isn’t automatically deployed in the prod account via CodePipeline and needs manual approval?
  3. Can I use a CodeDeploy group with an Auto Scaling group?
    • Yes. Minor changes required to the CodeDeploy group creation process. Refer to the following Solution Variations section for more information.
  4. Can I use this pattern for EC2 Windows instances?

Solution variations

In this section, we provide a few variations to our solution:

Author bio

author-pic

 Nitin Verma

Nitin is currently a Sr. Cloud Architect in the AWS Managed Services(AMS). He has many years of experience with DevOps-related tools and technologies. Speak to your AWS Managed Services representative to deploy this solution in AMS!

 

Send real-time alerts using Amazon Pinpoint

Post Syndicated from Dhiraj Thakur original https://aws.amazon.com/blogs/messaging-and-targeting/send-real-time-alerts-using-amazon-pinpoint/

Businesses need to send real-time notifications in order to take action when alerted of a critical situation. Examples could include anomaly detection, healthcare emergencies, operations failures, and fraud transactions. Email, SMS, and push notifications are often used to notify stakeholders in real-time. However, building a large-scale, real-time notification solution can be a complex and costly challenge for a business.

Amazon Pinpoint enables you to engage with your stakeholders in real-time by sending email, SMS and push notifications. Your app can use the Amazon Pinpoint API and the AWS SDKs to send direct messages. With transactional messages, you send alerts to specific recipients, as opposed to messages that you send to segments. There is no minimum fee, no setup cost, and no fixed monthly cost with Amazon Pinpoint.

In this blog, we explore a solution to notify stakeholders of a large customer transaction. This requires immediate attention as our stakeholders want to ensure that there is enough inventory available to deliver goods with no delay.

Solution Overview

The solution that we build to handle this use case can be deployed in one hour. The following diagram illustrates the AWS services integrated in this solution:

At a high level, the solution uses the following workflow:

1. Define large value transaction threshold in rule table in Amazon DynamoDB.
2. Setup Amazon Pinpoint and configure to send email and SMS.
3. Setup AWS Lambda and implement the logic to send SMS and email if a customer makes a large value order transaction.
4. Create a test transaction in an order table using Amazon DynamoDB.
5.  Check details of SMS and email received.

Setting up the solution

Step 1: Set up Amazon Pinpoint

The first step in setting up this solution is to create a new Amazon Pinpoint project and configure the SMS and Email channel.
1. Navigate to Services -> Pinpoint.
2.Click on create a project.
3. Provide a name and click on the Create button.
4. Select Email in left panel and click on Edit.
5.  Select “Enable the email channel for this project”. Select “Verify a new email address” and provide a default sender address. Click on verify email address. Click on save. You should get a verification mail. Verify the email by clicking on the verification link.

Once your email is verified, it should look like this:

6. Repeat the same step to verify receiver email address as well.

Please note: By enabling the email channel, we can send up to 200 email per day in sandbox mode. We must verify an email address or domain identity in order to send any email. While we remain in sandbox, we also must verify all recipient email addresses before we proceed — this does not apply in production.

7.  Select SMS and voice in left panel and select “Enable the SMS channel for this project”. Please make sure “Transactional” is selected for critical or time-sensitive messages. Amazon Pinpoint optimizes the delivery of these messages for highest reliability.

Please note: We do not need to verify any phone numbers for this channel. However, we can request dedicated codes (either short or long) for our own exclusive use. Otherwise AWS will automatically allocate codes when we send SMS.

8. Note down the project ID.

We have done all the configurations in Amazon Pinpoint. Now, it’s time to setup Amazon DynamoDB tables.

Step 2: Set up Amazon DynamoDB table

1. Navigate to Services -> DynamoDB.
2. Click Create table.
3. Create a record similar to this:

This table stores your definition of large-value order transaction.

{
		"transaction_type" : "premium"
		"min_transaction_value" : 10000
		"max_transaction_value" : 50000
		"send_notification" : ""
		"unit" : "$"
		"email" : "[email protected]"
		"phone" : "+91123456789"
		
	
	}

4. Create order_detail table. Provide order_id as partition key. You can select “String” as data type.
5. Now we must enable the stream in this table. Once we enable a stream on a table, Amazon DynamoDB captures information about every modification to data items in the table. We will integrate this table with AWS Lambda to validate the order value. Click on Manage Stream.
6. Select appropriate view type and click on Enable.

Step 3: Set up AWS Lambda

In this step, we create an AWS Lambda function and then integrate it with the order_detail table. After, we will check the order and send a notification if large enough.

1. Navigate to Service>Lambda.
2.Click on Create function.
3. Select runtime as Python 3.6. Select an execution role. Please make sure that your role gives read access to Amazon DynamoDB table.
4. Click on Add Trigger.
5.  Select the Amazon DynamoDB table from the drop-down. Select order_detail table from the drop-down. Keep everything default. This integrates order_detail streaming to our AWS Lambda function.

Our Lambda function is invoked every time a transaction happens in order_detail table.

6. Now copy the source code.

The AWS Lambda code is available on GitHub here: https://github.com/aws-samples/send-real-time-notification-using-amazon-pinpoint-samples 

Step 4: Testing

1. That’s it! We have built the solution. Now it’s time to do an end to end test by creating an order transaction in order_detail table.

2. Run the Amazon DynamoDB put-item command to create a new item over our threshhold, or replace an old item with a new one. In our case it will be a new order transaction in order_detail table. Please make sure you have configured AWS CLI. You can refer to the quickstart guide to learn more.

aws dynamodb put-item --table-name order_detail --item "{""order_id"":{""S"":""O0001""},""customer_id"":{""S"":""C0001""},""customer_name"":{""S"":""JOHN MILLER""},""order_date"":{""S"":""2020-06-13T17:42:34Z""},""item_id"":{""S"":""P0001""},""item_quantity"":{""N"":""12""},""order_value"":{""N"":""20000""},""unit"":{""S"":""$""},""delivery_date"":{""S"":""2020-06-20""}}" --return-consumed-capacity TOTAL

3. It returns a success message similar to this:

4.  The order value of $20,000 is falling in the range of our large value order transaction definition (between $10,000 to $50,000). You should receive an SMS on the mobile number you have provided in the transaction_alert_rule table.

5. You will also receive an email as per configuration in the transaction_alert_rule table.

Step 4: Clean up

You’ve now successfully built our real-time notification solution using Amazon Pinpoint. Delete the resource you created in this blog like the Amazon DynamoDB tables to avoid ongoing charges. You can use the AWS CLI, AWS Management Consoles, or the AWS APIs to perform the cleanup.

Conclusion

Customers can use Amazon Pinpoint to help scale communications across use cases, including real-time notifications. Amazon Pinpoint is a flexible and scalable outbound and inbound marketing communications service. You can connect with customers and stakeholders over channels like email, SMS, push, or voice.

 

Adding WhatsApp as an Amazon Pinpoint Channel

Post Syndicated from Edwin Bejarano original https://aws.amazon.com/blogs/messaging-and-targeting/adding-whatsapp-as-an-amazon-pinpoint-channel/

Amazon Pinpoint recently announced the general availability of custom channels. Custom channels enable you to extend the capabilities of Amazon Pinpoint via a webhook or AWS Lambda function. Among many other possibilities, you can use custom channels to send messages to your customers through any API-enabled service, for example WhatsApp. With these new channels, you have full control over the message delivery to the endpoints associated with each custom channel campaign.

In this post, we provide a quick overview of the features and capabilities of using a custom channel as part of campaigns. We also provide a blueprint that you can use to build your first integration with WhatsApp as a custom channel.

Note: Amazon Web Services isn’t responsible for any third-party service that you use to send messages with custom channels. Third-party services may be subject to additional terms. 

Prerequisites

Before creating your new custom channel, you must have the integration ready and an Amazon Identity and Account Management (IAM) user created with the necessary permissions. First set up the following:

  1. Create an IAM administrator. For more information, see Creating your first IAM admin user and group in the IAM User Guide. Specify the credentials of this IAM user when you set up the AWS Command Line Interface (CLI).
  2. Follow the steps at https://www.twilio.com/docs/whatsapp/api to create a free Twilio account and have your WhatsApp account join the sandbox to be able to receive messages. Note the account SID and Auth token as they are needed in a later step.
  3. Configure the AWS CLI. For more information about setting up the AWS CLI, see Configuring the AWS CLI.

whatsapp

Step 1: Create an Amazon Pinpoint project.

In this section, you create and configure a project in Amazon Pinpoint. Later, you use this data to create segments and campaigns.

To set up the Amazon Pinpoint project

  1. Sign in to the Amazon Pinpoint console at http://console.aws.amazon.com/pinpoint/.
  2. On the All projects page, choose Create a project. Enter a name for the project, and then choose Create.
  3. On the Configure features page, under SMS and Voice, choose Configure.
  4. Under General settings, select Enable the SMS channel for this project, and then choose Save changes.
  5. In the navigation pane, under Settings, choose General settings. In the Project details section, copy the value under Project ID. You need this value for later.

Step 2: Create an endpoint.

In Amazon Pinpoint, an endpoint represents a specific method of contacting a customer. This could be their email address (for email messages) or their phone number (for SMS messages) or a custom endpoint type. Endpoints can also contain custom attributes, and you can associate multiple endpoints with a single user. In this step, we create an SMS endpoint that is used to send a WhatsApp message.

To create an endpoint using AWS CLI, at the command line, enter the following command:

aws pinpoint update-endpoint --application-id <project-id> \
--endpoint-id 12456 --endpoint-request "Address='<mobile-number>', \
ChannelType='SMS',Attributes={username=['testUser'],integrations=['WhatsApp']}"

In the preceding example, replace <project-id> with the Amazon Pinpoint Project ID that you copied in step 1. Replace <mobile-number> with your phone number, formatted in E.164 format (for example, +12065550142). For the WhatsApp integration to work, you must use the mobile number that you registered in the WhatsApp sandbox in the prerequisite steps.

Step 3: Create an AWS Lambda.

You must create an AWS Lambda that has the code that calls Twilio and sends a message to the endpoint.

1. Open the AWS Lambda console at http://console.aws.amazon.com/AWSLambda, and then click on Create Function.

2. Choose Author from scratch.

3. For Function Name, enter ‘WhatsAppTest’.

4. For Runtime, select Python 3.7.

5. Click Create Function.

6. For the function code, copy the following and paste into the code browser in your AWS Lambda function:

import base64
import json
import os
import urllib
from urllib import request, parse
 
TWILIO_SMS_URL = "https://api.twilio.com/2010-04-01/Accounts/{}/Messages.json" 
TWILIO_ACCOUNT_SID = os.environ.get("TWILIO_ACCOUNT_SID")
TWILIO_AUTH_TOKEN = os.environ.get("TWILIO_AUTH_TOKEN")
TWILIO_FROM_NUMBER = os.environ.get("TWILIO_FROM_NUMBER")
 
def lambda_handler(event, context):
	if not TWILIO_ACCOUNT_SID:
		return "Unable to access Twilio Account SID."
	elif not TWILIO_AUTH_TOKEN:
		return "Unable to access Twilio Auth Token."
	elif not TWILIO_FROM_NUMBER:
		return "Unable to access Twilio WhatsApp From Number."
	# Lets print out the event for our logs 
	print("Received event: {}".format(event))
	# insert Twilio Account SID into the REST API URL 
	populated_url = TWILIO_SMS_URL.format(TWILIO_ACCOUNT_SID)
	
	for key in event['Endpoints'].keys(): 
		to_number = event['Endpoints'][key]['Address']
		# Example body and using an attribute from the endpoint
		username = event['Endpoints'][key]['Attributes']['username'][0]
		body = "Hello {}, here is your weekly 10% discount coupon: SAVE10".format(username)
		post_params = {"To": "whatsapp:" + to_number, "From": "whatsapp:" + TWILIO_FROM_NUMBER, "Body": body}
		# encode the parameters for Python's urllib 
		data = parse.urlencode(post_params).encode() 
		req = request.Request(populated_url)
		# add authentication header to request based on Account SID + Auth Token 
		authentication = "{}:{}".format(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN) 
		base64string = base64.b64encode(authentication.encode('utf-8')) 
		req.add_header("Authorization", "Basic %s" % base64string.decode('ascii')) 
		print("Request: " + str(request))
		try:
			# perform HTTP POST request
			with request.urlopen(req, data) as f:
				print("Twilio returned {}".format(str(f.read().decode('utf-8')))) 
		except Exception as e:
			# something went wrong!
			return e
	return "WhatsApp messages sent successfully"

7. Add environment variables for your Twilio Account SID, Auth token, and From Number that you acquired from the prerequisite steps.

environmentalvariables

8. Add permissions to your AWS Lambda to allow Amazon Pinpoint to invoke it:

aws lambda add-permission \
--function-name WhatsAppTest \
--statement-id sid \
--action lambda:InvokeFunction \
--principal pinpoint.us-east-1.amazonaws.com \
--source-arn arn:aws:mobiletargeting:us-east-1:<account-id>:apps/*

Step 4: Create a segment and campaign in Amazon Pinpoint.

Now that we have an endpoint, we must add it to a segment so that we can use it within a campaign. By sending a campaign, we can verify that our Amazon Pinpoint project is configured correctly, and that we created the endpoint correctly.

To create the segment and campaign:

1. Open the Amazon Pinpoint console at http://console.aws.amazon.com/pinpoint, and then choose the project that you created in step 1.

2. In the navigation pane, choose Segments, and then choose Create a segment.

3. Name the segment “WhatsAppTest.” Under Segment group 1, on the Add a filter menu, choose Filter by endpoint.

4. For Choose an endpoint attribute, choose integrations, then for values, choose WhatsApp.

Segmentfilter

5. Confirm that the Segment estimate section shows that there is one eligible endpoint, and then choose Create segment.

6. In the navigation pane, choose Campaigns, and then choose Create a campaign.

7. Name the campaign “WhatsAppTest.” Under Choose a channel for this campaign, choose Custom, and then choose Next.

8. On the Choose a segment page, choose the “WhatsAppTest” segment that you just created, and then choose Next.

9. In Create your message, choose the AWS Lambda function we just created, ‘WhatsAppTest.’ Select SMS in the Endpoint Options. On the Choose when to send the campaign page, keep all of the default values, and then choose Next. On the Review and launch page, choose Launch campaign.

Within a few seconds, you should receive a WhatsApp message at the phone number that you specified when you created the endpoint.

create message

Next steps

Your Custom channel solution for WhatsApp is now ready to use. There are several ways you can make this solution your own:

  • Customize your messaging: This post used an example message to be sent to your endpoints within the AWS Lambda. You can customize that message to fit your needs.
  • Expand endpoints in your application: This post only used one endpoint for the integration. You can use your WhatsApp integration with new endpoints by importing a segment that can be used with a new campaign. Learn how to import a segment here: https://docs.aws.amazon.com/pinpoint/latest/userguide/segments-importing.html
  • Use new integrations: This post focused on integrating your custom channel with WhatsApp but there are many other integrations that are possible when using AWS Lambda.

Amazon Pinpoint is a flexible and scalable outbound and inbound marketing communications service. Learn more here: https://aws.amazon.com/pinpoint/

New account review process in the AWS Support Center for Amazon SES and Amazon Pinpoint

Post Syndicated from Heidi Gloudemans original https://aws.amazon.com/blogs/messaging-and-targeting/new-account-review-process-in-the-aws-support-center-for-amazon-ses-and-amazon-pinpoint/

Note: This post was written by Dustin Taylor, an AWS Technical Program Manager for Amazon Pinpoint.


Amazon Simple Email Service (SES) and Amazon Pinpoint have processes and systems in place to help protect AWS customers from issues that could damage their reputations as email senders. These systems alert customers about sending issues as early as possible. Examples of these issues can include high bounce rates, high complaint rates, or a large number of messages sent to spamtraps. In some cases, AWS may pause a customer’s account if the issue represents a significant risk or damage to their sender reputation.

To resolve these sending issues, customers work directly with the AWS team. Previously, the customer communicated resolution plans with AWS over email. However, the contact used to manage this process was the root email address of the customer’s AWS account. Many customers told us this process was inconvenient.

In order to make this process easier, we have migrated all customer communication on account reviews to the AWS Support Center. Effective immediately, if an account is placed under review or is subject to a sending pause, the Amazon SES and Amazon Pinpoint teams automatically create a Support Center case. To access the AWS Support Center, start by signing in to the AWS Management Console at https://console.aws.amazon.com. Then, on the Support menu, choose Support Center.

If we open one of these cases on your account, you can use the Support Center to communicate with our team about the issue. In each case, we provide information about the nature of the issue. We also describe the information that we need you to provide us to resolve the issue. You can update the case with actions you have taken directly in the Support Center case.

These account reviews and sending pause events are rare. However, we hope that this improved process makes it easier to be successful with Amazon SES and Amazon Pinpoint.

 

Amazon Personalize optimizer using Amazon Pinpoint events

Post Syndicated from Heidi Gloudemans original https://aws.amazon.com/blogs/messaging-and-targeting/amazon-personalize-optimizer-using-amazon-pinpoint-events/

Note: This post was written by Ryan Lowe, an AWS Solution Architect and Chelsea Graf, Senior Pinpoint PM.


This is the second of two posts in our series on personalization with Amazon Pinpoint. Today’s customer expects a high level of personalization in order to continue engaging with an enterprise.  Rather than a generalized marketing campaign, enterprises use curated messaging to lower churn rates, increase consumer interaction, and drive higher conversion rates. Many enterprises are turning to machine learning to deliver personalized product recommendations or promotions at scale. In order to maintain an effective and relevant machine learning model, you need high volumes of recent behavioral data. Without an automated data pipeline, retraining is manual, inefficient and risks a model becoming outdated.

AWS offers the Amazon Personalize Optimizer Using Amazon Pinpoint Events solution to help you easily connect your Amazon Personalize campaigns and Amazon Pinpoint projects. This solution allows you to train and publish models quickly without support from an engineer. You can define the frequency and the type of data used to retrain your models. Using an automated retraining loop frees you to build new models and keep them relevant for your marketers.

The following diagram illustrates the flow of data in this solution.

The AWS CloudFormation template configures the event stream in an existing Amazon Pinpoint project to use Amazon Kinesis Data Firehose to export behavioral events such as email opens. It also configures the latter to store event data in Amazon Simple Storage Service (Amazon S3). The Amazon S3 data schema is stored in an AWS Glue Data Catalog enabling data queries.

There is a constant flow of real-time data moving from Amazon Pinpoint through Kinesis Data Firehose and being stored in Amazon S3. When you send a campaign, Amazon Pinpoint connects with Amazon Personalize to retrieve a personalized recommendation. This recommendation is based on the Amazon Pinpoint recommender model configuration for each user identified in the campaign.

The AWS CloudFormation template also deploys a daily batch process orchestrated by AWS Step Functions. The process begins when an Amazon CloudWatch time-based event triggers a series of AWS Lambda functions. The functions use an Amazon Athena query to query customer data stored in Amazon S3. The query result is then used to retrain Amazon Personalize by providing new interaction data from the Amazon Pinpoint event data.

To get started, visit the Amazon Personalize Optimizer Using Amazon Pinpoint Events solution page on the AWS Solutions site.

Template Personalization using Amazon Pinpoint

Post Syndicated from Heidi Gloudemans original https://aws.amazon.com/blogs/messaging-and-targeting/template-personalization-using-amazon-pinpoint/

Note: This post was written by Ryan Lowe, an AWS Solution Architect and Chelsea Graf, a Senior PM for Amazon Pinpoint. 


Today’s customer expects a high level of personalization in order to continue engaging with an enterprise. Using curated messaging, rather than a “one-size-fits-all” marketing campaign, an enterprise can lower churn rates, increase customer interaction, and drive higher conversion rates. Many enterprises are turning to machine learning to deliver personalized product recommendations or promotions at scale. With the growth of AI and ML tools such as Amazon Personalize, the barrier to create a machine learning model is lowering. However, training a model is only the first step toward personalized messaging. Once trained, a model must be integrated with a messaging service in order to get recommendations to customers. Then it is critical to use customers’ behavioral data to continue training and optimizing a model to ensure relevancy.

In this two-part post, we walk you through how to add personalized content to your Amazon Pinpoint templates using an Amazon Personalize campaign, and then how to use behavioral events, such as email opens, to retrain the campaign. In the first post, you will learn how to configure an integration between your Amazon Pinpoint account and your Amazon Personalize campaign. You will then learn how to populate dynamic attributes in your Amazon Pinpoint messaging templates with the recommendations returned by Amazon Personalize. In the second post, you learn how to use custom Amazon Pinpoint events to retrain your Amazon Personalize campaigns.

Template personalization using machine learning models

Using Amazon Pinpoint, you can now create a configuration connecting a recommender model you created in Amazon Personalize with your Amazon Pinpoint account. A recommender model is an ML model that’s designed to answer the question, “what may a user like or be interested in?” The recommender model uses customer demographics and behavioral data to predict a particular customer’s preferences from a given set of products or items. In return, it provides that information as a set of recommendations. By using recommender models with Amazon Pinpoint, you can send personalized recommendations to message recipients based on each recipient’s attributes and behavior.

To get started personalizing your messaging using machine learning, Amazon Personalize guides you through the process of creating and training a recommender model. It then guides you on preparing and deploying the model as an Amazon Personalize campaign. To create a model that can integrate with Amazon Pinpoint, create a solution using a USER_PERSONALIZATION recipe, then deploy your campaign. Be sure that you train your model using either an Amazon Pinpoint endpoint id (Id) or user id (EndpointUser.UserId). You need this identifier to retrieve recommendations during campaign execution.

Next, you define the configuration you want to use to call to your Amazon Personalize campaign and fetch recommendations for each of your customers. In the console, Amazon Pinpoint walks you through the process of defining your configuration. To start, be sure to provide a clear name and description of your model to allow you to differentiate this configuration from others when creating a template. You can then select the Amazon Personalize campaign you want to use, the IAM role that permits Amazon Pinpoint to call your Amazon Personalize campaign, and the identifier that Amazon Pinpoint is to pass to Amazon Personalize to retrieve the recommendation. To get the correct recommendations, it is critical that you select the identifier you used to train your Amazon Personalize campaign. As part of your configuration you also select how many items or products you would like to get back from your Amazon Personalize campaign. For example, if you are sending an email that you want to show three items to be recommended, you would set the Number of Recommendations per Message to three. Finally, you select how you want to process the recommendations you get back from Amazon Personalize.

An Amazon Personalize campaign returns a string based on the data you used to create your model. This can include a simple product ID, URL, or even an HTML blurb. You have two options for processing this string; using the value returned by the model or using an AWS Lambda function. If Amazon Personalize returns an HTML blurb, you may opt to just add that string to your Amazon Pinpoint template and send a message. All you would do is set the friendly name of your attribute and save your configuration. When using a product ID, you may want to make additional attributes available such as product name, price, or image. Then you would select Using a Lambda function and select the Lambda you want to pass the string to. The Lambda function accepts the Identifier and returns Amazon Pinpoint additional attributes. In the console, you can define up to 10 custom attributes for each item returned by your Amazon Personalize campaign. You can find an example Lambda function in the Amazon Pinpoint Developer Guide.

Once you have saved your configuration, it is accessible directly from the Attribute finder in the Amazon Pinpoint template editor. To add dynamic attributes to a template, start by creating or editing a template. In the Attribute Finder, select Recommended attributes and choose connect model. After selecting your model, you see a list of attributes that you can copy and paste directly into your template. If you opted to have more than one item returned per customer, you see a list of recommendations. These correspond to the order the items were returned from Amazon Personalize. Now you can click to copy an attribute and paste it into your message. Once you have added your desired attributes, be sure to also add default values to your message. To specify a default value for a message variable, expand the Default attribute values section of your template. Then, in the list of variables, enter the default value that you want to use for the variable. We recommend that you do this for each variable in the template. Once you have entered your defaults, you can create a new template, save a new version, or update an existing version of an existing template.

The final step is to create an Amazon Pinpoint campaign using your template and begin sending messages populated using machine learning.

In our next post, you can learn how to retrain your model using custom events.

 

AWS Alarms for Application Errors

Post Syndicated from Bozho original https://techblog.bozho.net/aws-alarms-for-application-errors/

Monitoring is key for any real-world application. You have to know what’s happening and be alerted in real time if something wrong is happening. AWS has CloudWatch for that, and gives you a lot of metrics automatically. But there are some that you have to define yourself. And then you need to define proper alarms.

Here I’ll focus on hour:

  • High number of application errors
  • High number of application warnings
  • High number of 5xx errors on the load balancer
  • High number of 4xx errors on the load balancer

First, the prerequisites:

  • You need to be using CloudFormation to automate everything. You can create all of those things manually, but automation is a big plus
  • If using CloudFormation, you’d preferably have a sub-stack for configuring alarms
  • You need to be collecting your logs with CloudWatch logs

If you are not using CloudWatch logs, here’s a simple config file and script to enable them:

{
  "agent": {
    "metrics_collection_interval": 10,
    "region": "eu-west-1",
    "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "{{logPath}}",
            "log_group_name": "{{logGroupName}}",
            "log_stream_name": "{instance_id}",
            "timestamp_format": "%Y-%m-%d %H:%M:%S"
          }
        ]
      }
    }
  }
}
# install AWS CloudWatch monitor
mkdir cloud-watch-agent
cd cloud-watch-agent
wget https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip
unzip AmazonCloudWatchAgent.zip
./install.sh

aws s3 cp s3://$BUCKET_NAME/cloudwatch-agent-config.json /var/config/cloudwatch-agent-config.json
sed -i -- 's|{{logPath}}|/var/log/application.log|g' /var/config/cloudwatch-agent-config.json
sed -i -- 's|{{logGroupName}}|app_node|g' /var/config/cloudwatch-agent-config.json

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/var/config/cloudwatch-agent-config.json -s

Now you have to define two things: Log metrics and alarms. The cloudformation code below creates both:

    "HighAppErrorsNotification": {
      "Type": "AWS::CloudWatch::Alarm",
      "Properties": {
        "AlarmActions": [
          {
            "Ref": "NotificationTopicId"
          }
        ],
        "InsufficientDataActions": [
          {
            "Ref": "NotificationTopicId"
          }
        ],
        "AlarmDescription": "Notify if there are too many application errors",
        "ComparisonOperator": "GreaterThanOrEqualToThreshold",
        "EvaluationPeriods": "1",
        "MetricName": "ApplicationErrors",
        "Namespace": "LogMetrics",
        "Period": "900",
        "Statistic": "Sum",
        "Threshold": "5",
        "TreatMissingData": "ignore"
      }
    },
    "ErrorMetricFilter": {
      "Type": "AWS::Logs::MetricFilter",
      "Properties": {
        "LogGroupName": "app_node",
        "FilterPattern": "ERROR",
        "MetricTransformations": [
          {
            "DefaultValue": 0,
            "MetricValue": "1",
            "MetricNamespace": "LogMetrics",
            "MetricName": "ApplicationErrors"
          }
        ]
      }
    },

If you need to do that manually, go to the CloudWatch logs homepage, select the log group (app_node) and use the button “Create metric filter” ontop. It lets you specify the pattern to look for (“ERROR” in this case). When you have that ready, you can create an Alarm based on it, through the Alarms -> Create alarm. Lookup the metric by name and select it to trigger the alarm (in the example above, it gets triggered if there are more than 5 errors within 900 seconds)

You can then create an identical alarm for warnings (pattern to look for: “WARN”). The threshold there might be higher, e.g. 10 or 20. But that depends on your application logging patterns.

Then there’s the error 5xx load balancer alarms. In CloudFormation it would look like this:

   "TooMany5xxErrorsWebAppAlarmNotification": {
      "Type": "AWS::CloudWatch::Alarm",
      "Properties": {
        "AlarmActions": [
          {
            "Ref": "NotificationTopicId"
          }
        ],
        "InsufficientDataActions": [
          {
            "Ref": "NotificationTopicId"
          }
        ],
        "AlarmDescription": "Notify if there are too many 5xx errors",
        "ComparisonOperator": "GreaterThanOrEqualToThreshold",
        "Dimensions": [
          {
            "Name": "LoadBalancer",
            "Value": {
              "Ref": "WebAppALBId"
            }
          }
        ],
        "TreatMissingData": "notBreaching",
        "EvaluationPeriods": "1",
        "MetricName": "HTTPCode_Target_5XX_Count",
        "Namespace": "AWS/ApplicationELB",
        "Period": "60",
        "Statistic": "Sum",
        "Threshold": "2"
      }
    }

You can again create that manually – look for the HTTPCode_Target_5XX_Count metric in the metric selection screen for the alarm. You have several options there, the most straightforward is to select the per AppELB metric. And again, the same approach can be used for 4xx errors (HTTPCode_Target_5XX_Count).

Getting this running with CloudFormation (and even manually) is not as straighforward as it seems. The right combination of metric names, namespaces and values is not obvious and the relevant documentation is not the first thing that pops up. So I decided to share something that works, as it may take some time experimenting before getting it to that state.

But even outside of CloudFormation or AWS context, monitoring and alerting in case of a high number of application errors, warnings and HTTP errors is a must. And automating the creation of those alarms is the recommended approach.

The post AWS Alarms for Application Errors appeared first on Bozho's tech blog.

Using Short Message Service (SMS) for Mass Communications

Post Syndicated from Heidi Gloudemans original https://aws.amazon.com/blogs/messaging-and-targeting/using-short-message-service-sms-for-mass-communications/

Short Message Service (SMS) is a text-based component of the global wireless telephone network. Most consumers use SMS as part of Person-to-Person (P2P) communications in their personal lives, but Application-to-Person (A2P) communications are on the rise. A2P messaging enables an application to send a SMS directly to a customer. In 2019, over 1.2 trillion A2P SMS messages were sent worldwide.1 Organizations, governments, and learning institutions use A2P SMS from Amazon Pinpoint to stay connected to their customers. Common use cases include transactional messages, targeted promotions to a specific audience, or even as a channel for mass communications.

Although there are over 5 billion mobile phones in the world today, little more than half are smartphones.2 The ubiquity of SMS worldwide makes it a natural choice in many mass communications scenarios. SMS is broadly available, without the need for Wi-Fi or data, can facilitate two-way communications, and has a high open rate of over 90%.3

Use cases

There are multiple scenarios where organizations, governments, and learning institutions need to rapidly outreach to their customers, consumers, constituents, or students through SMS. Here are a few examples:

  • Organizations
    • Special working hours
    • Time-based promotions
  • Governments
    • New program alerts or updates
    • Upcoming events
  • Learning Institutions
    • Daily or weekly digests
    • School closure announcements or updates

Getting started

You can use Amazon Pinpoint to immediately send transactional messages, or to send SMS messages as part of a campaign. When you send an SMS campaign in Amazon Pinpoint, you can choose a pre-configured template, create a targeted segment, and view response metrics after the campaign is sent. To help you get started, we’ve created the following tutorials:

  1. Delivery via Console: Creating a campaign in the Amazon Pinpoint console using SMS as a channel
  2. Immediate Delivery via API: Using the SMS APIs from your application for immediate delivery
  3. Campaign Delivery via API: Using the campaign APIs with SMS as a channel in your application

There are initial actions to be taken by organizations when deciding to use SMS as a mass communications channel. First and foremost is gathering the right contact information for intended recipients and permission to contact them. SMS is an opt-in communications channel that may be subject to country-by-country laws and regulations. In many cases, this means that organizations must gain explicit permission from their intended recipients to send them SMS.

One way to gain consent from your customers is to create a web-based registration page. The Amazon Pinpoint Developer Guide contains a tutorial for setting up an SMS registration system.  For SMS best practices, including how to design an SMS-based mass communication and obtain opt-in from your customers to send them SMS, please review SMS Best Practices.

Other important considerations

In the ideal SMS message for mass communications, content should be clear, concise, and under 160 characters. Although messages over 160 characters can be sent, they may be sent and charged as multiple messages. (Learn more here.) An example for a simple 1-way message could be a school snow day notification.

Two (2) hour snow delay on 1/5 (Jan 5). Follow up today at 9 PM PST. https://amzn.to/a - FROM Alpha School District, 1/4/2021.

As a scenario progresses, updates can be provided, but try not share redundant information.

School closure on 1/5 (Jan 5). https://amzn.to/b - FROM Alpha School District, 1/4/2021.

If there are common questions you would like to answer from recipients, you can configure an SMS bot. Learn more about 2-way SMS bots in this deep dive article.

While it may seem like these messages can be sent by any SMS service, there are some important considerations when choosing an SMS provider, such as the scale of the provider’s network. While many mass communications are driven locally, working with a provider that can scale globally is important for the deliverability of the message. Amazon Pinpoint is able to work across global provider networks and, if there are failures, fall back to redundant routes.

Conclusion

SMS is here to stay as a channel for customer engagement across use cases, including mass communication. Amazon Pinpoint is a marketing and developer communications service, powering global end customer messaging through channels that includes SMS, push, email, and voice. Get started today: https://aws.amazon.com/pinpoint

Other useful links

 

Learn more about Amazon Pinpoint SMS channel in this YouTube webinar:

 

 

References

  1. https://www.statista.com/statistics/712394/a2p-sms-traffic-worldwide/
  2. https://www.pewresearch.org/global/2019/02/05/smartphone-ownership-is-growing-rapidly-around-the-world-but-not-always-equally/
  3. https://www.business2community.com/digital-marketing/sms-versus-email-marketing-0957139#!bth7SG

Sandcastle – AWS S3 Bucket Enumeration Tool

Post Syndicated from Darknet original https://www.darknet.org.uk/2020/03/sandcastle-aws-s3-bucket-enumeration-tool/?utm_source=rss&utm_medium=social&utm_campaign=darknetfeed

Sandcastle – AWS S3 Bucket Enumeration Tool

Sandcastle is a Python-based Amazon AWS S3 Bucket Enumeration Tool, formerly known as bucketCrawler. The script takes a target’s name as the stem argument (e.g. shopify) and iterates through a file of bucket name permutations.

Amazon S3 [Simple Storage Service] is cloud storage for the Internet. To upload your data (photos, videos, documents etc.), you first create a bucket in one of the AWS Regions. You can then upload any number of objects to the bucket.

Read the rest of Sandcastle – AWS S3 Bucket Enumeration Tool now! Only available at Darknet.

Retrying Undelivered Voice Messages with Amazon Pinpoint

Post Syndicated from Heidi Gloudemans original https://aws.amazon.com/blogs/messaging-and-targeting/retrying-undelivered-voice-messages-with-amazon-pinpoint/

Note: This post was written by Murat Balkan, an AWS Senior Solutions Architect.


Many of our customers use voice notifications to deliver mission-critical and time-sensitive messages to their users. Customers often configure their systems to retry delivery when these voice messages aren’t delivered the first time around. Other customers set up their systems to fall back to another channel in this situation.

This blog post shows you how to retry the delivery of a voice message if the initial attempt wasn’t successful.

Architecture

By completing the steps in this post, you can create a system that uses the architecture illustrated in the following image:

 

First, a Lambda function calls the SendMessage operation in the Amazon Pinpoint API. The SendMessage operation then initiates a phone call to the recipient and generates a unique message ID, which is returned to the Lambda function. The Lambda function then adds this message ID to a DynamoDB table.

While Amazon Pinpoint attempts to deliver a message to a recipient, it emits several event records. These records indicate when the call is initiated, when the phone is ringing, when the call is answered, and so forth. Amazon Pinpoint publishes these events to an Amazon SNS topic. In this example, we’re only interested in the BUSY, FAILED, and NO_ANSWER event types, so we add some filtering criteria.

An Amazon SQS queue then subscribes to the Amazon SNS topic and monitors the incoming events. The Delivery Delay attribute of this queue is also set at the queue level. This configuration provides a back-off retry mechanism for failed voice messages.

When the Delivery Delay timer is reached, another Lambda function polls the queue and extracts the MessageId attribute from the polled message. It uses this attribute to locate the DynamoDB record for the original call. This record also tells us how many times Amazon Pinpoint has attempted to deliver the message.

The Lambda function compares the number of retries to a MAX_RETRY environment variable to determine whether it should attempt to send the message again.

Prerequisites

To start sending transactional voice messages, create an Amazon Pinpoint project and then request a phone number. Next, clone this GitHub repository to your local machine.

After you add a long code to your account, use AWS SAM to deploy the remaining parts of this serverless architecture. You provide the long number as an input parameter to this template.

The AWS SAM template creates the following resources:

  • A Lambda function (CallGenerator) that initiates a voice call using the Amazon Pinpoint API.
  • An Amazon SNS topic that collects state change events from Amazon Pinpoint.
  • An Amazon SQS queue that queues the messages.
  • A Lambda function (RetryCallGenerator) that polls the Amazon SQS queue and re-initiates the previously failed call attempt by calling the CallGenerator function.
  • A DynamoDB table that contains information about previous attempts to deliver the call.

The template also defines a custom Lambda resource, CustomResource, which creates a configuration set in Amazon Pinpoint. This configuration set specifies the events to send to Amazon SNS. A Lambda environment variable, CONFIG_SET_NAME, contains the name of the configuration set.

This architecture consists of two Lambda functions, which are represented as two different apps in the AWS SAM template. These functions are named CallGenerator and RetryCallGenerator. The CallGenerator function initiates the voice message with Amazon Pinpoint. The SendMessage API in Amazon Pinpoint returns a MessageId. The architecture uses this ID as a key to connect messages to the various events that they generate. The CallGenerator function also retains this ID in a DynamoDB table called call_attempts. The RetryCallGenerator function looks up the MessageId in the call_attempts table. If necessary, the function tries to send the message again by invoking the CallGenerator function.

 

Deploying and Testing

Start by downloading the template from the GitHub repository. AWS SAM requires you to specify an Amazon Simple Storage Service (Amazon S3) bucket to hold the deployment artifacts. If you haven’t already created a bucket for this purpose, create one now. The bucket should be reachable by a AWS Identity and Access Management (IAM) user.

At the command line, enter the following command to package the application:

sam package --template template.yaml --output-template-file output_template.yaml --s3-bucket BUCKET_NAME_HERE

In the preceding command, replace BUCKET_NAME_HERE with the name of the Amazon S3 bucket that should hold the deployment artifacts.

AWS SAM packages the application and copies it into the Amazon S3 bucket. This AWS SAM template requires you to specify three parameters: longCode, the phone number that’s used to make the outbound calls; maxRetry, which is used to set the MAX_RETRY environment variable for the RetryCallGenerator application; and retryDelaySeconds, which sets the delivery delay time for the Amazon SQS queue.

When the AWS SAM package command finishes running, enter the following command to deploy the package:

sam deploy --template-file output_template.yaml --stack-name blogstack --capabilities CAPABILITY_IAM --parameter-overrides maxRetry=2 longCode=LONG_CODE retryDelaySeconds=60

In the preceding command, replace LONG_CODE with the dedicated phone number that you acquired earlier.

When you run this command, AWS SAM shows the progress of the deployment. When the deployment finishes, you can test it by sending a sample event to the CallGenerator Lambda function. Use the following sample event to test the Lambda function:

{
"Message": "<speak>Thank you for visiting the AWS <emphasis>Messaging and Targeting Blog</emphasis>.</speak>",
"PhoneNumber": "DESTINATION_PHONE_NUMBER",
"RetryCount": 0
}

In the preceding event, replace DESTINATION_PHONE_NUMBER with the phone number to which you want to send a test message.

Important: Telecommunication providers take several steps to limit unsolicited voice messages. For example, providers in the United States only deliver a certain number of automated voice messages to each recipient per day. For this reason, you can only use Amazon Pinpoint to send 10 calls per day to each recipient. Keep this limit in mind during the testing process.

Conclusion

This architecture shows how Amazon Pinpoint can deliver state change events to Amazon SNS and how a serverless application can use it. You can adapt this architecture to apply to other use cases, such as call auditing, advanced call analytics and more.

An AWS Elasticsearch Post-Mortem

Post Syndicated from Bozho original https://techblog.bozho.net/aws-elasticsearch-post-mortem/

So it happened that we had a production issue on the SaaS version of LogSentinel – our Elasticsearch stopped indexing new data. There was no data loss, as elasticsearch is just a secondary storage, but it caused some issues for our customers (they could not see the real-time data on their dashboards). Below is a post-mortem analysis – what happened, why it happened, how we handled it and how we can prevent it.

Let me start with a background of how the system operates – we accept audit trail entries (logs) through a RESTful API (or syslog), and push them to a Kafka topic. Then the Kafka topic is consumed to store the data in the primary storage (Cassandra) and index it for better visualization and analysis in Elasticsearch. The managed AWS Elasticsearch service was chosen because it saves you all the overhead of cluster management, and as a startup we want to minimize our infrastructure management efforts. That’s a blessing and a curse, as we’ll see below.

We have alerting enabled on many elements, including the Elasticsearch storage space and the number of application errors in the log files. This allows us to respond quickly to issues. So the “high number of application errors” alarm triggered. Indexing was blocked due to FORBIDDEN/8/index write. We have a system call that enables it, so I tried to run it, but after less than a minute it was blocked again. This meant that our Kafka consumers failed to process the messages, which is fine, as we have a sufficient message retention period in Kafka, so no data can be lost.

I investigated the possible reasons for such a block. And there are two, according to Amazon – increased JVM memory pressure and low disk space. I checked the metrics and everything looked okay – JVM memory pressure was barely reaching 70% (and 75% is the threshold), and there was more than 200GiB free storage. There was only one WARN in the elasticsearch application logs (it was “node failure”, but after that there were no issues reported)

There was another strange aspect of the issue – there were twice as many nodes as configured. This usually happens during upgrades, as AWS is using blue/green deployment for Elasticsearch, but we haven’t done any upgrade recently. These additional nodes usually go away after a short period of time (after the redeployment/upgrade is ready), but they wouldn’t go away in this case.

Being unable to SSH into the actual machine, being unable to unblock the indexing through Elasticsearch means, and being unable to shut down or restart the nodes, I raised a ticket with support. And after a few ours and a few exchanged messages, the problem was clear and resolved.

The main reason for the issue is 2-fold. First, we had a configuration that didn’t reflect the cluster status – we had assumed a bit more nodes and our shared and replica configuration meant we have unassigned replicas (more on shards and replicas here and here). The best practice is to have nodes > number of replicas, so that each node gets one replica (plus the main shard). Having unassigned shard replicas is not bad per se, and there are legitimate cases for it. Our can probably be seen as misconfiguration, but not one with immediate negative effects. We chose those settings in part because it’s not possible to change some settings in AWS after a cluster is created. And opening and closing indexes is not supported.

The second issue is AWS Elasticsearch logic for calculating free storage in their circuit breaker that blocks indexing. So even though there were 200+ GiB free space on each of the existing nodes, AWS Elasticsearch thought we were out of space and blocked indexing. There was no way for us to see that, as we only see the available storage, not what AWS thinks is available. So, the calculation gets the total number of shards+replicas and multiplies it by the per-shard storage. Which means unassigned replicas that do not take actual space are calculated as if they take up space. That logic is counterintuitive (if not plain wrong), and there is hardly a way to predict it.

This logic appears to be triggered when blue/green deployment occurs – so in normal operation the actual remaining storage space is checked, but during upgrades, the shard-based check is triggered. That has blocked the entire cluster. But what triggered the blue/green deployment process?

We occasionally need access to Kibana, and because of our strict security rules it is not accessible to anyone by default. So we temporarily change the access policy to allow access from our office IP(s). This change is not expected to trigger a new deployment, and has never lead to that. AWS documentation, however, states:

In most cases, the following operations do not cause blue/green deployments: Changing access policy, Changing the automated snapshot hour, If your domain has dedicated master nodes, changing data instance count.
There are some exceptions. For example, if you haven’t reconfigured your domain since the launch of three Availability Zone support, Amazon ES might perform a one-time blue/green deployment to redistribute your dedicated master nodes across Availability Zones.

There are other exceptions, apparently, and one of them happened to us. That lead to the blue/green deployment, which in turn, because of our flawed configuration, triggered the index block based on the odd logic to assume unassigned replicas as taking up storage space.

How we fixed it – we recreated the index with fewer replicas and started a reindex (it takes data from the primary source and indexes it in batches). That reduced the size taken and AWS manually intervened to “unstuck” the blue/green deployment. Once the problem was known, the fix was easy (and we have to recreate the index anyway due to other index configuration changes). It’s appropriate to (once again) say how good AWS support is, in both fixing the issue and communicating it.

As I said in the beginning, this did not mean there’s data loss because we have Kafka keep the messages for a sufficient amount of time. However, once the index was writable, we expected the consumer to continue from the last successful message – we have specifically written transactional behaviour that committed the offsets only after successful storing in the primary storage and successful indexing. Unfortunately, the kafka client we are using had auto-commit turned on that we have overlooked. So the consumer has skipped past the failed messages. They are still in Kafka and we are processing them with a separate tool, but that showed us that our assumption was wrong and the fact that the code calls “commit” doesn’t actually mean something.

So, the morals of the story:

  • Monitor everything. Bad things happen, it’s good to learn about them quickly.
  • Check your production configuration and make sure it’s adequate to the current needs. Be it replicas, JVM sizes, disk space, number of retries, auto-scaling rules, etc.
  • Be careful with managed cloud services. They save a lot of effort but also take control away from you. And they may have issues for which your only choice is contacting support.
  • If providing managed services, make sure you show enough information about potential edge cases. An error console, an activity console, or something, that would allow the customer to know what is happening.
  • Validate your assumptions about default settings of your libraries. (Ideally, libraries should warn you if you are doing something not expected in the current state of configuration)
  • Make sure your application is fault-tolerant, i.e. that failure in one component doesn’t stop the world and doesn’t lead to data loss.

To sum it up, a rare event unexpectedly triggered a blue/green deployment, where a combination of flawed configuration and flawed free space calculation resulted in an unwritable cluster. Fortunately, no data is lost and at least I learned something.

The post An AWS Elasticsearch Post-Mortem appeared first on Bozho's tech blog.