Tag Archives: xml

Security updates for Tuesday

Post Syndicated from ris original https://lwn.net/Articles/734142/rss

Security updates have been issued by Arch Linux (apache and ettercap), Debian (gdk-pixbuf and newsbeuter), Red Hat (kernel), Slackware (httpd, libgcrypt, and ruby), SUSE (kernel), and Ubuntu (bind9, kernel, libidn2-0, libxml2, linux, linux-aws, linux-gke, linux-kvm, linux-raspi2, linux-snapdragon, linux, linux-raspi2, linux-hwe, linux-lts-trusty, and linux-lts-xenial).

How to Provision Complex, On-Demand Infrastructures by Using Amazon API Gateway and AWS Lambda

Post Syndicated from Ben Eichorst original https://aws.amazon.com/blogs/compute/how-to-provision-complex-on-demand-infrastructures-by-using-amazon-api-gateway-and-aws-lambda/

Many AWS customers are using the power of AWS CloudFormation to customize complex infrastructures. At the same time, they are moving towards self-service for their expanding customer bases. How can complex infrastructure be provisioned on-demand while minimizing customer use of the AWS Management Console?

Let’s say AnyCompany uses AWS services to process sensitive datasets owned by its customers. These customers need to be able to provision their own processing infrastructure on demand. However, AnyCompany doesn’t want the customers to access AWS CloudFormation directly, or see how customer data is processed. How can AnyCompany create resources with CloudFormation templates without exposing proprietary processing methods?

You can use Amazon API Gateway and AWS Lambda to provide complex, on-demand, data-processing infrastructure to users who have no need to access the AWS Management Console. In this post, we walk through the setup and configuration of working code examples that you (and your customers) can use to provision on-demand infrastructure. This post’s solution combines principles of immutable computing with a method to provide granular access to powerful capabilities available in the AWS Cloud. In this post, you create immutability for an Amazon EC2 instance by provisioning it without an SSH key or any other access mechanism. The instance can’t be altered after it’s launched.

Solution architecture

API Gateway simplifies the creation, management, and deployment of APIs. Integration of API Gateway with Lambda, the AWS serverless compute service, allows for further interaction with the larger family of AWS services. In this post, the Lambda function provisions on-demand CloudFormation infrastructure stacks for our example service’s primary business function:  the calculation of pi.

Two CloudFormation templates are used to provision infrastructure stacks:

  • The primary template
  • The business-function template

The primary CloudFormation template creates the base infrastructure that is shown in the following diagram.

Architecture Diagram

The primary template creates a stack of resources:

  • API Gateway resources
  • The associated AWS Identity and Access Management (IAM) security roles
  • An Amazon S3 bucket for the results of the data processed by the business-function template
  • A Lambda function specifically for the purpose of creating multiple iterations of a second repeatable infrastructure. This repeatable infrastructure, contained within the second CloudFormation template, is referred to as the business-function template.

The API created by the primary template allows system users to pass selected parameters to the business-function template, such as cost-center tags, a unique name, and data-processing parameters. The ability to pass parameters allows the business-function infrastructure to adapt to the specific needs of each request. In this post’s example—calculating the first 15,000 digits of pi—the number of digits calculated can be specified in the request initiated by the customer. For your own business function templates, this could be the location of input data or an email address to which results are sent.

In the example business-function template, the template provisions an immutable EC2 instance that calculates a specified number of digits of pi and uploads the results to the S3 bucket. The business-function stack then self-terminates, reducing any potential extra costs.

This architecture allows you, at multiple points, to control access, processing parameters, and workflow.

In the earlier diagram, (1) all access to the AWS Management Console and API for your account’s infrastructure is segregated through the use of API Gateway. Then, (2) IAM role–based restrictions are in place for API Gateway to call the Lambda function. The Lambda function serves to selectively add, subtract, or alter input parameters from external users. The code for this function is embedded in the primary CloudFormation template. You can modify the code to add specific tags for billing information or call other AWS services such as Amazon DynamoDB or Amazon RDS to keep a record of which infrastructure is instantiated by which API users.

In our example template, the Lambda function is populated to pass only the required parameters for the business-function template’s calculation of pi. The use of the function restricts the ability of customers to inject unexpected or undesirable parameters. The solution then (3) uses a Lambda execution role to control the Lambda function’s access. Finally, (4) the Lambda function initiates the CloudFormation business-function template by using another IAM role that is restricted to instantiating only the resources required.

The use of an intermediate Lambda function here allows for heavy customization and filtering of the input requests from the customer-facing API Gateway.

Solution Features

By using this API Gateway–based solution, you can benefit in a number of ways:

  • User requests are decoupled from the infrastructure that fulfills those requests.

    This post’s solution removes the need for users to have access to the AWS Management Console. Users can accomplish their infrastructure-backed requests on demand without having any direct access to the console. Additionally, processing methods and infrastructure can be switched in and out without any change to the external appearance of your service. For example, in this post, I could have a containerized solution such as Amazon EC2 Container Services (Amazon ECS) process my pi-calculation service without any visibility to users.

  • Proprietary processing methods and infrastructure are obscured from users.

    The solution in this post can protect the proprietary secrets of your company’s data processing service by obscuring which processing methods you are using and which infrastructure performs the processing. The solution also isolates users from viewing each other’s operations. Without having console access, users cannot determine which running processing stacks are initiated by other users.

  • On-demand infrastructure in specific configurations is created without allowing users to provision arbitrary infrastructure.

    The solution in this post complements the functionality of the AWS Service Catalog. Beyond allowing only preapproved infrastructure and preventing unapproved resources from executing outside your company’s policies, this API Gateway–driven method delivers a specific, processed result. For ease of troubleshooting and integration with DevOps workflows, you can standardize, version, and deploy complicated CloudFormation stacks for networking, compute, storage, and big data processing.

  • Simplification of the user experience.

    This solution also simplifies the user experience. Commands that request a specific result are limited to a single, familiar REST API interface call that delivers only that result.

  • Cost savings through self-terminating infrastructure.

    The architecture of this solution is augmented through the use of self-terminating CloudFormation stacks. Stacks can spin up expensive infrastructure to perform data processing on demand, and self-terminate as soon as the task has completed. This can save you infrastructure costs associated with idle resources.

Walkthrough: Deploy the solution

This post’s example templates create the base infrastructure and a business-function process. The business-function process can instantiate a single EC2 instance to calculate a specified number of digits of pi and deposit the results in an S3 bucket. As noted previously, this example includes built-in logic to automatically self-terminate the business-function template’s CloudFormation stacks after instantiation.

The following steps walk through how to provision this solution for creating on-demand business-function CloudFormation stacks through API Gateway. You can modify the business-function template code for your specialized infrastructure needs. However, you must upload the modified template to your own S3 bucket before completing Step 1 because the business-function template location is required for the primary template’s parameters.

Both templates are designed for use in the us-west-2 region and may require minor modifications to function properly in other regions.

Costs

API Gateway costs are based on the number of API calls requested, and the single call here leads to a near-zero cost. For current pricing, see Amazon API Gateway Pricing. The associated S3 costs and Lambda costs for this post also are negligible because you are not transferring much data or spending a significant amount of time processing serverless code. The most expensive part of this post’s solution is the on-demand EC2 instance cost invoked in each business-function template instantiation. This cost is approximately $0.012 per hour for each run with the default t2.micro EC2 instance. In total, running the solution as presented in this post should cost you less than $0.10 in order to test multiple calculations of pi.

 

1. Deploy the CloudFormation template

First, deploy the primary CloudFormation template for API Gateway and other resources. This template creates IAM policies, IAM roles, API Gateway resources, a new S3 bucket, and a Lambda function.

The primary template requires you to have enabled API Gateway Usage Plans.

You can deploy the primary template and launch the primary stack in the us-west-2 region by selecting the following button. Aside from Stack name, the only required parameters are the exact location of the business-function template in S3 and a unique S3 bucket name as a location for template results.

  1. In the CloudFormation console, on the Specify Details page, type a stack name.
  2. For NewS3BucketName, type a unique name.
  3. For S3CFTLocation, leave the default value. If you adapt the primary template for your own customized business-function template, be sure to include the entire URI of the S3 location (for example, https://s3-us-west-2.amazonaws.com/mys3bucket/businessFunctionTemplate.yaml).

Create Stack Screen

  1. On the Options page, no changes are required, but you can specify additional tags if desired.
  2. On the Capabilities page, acknowledge that CloudFormation might create IAM resources, as shown in the following screenshot. To finish creating the stack, choose Create.

 

2. Create an API key for the API Gateway

Though you could modify the example primary template for public use, this example restricts access to execute API calls through the use of API keys. After you create the API key, you must associate the new API key with the usage plan created by the primary template.

  1. In the API Gateway console, choose API Keys.
  2. Choose Actions, Create API key.
  3. For Name, type a name for the key. For API Key, leave the default Auto Generate
  4. (Optional) For Description, type a value.
  5. Choose Save.
  6. Choose Add to Usage Plan and type WrappingApiUsagePlan. This usage plan defaults your Stage to PROD because it is the only configured stage.
  7. To add the key to the usage plan, choose the green check icon. Your key should resemble the following screenshot.
  8. Choose Show and note the API key value for the next step.

3. Initiate a REST API query

Now, initiate a REST API query against the API that you created. in the example curl command below, replace the placeholder API key specified in the x-api-key HTTP header with your API key.

You also need to replace the example URI path with your API’s specific URI. Create the full URI by concatenating the base URI listed in the CloudFormation output for the primary template stack with the primary template’s API call path, which is /PROD/PerRunExecute. This full URI for your specific API call should closely resemble the URI path (e.g. https://abcdefghi.execute-api.us-west-2.amazonaws.com/PROD/PerRunExecute) in the following example curl command.

curl -v -X POST -i -H "x-api-key: 012345ABCDefGHIjkLMS20tGRJ7othuyag" https://abcdefghi.execute-api.us-west-2.amazonaws.com/PROD/PerRunExecute -d '{
  "StackName":"MyCalculatePiStack",
  "CostCenter":"12345",
  "DigitsToCalculate": "15000"
}'

When you run the curl command, it executes a POST query to the REST API to instantiate the business-function template and calculate the first 15,000 digits of pi. A successful API query returns information on the curl request as well as CloudFormation API output similar to the following, indicating an “HTTPStatusCode” of 200.

{"StackId": "arn:aws:cloudformation:us-west-2:123456789012:stack/MyCalculatePiStack/1234debf-9423-11e7-9a08-50399b8d8a8c", "ResponseMetadata": {"RetryAttempts": 0, "HTTPStatusCode": 200, "RequestId": "0cbabc97-9423-11e7-97b3-395d92028ea7", "HTTPHeaders": {"x-amzn-requestid": "0cbabc97-9423-11e7-97b3-395d92028ea7", "date": "Thu, 07 Sep 2017 23:19:52 GMT", "content-length": "388", "content-type": "text/xml"}}}


4. Examine the output


After the EC2 instance completes its processing and uploads the results to the S3 bucket created by the primary template, you can examine the output. Using the configured t2.micro EC2 instance, instantiation and calculation for the first 15,000 digits of pi takes approximately 5–10 minutes. You can monitor progress in the EC2 console (where you see a new EC2 instance running) or the CloudFormation console (where you see the provisioned stack in a “created” state).


After the first 15,000 digits of pi have been uploaded to the S3 bucket, the EC2 instance self-terminates the stack. You can then download the results file from the S3 bucket, as shown below. The naming convention for pi calculating runs is “PiCalculation” concatenated with an epoch timestamp from the time of execution.




This file contains the instance ID that performed the processing, a time stamp, and the first 15,000 digits of pi. The output should look similar to the following screenshot.




 


5. Clean up


To help minimize your costs, clean up any infrastructure that resulted from following the solution in this post. Remember that you first have to detach the API key that you created from the usage plan in order to delete the primary CloudFormation stack. You can do this by clicking the X for associated usage plans for your API key, as shown in the following screenshot.




You also must empty the S3 results bucket of all files for the bucket to be deleted when the CloudFormation stack self-terminates.


This solution can be adapted to any CloudFormation template encompassing a business function for your service.


Adapt to Your Services


To adapt this example to your own business-function:


    

  1. Write a CloudFormation template that implements your new business function.
    2. 

  2. In the primary template, customize the Lambda resource object code for “PerRunCloudFormationExecutionLambda” to accept and pass on the specific parameters needed by your business function template. Save it with a new name.
    3. 

  3. Upload your custom business-function CloudFormation template developed in step 1 to an S3 bucket with permissions that allow the CloudFormation service to access the template objects. For more information about S3 access control, see Managing Access Permissions to Your Amazon S3 Resource.
    4. 

  4. Start your new stack with the modified primary CloudFormation template. Pass it, as a parameter, the name of your new business function template as a parameter. You can reuse the same API Gateway usage plan created previously, if desired.
    5. 

  5. Test, using your new stack name in place of the bolded content above, and posting an API Gateway query with parameters needed by your new business function.
    6. 



Conclusion


By using API Gateway, you can control complex, on-demand CloudFormation stacks that create AWS infrastructure. When you combine this functionality with self-terminating templates, you can realize significant cost savings. API Gateway also allows for standardization of infrastructure. It can enable your users to instantiate complex and costly architectures on demand and only for as long as your users need them.


Though the solution in this post assumes that the business function is to calculate the first 15,000 digits of pi (a less than profitable venture these days), you can adopt the solution to deliver significant cost savings. For example, instead of a single EC2 instance, the business-function template could instantiate Amazon Redshift (a petabyte-scale data warehouse), Amazon EMR, or any complex processing infrastructure. As with all on-demand AWS services, you pay for the infrastructure only when it is running.


If you have comments about any of this content, submit them in the “Comments” section below. If you have questions about implementing this solution, start a new thread on the API Gateway forum.

	

	
	








	

		
Security updates for Tuesday

		

			 		

	


		

		
Post Syndicated from corbet original https://lwn.net/Articles/731678/rss


      Security updates have been issued by Debian (extplorer and libraw), Fedora (mingw-libsoup, python-tablib, ruby, and subversion), Mageia (avidemux, clamav, nasm, php-pear-CAS, and shutter), Oracle (xmlsec1), Red Hat (openssl tomcat), Scientific Linux (authconfig, bash, curl, evince, firefox, freeradius, gdm gnome-session, ghostscript, git, glibc, gnutls, groovy, GStreamer, gtk-vnc, httpd, java-1.7.0-openjdk, kernel, libreoffice, libsoup, libtasn1, log4j, mariadb, mercurial, NetworkManager, openldap, openssh, pidgin, pki-core, postgresql, python, qemu-kvm, samba, spice, subversion, tcpdump, tigervnc fltk, tomcat, X.org, and xmlsec1), SUSE (git), and Ubuntu (augeas, cvs, and texlive-base).

	

	
	




	

		
Security updates for Monday

		

			 		

	


		

		
Post Syndicated from jake original https://lwn.net/Articles/731567/rss


      Security updates have been issued by Arch Linux (newsbeuter), Debian (augeas, curl, ioquake3, libxml2, newsbeuter, and strongswan), Fedora (bodhi, chicken, chromium, cryptlib, cups-filters, cyrus-imapd, glibc, mingw-openjpeg2, mingw-postgresql, qpdf, and torbrowser-launcher), Gentoo (bzip2, evilvte, ghostscript-gpl, Ked Password Manager, and rar), Mageia (curl, cvs, fossil, jetty, kernel, kernel-linus, kernel-tmb, libmspack, mariadb, mercurial, potrace, ruby, and taglib), Oracle (kernel), Red Hat (xmlsec1), and Ubuntu (graphite2 and strongswan).

	

	
	






	

		
Security updates for Thursday

		

			 		

	


		

		
Post Syndicated from jake original https://lwn.net/Articles/731309/rss


      Security updates have been issued by CentOS (git), Debian (firefox-esr and mariadb-10.0), Gentoo (bind and tnef), Mageia (kauth, kdelibs4, poppler, subversion, and vim), openSUSE (fossil, git, libheimdal, libxml2, minicom, nodejs4, nodejs6, openjpeg2, openldap2, potrace, subversion, and taglib), Oracle (git and kernel), Red Hat (git, groovy, httpd24-httpd, and mercurial), Scientific Linux (git), and SUSE (freeradius-server, ImageMagick, and subversion).

	

	
	




	

		
Wanted: Front End Developer

		

			 		

	


		

		
Post Syndicated from Yev original https://www.backblaze.com/blog/wanted-front-end-developer/




Want to work at a company that helps customers in over 150 countries around the world protect the memories they hold dear? Do you want to challenge yourself with a business that serves consumers, SMBs, Enterprise, and developers? If all that sounds interesting, you might be interested to know that Backblaze is looking for a Front End Developer​!


Backblaze is a 10 year old company. Providing great customer experiences is the “secret sauce” that enables us to successfully compete against some of technology’s giants. We’ll finish the year at ~$20MM ARR and are a profitable business. This is an opportunity to have your work shine at scale in one of the fastest growing verticals in tech – Cloud Storage.


You will utilize HTML, ReactJS, CSS and jQuery to develop intuitive, elegant user experiences. As a member of our Front End Dev team, you will work closely with our web development, software design, and marketing teams.


On a day to day basis, you must be able to convert image mockups to HTML or ReactJS – There’s some production work that needs to get done. But you will also be responsible for helping build out new features, rethink old processes, and enabling third party systems to empower our marketing/sales/ and support teams.


Our Front End Developer must be proficient in:


    

  • HTML, ReactJS
    • 

  • UTF-8, Java Properties, and Localized HTML (Backblaze runs in 11 languages!)
    • 

  • JavaScript, CSS, Ajax
    • 

  • jQuery, Bootstrap
    • 

  • JSON, XML
    • 

  • Understanding of cross-browser compatibility issues and ways to work around them
    • 

  • Basic SEO principles and ensuring that applications will adhere to them
    • 

  • Learning about third party marketing and sales tools through reading documentation. Our systems include Google Tag Manager, Google Analytics, Salesforce, and Hubspot
    • 



Struts, Java, JSP, Servlet and Apache Tomcat are a plus, but not required.


We’re looking for someone that is:


    

  • Passionate about building friendly, easy to use Interfaces and APIs.
    • 

  • Likes to work closely with other engineers, support, and marketing to help customers.
    • 

  • Is comfortable working independently on a mutually agreed upon prioritization queue (we don’t micromanage, we do make sure tasks are reasonably defined and scoped).
    • 

  • Diligent with quality control. Backblaze prides itself on giving our team autonomy to get work done, do the right thing for our customers, and keep a pace that is sustainable over the long run. As such, we expect everyone that checks in code that is stable. We also have a small QA team that operates as a secondary check when needed.
    • 



Backblaze Employees Have:


    

  • Good attitude and willingness to do whatever it takes to get the job done
    • 

  • Strong desire to work for a small fast, paced company
    • 

  • Desire to learn and adapt to rapidly changing technologies and work environment
    • 

  • Comfort with well behaved pets in the office
    • 



This position is located in San Mateo, California. Regular attendance in the office is expected. Backblaze is an Equal Opportunity Employer and we offer competitive salary and benefits, including our no policy vacation policy. 


If this sounds like you

Send an email to [email protected] with:


    

  1. Front End Dev​ in the subject line
    2. 

  2. Your resume attached
    3. 

  3. An overview of your relevant experience
    4. 



The post Wanted: Front End Developer appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

	

	
	




	

		
Security updates for Friday

		

			 		

	


		

		
Post Syndicated from ris original https://lwn.net/Articles/730629/rss


      Security updates have been issued by Arch Linux (firefox, flashplugin, lib32-flashplugin, libsoup, and varnish), Debian (freeradius, git, libsoup2.4, pjproject, postgresql-9.1, postgresql-9.4, postgresql-9.6, subversion, and xchat), Fedora (gsoap, irssi, knot-resolver, php-horde-horde, php-horde-Horde-Core, php-horde-Horde-Form, php-horde-Horde-Url, php-horde-kronolith, php-horde-nag, and php-horde-turba), Mageia (perl-XML-LibXML), Oracle (libsoup), Red Hat (firefox and libsoup), SUSE (kernel and libsoup), and Ubuntu (git, kernel, libsoup2.4, linux, linux-aws, linux-gke, linux-raspi2, linux-snapdragon, linux, linux-raspi2, linux-hwe, linux-lts-trusty, linux-lts-xenial, php5, php7.0, and subversion).

	

	
	






	

		
Deploying an NGINX Reverse Proxy Sidecar Container on Amazon ECS

		

			 		

	


		

		
Post Syndicated from Nathan Peck original https://aws.amazon.com/blogs/compute/nginx-reverse-proxy-sidecar-container-on-amazon-ecs/


Reverse proxies are a powerful software architecture primitive for fetching resources from a server on behalf of a client. They serve a number of purposes, from protecting servers from unwanted traffic to offloading some of the heavy lifting of HTTP traffic processing.


This post explains the benefits of a reverse proxy, and explains how to use NGINX and Amazon EC2 Container Service (Amazon ECS) to easily implement and deploy a reverse proxy for your containerized application.


Components


NGINX is a high performance HTTP server that has achieved significant adoption because of its asynchronous event driven architecture. It can serve thousands of concurrent requests with a low memory footprint. This efficiency also makes it ideal as a reverse proxy.


Amazon ECS is a highly scalable, high performance container management service that supports Docker containers. It allows you to run applications easily on a managed cluster of Amazon EC2 instances. Amazon ECS helps you get your application components running on instances according to a specified configuration. It also helps scale out these components across an entire fleet of instances.


Sidecar containers are a common software pattern that has been embraced by engineering organizations. It’s a way to keep server side architecture easier to understand by building with smaller, modular containers that each serve a simple purpose. Just like an application can be powered by multiple microservices, each microservice can also be powered by multiple containers that work together. A sidecar container is simply a way to move part of the core responsibility of a service out into a containerized module that is deployed alongside a core application container.


The following diagram shows how an NGINX reverse proxy sidecar container operates alongside an application server container:




In this architecture, Amazon ECS has deployed two copies of an application stack that is made up of an NGINX reverse proxy side container and an application container. Web traffic from the public goes to an Application Load Balancer, which then distributes the traffic to one of the NGINX reverse proxy sidecars. The NGINX reverse proxy then forwards the request to the application server and returns its response to the client via the load balancer.


Reverse proxy for security


Security is one reason for using a reverse proxy in front of an application container. Any web server that serves resources to the public can expect to receive lots of unwanted traffic every day. Some of this traffic is relatively benign scans by researchers and tools, such as Shodan or nmap:




[18/May/2017:15:10:10 +0000] "GET /YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann HTTP/1.1" 404 1389 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
[18/May/2017:18:19:51 +0000] "GET /clientaccesspolicy.xml HTTP/1.1" 404 322 - Cloud mapping experiment. Contact [email protected]


But other traffic is much more malicious. For example, here is what a web server sees while being scanned by the hacking tool ZmEu, which scans web servers trying to find PHPMyAdmin installations to exploit:




[18/May/2017:16:27:39 +0000] "GET /mysqladmin/scripts/setup.php HTTP/1.1" 404 391 - ZmEu
[18/May/2017:16:27:39 +0000] "GET /web/phpMyAdmin/scripts/setup.php HTTP/1.1" 404 394 - ZmEu
[18/May/2017:16:27:39 +0000] "GET /xampp/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /apache-default/phpmyadmin/scripts/setup.php HTTP/1.1" 404 405 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /phpMyAdmin-2.10.0.0/scripts/setup.php HTTP/1.1" 404 397 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /mysql/scripts/setup.php HTTP/1.1" 404 386 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /admin/scripts/setup.php HTTP/1.1" 404 386 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /forum/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /typo3/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:42 +0000] "GET /phpMyAdmin-2.10.0.1/scripts/setup.php HTTP/1.1" 404 399 - ZmEu
[18/May/2017:16:27:44 +0000] "GET /administrator/components/com_joommyadmin/phpmyadmin/scripts/setup.php HTTP/1.1" 404 418 - ZmEu
[18/May/2017:18:34:45 +0000] "GET /phpmyadmin/scripts/setup.php HTTP/1.1" 404 390 - ZmEu
[18/May/2017:16:27:45 +0000] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 401 - ZmEu




In addition, servers can also end up receiving unwanted web traffic that is intended for another server. In a cloud environment, an application may end up reusing an IP address that was formerly connected to another service. It’s common for misconfigured or misbehaving DNS servers to send traffic intended for a different host to an IP address now connected to your server.


It’s the responsibility of anyone running a web server to handle and reject potentially malicious traffic or unwanted traffic. Ideally, the web server can reject this traffic as early as possible, before it actually reaches the core application code. A reverse proxy is one way to provide this layer of protection for an application server. It can be configured to reject these requests before they reach the application server.


Reverse proxy for performance


Another advantage of using a reverse proxy such as NGINX is that it can be configured to offload some heavy lifting from your application container. For example, every HTTP server should support gzip. Whenever a client requests gzip encoding, the server compresses the response before sending it back to the client. This compression saves network bandwidth, which also improves speed for clients who now don’t have to wait as long for a response to fully download.


NGINX can be configured to accept a plaintext response from your application container and gzip encode it before sending it down to the client. This allows your application container to focus 100% of its CPU allotment on running business logic, while NGINX handles the encoding with its efficient gzip implementation.


An application may have security concerns that require SSL termination at the instance level instead of at the load balancer. NGINX can also be configured to terminate SSL before proxying the request to a local application container. Again, this also removes some CPU load from the application container, allowing it to focus on running business logic. It also gives you a cleaner way to patch any SSL vulnerabilities or update SSL certificates by updating the NGINX container without needing to change the application container.


NGINX configuration


Configuring NGINX for both traffic filtering and gzip encoding is shown below:




http {
  # NGINX will handle gzip compression of responses from the app server
  gzip on;
  gzip_proxied any;
  gzip_types text/plain application/json;
  gzip_min_length 1000;
 
  server {
    listen 80;
 
    # NGINX will reject anything not matching /api
    location /api {
      # Reject requests with unsupported HTTP method
      if ($request_method !~ ^(GET|POST|HEAD|OPTIONS|PUT|DELETE)$) {
        return 405;
      }
 
      # Only requests matching the whitelist expectations will
      # get sent to the application server
      proxy_pass http://app:3000;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection 'upgrade';
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_cache_bypass $http_upgrade;
    }
  }
}




The above configuration only accepts traffic that matches the expression /api and has a recognized HTTP method. If the traffic matches, it is forwarded to a local application container accessible at the local hostname app. If the client requested gzip encoding, the plaintext response from that application container is gzip-encoded.


Amazon ECS configuration


Configuring ECS to run this NGINX container as a sidecar is also simple. ECS uses a core primitive called the task definition. Each task definition can include one or more containers, which can be linked to each other:


 {
  "containerDefinitions": [
     {
       "name": "nginx",
       "image": "<NGINX reverse proxy image URL here>",
       "memory": "256",
       "cpu": "256",
       "essential": true,
       "portMappings": [
         {
           "containerPort": "80",
           "protocol": "tcp"
         }
       ],
       "links": [
         "app"
       ]
     },
     {
       "name": "app",
       "image": "<app image URL here>",
       "memory": "256",
       "cpu": "256",
       "essential": true
     }
   ],
   "networkMode": "bridge",
   "family": "application-stack"
}



This task definition causes ECS to start both an NGINX container and an application container on the same instance. Then, the NGINX container is linked to the application container. This allows the NGINX container to send traffic to the application container using the hostname app.


The NGINX container has a port mapping that exposes port 80 on a publically accessible port but the application container does not. This means that the application container is not directly addressable. The only way to send it traffic is to send traffic to the NGINX container, which filters that traffic down. It only forwards to the application container if the traffic passes the whitelisted rules.


Conclusion


Running a sidecar container such as NGINX can bring significant benefits by making it easier to provide protection for application containers. Sidecar containers also improve performance by freeing your application container from various CPU intensive tasks. Amazon ECS makes it easy to run sidecar containers, and automate their deployment across your cluster.


To see the full code for this NGINX sidecar reference, or to try it out yourself, you can check out the open source NGINX reverse proxy reference architecture on GitHub.


– Nathan
  @nathankpeck



	

	
	




	

		
LibreOffice 5.4 released with new features for Writer, Calc and Impress

		

			 		

	


		

		
Post Syndicated from ris original https://lwn.net/Articles/729161/rss


      The Document Foundation has announced LibreOffice 5.4, the last major

release of the LibreOffice 5.x family.  There are some new features in

every module and a number of incremental improvements to Microsoft Office

file compatibility.  “Thanks to the efforts of developers, the XML

description of a new document

written by LibreOffice is 50% smaller in the case of ODF (ODT), and around

90% smaller in the case of OOXML (DOCX), in comparison with the same

document generated by the leading proprietary office suite.”

	

	
	




	

		
Run Common Data Science Packages on Anaconda and Oozie with Amazon EMR

		

			 		

	


		

		
Post Syndicated from John Ohle original https://aws.amazon.com/blogs/big-data/run-common-data-science-packages-on-anaconda-and-oozie-with-amazon-emr/


In the world of data science, users must often sacrifice cluster set-up time to allow for complex usability scenarios. Amazon EMR allows data scientists to spin up complex cluster configurations easily, and to be up and running with complex queries in a matter of minutes.


Data scientists often use scheduling applications such as Oozie to run jobs overnight. However, Oozie can be difficult to configure when you are trying to use popular Python packages (such as “pandas,” “numpy,” and “statsmodels”), which are not included by default.


One such popular platform that contains these types of packages (and more) is Anaconda. This post focuses on setting up an Anaconda platform on EMR, with an intent to use its packages with Oozie. I describe how to run jobs using a popular open source scheduler like Oozie.


Walkthrough


For this post, you walk through the following tasks:


    

  • Create an EMR cluster.
    • 

  • Download Anaconda on your master node.
    • 

  • Configure Oozie.
    • 

  • Test the steps.
    • 



Create an EMR cluster


Spin up an Amazon EMR cluster using the console or the AWS CLI. Use the latest release, and include Apache Hadoop, Apache Spark, Apache Hive, and Oozie.






To create a three-node cluster in the us-east-1 region, issue an AWS CLI command such as the following. This command must be typed as one line, as shown below. It is shown here separated for readability purposes only.




aws emr create-cluster \ 
--release-label emr-5.7.0 \ 
 --name '<YOUR-CLUSTER-NAME>' \
 --applications Name=Hadoop Name=Oozie Name=Spark Name=Hive \ 
 --ec2-attributes '{"KeyName":"<YOUR-KEY-PAIR>","SubnetId":"<YOUR-SUBNET-ID>","EmrManagedSlaveSecurityGroup":"<YOUR-EMR-SLAVE-SECURITY-GROUP>","EmrManagedMasterSecurityGroup":"<YOUR-EMR-MASTER-SECURITY-GROUP>"}' \ 
 --use-default-roles \ 
 --instance-groups '[{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"<YOUR-INSTANCE-TYPE>","Name":"Master - 1"},{"InstanceCount":<YOUR-CORE-INSTANCE-COUNT>,"InstanceGroupType":"CORE","InstanceType":"<YOUR-INSTANCE-TYPE>","Name":"Core - 2"}]'





One-line version for reference:




aws emr create-cluster --release-label emr-5.7.0 --name '<YOUR-CLUSTER-NAME>' --applications Name=Hadoop Name=Oozie Name=Spark Name=Hive --ec2-attributes '{"KeyName":"<YOUR-KEY-PAIR>","SubnetId":"<YOUR-SUBNET-ID>","EmrManagedSlaveSecurityGroup":"<YOUR-EMR-SLAVE-SECURITY-GROUP>","EmrManagedMasterSecurityGroup":"<YOUR-EMR-MASTER-SECURITY-GROUP>"}' --use-default-roles --instance-groups '[{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"<YOUR-INSTANCE-TYPE>","Name":"Master - 1"},{"InstanceCount":<YOUR-CORE-INSTANCE-COUNT>,"InstanceGroupType":"CORE","InstanceType":"<YOUR-INSTANCE-TYPE>","Name":"Core - 2"}]'




Download Anaconda


SSH into your EMR master node instance and download the official Anaconda installer:




wget https://repo.continuum.io/archive/Anaconda2-4.4.0-Linux-x86_64.sh






At the time of publication, Anaconda 4.4 is the most current version available. For the download link location for the latest Python 2.7 version (Python 3.6 may encounter issues), see https://www.continuum.io/downloads.  Open the context (right-click) menu for the Python 2.7 download link, choose Copy Link Location, and use this value in the previous wget command.




This post used the Anaconda 4.4 installation. If you have a later version, it is shown in the downloaded filename:  “anaconda2-<version number>-Linux-x86_64.sh”.


Run this downloaded script and follow the on-screen installer prompts.




chmod u+x Anaconda2-4.4.0-Linux-x86_64.sh
./Anaconda2-4.4.0-Linux-x86_64.sh




For an installation directory, select somewhere with enough space on your cluster, such as “/mnt/anaconda/”.




The process should take approximately 1–2 minutes to install. When prompted if you “wish the installer to prepend the Anaconda2 install location”, select the default option of [no].


After you are done, export the PATH to include this new Anaconda installation:




export PATH=/mnt/anaconda/bin:$PATH




Zip up the Anaconda installation:




cd /mnt/anaconda/
zip -r anaconda.zip .




The zip process may take 4–5 minutes to complete.


(Optional) Upload this anaconda.zip file to your S3 bucket for easier inclusion into future EMR clusters. This removes the need to repeat the previous steps for future EMR clusters.




Configure Oozie


Next, you configure Oozie to use Pyspark and the Anaconda platform.


Get the location of your Oozie sharelibupdate folder. Issue the following command and take note of the “sharelibDirNew” value:




oozie admin -sharelibupdate




For this post, this value is “hdfs://ip-192-168-4-200.us-east-1.compute.internal:8020/user/oozie/share/lib/lib_20170616133136”.




Pass in the required Pyspark files into Oozies sharelibupdate location. The following files are required for Oozie to be able to run Pyspark commands:


    

  • pyspark.zip
    • 

  • py4j-0.10.4-src.zip
    • 



These are located on the EMR master instance in the location “/usr/lib/spark/python/lib/”, and must be put into the Oozie sharelib spark directory. This location is the value of the sharelibDirNew parameter value (shown above) with “/spark/” appended, that is, “hdfs://ip-192-168-4-200.us-east-1.compute.internal:8020/user/oozie/share/lib/lib_20170616133136/spark/”.


To do this, issue the following commands:




hdfs dfs -put /usr/lib/spark/python/lib/py4j-0.10.4-src.zip hdfs://ip-192-168-4-200.us-east-1.compute.internal:8020/user/oozie/share/lib/lib_20170616133136/spark/
hdfs dfs -put /usr/lib/spark/python/lib/pyspark.zip hdfs://ip-192-168-4-200.us-east-1.compute.internal:8020/user/oozie/share/lib/lib_20170616133136/spark/





After you’re done, Oozie can use Pyspark in its processes.


Pass the anaconda.zip file into HDFS as follows:




hdfs dfs -put /mnt/anaconda/anaconda.zip /tmp/myLocation/anaconda.zip




(Optional) Verify that it was transferred successfully with the following command:




hdfs dfs -ls /tmp/myLocation/




On your master node, execute the following command:




export PYSPARK_PYTHON=/mnt/anaconda/bin/python




Set the PYSPARK_PYTHON environment variable on the executor nodes. Put the following configurations in your “spark-opts” values in your Oozie workflow.xml file:




–conf spark.executorEnv.PYSPARK_PYTHON=./anaconda_remote/bin/python
–conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./anaconda_remote/bin/python




This is referenced from the Oozie job in the following line in your workflow.xml file, also included as part of your “spark-opts”:




--archives hdfs:///tmp/myLocation/anaconda.zip#anaconda_remote




Your Oozie workflow.xml file should now look something like the following:




<workflow-app name="spark-wf" xmlns="uri:oozie:workflow:0.5">
<start to="start_spark" />
<action name="start_spark">
    <spark xmlns="uri:oozie:spark-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <prepare>
            <delete path="/tmp/test/spark_oozie_test_out3"/>
        </prepare>
        <master>yarn-cluster</master>
        <mode>cluster</mode>
        <name>SparkJob</name>
        <class>clear</class>
        <jar>hdfs:///user/oozie/apps/myPysparkProgram.py</jar>
        <spark-opts>--queue default
            --conf spark.ui.view.acls=*
            --executor-memory 2G --num-executors 2 --executor-cores 2 --driver-memory 3g
            --conf spark.executorEnv.PYSPARK_PYTHON=./anaconda_remote/bin/python
            --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./anaconda_remote/bin/python
            --archives hdfs:///tmp/myLocation/anaconda.zip#anaconda_remote
        </spark-opts>
    </spark>
    <ok to="end"/>
    <error to="kill"/>
</action>
        <kill name="kill">
                <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
        </kill>
        <end name="end"/>
</workflow-app>




Test steps


To test this out, you can use the following job.properties and myPysparkProgram.py file, along with the following steps:


job.properties




masterNode ip-xxx-xxx-xxx-xxx.us-east-1.compute.internal
nameNode hdfs://${masterNode}:8020
jobTracker ${masterNode}:8032
master yarn
mode cluster
queueName default
oozie.libpath ${nameNode}/user/oozie/share/lib
oozie.use.system.libpath true
oozie.wf.application.path ${nameNode}/user/oozie/apps/





Note: You can get your master node IP address (denoted as “ip-xxx-xxx-xxx-xxx” here) from the value for the sharelibDirNew parameter noted earlier.


myPysparkProgram.py




from pyspark import SparkContext, SparkConf
import numpy
import sys

conf = SparkConf().setAppName('myPysparkProgram')
sc = SparkContext(conf=conf)

rdd = sc.textFile("/user/hadoop/input.txt")

x = numpy.sum([3,4,5]) #total = 12

rdd = rdd.map(lambda line: line + str(x))
rdd.saveAsTextFile("/user/hadoop/output")




Put the “myPysparkProgram.py” into the location mentioned between the “<jar>xxxxx</jar>” tags in your workflow.xml. In this example, the location is “hdfs:///user/oozie/apps/”. Use the following command to move the “myPysparkProgram.py” file to the correct location:




hdfs dfs -put myPysparkProgram.py /user/oozie/apps/




Put the above workflow.xml file into the “/user/oozie/apps/” location in hdfs:




hdfs dfs –put workflow.xml /user/oozie/apps/




Note: The job.properties file is run locally from the EMR master node.


Create a sample input.txt file with some data in it. For example:


input.txt




This is a sentence.
So is this. 
This is also a sentence.




Put this file into hdfs:




hdfs dfs -put input.txt /user/hadoop/




Execute the job in Oozie with the following command. This creates an Oozie job ID.




oozie job -config job.properties -run







You can check the Oozie job state with the command:




oozie job -info <Oozie job ID>







    

  1. When the job is successfully finished, the results are located at:
    2. 







/user/hadoop/output/part-00000
/user/hadoop/output/part-00001







    

  1. Run the following commands to view the output:
    2. 





hdfs dfs -cat /user/hadoop/output/part-00000
hdfs dfs -cat /user/hadoop/output/part-00001




The output will be:




This is a sentence. 12
So is this 12
This is also a sentence 12




Summary


The myPysparkProgram.py has successfully imported the numpy library from the Anaconda platform and has produced some output with it. If you tried to run this using standard Python, you’d encounter the following error:




Now when your Python job runs in Oozie, any imported packages that are implicitly imported by your Pyspark script are imported into your job within Oozie directly from the Anaconda platform. Simple!


If you have questions or suggestions, please leave a comment below.




Additional Reading


Learn how to use Apache Oozie workflows to automate Apache Spark jobs on Amazon EMR.




 




About the Author


John Ohle is an AWS BigData Cloud Support Engineer II for the BigData team in Dublin. He works to provide advice and solutions to our customers on their Big Data projects and workflows on AWS. In his spare time, he likes to play music, learn, develop tools and write documentation to further help others – both colleagues and customers alike.


 


 


 

	

	
	




	

		
Security updates for Wednesday

		

			 		

	


		

		
Post Syndicated from ris original https://lwn.net/Articles/728925/rss


      Security updates have been issued by Debian (bind9, icedove, openjdk-8, qemu, and rkhunter), Fedora (krb5, libmwaw, perl-XML-LibXML, qemu, subversion, and webkitgtk4), Mageia (cinnamon-settings-daemon, graphite2, gsoap, libquicktime, and wireshark), openSUSE (catdoc, gsoap, jasper, and Wireshark), and Ubuntu (linux-aws, linux-gke and ruby1.9.1, ruby2.0, ruby2.3).

	

	
	




	

		
Security updates for Monday

		

			 		

	


		

		
Post Syndicated from ris original https://lwn.net/Articles/728666/rss


      Security updates have been issued by CentOS (graphite2 and java-1.8.0-openjdk), Debian (atril, bind9, catdoc, and qemu), Fedora (glpi, GraphicsMagick, heimdal, kernel, nodejs, perl-XML-LibXML, and qt5-qtwebengine), Gentoo (adobe-flash), Mageia (c-ares, expat, flash-player-plugin, gnutls, libgcrypt, libtiff, sane, and tnef), openSUSE (evince and xorg-x11-server), Scientific Linux (graphite2), Slackware (seamonkey), and Ubuntu (heimdal and linux-lts-trusty).

	

	
	




	

		
Deploying Java Microservices on Amazon EC2 Container Service

		

			 		

	


		

		
Post Syndicated from Nathan Taber original https://aws.amazon.com/blogs/compute/deploying-java-microservices-on-amazon-ec2-container-service/


This post and accompanying code graciously contributed by:












Huy Huynh
 Sr. Solutions Architect		
Magnus Bjorkman
 Solutions Architect		






Java is a popular language used by many enterprises today. To simplify and accelerate Java application development, many companies are moving from a monolithic to microservices architecture. For some, it has become a strategic imperative. Containerization technology, such as Docker, lets enterprises build scalable, robust microservice architectures without major code rewrites.


In this post, I cover how to containerize a monolithic Java application to run on Docker. Then, I show how to deploy it on AWS using Amazon EC2 Container Service (Amazon ECS), a high-performance container management service. Finally, I show how to break the monolith into multiple services, all running in containers on Amazon ECS.




Application Architecture


For this example, I use the Spring Pet Clinic, a monolithic Java application for managing a veterinary practice. It is a simple REST API, which allows the client to manage and view Owners, Pets, Vets, and Visits.




It is a simple three-tier architecture:


    

  • Client
     You simulate this by using curl commands.
    • 

  • Web/app server
     This is the Java and Spring-based application that you run using the embedded Tomcat. As part of this post, you run this within Docker containers.
    • 

  • Database server
     This is the relational database for your application that stores information about owners, pets, vets, and visits. For this post, use MySQL RDS.
    • 





I decided to not put the database inside a container as containers were designed for applications and are transient in nature. The choice was made even easier because you have a fully managed database service available with Amazon RDS.


RDS manages the work involved in setting up a relational database, from provisioning the infrastructure capacity that you request to installing the database software. After your database is up and running, RDS automates common administrative tasks, such as performing backups and patching the software that powers your database. With optional Multi-AZ deployments, Amazon RDS also manages synchronous data replication across Availability Zones with automatic failover.


Walkthrough


You can find the code for the example covered in this post at amazon-ecs-java-microservices on GitHub.


Prerequisites


You need the following to walk through this solution:


    

  • An AWS account
    • 

  • An access key and secret key for a user in the account
    • 

  • The AWS CLI installed
    • 



Also, install the latest versions of the following:


    

  • Java
    • 

  • Maven
    • 

  • Python
    • 

  • Docker
    • 



Step 1: Move the existing Java Spring application to a container deployed using Amazon ECS


First, move the existing monolith application to a container and deploy it using Amazon ECS. This is a great first step before breaking the monolith apart because you still get some benefits before breaking apart the monolith:


    

  • An improved pipeline. The container also allows an engineering organization to create a standard pipeline for the application lifecycle.
    • 

  • No mutations to machines.
    • 



You can find the monolith example at 1_ECS_Java_Spring_PetClinic.


Container deployment overview


The following diagram is an overview of what the setup looks like for Amazon ECS and related services:




This setup consists of the following resources:


    

  • The client application that makes a request to the load balancer.
    • 

  • The load balancer that distributes requests across all available ports and instances registered in the application’s target group using round-robin.
    • 

  • The target group that is updated by Amazon ECS to always have an up-to-date list of all the service containers in the cluster. This includes the port on which they are accessible.
    • 

  • One Amazon ECS cluster that hosts the container for the application.
    • 

  • A VPC network to host the Amazon ECS cluster and associated security groups.
    • 



Each container has a single application process that is bound to port 8080 within its namespace. In reality, all the containers are exposed on a different, randomly assigned port on the host.


The architecture is containerized but still monolithic because each container has all the same features of the rest of the containers


The following is also part of the solution but not depicted in the above diagram:


    

  • One Amazon EC2 Container Registry (Amazon ECR) repository for the application.
    • 

  • A service/task definition that spins up containers on the instances of the Amazon ECS cluster.
    • 

  • A MySQL RDS instance that hosts the applications schema. The information about the MySQL RDS instance is sent in through environment variables to the containers, so that the application can connect to the MySQL RDS instance.
    • 



I have automated setup with the 1_ECS_Java_Spring_PetClinic/ecs-cluster.cf AWS CloudFormation template and a Python script.


The Python script calls the CloudFormation template for the initial setup of the VPC, Amazon ECS cluster, and RDS instance. It then extracts the outputs from the template and uses those for API calls to create Amazon ECR repositories, tasks, services, Application Load Balancer, and target groups.


Environment variables and Spring properties binding


As part of the Python script, you pass in a number of environment variables to the container as part of the task/container definition:


'environment': [
{
'name': 'SPRING_PROFILES_ACTIVE',
'value': 'mysql'
},
{
'name': 'SPRING_DATASOURCE_URL',
'value': my_sql_options['dns_name']
},
{
'name': 'SPRING_DATASOURCE_USERNAME',
'value': my_sql_options['username']
},
{
'name': 'SPRING_DATASOURCE_PASSWORD',
'value': my_sql_options['password']
}
],


The preceding environment variables work in concert with the Spring property system. The value in the variable SPRING_PROFILES_ACTIVE, makes Spring use the MySQL version of the application property file. The other environment files override the following properties in that file:


    

  • spring.datasource.url
    • 

  • spring.datasource.username
    • 

  • spring.datasource.password
    • 



Optionally, you can also encrypt sensitive values by using Amazon EC2 Systems Manager Parameter Store. Instead of handing in the password, you pass in a reference to the parameter and fetch the value as part of the container startup. For more information, see Managing Secrets for Amazon ECS Applications Using Parameter Store and IAM Roles for Tasks.


Spotify Docker Maven plugin


Use the Spotify Docker Maven plugin to create the image and push it directly to Amazon ECR. This allows you to do this as part of the regular Maven build. It also integrates the image generation as part of the overall build process. Use an explicit Dockerfile as input to the plugin.


FROM frolvlad/alpine-oraclejdk8:slim
VOLUME /tmp
ADD spring-petclinic-rest-1.7.jar app.jar
RUN sh -c 'touch /app.jar'
ENV JAVA_OPTS=""
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]


The Python script discussed earlier uses the AWS CLI to authenticate you with AWS. The script places the token in the appropriate location so that the plugin can work directly against the Amazon ECR repository.


Test setup


You can test the setup by running the Python script:
 python setup.py -m setup -r <your region>


After the script has successfully run, you can test by querying an endpoint:
 curl <your endpoint from output above>/owner


You can clean this up before going to the next section:
 python setup.py -m cleanup -r <your region>


Step 2: Converting the monolith into microservices running on Amazon ECS


The second step is to convert the monolith into microservices. For a real application, you would likely not do this as one step, but re-architect an application piece by piece. You would continue to run your monolith but it would keep getting smaller for each piece that you are breaking apart.


By migrating microservices, you would get four benefits associated with microservices:


    

  • Isolation of crashes
     If one microservice in your application is crashing, then only that part of your application goes down. The rest of your application continues to work properly.
    • 

  • Isolation of security
     When microservice best practices are followed, the result is that if an attacker compromises one service, they only gain access to the resources of that service. They can’t horizontally access other resources from other services without breaking into those services as well.
    • 

  • Independent scaling
     When features are broken out into microservices, then the amount of infrastructure and number of instances of each microservice class can be scaled up and down independently.
    • 

  • Development velocity
     In a monolith, adding a new feature can potentially impact every other feature that the monolith contains. On the other hand, a proper microservice architecture has new code for a new feature going into a new service. You can be confident that any code you write won’t impact the existing code at all, unless you explicitly write a connection between two microservices.
    • 



Find the monolith example at 2_ECS_Java_Spring_PetClinic_Microservices.
 You break apart the Spring Pet Clinic application by creating a microservice for each REST API operation, as well as creating one for the system services.


Java code changes


Comparing the project structure between the monolith and the microservices version, you can see that each service is now its own separate build.
 First, the monolith version:
 
 You can clearly see how each API operation is its own subpackage under the org.springframework.samples.petclinic package, all part of the same monolithic application.
 This changes as you break it apart in the microservices version:
 
 Now, each API operation is its own separate build, which you can build independently and deploy. You have also duplicated some code across the different microservices, such as the classes under the model subpackage. This is intentional as you don’t want to introduce artificial dependencies among the microservices and allow these to evolve differently for each microservice.


Also, make the dependencies among the API operations more loosely coupled. In the monolithic version, the components are tightly coupled and use object-based invocation.


Here is an example of this from the OwnerController operation, where the class is directly calling PetRepository to get information about pets. PetRepository is the Repository class (Spring data access layer) to the Pet table in the RDS instance for the Pet API:


@RestController
class OwnerController {

    @Inject
    private PetRepository pets;
    @Inject
    private OwnerRepository owners;
    private static final Logger logger = LoggerFactory.getLogger(OwnerController.class);

    @RequestMapping(value = "/owner/{ownerId}/getVisits", method = RequestMethod.GET)
    public ResponseEntity<List<Visit>> getOwnerVisits(@PathVariable int ownerId){
        List<Pet> petList = this.owners.findById(ownerId).getPets();
        List<Visit> visitList = new ArrayList<Visit>();
        petList.forEach(pet -> visitList.addAll(pet.getVisits()));
        return new ResponseEntity<List<Visit>>(visitList, HttpStatus.OK);
    }
}


In the microservice version, call the Pet API operation and not PetRepository directly. Decouple the components by using interprocess communication; in this case, the Rest API. This provides for fault tolerance and disposability.


@RestController
class OwnerController {

    @Value("#{environment['SERVICE_ENDPOINT'] ?: 'localhost:8080'}")
    private String serviceEndpoint;

    @Inject
    private OwnerRepository owners;
    private static final Logger logger = LoggerFactory.getLogger(OwnerController.class);

    @RequestMapping(value = "/owner/{ownerId}/getVisits", method = RequestMethod.GET)
    public ResponseEntity<List<Visit>> getOwnerVisits(@PathVariable int ownerId){
        List<Pet> petList = this.owners.findById(ownerId).getPets();
        List<Visit> visitList = new ArrayList<Visit>();
        petList.forEach(pet -> {
            logger.info(getPetVisits(pet.getId()).toString());
            visitList.addAll(getPetVisits(pet.getId()));
        });
        return new ResponseEntity<List<Visit>>(visitList, HttpStatus.OK);
    }

    private List<Visit> getPetVisits(int petId){
        List<Visit> visitList = new ArrayList<Visit>();
        RestTemplate restTemplate = new RestTemplate();
        Pet pet = restTemplate.getForObject("http://"+serviceEndpoint+"/pet/"+petId, Pet.class);
        logger.info(pet.getVisits().toString());
        return pet.getVisits();
    }
}


You now have an additional method that calls the API. You are also handing in the service endpoint that should be called, so that you can easily inject dynamic endpoints based on the current deployment.


Container deployment overview


Here is an overview of what the setup looks like for Amazon ECS and the related services:
 
 This setup consists of the following resources:


    

  • The client application that makes a request to the load balancer.
    • 

  • The Application Load Balancer that inspects the client request. Based on routing rules, it directs the request to an instance and port from the target group that matches the rule.
    • 

  • The Application Load Balancer that has a target group for each microservice. The target groups are used by the corresponding services to register available container instances. Each target group has a path, so when you call the path for a particular microservice, it is mapped to the correct target group. This allows you to use one Application Load Balancer to serve all the different microservices, accessed by the path. For example, https:///owner/* would be mapped and directed to the Owner microservice.
    • 

  • One Amazon ECS cluster that hosts the containers for each microservice of the application.
    • 

  • A VPC network to host the Amazon ECS cluster and associated security groups.
    • 



Because you are running multiple containers on the same instances, use dynamic port mapping to avoid port clashing. By using dynamic port mapping, the container is allocated an anonymous port on the host to which the container port (8080) is mapped. The anonymous port is registered with the Application Load Balancer and target group so that traffic is routed correctly.


The following is also part of the solution but not depicted in the above diagram:


    

  • One Amazon ECR repository for each microservice.
    • 

  • A service/task definition per microservice that spins up containers on the instances of the Amazon ECS cluster.
    • 

  • A MySQL RDS instance that hosts the applications schema. The information about the MySQL RDS instance is sent in through environment variables to the containers. That way, the application can connect to the MySQL RDS instance.
    • 



I have again automated setup with the 2_ECS_Java_Spring_PetClinic_Microservices/ecs-cluster.cf CloudFormation template and a Python script.


The CloudFormation template remains the same as in the previous section. In the Python script, you are now building five different Java applications, one for each microservice (also includes a system application). There is a separate Maven POM file for each one. The resulting Docker image gets pushed to its own Amazon ECR repository, and is deployed separately using its own service/task definition. This is critical to get the benefits described earlier for microservices.


Here is an example of the POM file for the Owner microservice:


<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.springframework.samples</groupId>
    <artifactId>spring-petclinic-rest</artifactId>
    <version>1.7</version>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.2.RELEASE</version>
    </parent>
    <properties>
        <!-- Generic properties -->
        <java.version>1.8</java.version>
        <docker.registry.host>${env.docker_registry_host}</docker.registry.host>
    </properties>
    <dependencies>
        <dependency>
            <groupId>javax.inject</groupId>
            <artifactId>javax.inject</artifactId>
            <version>1</version>
        </dependency>
        <!-- Spring and Spring Boot dependencies -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-rest</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-cache</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <!-- Databases - Uses HSQL by default -->
        <dependency>
            <groupId>org.hsqldb</groupId>
            <artifactId>hsqldb</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <scope>runtime</scope>
        </dependency>
        <!-- caching -->
        <dependency>
            <groupId>javax.cache</groupId>
            <artifactId>cache-api</artifactId>
        </dependency>
        <dependency>
            <groupId>org.ehcache</groupId>
            <artifactId>ehcache</artifactId>
        </dependency>
        <!-- end of webjars -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>com.spotify</groupId>
                <artifactId>docker-maven-plugin</artifactId>
                <version>0.4.13</version>
                <configuration>
                    <imageName>${env.docker_registry_host}/${project.artifactId}</imageName>
                    <dockerDirectory>src/main/docker</dockerDirectory>
                    <useConfigFile>true</useConfigFile>
                    <registryUrl>${env.docker_registry_host}</registryUrl>
                    <!--dockerHost>https://${docker.registry.host}</dockerHost-->
                    <resources>
                        <resource>
                            <targetPath>/</targetPath>
                            <directory>${project.build.directory}</directory>
                            <include>${project.build.finalName}.jar</include>
                        </resource>
                    </resources>
                    <forceTags>false</forceTags>
                    <imageTags>
                        <imageTag>${project.version}</imageTag>
                    </imageTags>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>


Test setup


You can test this by running the Python script:


python setup.py -m setup -r <your region>


After the script has successfully run, you can test by querying an endpoint:


curl <your endpoint from output above>/owner


Conclusion


Migrating a monolithic application to a containerized set of microservices can seem like a daunting task. Following the steps outlined in this post, you can begin to containerize monolithic Java apps, taking advantage of the container runtime environment, and beginning the process of re-architecting into microservices. On the whole, containerized microservices are faster to develop, easier to iterate on, and more cost effective to maintain and secure.


This post focused on the first steps of microservice migration. You can learn more about optimizing and scaling your microservices with components such as service discovery, blue/green deployment, circuit breakers, and configuration servers at http://aws.amazon.com/containers.


If you have questions or suggestions, please comment below.

	

	
	






	

		
Security updates for Monday

		

			 		

	


		

		
Post Syndicated from ris original https://lwn.net/Articles/727038/rss


      Security updates have been issued by Arch Linux (libgcrypt and systemd), Debian (apache2, icedove, libgcrypt20, libxml2, and vorbis-tools), Fedora (openvpn, systemd, xen, and zabbix), Mageia (bitlbee and libtiff), openSUSE (kdepim, messagelib, kdepim4, libxml2, and php5), Oracle (kernel), Slackware (glibc and kernel), and SUSE (python-pycrypto, unrar, and xen).

	

	
	




	

		
Kotlin and Groovy JVM Languages with AWS Lambda

		

			 		

	


		

		
Post Syndicated from Juan Villa original https://aws.amazon.com/blogs/compute/kotlin-and-groovy-jvm-languages-with-aws-lambda/



 Juan Villa – Partner Solutions Architect


 


When most people hear “Java” they think of Java the programming language. Java is a lot more than a programming language, it also implies a larger ecosystem including the Java Virtual Machine (JVM). Java, the programming language, is just one of the many languages that can be compiled to run on the JVM. Some of the most popular JVM languages, other than Java, are Clojure, Groovy, Scala, Kotlin, JRuby, and Jython (see this link for a list of more JVM languages).


Did you know that you can compile and subsequently run all these languages on AWS Lambda?


AWS Lambda supports the Java 8 runtime, but this does not mean you are limited to the Java language. The Java 8 runtime is capable of running JVM languages such as Kotlin and Groovy once they have been compiled and packaged as a “fat” JAR (a JAR file containing all necessary dependencies and classes bundled in).


In this blog post we’ll work through building AWS Lambda functions in both Kotlin and Groovy programming languages. To compile and package our projects we will use Gradle build tool.




To follow along, please clone the Git repository available at GitHub here. Also, I recommend using an Integrated Development Environment (IDE) such as JetBrain’s IntelliJ IDEA, this is the IDE I used while working on these projects.


Kotlin


Kotlin is a statically-typed JVM language designed and developed by JetBrains (one of our Amazon Partner Network Technology partners) and the open source community. Compared to Java the programming language, Kotlin has additional powerful language features such as: Data Classes, Default Arguments, Extensions, Elvis Operator, and Destructuring Declarations. This is a just a short list of Kotlin’s powerful language features. For a more thorough list of features, and how to use them, refer to the full documentation of the Kotlin language.


Let’s jump right into the code and see what an AWS Lambda function looks like in Kotlin.


package com.aws.blog.jvmlangs.kotlin

import java.io.*
import com.fasterxml.jackson.module.kotlin.*

data class HandlerInput(val who: String)
data class HandlerOutput(val message: String)

class Main {
    val mapper = jacksonObjectMapper()

    fun handler(input: InputStream, output: OutputStream): Unit {
        val inputObj = mapper.readValue<HandlerInput>(input)
        mapper.writeValue(output, HandlerOutput("Hello ${inputObj.who}"))
    }
}


The above example is a very simple Hello World application that accepts as an input a JSON object containing a key called “who” and returns a JSON object containing a key called “message” with a value of “Hello {who}”.


AWS Lambda does not support serializing JSON objects into Kotlin data classes, but don’t worry! AWS Lambda supports passing an input object as a Stream, and also supports an output Stream for returning a result (see this link for more information). Combined with the Input/Output Stream form of the handler function, we are using the Jackson library with a Kotlin extension function to support serialization and deserialization of Kotlin data class types.


To get started with this example, let’s first compile and package the Kotlin project.


git clone https://github.com/awslabs/lambda-kotlin-groovy-example
cd lambda-kotlin-groovy-example/kotlin
./gradlew shadowJar


Once packaged, a JAR file containing all necessary dependencies will be available at “build/libs/ jvmlangs-kotlin-1.0-SNAPSHOT-all.jar”. Now let’s deploy this package to AWS Lambda.


To deploy the lambda function, we will be using the AWS Command Line Interface (CLI). You can find information on how to set up the AWS CLI here. This tool allows you to set up and manage AWS services via the command line.


aws lambda create-function --region us-east-1 --function-name kotlin-hello \
--zip-file fileb://build/libs/jvmlangs-kotlin-1.0-SNAPSHOT-all.jar \
--role arn:aws:iam::<account_id>:role/lambda_basic_execution \
--handler com.aws.blog.jvmlangs.kotlin.Main::handler --runtime java8 \
--timeout 15 --memory-size 128


Once deployed, we can test the function by invoking the lambda function from the CLI.


aws lambda invoke --function-name kotlin-hello --payload '{"who": "AWS Fan"}' output.txt
cat output.txt


If successful, you’ll see an output of “{"message":"Hello AWS Fan"}”.


Groovy


Groovy is an optionally typed JVM language with both dynamic and static typing capabilities. Groovy is currently being supported by the Apache Software Foundation. Like Kotlin, Groovy also packs a lot of powerful features such as: Closures, Dynamic Typing, Collection Literals, String Interpolation, and Elvis Operator. This is just a short list, see the full documentation for a list of features and how to use them.


Once again, let’s jump right into the code.


package com.aws.blog.jvmlangs.groovy

class HandlerInput {
    String who
}
class HandlerOutput {
    String message
}

class Main {
    def handler(HandlerInput input) {
        return new HandlerOutput(message: "Hello ${input.who}")
    }
}


Just like the Kotlin example, we have defined a function that takes a simple JSON object containing a “who” key value and build a response containing a “message” key. Note that in this case we are not using the Input/Output Stream form of the handler function, but rather we are letting AWS Lambda serialize the input JSON object into the type HandlerInput. To accomplish this, AWS Lambda uses the Jackson library and handles the serialization for us.


Let’s go ahead and compile and package this Groovy example.


git clone https://github.com/awslabs/lambda-kotlin-groovy-example
cd lambda-kotlin-groovy-example/groovy
./gradlew shadowJar


Once packaged, a JAR file containing all necessary dependencies will be available at “build/libs/ jvmlangs-groovy-1.0-SNAPSHOT-all.jar”. Now let’s deploy this package to AWS Lambda.


aws lambda create-function --region us-east-1 --function-name groovy-hello \
--zip-file fileb://build/libs/jvmlangs-groovy-1.0-SNAPSHOT-all.jar \
--role arn:aws:iam::<account_id>:role/lambda_basic_execution \
--handler com.aws.blog.jvmlangs.groovy.Main::handler --runtime java8 \
--timeout 15 --memory-size 128


Once deployed, we can test the function by invoking the lambda function from the CLI.


aws lambda invoke --function-name groovy-hello --payload '{"who": "AWS Fan"}' output.txt
cat output.txt


If successful, you’ll see an output of “{"message":"Hello AWS Fan"}”.


Gradle Build Tool


Finally, let’s touch up on how we built the JAR package from the Kotlin and Groovy sources above. To build the JARs we used the Gradle build tool. Gradle builds a project by reading instructions from a file called “build.gradle”. This is a file written in Gradle’s Groovy Domain Specific Langauge (DSL). You can find more information on the gradle build file by looking at their documentation. Let’s take a look at the Gradle build files we used for this post.


For the Kotlin example, this is the build file we used.


buildscript {
    repositories {
        mavenCentral()
        jcenter()
    }
    dependencies {
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
        classpath "com.github.jengelman.gradle.plugins:shadow:1.2.3"
    }
}

group 'com.aws.blog.jvmlangs.kotlin'
version '1.0-SNAPSHOT'

apply plugin: 'kotlin'
apply plugin: 'com.github.johnrengelman.shadow'

repositories {
    mavenCentral()
}

dependencies {
    compile "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
    compile "com.fasterxml.jackson.module:jackson-module-kotlin:2.8.2"
}


For the Groovy example this is the build file we used.


buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.3'
    }
}

group 'com.aws.blog.jvmlangs.groovy'
version '1.0-SNAPSHOT'

apply plugin: 'groovy'
apply plugin: 'com.github.johnrengelman.shadow'

repositories {
    mavenCentral()
}

dependencies {
    compile 'org.codehaus.groovy:groovy-all:2.3.11'
    testCompile group: 'junit', name: 'junit', version: '4.11'
}


As you can see, the build files for both Kotlin and Groovy files are very similar. For the Kotlin project we define a dependency on the Jackson Kotlin module. Also, for each respective language we include the language supporting libraries (kotlin-stdlib and groovy-all respectively).


In addition, you will notice that we are using a plugin called “shadow”. We use this plugin to package all the project dependencies into one JAR by using the Gradle task “shadowJar”. You can find more information on Shadow in their documentation.


Final Words


Don’t stop here though! Take a look at other JVM languages and get them running on AWS Lambda with the Java 8 runtime. Maybe start with Clojure? or Scala?


Also take a look AWS Lambda Java libraries provided by AWS. They provide interfaces and models to make handling events from event sources easier to handle.