Post Syndicated from Rajdeep Mukherjee original https://aws.amazon.com/blogs/devops/improving-aws-java-applications-with-amazon-codeguru-reviewer/
Amazon CodeGuru Reviewer is a machine learning (ML)-based AWS service for providing automated code reviews comments on your Java and Python applications. Powered by program analysis and ML, CodeGuru Reviewer detects hard-to-find bugs and inefficiencies in your code and leverages best practices learned from across millions of lines of open-source and Amazon code. You can start analyzing your code through pull requests and full repository analysis (for more information, see Automating code reviews and application profiling with Amazon CodeGuru).
The recommendations generated by CodeGuru Reviewer for Java fall into the following categories:
- AWS best practices
- Concurrency
- Security
- Resource leaks
- Other specialized categories such as sensitive information leaks, input validation, and code clones
- General best practices on data structures, control flow, exception handling, and more
We expect the recommendations to benefit beginners as well as expert Java programmers.
In this post, we showcase CodeGuru Reviewer recommendations related to using the AWS SDK for Java. For in-depth discussion of other specialized topics, see our posts on concurrency, security, and resource leaks. For Python applications, see Raising Python code quality using Amazon CodeGuru.
The AWS SDK for Java simplifies the use of AWS services by providing a set of features that are consistent and familiar for Java developers. The SDK has more than 250 AWS service clients, which are available on GitHub. Service clients include services like Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, Amazon Kinesis, Amazon Elastic Compute Cloud (Amazon EC2), AWS IoT, and Amazon SageMaker. These services constitute more than 6,000 operations, which you can use to access AWS services. With such rich and diverse services and APIs, developers may not always be aware of the nuances of AWS API usage. These nuances may not be important at the beginning, but become critical as the scale increases and the application evolves or becomes diverse. This is why CodeGuru Reviewer has a category of recommendations: AWS best practices. This category of recommendations enables you to become aware of certain features of AWS APIs so your code can be more correct and performant.
The first part of this post focuses on the key features of the AWS SDK for Java as well as API patterns in AWS services. The second part of this post demonstrates using CodeGuru Reviewer to improve code quality for Java applications that use the AWS SDK for Java.
AWS SDK for Java
The AWS SDK for Java supports higher-level abstractions for simplified development and provides support for cross-cutting concerns such as credential management, retries, data marshaling, and serialization. In this section, we describe a few key features that are supported in the AWS SDK for Java. Additionally, we discuss some key API patterns such as batching, and pagination, in AWS services.
The AWS SDK for Java has the following features:
- Waiters – Waiters are utility methods that make it easy to wait for a resource to transition into a desired state. Waiters makes it easier to abstract out the polling logic into a simple API call. The waiters interface provides a custom delay strategy to control the sleep time between retries, as well as a custom condition on whether polling of a resource should be retried. The AWS SDK for Java also offer an async variant of waiters.
- Exceptions – The AWS SDK for Java uses runtime (or unchecked) exceptions instead of checked exceptions in order to give you fine-grained control over the errors you want to handle and to prevent scalability issues inherent with checked exceptions in large applications. Broadly, the AWS SDK for Java has two types of exceptions:
- AmazonClientException – Indicates that a problem occurred inside the Java client code, either while trying to send a request to AWS or while trying to parse a response from AWS. For example, the AWS SDK for Java throws an AmazonClientException if no network connection is available when you try to call an operation on one of the clients.
- AmazonServiceException – Represents an error response from an AWS service. For example, if you try to end an EC2 instance that doesn’t exist, Amazon EC2 returns an error response, and all the details of that response are included in the AmazonServiceException that’s thrown. For some cases, a subclass of AmazonServiceException is thrown to allow you fine-grained control over handling error cases through catch blocks.
The API has the following patterns:
- Batching – A batch operation provides you with the ability to perform a single CRUD operation (create, read, update, delete) on multiple resources. Some typical use cases include the following:
- Pagination – Many AWS operations return paginated results when the response object is too large to return in a single response. To enable you to perform pagination, the request and response objects for many service clients in the SDK provide a continuation token (typically named NextToken) to indicate additional results.
AWS best practices
Now that we have summarized the SDK-specific features and API patterns, let’s look at the CodeGuru Reviewer recommendations on AWS API use.
The CodeGuru Reviewer recommendations for the AWS SDK for Java range from detecting outdated or deprecated APIs to warning about API misuse, missing pagination, authentication and exception scenarios, and using efficient API alternatives. In this section, we discuss a few examples patterned after real code.
Handling pagination
Over 1,000 APIs from more than 150 AWS services have pagination operations. The pagination best practice rule in CodeGuru covers all the pagination operations. In particular, the pagination rule checks if the Java application correctly fetches all the results of the pagination operation.
The response of a pagination operation in AWS SDK for Java 1.0 contains a token that has to be used to retrieve the next page of results. In the following code snippet, you make a call to listTables(), a DynamoDB ListTables operation, which can only return up to 100 table names per page. This code might not produce complete results because the operation returns paginated results instead of all results.
public void getDynamoDbTable() {
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
List<String> tables = dynamoDbClient.listTables().getTableNames();
System.out.println(tables)
}
CodeGuru Reviewer detects the missing pagination in the code snippet and makes the following recommendation to add another call to check for additional results.
You can accept the recommendation and add the logic to get the next page of table names by checking if a token (LastEvaluatedTableName in ListTablesResponse) is included in each response page. If such a token is present, it’s used in a subsequent request to fetch the next page of results. See the following code:
public void getDynamoDbTable() {
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
ListTablesRequest listTablesRequest = ListTablesRequest.builder().build();
boolean done = false;
while (!done) {
ListTablesResponse listTablesResponse = client.listTables(listTablesRequest);
System.out.println(listTablesResponse.tableNames());
if (listTablesResponse.lastEvaluatedTableName() == null) {
done = true;
}
listTablesRequest = listTablesRequest.toBuilder()
.exclusiveStartTableName(listTablesResponse.lastEvaluatedTableName())
.build();
}
}
Handling failures in batch operation calls
Batch operations are common with many AWS services that process bulk requests. Batch operations can succeed without throwing exceptions even if some items in the request fail. Therefore, a recommended practice is to explicitly check for any failures in the result of the batch APIs. Over 40 APIs from more than 20 AWS services have batch operations. The best practice rule in CodeGuru Reviewer covers all the batch operations. In the following code snippet, you make a call to sendMessageBatch, a batch operation from Amazon SQS, but it doesn’t handle any errors returned by that batch operation:
public void flush(final String sqsEndPoint,
final List<SendMessageBatchRequestEntry> batch) {
AwsSqsClientBuilder awsSqsClientBuilder;
AmazonSQS sqsClient = awsSqsClientBuilder.build();
if (batch.isEmpty()) {
return;
}
sqsClient.sendMessageBatch(sqsEndPoint, batch);
}
CodeGuru Reviewer detects this issue and makes the following recommendation to check the return value for failures.
You can accept this recommendation and add logging for the complete list of messages that failed to send, in addition to throwing an SQSUpdateException. See the following code:
public void flush(final String sqsEndPoint,
final List<SendMessageBatchRequestEntry> batch) {
AwsSqsClientBuilder awsSqsClientBuilder;
AmazonSQS sqsClient = awsSqsClientBuilder.build();
if (batch.isEmpty()) {
return;
}
SendMessageBatchResult result = sqsClient.sendMessageBatch(sqsEndPoint, batch);
final List<BatchResultErrorEntry> failed = result.getFailed();
if (!failed.isEmpty()) {
final String failedMessage = failed.stream()
.map(batchResultErrorEntry ->
String.format("…", batchResultErrorEntry.getId(),
batchResultErrorEntry.getMessage()))
.collect(Collectors.joining(","));
throw new SQSUpdateException("Error occurred while sending
messages to SQS::" + failedMessage);
}
}
Exception handling best practices
Amazon S3 is one of the most popular AWS services with our customers. A frequent operation with this service is to upload a stream-based object through an Amazon S3 client. Stream-based uploads might encounter occasional network connectivity or timeout issues, and the best practice to address such a scenario is to properly handle the corresponding ResetException error. ResetException extends SdkClientException, which subsequently extends AmazonClientException. Consider the following code snippet, which lacks such exception handling:
private void uploadInputStreamToS3(String bucketName,
InputStream input,
String key, ObjectMetadata metadata)
throws SdkClientException {
final AmazonS3Client amazonS3Client;
PutObjectRequest putObjectRequest =
new PutObjectRequest(bucketName, key, input, metadata);
amazonS3Client.putObject(putObjectRequest);
}
In this case, CodeGuru Reviewer correctly detects the missing handling of the ResetException error and suggests possible solutions.
This recommendation is rich in that it provides alternatives to suit different use cases. The most common handling uses File or FileInputStream objects, but in other cases explicit handling of mark and reset operations are necessary to reliably avoid a ResetException.
You can fix the code by explicitly setting a predefined read limit using the setReadLimit method of RequestClientOptions. Its default value is 128 KB. Setting the read limit value to one byte greater than the size of stream reliably avoids a ResetException.
For example, if the maximum expected size of a stream is 100,000 bytes, set the read limit to 100,001 (100,000 + 1) bytes. The mark and reset always work for 100,000 bytes or less. However, this might cause some streams to buffer that number of bytes into memory.
The fix reliably avoids ResetException when uploading an object of type InputStream to Amazon S3:
private void uploadInputStreamToS3(String bucketName, InputStream input,
String key, ObjectMetadata metadata)
throws SdkClientException {
final AmazonS3Client amazonS3Client;
final Integer READ_LIMIT = 10000;
PutObjectRequest putObjectRequest =
new PutObjectRequest(bucketName, key, input, metadata);
putObjectRequest.getRequestClientOptions().setReadLimit(READ_LIMIT);
amazonS3Client.putObject(putObjectRequest);
}
Replacing custom polling with waiters
A common activity when you’re working with services that are eventually consistent (such as DynamoDB) or have a lead time for creating resources (such as Amazon EC2) is to wait for a resource to transition into a desired state. The AWS SDK provides the Waiters API, a convenient and efficient feature for waiting that abstracts out the polling logic into a simple API call. If you’re not aware of this feature, you might come up with a custom, and potentially inefficient polling logic to determine whether a particular resource had transitioned into a desired state.
The following code appears to be waiting for the status of EC2 instances to change to shutting-down or terminated inside a while (true) loop:
private boolean terminateInstance(final String instanceId, final AmazonEC2 ec2Client)
throws InterruptedException {
long start = System.currentTimeMillis();
while (true) {
try {
DescribeInstanceStatusResult describeInstanceStatusResult =
ec2Client.describeInstanceStatus(new DescribeInstanceStatusRequest()
.withInstanceIds(instanceId).withIncludeAllInstances(true));
List<InstanceStatus> instanceStatusList =
describeInstanceStatusResult.getInstanceStatuses();
long finish = System.currentTimeMillis();
long timeElapsed = finish - start;
if (timeElapsed > INSTANCE_TERMINATION_TIMEOUT) {
break;
}
if (instanceStatusList.size() < 1) {
Thread.sleep(WAIT_FOR_TRANSITION_INTERVAL);
continue;
}
currentState = instanceStatusList.get(0).getInstanceState().getName();
if ("shutting-down".equals(currentState) || "terminated".equals(currentState)) {
return true;
} else {
Thread.sleep(WAIT_FOR_TRANSITION_INTERVAL);
}
} catch (AmazonServiceException ex) {
throw ex;
}
…
}
CodeGuru Reviewer detects the polling scenario and recommends you use the waiters feature to help improve efficiency of such programs.
Based on the recommendation, the following code uses the waiters function that is available in the AWS SDK for Java. The polling logic is replaced with the waiters() function, which is then run with the call to waiters.run(…), which accepts custom provided parameters, including the request and optional custom polling strategy. The run function polls synchronously until it’s determined that the resource transitioned into the desired state or not. The SDK throws a WaiterTimedOutException if the resource doesn’t transition into the desired state even after a certain number of retries. The fixed code is more efficient, simple, and abstracts the polling logic to determine whether a particular resource had transitioned into a desired state into a simple API call:
public void terminateInstance(final String instanceId, final AmazonEC2 ec2Client)
throws InterruptedException {
Waiter<DescribeInstancesRequest> waiter = ec2Client.waiters().instanceTerminated();
ec2Client.terminateInstances(new TerminateInstancesRequest().withInstanceIds(instanceId));
try {
waiter.run(new WaiterParameters()
.withRequest(new DescribeInstancesRequest()
.withInstanceIds(instanceId))
.withPollingStrategy(new PollingStrategy(new MaxAttemptsRetryStrategy(60),
new FixedDelayStrategy(5))));
} catch (WaiterTimedOutException e) {
List<InstanceStatus> instanceStatusList = ec2Client.describeInstanceStatus(
new DescribeInstanceStatusRequest()
.withInstanceIds(instanceId)
.withIncludeAllInstances(true))
.getInstanceStatuses();
String state;
if (instanceStatusList != null && instanceStatusList.size() > 0) {
state = instanceStatusList.get(0).getInstanceState().getName();
}
}
}
Service-specific best practice recommendations
In addition to the SDK operation-specific recommendations in the AWS SDK for Java we discussed, there are various AWS service-specific best practice recommendations pertaining to service APIs for services such as Amazon S3, Amazon EC2, DynamoDB, and more, where CodeGuru Reviewer can help to improve Java applications that use AWS service clients. For example, CodeGuru can detect the following:
- Resource leaks in Java applications that use high-level libraries, such as the Amazon S3 TransferManager
- Deprecated methods in various AWS services
- Missing null checks on the response of the GetItem API call in DynamoDB
- Missing error handling in the output of the PutRecords API call in Kinesis
- Anti-patterns such as binding the SNS subscribe or createTopic operation with Publish operation
Conclusion
This post introduced how to use CodeGuru Reviewer to improve the use of the AWS SDK in Java applications. CodeGuru is now available for you to try. For pricing information, see Amazon CodeGuru pricing.
