Tag Archives: Sync

GNOME 3.26 released

Post Syndicated from ris original https://lwn.net/Articles/733587/rss

The GNOME Project has announced the release of GNOME 3.26 “Manchester”.
This release brings refinements to the system search, animations for
maximizing and unmaximizing windows and support for color Emoji.

Improvements to core GNOME applications include a redesigned Settings
application, a new display settings panel, Firefox sync in the Web browser,
and many more.” There are openSUSE nightly live images that include
GNOME 3.26.

Strategies for Backing Up Windows Computers

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/strategies-for-backing-up-windows-computers/

Windows 7, Windows 8, Windows 10 logos

There’s a little company called Apple making big announcements this week, but about 45% of you are on Windows machines, so we thought it would be a good idea to devote a blog post today to Windows users and the options they have for backing up Windows computers.

We’ll be talking about the various options for backing up Windows desktop OS’s 7, 8, and 10, and Windows servers. We’ve written previously about this topic in How to Back Up Windows, and Computer Backup Options, but we’ll be covering some new topics and ways to combine strategies in this post. So, if you’re a Windows user looking for shelter from all the Apple hoopla, welcome to our Apple Announcement Day Windows Backup Day post.

Windows laptop

First, Let’s Talk About What We Mean by Backup

This might seem to our readers like an unneeded appetizer on the way to the main course of our post, but we at Backblaze know that people often mean very different things when they use backup and related terms. Let’s start by defining what we mean when we say backup, cloud storage, sync, and archive.

Backup
A backup is an active copy of the system or files that you are using. It is distinguished from an archive, which is the storing of data that is no longer in active use. Backups fall into two main categories: file and image. File backup software will back up whichever files you designate by either letting you include files you wish backed up or by excluding files you don’t want backed up, or both. An image backup, sometimes called a disaster recovery backup or a system clone, is useful if you need to recreate your system on a new drive or computer.
The first backup generally will be a full backup of all files. After that, the backup will be incremental, meaning that only files that have been changed since the full backup will be added. Often, the software will keep changed versions of the files for some period of time, so you can maintain a number of previous revisions of your files in case you wish to return to something in an earlier version of your file.
The destination for your backup could be another drive on your computer, an attached drive, a network-attached drive (NAS), or the cloud.
Cloud Storage
Cloud storage vendors supply data storage just as a utility company supplies power, gas, or water. Cloud storage can be used for data backups, but it can also be used for data archives, application data, records, or libraries of photos, videos, and other media.
You contract with the service for storing any type of data, and the storage location is available to you via the internet. Cloud storage providers generally charge by some combination of data ingress, egress, and the amount of data stored.
Sync
File sync is useful for files that you wish to have access to from different places or computers, or for files that you wish to share with others. While sync has its uses, it has limitations for keeping files safe and how much it could cost you to store large amounts of data. As opposed to backup, which keeps revision of files, sync is designed to keep two or more locations exactly the same. Sync costs are based on how much data you sync and can get expensive for large amounts of data.
Archive
A data archive is for data that is no longer in active use but needs to be saved, and may or may not ever be retrieved again. In old-style storage parlance, it is called cold storage. An archive could be stored with a cloud storage provider, or put on a hard drive or flash drive that you disconnect and put in the closet, or mail to your brother in Idaho.

What’s the Best Strategy for Backing Up?

Now that we’ve got our terminology clear, let’s talk backup strategies for Windows.

At Backblaze, we advocate the 3-2-1 strategy for safeguarding your data, which means that you should maintain three copies of any valuable data — two copies stored locally and one stored remotely. I follow this strategy at home by working on the active data on my Windows 10 desktop computer (copy one), which is backed up to a Drobo RAID device attached via USB (copy two), and backing up the desktop to Backblaze’s Personal Backup in the cloud (copy three). I also keep an image of my primary disk on a separate drive and frequently update it using Windows 10’s image tool.

I use Dropbox for sharing specific files I am working on that I might wish to have access to when I am traveling or on another computer. Once my subscription with Dropbox expires, I’ll use the latest release of Backblaze that has individual file preview with sharing built-in.

Before you decide which backup strategy will work best for your situation, you’ll need to ask yourself a number of questions. These questions include where you wish to store your backups, whether you wish to supply your own storage media, whether the backups will be manual or automatic, and whether limited or unlimited data storage will work best for you.

Strategy 1 — Back Up to a Local or Attached Drive

The first copy of the data you are working on is often on your desktop or laptop. You can create a second copy of your data on another drive or directory on your computer, or copy the data to a drive directly attached to your computer, such as via USB.

external hard drive and RAID NAS devices

Windows has built-in tools for both file and image level backup. Depending on which version of Windows you use, these tools are called Backup and Restore, File History, or Image. These tools enable you to set a schedule for automatic backups, which ensures that it is done regularly. You also have the choice to use Windows Explorer (aka File Explorer) to manually copy files to another location. Some external disk drives and USB Flash Drives come with their own backup software, and other backup utilities are available for free or for purchase.

Windows Explorer File History screenshot

This is a supply-your-own media solution, meaning that you need to have a hard disk or other medium available of sufficient size to hold all your backup data. When a disk becomes full, you’ll need to add a disk or swap out the full disk to continue your backups.

We’ve written previously on this strategy at Should I use an external drive for backup?

Strategy 2 — Back Up to a Local Area Network (LAN)

Computers, servers, and network-attached-storage (NAS) on your local network all can be used for backing up data. Microsoft’s built-in backup tools can be used for this job, as can any utility that supports network protocols such as NFS or SMB/CIFS, which are common protocols that allow shared access to files on a network for Windows and other operatings systems. There are many third-party applications available as well that provide extensive options for managing and scheduling backups and restoring data when needed.

NAS cloud

Multiple computers can be backed up to a single network-shared computer, server, or NAS, which also could then be backed up to the cloud, which rounds out a nice backup strategy, because it covers both local and remote copies of your data. System images of multiple computers on the LAN can be included in these backups if desired.

Again, you are managing the backup media on the local network, so you’ll need to be sure you have sufficient room on the destination drives to store all your backup data.

Strategy 3 — Back Up to Detached Drive at Another Location

You may have have read our recent blog post, Getting Data Archives Out of Your Closet, in which we discuss the practice of filling hard drives and storing them in a closet. Of course, to satisfy the off-site backup guideline, these drives would need to be stored in a closet that’s in a different geographical location than your main computer. If you’re willing to do all the work of copying the data to drives and transporting them to another location, this is a viable option.

stack of hard drives

The only limitation to the amount of backup data is the number of hard drives you are willing to purchase — and maybe the size of your closet.

Strategy 4 — Back Up to the Cloud

Backing up to the cloud has become a popular option for a number of reasons. Internet speeds have made moving large amounts of data possible, and not having to worry about supplying the storage media simplifies choices for users. Additionally, cloud vendors implement features such as data protection, deduplication, and encryption as part of their services that make cloud storage reliable, secure, and efficient. Unlimited cloud storage for data from a single computer is a popular option.

A backup vendor likely will provide a software client that runs on your computer and backs up your data to the cloud in the background while you’re doing other things, such as Backblaze Personal Backup, which has clients for Windows computers, Macintosh computers, and mobile apps for both iOS and Android. For restores, Backblaze users can download one or all of their files for free from anywhere in the world. Optionally, a 128 GB flash drive or 4 TB drive can be overnighted to the customer, with a refund available if the drive is returned.

Storage Pod in the cloud

Backblaze B2 Cloud Storage is an option for those who need capabilities beyond Backblaze’s Personal Backup. B2 provides cloud storage that is priced based on the amount of data the customer uses, and is suitable for long-term data storage. B2 supports integrations with NAS devices, as well as Windows, Macintosh, and Linux computers and servers.

Services such as BackBlaze B2 are often called Cloud Object Storage or IaaS (Infrastructure as a Service), because they provide a complete solution for storing all types of data in partnership with vendors who integrate various solutions for working with B2. B2 has its own API (Application Programming Interface) and CLI (Command-line Interface) to work with B2, but B2 becomes even more powerful when paired with any one of a number of other solutions for data storage and management provided by third parties who offer both hardware and software solutions.

Backing Up Windows Servers

Windows Servers are popular workstations for some users, and provide needed network services for others. They also can be used to store backups from other computers on the network. They, in turn, can be backed up to attached drives or the cloud. While our Personal Backup client doesn’t support Windows servers, our B2 Cloud Storage has a number of integrations with vendors who supply software or hardware for storing data both locally and on B2. We’ve written a number of blog posts and articles that address these solutions, including How to Back Up your Windows Server with B2 and CloudBerry.

Sometimes the Best Strategy is to Mix and Match

The great thing about computers, software, and networks is that there is an endless number of ways to combine them. Our users and hardware and software partners are ingenious in configuring solutions that save data locally, copy it to an attached or network drive, and then store it to the cloud.

image of cloud backup

Among our B2 partners, Synology, CloudBerry Archiware, QNAP, Morro Data, and GoodSync have integrations that allow their NAS devices to store and retrieve data to and from B2 Cloud Storage. For a drag-and-drop experience on the desktop, take a look at CyberDuck, MountainDuck, and Dropshare, which provide users with an easy and interactive way to store and use data in B2.

If you’d like to explore more options for combining software, hardware, and cloud solutions, we invite you to browse the integrations for our many B2 partners.

Have Questions?

Windows versions, tools, and backup terminology all can be confusing, and we know how hard it can be to make sense of all of it. If there’s something we haven’t addressed here, or if you have a question or contribution, please let us know in the comments.

And happy Windows Backup Day! (Just don’t tell Apple.)

The post Strategies for Backing Up Windows Computers appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Parallel Processing in Python with AWS Lambda

Post Syndicated from Oz Akan original https://aws.amazon.com/blogs/compute/parallel-processing-in-python-with-aws-lambda/

If you develop an AWS Lambda function with Node.js, you can call multiple web services without waiting for a response due to its asynchronous nature.  All requests are initiated almost in parallel, so you can get results much faster than a series of sequential calls to each web service. Considering the maximum execution duration for Lambda, it is beneficial for I/O bound tasks to run in parallel.

If you develop a Lambda function with Python, parallelism doesn’t come by default. Lambda supports Python 2.7 and Python 3.6, both of which have multiprocessing and threading modules. The multiprocessing module supports multiple cores so it is a better choice, especially for CPU intensive workloads. With the threading module, all threads are going to run on a single core though performance difference is negligible for network-bound tasks.

In this post, I demonstrate how the Python multiprocessing module can be used within a Lambda function to run multiple I/O bound tasks in parallel.

Example use case

In this example, you call Amazon EC2 and Amazon EBS API operations to find the total EBS volume size for all your EC2 instances in a region.

This is a two-step process:

  • The Lambda function calls EC2 to list all EC2 instances
  • The function calls EBS for each instance to find attached EBS volumes

Sequential Execution

If you make these calls sequentially, during the second step, your code has to loop over all the instances and wait for each response before moving to the next request.

The class named VolumesSequential has the following methods:

  • __init__ creates an EC2 resource.
  • total_size returns all EC2 instances and passes these to the instance_volumes method.
  • instance_volumes finds the total size of EBS volumes for the instance.
  • total_size adds all sizes from all instances to find total size for the EBS volumes.

Source Code for Sequential Execution

import time
import boto3

class VolumesSequential(object):
    """Finds total volume size for all EC2 instances"""
    def __init__(self):
        self.ec2 = boto3.resource('ec2')

    def instance_volumes(self, instance):
        """
        Finds total size of the EBS volumes attached
        to an EC2 instance
        """
        instance_total = 0
        for volume in instance.volumes.all():
            instance_total += volume.size
        return instance_total

    def total_size(self):
        """
        Lists all EC2 instances in the default region
        and sums result of instance_volumes
        """
        print "Running sequentially"
        instances = self.ec2.instances.all()
        instances_total = 0
        for instance in instances:
            instances_total += self.instance_volumes(instance)
        return instances_total

def lambda_handler(event, context):
    volumes = VolumesSequential()
    _start = time.time()
    total = volumes.total_size()
    print "Total volume size: %s GB" % total
    print "Sequential execution time: %s seconds" % (time.time() - _start)

Parallel Execution

The multiprocessing module that comes with Python 2.7 lets you run multiple processes in parallel. Due to the Lambda execution environment not having /dev/shm (shared memory for processes) support, you can’t use multiprocessing.Queue or multiprocessing.Pool.

If you try to use multiprocessing.Queue, you get an error similar to the following:

[Errno 38] Function not implemented: OSError
…
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented

On the other hand, you can use multiprocessing.Pipe instead of multiprocessing.Queue to accomplish what you need without getting any errors during the execution of the Lambda function.

The class named VolumeParallel has the following methods:

  • __init__ creates an EC2 resource
  • instance_volumes finds the total size of EBS volumes attached to an instance
  • total_size finds all instances and runs instance_volumes for each to find the total size of all EBS volumes attached to all EC2 instances.

Source Code for Parallel Execution

import time
from multiprocessing import Process, Pipe
import boto3

class VolumesParallel(object):
    """Finds total volume size for all EC2 instances"""
    def __init__(self):
        self.ec2 = boto3.resource('ec2')

    def instance_volumes(self, instance, conn):
        """
        Finds total size of the EBS volumes attached
        to an EC2 instance
        """
        instance_total = 0
        for volume in instance.volumes.all():
            instance_total += volume.size
        conn.send([instance_total])
        conn.close()

    def total_size(self):
        """
        Lists all EC2 instances in the default region
        and sums result of instance_volumes
        """
        print "Running in parallel"

        # get all EC2 instances
        instances = self.ec2.instances.all()
        
        # create a list to keep all processes
        processes = []

        # create a list to keep connections
        parent_connections = []
        
        # create a process per instance
        for instance in instances:            
            # create a pipe for communication
            parent_conn, child_conn = Pipe()
            parent_connections.append(parent_conn)

            # create the process, pass instance and connection
            process = Process(target=self.instance_volumes, args=(instance, child_conn,))
            processes.append(process)

        # start all processes
        for process in processes:
            process.start()

        # make sure that all processes have finished
        for process in processes:
            process.join()

        instances_total = 0
        for parent_connection in parent_connections:
            instances_total += parent_connection.recv()[0]

        return instances_total


def lambda_handler(event, context):
    volumes = VolumesParallel()
    _start = time.time()
    total = volumes.total_size()
    print "Total volume size: %s GB" % total
    print "Sequential execution time: %s seconds" % (time.time() - _start)

Performance

There are a few differences between two Lambda functions when it comes to the execution environment. The parallel function requires more memory than the sequential one. You may run the parallel Lambda function with a relatively large memory setting to see how much memory it uses. The amount of memory required by the Lambda function depends on what the function does and how many processes it runs in parallel. To restrict maximum memory usage, you may want to limit the number of parallel executions.

In this case, when you give 1024 MB for both Lambda functions, the parallel function runs about two times faster than the sequential function. I have a handful of EC2 instances and EBS volumes in my account so the test ran way under the maximum execution limit for Lambda. Remember that parallel execution doesn’t guarantee that the runtime for the Lambda function will be under the maximum allowed duration but does speed up the overall execution time.

Sequential Run Time Output

START RequestId: 4c370b12-f9d3-11e6-b46b-b5d41afd648e Version: $LATEST
Running sequentially
Total volume size: 589 GB
Sequential execution time: 3.80066084862 seconds
END RequestId: 4c370b12-f9d3-11e6-b46b-b5d41afd648e
REPORT RequestId: 4c370b12-f9d3-11e6-b46b-b5d41afd648e Duration: 4091.59 ms Billed Duration: 4100 ms  Memory Size: 1024 MB Max Memory Used: 46 MB

Parallel Run Time Output

START RequestId: 4f1328ed-f9d3-11e6-8cd1-c7381c5c078d Version: $LATEST
Running in parallel
Total volume size: 589 GB
Sequential execution time: 1.89170885086 seconds
END RequestId: 4f1328ed-f9d3-11e6-8cd1-c7381c5c078d
REPORT RequestId: 4f1328ed-f9d3-11e6-8cd1-c7381c5c078d Duration: 2069.33 ms Billed Duration: 2100 ms  Memory Size: 1024 MB Max Memory Used: 181 MB 

Summary

In this post, I demonstrated how to run multiple I/O bound tasks in parallel by developing a Lambda function with the Python multiprocessing module. With the help of this module, you freed the CPU from waiting for I/O and fired up several tasks to fit more I/O bound operations into a given time frame. This might be the trick to reduce the overall runtime of a Lambda function especially when you have to run so many and don’t want to split the work into smaller chunks.

The 4.13 kernel is out

Post Syndicated from corbet original https://lwn.net/Articles/732793/rss

Linus has released the 4.13 kernel, right on schedule.
Headline features in this release include
kernel hardening via structure layout
randomization
,
native TLS protocol support,
better huge-page swapping,
improved handling of writeback errors,
better asynchronous I/O support,
better power management via next-interrupt
prediction
,
the elimination of the DocBook toolchain for formatted documentation,
and more. There is one other change that is called out explicitly in the
announcement: “The change in question is simply changing the default cifs behavior:
instead of defaulting to SMB 1.0 (which you really should not use:
just google for ‘stop using SMB1’ or similar), the default cifs mount
now defaults to a rather more modern SMB 3.0.

Deadline 10 – Launch a Rendering Fleet in AWS

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/deadline-10-launch-a-rendering-fleet-in-aws/

Graphical rendering is a compute-intensive task that is, as they say, embarrassingly parallel. Looked at another way, this means that there’s a more or less linear relationship between the number of processors that are working on the problem and the overall wall-clock time that it takes to complete the task. In a creative endeavor such as movie-making, getting the results faster spurs creativity, improves the feedback loop, gives you time to make more iterations and trials, and leads to a better result. Even if you have a render farm in-house, you may still want to turn to the cloud in order to gain access to more compute power at peak times. Once you do this, the next challenge is to manage the combination of in-house resources, cloud resources, and the digital assets in a unified fashion.

Deadline 10
Earlier this week we launched Deadline 10, a powerful render management system. Building on technology that we brought on board with the acquisition of Thinkbox Software, Deadline 10 is designed to extend existing on-premises rendering into the AWS Cloud, giving you elasticity and flexibility while remaining simple and easy to use. You can set up and manage large-scale distributed jobs that span multiple AWS regions and benefit from elastic, usage-based AWS licensing for popular applications like Deadline for Autodesk 3ds Max, Maya, Arnold, and dozens more, all available from the Thinkbox Marketplace. You can purchase software licenses from the marketplace, use your existing licenses, or use them together.

Deadline 10 obtains cloud-based compute resources by managing bids for EC2 Spot Instances, providing you with access to enough low-cost compute capacity to let your imagination run wild! It uses your existing AWS account, tags EC2 instances for tracking, and synchronizes your local assets to the cloud before rendering begins.

A Quick Tour
Let’s take a quick tour of Deadline 10 and see how it makes use of AWS. The AWS Portal is available from the View menu:

The first step is to log in to my AWS account:

Then I configure the connection server, license server, and the S3 bucket that will be used to store rendering assets:

Next, I set up my Spot fleet, establishing a maximum price per hour for each EC2 instance, setting target capacity, and choosing the desired rendering application:

I can also choose any desired combination of EC2 instance types:

When I am ready to render I click on Start Spot Fleet:

This will initiate the process of bidding for and managing Spot Instances. The running instances are visible from the Portal:

I can monitor the progress of my rendering pipeline:

I can stop my Spot fleet when I no longer need it:

Deadline 10 is now available for usage based license customers; a new license is needed for traditional floating license users. Pricing for yearly Deadline licenses has been reduced to $48 annually. If you are already using an earlier version of Deadline, feel free to contact us to learn more about licensing options.

Jeff;

Philips Wants to Stop Piracy With ‘Ambilight’ Technology

Post Syndicated from Ernesto original https://torrentfreak.com/philips-wants-to-stop-piracy-with-ambilight-technology-170729/

The movie industry sees illegally recorded movies as one of the biggest piracy threats and goes to extremes to stop it.

Over the years, movie theaters have used audio and video watermarking tools to detect pirates, for example. And during prominent events, night-vision goggles and other spy tech is used to monitor movie goers.

Most anti-piracy efforts are not particularly pleasing for the public, but a new technology from Philips hopes to change this. In a recent patent application the technology company proposes an ambient lighting system for movie theaters that will hinder pirate recordings.

“The major motion picture studios are losing large amounts of money due to piracy. The worldwide motion picture industry, distributors, theaters, video stores and pay-per-view operators, lose each year billions of dollars as a result of piracy,” the company begins.

Philips notes that many of the current security measures, such as watermarking, make piracy traceable. However, they don’t prevent actual recording and distribution. Watermarks can be simply rendered useless if pirates remove them afterwards, for example.

“The problem with current security measures in digital cinema is that they do not prevent pirates from filming the content from the cinema screen. Watermarks can help to identify the source of pirate content, but it does not prevent the actual copying and spreading of the content.”

The ambient lighting solution is different and Philips believes that it could reduce piracy substantially.

The concept is quite easy. By shining light on the visitors from the side, at a certain frequency, the pirate recordings will be severely degraded up to a point where they’ll become completely unwatchable.

“The solution proposed here is to shine visible light from an (additional) light source with a pre-determined modulation frequency at the cinema screen while the movie plays,” the patent application reads.

“When the frequency is chosen well, a camera will not be able to properly film the content, because the light source runs out of sync with the camera resulting in stripes running through the movie content.”

The idea is partly inspired by Philips existing Ambilight technology for TVs. This generates lighting effects around the TV that correspond to the video content. The cinema setup will do the same while hindering pirates in the process. A win-win situation, according to Philips.

“It is yet a further aspect to provide a more immersive experience when watching video content on a display device in combination with anti-piracy measures,” the patent application reads.

It’s unclear whether there are concrete plans to roll out the technology at movie theaters. Philips, however, is no stranger to these type of anti-piracy solutions. The company previously rolled out CineFence, a forensic marking technology for both image and sound, targeted at digital cinema content.

In any case, it’s a good marketing message to get movie theaters to try ambient lighting. Whether it’s really effective has yet to be seen though.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

How to Migrate All of Your Data from CrashPlan

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/how-to-migrate-your-data-from-crashplan/

Migrating from Crashplan

With CrashPlan deciding to leave the consumer backup space, ex-customers are faced with having to migrate their data to a new cloud backup service. Uploading your data from your computer to a new service is onerous enough, but one thing that seems to be getting overlooked is the potential for the files that reside in CrashPlan Central, but not on your computer, to be lost during the migration to a new provider. Here’s an overview of the migration process to make sure you don’t lose data you wish to keep.

Why would you lose files?

By default CrashPlan for Home does not delete files from CrashPlan Central (their cloud storage servers) after they are uploaded from your computer. Unless you changed your CrashPlan “Frequency and versions” settings, all of the files you uploaded are still there. This includes all the files you deleted from your computer. For example, you may have a folder of old videos that you uploaded to CrashPlan and then deleted from your computer because of space concerns. This folder of old video files is still in your CrashPlan archive. It is very likely you have files stored in CrashPlan Central that are not on your computer. Such files are now in migration limbo, and we’ll get to those files in a minute, but first…

Get Started Now

CrashPlan was kind enough to make sure that everyone will have at least 60 days from August 22nd, 2017 to transfer their data. Most people will have more time, but everyone must be migrated by the end of October 2018.

Regardless, it’s better to get started now as it can take some time to upload your data to another backup provider. The first step in migrating your files is to choose a new cloud backup provider. Let’s assume you choose Backblaze Personal Backup.

Crashplan Migration Steps

The first step is to migrate all the data that is currently on your computer to Backblaze. Once you install Backblaze on your computer, it will automatically scan your system to locate the data to upload to Backblaze. The upload will continue automatically. You can speed up or slow down how quickly Backblaze will upload files by adjusting your performance settings for your Mac or for your Windows PC. In addition, any changes and new files are automatically uploaded as well. Backblaze keeps up to 30 days’ worth of file versions and always keeps the most recent version of every data file currently on your computer.

Question — Should you remove CrashPlan from your computer before migrating to Backblaze?
Answer — No.

If your computer fails during the upload to Backblaze, you’ll still have a full backup with CrashPlan. During the upload period you may want to decrease the resources (CPU and Network) used by CrashPlan and increase the resources available to Backblaze. You can “pause” CrashPlan for up to 24 hours, but that is a manual operation and may not be practical. In any case, you’ll also need to have CrashPlan around to recover those files in migration limbo.

Saving the Files in Migration Limbo

Let’s divide this process into two major parts: recovering the files and getting them stored somewhere else.

    Recovering Files in Limbo

    1. Choose a recovery device — Right now you don’t know how many files you will need to recover, but once you know that information, you’ll need a device to hold them. We recommend that you use an external USB hard drive as your recovery device. If you believe you will only have a small number of limbo files, then a thumb drive will work.
    2. Locate the Limbo files — Open the CrashPlan App on your computer and select the “Restore” menu item on the left. As an example, you can navigate to a given folder and see the files in that folder as shown below:

    Restore files from Crashplan

    1. Click on the “Show deleted files” box as shown below to display all the files, including those that are deleted. As an example, the same files listed above are shown below, and the list now includes the deleted file IMG_6533.JPG.

    Finding deleted file in Crasphlan Central

    1. Deleted files can be visually identified via the different icon and the text shown grayed out. Navigate through your folder/directory structure and select the files you wish to recover. Yes, this can take a while. You only need to click on the deleted files as the other files are currently still on your computer and being backed up directly to Backblaze.
    2. Make sure you change the restore location. By default this is set to “Desktop.” Click on the word “Desktop” to toggle through your options. Click on the option, and you’ll be able to change your backup destination to any mounted device connected to your system. As an example, we’ve chosen to restore the deleted files to the USB external drive named “Backblaze.”
    3. Click “Restore” to restore the files you have selected.

    Storing the Restored Limbo Files

    Now that you have an external USB hard drive with the recovered Limbo files, let’s get them saved to the cloud. With Backblaze you have two options. The first option is to make the Limbo files part of your Backblaze backup. You can do this in two ways.

    1. Copy the Limbo files to your computer and they will be automatically backed up to Backblaze with the rest of your files.
    2. – or –

    3. Connect the external USB Hard Drive to your computer and configure Backblaze to back up that device. This device should remain connected to the computer while the backup occurs, and then once every couple of weeks to make sure that nothing has changed on the hard drive.

    If neither of the above solutions works for you, the other option is to use the Backblaze B2 Cloud Storage service.

What is Backblaze B2 Cloud Storage?

B2 Cloud Storage is a service for storing files in the cloud. Files are available for download at any time, either through the API or through a browser-compatible URL. Files stored in the B2 cloud are not deleted unless you explicitly delete them. In that way it is very similar to CrashPlan. Here’s some help, if you are unsure about the difference between Backblaze Personal Backup and Backblaze B2.

There are four ways to access B2: 1) a Web GUI, 2) a Command-line interface (CLI), 3) an API, and 4) via partner integrations, such a CloudBerry, Synology, Arq, QNAP, GoodSync and many more you can find on our B2 integrations page. Most CrashPlan users will find either the Web GUI or a partner integration to be the way to go. Note: There is an additional cost to use the B2 service, and we’ll get to that shortly.

  1. Since you already have a Backblaze account, you just have to log in to your account. Click on “My Settings” on the left hand navigation and enable B2 Cloud Storage. If you haven’t already done so you will be asked to provide a Mobile number for contact and authentication purposes.
  2. To use the B2 Web GUI, you create a B2 “bucket” and then drag-and-drop the files into the B2 bucket.
  3. You can also choose to use a B2 partner integration to store your data into B2.

If you use B2 to store your Limbo files rescued from CrashPlan and you use Backblaze to back up your computer, you will be able to access and manage all of your data from your one Backblaze account.

What does all this cost?

If you are only going to use Backblaze Personal Backup to back up your computer, then you will pay $50/year per computer.

If you decide to combine the use of Backblaze Personal Backup and Backblaze B2, let’s assume you have 500 GB of data to back up from your computer to Backblaze. Let’s also assume you have to store 100 GB of data in Backblaze B2 that you rescued from CrashPlan limbo. Your annual cost would be:

    To back up 500 GB:

    1. — Backblaze Personal Backup — 1 year/1 computer — $50.00

    To archive 100 GB:

    1. — Backblaze B2 — 100 GB @ $0.005/GB/month for 12 months — $6.00

    The Total Annual Cost to store your CrashPlan data in Backblaze, including your recovered deleted files, is $56.00.

Migrating from CrashPlan to Carbonite

If you are considering migrating your CrashPlan for Home account to Carbonite, you will still have to upload your data to Carbonite. There is no automatic process to copy the files from CrashPlan to Carbonite. You will also have to recover the Limbo files we’ve been speaking about using the process we’ve outlined above. In summary, when moving from CrashPlan for Home to any other vendor you will have to reupload your data to the new vendor.

One More Option

There is one more option you can use when you move your data from CrashPlan to another cloud service. You can download all of your data from CrashPlan, including the active and deleted files, to a local computer or device such an external USB Hard Drive. Then you can upload all that data to the new cloud backup provider. Of course this will mean all that data makes two trips through your local network — down and then back up. This will take time and could be very taxing on any bandwidth limits you may have in place from your network provider.

If you have the bandwidth and the time, this can be a good option, as all your files stored in CrashPlan Central are included in your backup. But, if you have a lot of data and/or a slow internet connection, this can take a really, really long time.

Join Our Webinar for More Information

You can sign up for our upcoming webinar, “Migrating from CrashPlan for Home to Backblaze” on September 7th at 10:00 am PDT if you’d like to learn more about the migration methods we covered today. Please note, you will need to register for this webinar by either signing up for a Backblaze BrightTALK channel account or using your existing BrightTALK account.

CrashPlan Replacement

Now that you are faced with replacing your CrashPlan for Home account, don’t wait until your contract is about to run out. Give yourself at least a couple of months to make sure all the data, including the Limbo data, is safely migrated somewhere else.

Also, regardless of which option you chose for migrating your data from CrashPlan to a new cloud backup service, once everything is moved and you’ve checked to make sure you got everything, then and only then should you turn off your CrashPlan account and uninstall CrashPlan.

An Invitation

If you are a CrashPlan for Home user going through the migration to a new cloud backup service, and have ideas to help other users through the migration process, let us know in the comments. We’ll update this post with any relevant ideas from the community.

The post How to Migrate All of Your Data from CrashPlan appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Amazon AppStream 2.0 Launch Recap – Domain Join, Simple Network Setup, and Lots More

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-appstream-2-0-launch-recap-domain-join-simple-network-setup-and-lots-more/

We (the AWS Blog Team) work to maintain a delicate balance between coverage and volume! On the one hand, we want to make sure that you are aware of as many features as possible. On the other, we don’t want to bury you in blog posts. As a happy medium between these two extremes we sometimes let interesting new features pile up for a couple of weeks and then pull them together in the form of a recap post such as this one.

Today I would like to tell you about the latest and greatest additions to Amazon AppStream 2.0, our application streaming service (read Amazon AppStream 2.0 – Stream Desktop Apps from AWS to learn more). We launched GPU-powered streaming instances just a month ago and have been adding features rapidly; here are some recent launches that did not get covered in individual posts at launch time:

  • Microsoft Active Directory Domains – Connect AppStream 2.0 streaming instances to your Microsoft Active Directory domain.
  • User Management & Web Portal – Create and manage users from within the AppStream 2.0 management console.
  • Persistent Storage for User Files – Use persistent, S3-backed storage for user home folders.
  • Simple Network Setup – Enable Internet access for image builder and instance fleets more easily.
  • Custom VPC Security Groups – Use VPC security groups to control network traffic.
  • Audio-In – Use microphones with your streaming applications.

These features were prioritized based on early feedback from AWS customers who are using or are considering the use of AppStream 2.0 in their enterprises. Let’s take a quick look at each one.

Domain Join
This much-requested feature allows you to connect your AppStream 2.0 streaming instances to your Microsoft Active Directory (AD) domain. After you do this you can apply existing policies to your streaming instances, and provide your users with single sign-on access to intranet resources such as web sites, printers, and file shares. Your users are authenticated using the SAML 2.0 provider of your choice, and can access applications that require a connection to your AD domain.

To get started, visit the AppStream 2.0 Console, create and store a Directory Configuration:

Newly created image builders and newly launched fleets can then use the stored Directory Configuration to join the AD domain in an Organizational Unit (OU) that you provide:

To learn more, read Using Active Directory Domains with AppStream 2.0 and follow the Setting Up the Active Directory tutorial. You can also learn more in the What’s New.

User Management & Web Portal
This feature makes it easier for you to give new users access to the applications that you are streaming with AppStream 2.0 if you are not using the Domain Join feature that I described earlier.

You can create and manage users, give them access to applications through a web portal, and send them welcome emails, all with a couple of clicks:

AppStream 2.0 sends each new user a welcome email that directs them to a web portal where they will be prompted to create a permanent password. Once they are logged in they are able to access the applications that have been assigned to them.

To learn more, read Using the AppStream 2.0 User Pool and the What’s New.

Persistent Storage
This feature allows users of streaming applications to store files for use in later AppStream 2.0 sessions. Each user is given a home folder which is stored in Amazon Simple Storage Service (S3) between sessions. The folder is made available to the streaming instance at the start of the session and changed files are periodically synced back to S3. To enable this feature, simply check Enable Home Folders when you create your next fleet:

All folders (and the files within) are stored in an S3 bucket that is automatically created within your account when the feature is enabled. There is no limit on total file storage but we recommend that individual files be limited to 5 gigabytes.

Regular S3 pricing applies; to learn more about this feature read about Persistent Storage with AppStream 2.0 Home Folders and check out the What’s New.

Simple Network Setup
Setting up Internet access for your image builder and your streaming instances was once a multi-step process. You had to create a Network Address Translation (NAT) gateway in a public subnet of one of your VPCs and configure traffic routing rules.

Now, you can do this by marking the image builder or the fleet for Internet access, selecting a VPC that has at least one public subnet, and choosing the public subnet(s), all from the AppStream 2.0 Console:

To learn more, read Network Settings for Fleet and Image Builder Instances and Enabling Internet Access Using a Public Subnet and check out the What’s New.

Custom VPC Security Groups
You can create VPC security groups and associate them with your image builders and your fleets. This gives you fine-grained control over inbound and outbound traffic to databases, license servers, file shares, and application servers. Read the What’s New to learn more.

Audio-In
You can use analog and USB microphones, mixing consoles, and other audio input devices with your streaming applications. Simply click on Enable Microphone in the AppStream 2.0 toolbar to get started. Read the What’s New to learn more.

Available Now
All of these features are available now and you can start using them today in all AWS Regions where Amazon AppStream 2.0 is available.

Jeff;

PS – If you are new to AppStream 2.0, try out some pre-installed applications. No setup needed and you’ll get to experience the power of streaming applications first-hand.

An Invitation for CrashPlan Customers: Try Backblaze

Post Syndicated from Gleb Budman original https://www.backblaze.com/blog/crashplan-alternative-backup-solution/

Welcome CrashPlan Users
With news coming out this morning of CrashPlan exiting the consumer market, we know some of you may be considering which backup provider to call home. We welcome you to try us.

For over a decade, Backblaze has provided unlimited cloud backup for Windows and Macintosh computers at $5 per month (or $50 per year).

Backblaze is excellent if you’re looking for the cheapest online backup option that still offers serious file protection.” — Dann Berg, Tom’s Guide.

That’s it. Ready to make sure your data is safe? Try Backblaze for free — it’ll take you less than a minute and you don’t need a credit card to start protecting your data.

Our customers don’t have to choose between competing feature sets or hard to understand fine print. There are no extra charges and no limits on the size of your files — no matter how many videos you want to back up. And when we say unlimited, we mean unlimited; there are no restrictions on files, gigabytes, or restores. Customers also love the choices they have for getting their data back — web, mobile apps, and our free Restore by Mail option. We’re also the fastest to back up your data. While other services throttle your upload speeds, we want to get you protected as quickly as possible.

Backblaze vs. Carbonite

We know that CrashPlan is encouraging customers to look at Carbonite as an alternative. We would like to offer you another option: Backblaze. We cost less, we offer more, we store over 350 Petabytes of data, we have restored over 20 billion files, and customers in over 120 countries around the world trust us with their data.

Backblaze Carbonite Basic Carbonite Prime
Price per Computer $50/year $59.99/year $149.99/year
Back Up All User Data By Default – No Picking And Choosing Yes No No
Automatically Back Up Files Of Any Size, Including Videos Yes No Yes1
Back Up Multiple USB External Hard Drives Yes No No
Restore by Mail for Free Yes No No
Locate Computer Yes No No
Manage Families & Teams Yes No No
Protect Accounts Via Two Factor VerificationSMS & Authenticator Apps Yes No No
Protect Data Via Private Encryption Key Yes No No2
(1) All videos and files over 4GB require manual selection.  (2) Available on Windows Only

To get just some of the features offered by Backblaze for $50/year, you would need to purchase Carbonite Prime at $149.99/year.

Reminder: Sync is Not Backup

“Backblaze is my favorite online backup service, mostly because everything about it is so simple, especially its pricing and software.“ Tim Fisher — Lifewire: 22 Online Backup Services Reviewed

Of course, there are plenty of options in the marketplace. We encourage you to choose one to make sure you stay backed up. One thing we tell our own friends and family: sync is not backup.

If you’re considering using a sync service — Dropbox, Google Drive, OneDrive, iCloud, etc. — you should know that these services are not designed to back up all your data. Typically, they only sync data from a specific directory or folder. If the service detects a file was deleted from your sync folder, it also will delete it from their server, and you’re out of luck. In addition, most don’t support external drives and have tiered pricing that gets quite expensive.

Backblaze is the Simple, Reliable, and Affordable Choice for Unlimited Backup of All Your Data
People have trusted Backblaze to protect their digital photos, music, movies, and documents for the past 10 years. We look forward to doing the same for your valuable data.

Your CrashPlan service may not be getting shut off today. But there’s no reason to wait until your data is at risk. Try Backblaze for FREE today — all you need to do is pick an email/password and click download.

The post An Invitation for CrashPlan Customers: Try Backblaze appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

AWS CloudHSM Update – Cost Effective Hardware Key Management at Cloud Scale for Sensitive & Regulated Workloads

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-cloudhsm-update-cost-effective-hardware-key-management/

Our customers run an incredible variety of mission-critical workloads on AWS, many of which process and store sensitive data. As detailed in our Overview of Security Processes document, AWS customers have access to an ever-growing set of options for encrypting and protecting this data. For example, Amazon Relational Database Service (RDS) supports encryption of data at rest and in transit, with options tailored for each supported database engine (MySQL, SQL Server, Oracle, MariaDB, PostgreSQL, and Aurora).

Many customers use AWS Key Management Service (KMS) to centralize their key management, with others taking advantage of the hardware-based key management, encryption, and decryption provided by AWS CloudHSM to meet stringent security and compliance requirements for their most sensitive data and regulated workloads (you can read my post, AWS CloudHSM – Secure Key Storage and Cryptographic Operations, to learn more about Hardware Security Modules, also known as HSMs).

Major CloudHSM Update
Today, building on what we have learned from our first-generation product, we are making a major update to CloudHSM, with a set of improvements designed to make the benefits of hardware-based key management available to a much wider audience while reducing the need for specialized operating expertise. Here’s a summary of the improvements:

Pay As You Go – CloudHSM is now offered under a pay-as-you-go model that is simpler and more cost-effective, with no up-front fees.

Fully Managed – CloudHSM is now a scalable managed service; provisioning, patching, high availability, and backups are all built-in and taken care of for you. Scheduled backups extract an encrypted image of your HSM from the hardware (using keys that only the HSM hardware itself knows) that can be restored only to identical HSM hardware owned by AWS. For durability, those backups are stored in Amazon Simple Storage Service (S3), and for an additional layer of security, encrypted again with server-side S3 encryption using an AWS KMS master key.

Open & Compatible  – CloudHSM is open and standards-compliant, with support for multiple APIs, programming languages, and cryptography extensions such as PKCS #11, Java Cryptography Extension (JCE), and Microsoft CryptoNG (CNG). The open nature of CloudHSM gives you more control and simplifies the process of moving keys (in encrypted form) from one CloudHSM to another, and also allows migration to and from other commercially available HSMs.

More Secure – CloudHSM Classic (the original model) supports the generation and use of keys that comply with FIPS 140-2 Level 2. We’re stepping that up a notch today with support for FIPS 140-2 Level 3, with security mechanisms that are designed to detect and respond to physical attempts to access or modify the HSM. Your keys are protected with exclusive, single-tenant access to tamper-resistant HSMs that appear within your Virtual Private Clouds (VPCs). CloudHSM supports quorum authentication for critical administrative and key management functions. This feature allows you to define a list of N possible identities that can access the functions, and then require at least M of them to authorize the action. It also supports multi-factor authentication using tokens that you provide.

AWS-Native – The updated CloudHSM is an integral part of AWS and plays well with other tools and services. You can create and manage a cluster of HSMs using the AWS Management Console, AWS Command Line Interface (CLI), or API calls.

Diving In
You can create CloudHSM clusters that contain 1 to 32 HSMs, each in a separate Availability Zone in a particular AWS Region. Spreading HSMs across AZs gives you high availability (including built-in load balancing); adding more HSMs gives you additional throughput. The HSMs within a cluster are kept in sync: performing a task or operation on one HSM in a cluster automatically updates the others. Each HSM in a cluster has its own Elastic Network Interface (ENI).

All interaction with an HSM takes place via the AWS CloudHSM client. It runs on an EC2 instance and uses certificate-based mutual authentication to create secure (TLS) connections to the HSMs.

At the hardware level, each HSM includes hardware-enforced isolation of crypto operations and key storage. Each customer HSM runs on dedicated processor cores.

Setting Up a Cluster
Let’s set up a cluster using the CloudHSM Console:

I click on Create cluster to get started, select my desired VPC and the subnets within it (I can also create a new VPC and/or subnets if needed):

Then I review my settings and click on Create:

After a few minutes, my cluster exists, but is uninitialized:

Initialization simply means retrieving a certificate signing request (the Cluster CSR):

And then creating a private key and using it to sign the request (these commands were copied from the Initialize Cluster docs and I have omitted the output. Note that ID identifies the cluster):

$ openssl genrsa -out CustomerRoot.key 2048
$ openssl req -new -x509 -days 365 -key CustomerRoot.key -out CustomerRoot.crt
$ openssl x509 -req -days 365 -in ID_ClusterCsr.csr   \
                              -CA CustomerRoot.crt    \
                              -CAkey CustomerRoot.key \
                              -CAcreateserial         \
                              -out ID_CustomerHsmCertificate.crt

The next step is to apply the signed certificate to the cluster using the console or the CLI. After this has been done, the cluster can be activated by changing the password for the HSM’s administrative user, otherwise known as the Crypto Officer (CO).

Once the cluster has been created, initialized and activated, it can be used to protect data. Applications can use the APIs in AWS CloudHSM SDKs to manage keys, encrypt & decrypt objects, and more. The SDKs provide access to the CloudHSM client (running on the same instance as the application). The client, in turn, connects to the cluster across an encrypted connection.

Available Today
The new HSM is available today in the US East (Northern Virginia), US West (Oregon), US East (Ohio), and EU (Ireland) Regions, with more in the works. Pricing starts at $1.45 per HSM per hour.

Jeff;

How To Send Ethereum Transactions With Java

Post Syndicated from Bozho original https://techblog.bozho.net/send-ethereum-transactions-java/

After I’ve expressed my concerns about the blockchain technology, let’s get a bit more practical with the blockchain. In particular, with Ethereum.

I needed to send a transaction with Java, so I looked at EthereumJ. You have three options:

  • Full node – you enable syncing, which means the whole blockchain gets downloaded. It takes a lot of time, so I abandoned that approach
  • “Light” node – you disable syncing, so you just become part of the network, but don’t fetch any parts of the chain. Not entirely sure, but I think this corresponds to the “light” mode of geth (the ethereum CLI). You are able to send messages (e.g. transaction messages) to other peers to process and store on the blockchain, but you yourself do not have the blockchain.
  • Offline (no node) – just create and sign the transaction, compute its raw representation (in the ethereum RLP format) and push it to the blockchain via a centralized API, e.g. the etherscan.io API. Etherscan is itself a node on the network and it can perform all of the operations (so it serves as a proxy)

Before going further, maybe it’s worth pointing out a few general properties of the blockchain (the ethereum one and popular cryptocurrencies at least) – it is a distributed database, relying on a peer-to-peer (overlay) network, formed by whoever has a client software running (wallet or otherwise). Transactions are in the form of “I (private key owner) want to send this amount to that address”. Transactions can have additional data stored inside them, e.g. representing what they are about. Transactions then get verified by peers (currently using a Proof-of-work based consensus) and get stored on the blockchain, which means every connected peer gets the newly created blocks (each block consisting of multiple transactions). That’s the blockchain in short, and Ethereum is no exception.

Why you may want to send transactions? I can’t think of a simple and obvious use-case, maybe you just want to implement a better wallet than the existing ones. For example in my case I wanted to store the head of a hash chain on the blockchain so that it cannot be tampered with.

In my particular case I was more interested in storing a particular piece of data as part of the transaction, rather than the transaction itself, so I had two nodes that sent very small transactions to each other (randomly choosing sender and recipient). I know I could probably have done that with a smart contract instead, but “one step at a time”. The initial code can be found here, and is based heavily on the EthereumJ samples. Since EthereumJ uses spring internally, and my application uses spring, it took some extra effort to allow for two nodes, but that’s not so relevant to the task at hand. The most important piece of the code can be seen further below in this post, only slightly modified.

You should have a user.conf file on the classpath with some defaults, and it can be based on the default ethereumj config. The more important part is the external user1 and user2 conf files (which in the general scenario can just be one conf file). Here’s a sample one, with the following important parameters:

  • peer.networkId – whether you are using the real production network (=1), or a test network (=3). Obviously, for anything than production you’d want a test network. On test networks you can get free ether by utilizing a faucet. In order to use a test network there are two more parameters below – blockchain.config.name = ropsten and genesis = ropsten.json. Note that there are more test networks at the moment, for experimenting with alternatives to proof-of-work.
  • peer.privateKey – this is the most important bit. It is your secret key which gives you control over your blockchain “account”. Only using that private key you can sign transactions (using an ellptic curve algorithm). The private key has a corresponding public key, which is basically your address on the network – if anyone wants to send funds, he sends them to your public key. But only you can then send funds from your account, as nobody else owns the private key. Which means you have to protect it. In this case it’s in plaintext in a file, which may not be ideal if you operate with big amounts of ether. Consider using some key-management solution (as outlined here)
  • peer.ip.list – this is optional, but preferable – you need to have a list of peers to connect to in order to bootstrap your client and make it part of the network. The peers there are connected to other peers, and so on, and so forth, so in the end it’s a single interconnected network. Note that in combination with the port number, that requires some additional network configuration if you are using that on a server/cluster/stack – you’d have to open some ports and allow outgoing and incoming connections.
  • database.dir – this is the directory where the blockchain and the list of discovered peers will be stored. It uses leveldb, and what I found out is that ethereumj uses an outdated leveldb which didn’t work on my machine. So I excluded them and manually used newer versions
  • sync.enabled – whether you want to fetch the blockchain or not. Normally you don’t need to, as it takes a lot of time, but that way you are not a full node and don’t contribute to the network.

As I noted earlier, I didn’t need a full node, I just needed to send a transaction. The light node would do (the difference should be simply switching sync.enabled from true to false), but after initially successfully connecting to peers, I started getting weird exceptions I didn’t have time to go into, so I couldn’t join the network anymore (maybe because of the crappy wifi I’m currently using).

Fortunately, there is a completely “offline” approach – use an external API to publish your transactions. All you need is your private key and a library (EthereumJ in this case) to prepare your transaction. So you can forget everything you read in the previous paragraphs. What you need is just the RLP encoded transaction after you have signed it. E.g.:

byte[] nonce = ByteUtil.intToBytesNoLoadZeroes(getTransactionCount(senderAddress) + 1);
byte[] gasPrice = getGasPrice();
Transaction tx = new Transaction(
    nonce,
    gasPrice,
    ByteUtil.longToBytesNoLeadZeroes(200000),
    receiverAddress,
    ByteUtil.bigIntegerToBytes(BigInteger.valueOf(1)),  // 1 gwei
    data.getBytes(StandardCharsets.UTF_8),
    CHAIN_ID);
            
tx.sign(ECKey.fromPrivate(senderPrivateKey));
            
byte[] rawTx = tx.getEncoded();
            
restTemplate.getForObject(etherscanUrl, String.class, "0x" + BaseEncoding.base16().encode(rawTx));

In this example, I use the Etherscan.io API (there’s also a test one for the Ropsten network). Note: it doesn’t seem to be documented, but you have to pass a User-Agent header that matches your application name. It also has a manual entry form to test your transactions (the link is for the Ropsten test network).

What are the parameters above?

  • nonce – this is a sequence number for transactions per user (=per private key). Each subsequent transaction should have a nonce that is the nonce of the previous + 1. That way nobody can replay the same transaction and drain the funds of the sender (the transaction that gets signed contains the nonce, so you cannot use the same raw transaction representation and just resubmit it). How to obtain the nonce? If you are connected to the Ethereum network, there’s a ethereum.getRepository().getNonce(fromAddress);. However, in a disconnected scenario, you’d need to obtain the current number of transactions for the sender, and then increment it. This is done via the eth_getTransactionCount endpoint. Note that it’s returned as hexadecimal, so you have to parse it, e.g. {"jsonrpc":"2.0","result":"0x1","id":73}
  • gas price, maximum gas price – these are used to cover the transaction costs (sending isn’t for free). You can read more here. You can obtain the current gas price by calling the “eth_gasPrice” API endpoint. Probably it’s a good idea to actually fetch the gas price periodically and cache it for a short period, rather than fetching it for every transaction. If you are connected to the network, you can obtain the gas price automatically.
  • receiverAddress – a byte array representing the public key of the recipient
  • value – how much ether you want to send. The smallest unit is actually a “gwei”, and the value is specified in gweis (a fraction of 1 ETH)
  • data – any additional data that you want to put in the transaction.
  • chainId – this is again related to which network you are using. Production=1, Ropsten test network=3. If you are curious why you have to encode it in a transaction, you can read here.

After that you sign the raw representation of the transaction with your private key (the raw representation is RLP (Recursive Length Prefix)). And then you send it to the API (you’d need a key for that, which you can get at Etherscan and include it in the URL). It’s almost identical to what you would’ve done if you were connected. But now you are relying on a central party (Etherscan) instead of becoming part of the network.

It may look “easy”, and when you’ve already done it and grasped it, it sounds like a piece of cake, but there are too many details that nobody abstracts from you, so you have to have the full picture before even being able to push a single transaction. What a nonce is, what a chainId is, what a test network is, how to get test ether (the top google result for a ropsten faucet doesn’t work at the moment, so you have to figure that out as well), then figure out whether you want to sync the chain or not, to be part of the network or not, to resolve weird connectivity issues and network configuration. And that’s not even mentioning smart contracts. I’m not saying it’s bad, it’s just not simple enough and that’s a barrier to wider adoption. That probably applies to most of programming, though. Anyway, I hope the above examples can get people started more easily.

The post How To Send Ethereum Transactions With Java appeared first on Bozho's tech blog.

Deploying an NGINX Reverse Proxy Sidecar Container on Amazon ECS

Post Syndicated from Nathan Peck original https://aws.amazon.com/blogs/compute/nginx-reverse-proxy-sidecar-container-on-amazon-ecs/

Reverse proxies are a powerful software architecture primitive for fetching resources from a server on behalf of a client. They serve a number of purposes, from protecting servers from unwanted traffic to offloading some of the heavy lifting of HTTP traffic processing.

This post explains the benefits of a reverse proxy, and explains how to use NGINX and Amazon EC2 Container Service (Amazon ECS) to easily implement and deploy a reverse proxy for your containerized application.

Components

NGINX is a high performance HTTP server that has achieved significant adoption because of its asynchronous event driven architecture. It can serve thousands of concurrent requests with a low memory footprint. This efficiency also makes it ideal as a reverse proxy.

Amazon ECS is a highly scalable, high performance container management service that supports Docker containers. It allows you to run applications easily on a managed cluster of Amazon EC2 instances. Amazon ECS helps you get your application components running on instances according to a specified configuration. It also helps scale out these components across an entire fleet of instances.

Sidecar containers are a common software pattern that has been embraced by engineering organizations. It’s a way to keep server side architecture easier to understand by building with smaller, modular containers that each serve a simple purpose. Just like an application can be powered by multiple microservices, each microservice can also be powered by multiple containers that work together. A sidecar container is simply a way to move part of the core responsibility of a service out into a containerized module that is deployed alongside a core application container.

The following diagram shows how an NGINX reverse proxy sidecar container operates alongside an application server container:

In this architecture, Amazon ECS has deployed two copies of an application stack that is made up of an NGINX reverse proxy side container and an application container. Web traffic from the public goes to an Application Load Balancer, which then distributes the traffic to one of the NGINX reverse proxy sidecars. The NGINX reverse proxy then forwards the request to the application server and returns its response to the client via the load balancer.

Reverse proxy for security

Security is one reason for using a reverse proxy in front of an application container. Any web server that serves resources to the public can expect to receive lots of unwanted traffic every day. Some of this traffic is relatively benign scans by researchers and tools, such as Shodan or nmap:

[18/May/2017:15:10:10 +0000] "GET /YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann HTTP/1.1" 404 1389 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
[18/May/2017:18:19:51 +0000] "GET /clientaccesspolicy.xml HTTP/1.1" 404 322 - Cloud mapping experiment. Contact [email protected]

But other traffic is much more malicious. For example, here is what a web server sees while being scanned by the hacking tool ZmEu, which scans web servers trying to find PHPMyAdmin installations to exploit:

[18/May/2017:16:27:39 +0000] "GET /mysqladmin/scripts/setup.php HTTP/1.1" 404 391 - ZmEu
[18/May/2017:16:27:39 +0000] "GET /web/phpMyAdmin/scripts/setup.php HTTP/1.1" 404 394 - ZmEu
[18/May/2017:16:27:39 +0000] "GET /xampp/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /apache-default/phpmyadmin/scripts/setup.php HTTP/1.1" 404 405 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /phpMyAdmin-2.10.0.0/scripts/setup.php HTTP/1.1" 404 397 - ZmEu
[18/May/2017:16:27:40 +0000] "GET /mysql/scripts/setup.php HTTP/1.1" 404 386 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /admin/scripts/setup.php HTTP/1.1" 404 386 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /forum/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:41 +0000] "GET /typo3/phpmyadmin/scripts/setup.php HTTP/1.1" 404 396 - ZmEu
[18/May/2017:16:27:42 +0000] "GET /phpMyAdmin-2.10.0.1/scripts/setup.php HTTP/1.1" 404 399 - ZmEu
[18/May/2017:16:27:44 +0000] "GET /administrator/components/com_joommyadmin/phpmyadmin/scripts/setup.php HTTP/1.1" 404 418 - ZmEu
[18/May/2017:18:34:45 +0000] "GET /phpmyadmin/scripts/setup.php HTTP/1.1" 404 390 - ZmEu
[18/May/2017:16:27:45 +0000] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 401 - ZmEu

In addition, servers can also end up receiving unwanted web traffic that is intended for another server. In a cloud environment, an application may end up reusing an IP address that was formerly connected to another service. It’s common for misconfigured or misbehaving DNS servers to send traffic intended for a different host to an IP address now connected to your server.

It’s the responsibility of anyone running a web server to handle and reject potentially malicious traffic or unwanted traffic. Ideally, the web server can reject this traffic as early as possible, before it actually reaches the core application code. A reverse proxy is one way to provide this layer of protection for an application server. It can be configured to reject these requests before they reach the application server.

Reverse proxy for performance

Another advantage of using a reverse proxy such as NGINX is that it can be configured to offload some heavy lifting from your application container. For example, every HTTP server should support gzip. Whenever a client requests gzip encoding, the server compresses the response before sending it back to the client. This compression saves network bandwidth, which also improves speed for clients who now don’t have to wait as long for a response to fully download.

NGINX can be configured to accept a plaintext response from your application container and gzip encode it before sending it down to the client. This allows your application container to focus 100% of its CPU allotment on running business logic, while NGINX handles the encoding with its efficient gzip implementation.

An application may have security concerns that require SSL termination at the instance level instead of at the load balancer. NGINX can also be configured to terminate SSL before proxying the request to a local application container. Again, this also removes some CPU load from the application container, allowing it to focus on running business logic. It also gives you a cleaner way to patch any SSL vulnerabilities or update SSL certificates by updating the NGINX container without needing to change the application container.

NGINX configuration

Configuring NGINX for both traffic filtering and gzip encoding is shown below:

http {
  # NGINX will handle gzip compression of responses from the app server
  gzip on;
  gzip_proxied any;
  gzip_types text/plain application/json;
  gzip_min_length 1000;
 
  server {
    listen 80;
 
    # NGINX will reject anything not matching /api
    location /api {
      # Reject requests with unsupported HTTP method
      if ($request_method !~ ^(GET|POST|HEAD|OPTIONS|PUT|DELETE)$) {
        return 405;
      }
 
      # Only requests matching the whitelist expectations will
      # get sent to the application server
      proxy_pass http://app:3000;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection 'upgrade';
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_cache_bypass $http_upgrade;
    }
  }
}

The above configuration only accepts traffic that matches the expression /api and has a recognized HTTP method. If the traffic matches, it is forwarded to a local application container accessible at the local hostname app. If the client requested gzip encoding, the plaintext response from that application container is gzip-encoded.

Amazon ECS configuration

Configuring ECS to run this NGINX container as a sidecar is also simple. ECS uses a core primitive called the task definition. Each task definition can include one or more containers, which can be linked to each other:

 {
  "containerDefinitions": [
     {
       "name": "nginx",
       "image": "<NGINX reverse proxy image URL here>",
       "memory": "256",
       "cpu": "256",
       "essential": true,
       "portMappings": [
         {
           "containerPort": "80",
           "protocol": "tcp"
         }
       ],
       "links": [
         "app"
       ]
     },
     {
       "name": "app",
       "image": "<app image URL here>",
       "memory": "256",
       "cpu": "256",
       "essential": true
     }
   ],
   "networkMode": "bridge",
   "family": "application-stack"
}

This task definition causes ECS to start both an NGINX container and an application container on the same instance. Then, the NGINX container is linked to the application container. This allows the NGINX container to send traffic to the application container using the hostname app.

The NGINX container has a port mapping that exposes port 80 on a publically accessible port but the application container does not. This means that the application container is not directly addressable. The only way to send it traffic is to send traffic to the NGINX container, which filters that traffic down. It only forwards to the application container if the traffic passes the whitelisted rules.

Conclusion

Running a sidecar container such as NGINX can bring significant benefits by making it easier to provide protection for application containers. Sidecar containers also improve performance by freeing your application container from various CPU intensive tasks. Amazon ECS makes it easy to run sidecar containers, and automate their deployment across your cluster.

To see the full code for this NGINX sidecar reference, or to try it out yourself, you can check out the open source NGINX reverse proxy reference architecture on GitHub.

– Nathan
 @nathankpeck

A new twist on data backup: CloudNAS

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/cloudnas-backup/

Morro CacheDrive

There are many ways for SMBs, professionals, and advanced users to back up their data. The process can be as simple as copying files to a flash drive or an external drive, or as sophisticated as using a Synology or QNAP NAS device as your primary storage device and syncing the files to a cloud storage service such as Backblaze B2.

A recent entry into the backup arena is Morro Data and their CloudNAS solution, where files are stored in the cloud, cached locally as needed, and synced globally among the other CloudNAS systems in a given organization. There are three components to the solution:

  • A Morro CacheDrive — This resides on your internal network like a NAS device and stores from 1- to 8 TB of data depending on the model
  • The CloudNAS service — This software runs on the Morro CacheDrive to keep track of and manage the data
  • Backblaze B2 Cloud Storage — Where the data is stored in the cloud

The Morro CacheDrive is installed on your local network and looks like a network share. On Windows, the share can be mounted as a letter device, M:, for example. On the Mac, the device is mounted as a Shared device (Databank in the example below).

CloudNAS software dashboard

In either case, the device works like a folder/directory, typically on your desktop. You then either drag-and-drop or save a file to the folder/directory. This places the file on the CacheDrive. Once there, the file is automatically backed up to the cloud. In the case of CloudNAS solution, that cloud is Backblaze B2.

All that sounds pretty straight-forward, but what makes the CloudNAS solution unique is the solution allows you to have unlimited storage space. For example, you can access 5 TB of data from a 1 TB CacheDrive. Confused? Let me explain. All 5 TB of the data is stored in B2, having been uploaded to B2 each time you stored data on the CacheDrive. The 1 TB CacheDrive keeps (caches) the most recent or most often used files on the CacheDrive. When you need a file not currently stored on the CacheDrive, the CloudNAS software automatically downloads the file from the B2 cloud to the CacheDrive and makes it available to use as desired.

Things to know about the CloudNAS solution

  • Sharing Systems: Multiple users can mount the same CacheDrive with each being able to update and share the files.
  • Synced Systems: If you have two or more CloudNAS systems on your network, they will keep the B2 directory of files synced between all of the systems. Everyone on the network sees the same file list.
  • Unlimited Data: Regardless of the size of the CacheDrive device you purchase, you will not run out of space as Backblaze B2 will contain all of your data. That said, you should choose the size of your CacheDrive that fits your operational environment.
  • Network Speed: Files are initially stored on the CacheDrive, then copied to B2. Local network connections are typically much faster than internet network speeds. This means your files are uploaded to the CacheDrive fast then transferred to B2 as time allows at the speed of your internet connection, all without slowing you down. This should be interesting to those of you who have slower internet connections.
  • Access: The files stored using the Cloud NAS solution can be accessed through the shared folder/directory on your desktop as well as through a web-based Team Portal.

Getting Started

To start, you purchase a Morro CacheDrive. The price starts at $499.00 for a unit with 1 TB of cache storage. Next you choose a CloudNAS subscription. This starts at $10/month for the Standard plan, and lets you manage up to 10 TB of data. Finally, you connect Backblaze B2 to the Morro system to finish the set-up process. You pay Backblaze each month for the data you store in and download from B2 while using the Morro solution.

The CloudNAS solution is certainly a different approach to storing your data. You get the ability to store a nearly unlimited amount of data without having to upgrade your hardware as you go, and all of your data is readily available with just a few clicks. For users who need to store terabytes of data that needs be available anytime, the CloudNAS solution is worth a look.

The post A new twist on data backup: CloudNAS appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Announcing the Raspberry Jam Big Birthday Weekend 2018

Post Syndicated from Ben Nuttall original https://www.raspberrypi.org/blog/raspberry-jam-big-birthday-weekend-2018/

For the last few years, we have held a big Raspberry Pi community event in Cambridge around Raspberry Pi’s birthday, where people have come together for a huge party with talks, workshops, and more. We want more people to have the chance to join in with our birthday celebrations next year, so we’re going to be coordinating Raspberry Jams all over the world to take place over the Raspberry Jam Big Birthday Weekend, 3–4 March 2018.

Raspberry Pi Big Birthday Weekend 2018. GIF with confetti and bopping JAM balloons

Big Birthday fun!

Whether you’ve run a Raspberry Jam before, or you’d like to start a new Jam in your area, we invite you to join us for our Big Birthday Weekend, wherever you are in the world. This event will be a community-led, synchronised, global mega-Jam in celebration of our sixth birthday and the digital making community! Members of the Raspberry Pi Foundation team will be attending Jams far and wide to celebrate with you during the weekend.

Jams across the world will receive a special digital pack – be sure to register your interest so we can get your pack to you! We’ll also be sending out party kits to registered Jams – more info on this below.

Need help getting started?

First of all, check out the Raspberry Jam page to read all about Jams, and take a look at our recent blog post explaining the support for Jams that we offer.

If there’s no Jam near you yet, the Raspberry Jam Big Birthday Weekend is the perfect opportunity to start one yourself! If you’d like some help getting your Jam off the ground, there are a few places you can get support:

  • The Raspberry Jam Guidebook is full of advice gathered from the amazing people who run Jams in the UK.
  • The Raspberry Jam Slack team is available for Jam organisers to chat, share ideas, and get help from each other. Just email jam [at] raspberrypi.org and ask to be invited.
  • Attend a Jam! Find an upcoming Jam near you, and go along to get an idea of what it’s like.
  • Email us – if you have more queries, you can email jam [at] raspberrypi.org and we’ll do what we can to help.

Raspberry Jam

Get involved

If you’re keen to start a new Jam, there’s no need to wait until March – why not get up and running over the summer? Then you’ll be an expert by the time the Raspberry Jam Big Birthday Weekend comes around. Check out the guidebook, join the Jam Slack, and submit your event to the map when you’re ready.

Like the idea of running a Jam, but don’t want to do it by yourself? Then feel free to email us, and we’ll try and help you find someone to co-organise it.

If you don’t fancy organising a Jam for our Big Birthday Weekend, but would like to celebrate with us, keep an eye on our website for an update early next year. We’ll publish a full list of Jams participating in the festivities so you can find one near you. And if you’ve never attended a Jam before, there’s no need to wait: find one to join on the map here.

Raspberry Jam

Register your interest

If you think you’d like to run a Jam as part of the Big Birthday Weekend, register your interest now, and you’ll be the first to receive updates. Don’t worry if you don’t have the venue or logistics in place yet – this is just to let us know you’re keen, and to give us an idea about how big our party is going to be.

We will contact you in autumn to give you more information, as well as some useful resources. On top of our regular Raspberry Jam branding pack, we’ll provide a special digital Big Birthday Weekend pack to help you celebrate and tell everyone about your Jam!

Then, once you have confirmed you’re taking part, you’ll be able to register your Jam on our website. This will make sure that other people interested in joining the party can find your event. If your Jam is among the first 150 to be registered for a Big Birthday Weekend event, we will send you a free pack of goodies to use on the big day!

Go fill in the form, and we’ll be in touch!

 

PS: We’ll be running a big Cambridge event in the summer on the weekend of 30 June–1 July 2018. Put it in your diary – we’ll say more about it as we get closer to the date.

The post Announcing the Raspberry Jam Big Birthday Weekend 2018 appeared first on Raspberry Pi.

Concerns About The Blockchain Technology

Post Syndicated from Bozho original https://techblog.bozho.net/concerns-blockchain-technology/

The so-called (and marketing-branded) “blockchain technology” is promised to revolutionize every industry. Anything, they say, will become decentralized, free from middle men or government control. Services will thrive on various installments of the blockchain, and smart contracts will automatically enforce any logic that is related to the particular domain.

I don’t mind having another technological leap (after the internet), and given that I’m technically familiar with the blockchain, I may even be part of it. But I’m not convinced it will happen, and I’m not convinced it’s going to be the next internet.

If we strip the hype, the technology behind Bitcoin is indeed a technical masterpiece. It combines existing techniques (likes hash chains and merkle trees) with a very good proof-of-work based consensus algorithm. And it creates a digital currency, which ontop of being worth billions now, is simply cool.

But will this technology be mass-adopted, and will mass adoption allow it to retain the technological benefits it has?

First, I’d like to nitpick a little bit – if anyone is speaking about “decentralized software” when referring to “the blockchain”, be suspicious. Bitcon and other peer-to-peer overlay networks are in fact “distributed” (see the pictures here). “Decentralized” means having multiple providers, but doesn’t mean each user will be full-featured nodes on the network. This nitpicking is actually part of another argument, but we’ll get to that.

If blockchain-based applications want to reach mass adoption, they have to be user-friendly. I know I’m being captain obvious here (and fortunately some of the people in the area have realized that), but with the current state of the technology, it’s impossible for end users to even get it, let alone use it.

My first serious concern is usability. To begin with, you need to download the whole blockchain on your machine. When I got my first bitcoin several years ago (when it was still 10 euro), the blockchain was kind of small and I didn’t notice that problem. Nowadays both the Bitcoin and Ethereum blockchains take ages to download. I still haven’t managed to download the ethereum one – after several bugs and reinstalls of the client, I’m still at 15%. And we are just at the beginning. A user just will not wait for days to download something in order to be able to start using a piece of technology.

I recently proposed downloading snapshots of the blockchain via bittorrent to be included in the Ethereum protocol itself. I know that snapshots of the Bitcoin blockchain have been distributed that way, but it has been a manual process. If a client can quickly download the huge file up to a recent point, and then only donwload the latest ones in the the traditional way, starting up may be easier. Of course, the whole chain would have to be verified, but maybe that can be a background process that doesn’t stop you from using whatever is built ontop of the particular blockchain. (I’m not sure if that will be secure enough, and that, say potential Sybil attacks on the bittorrent part won’t make it undesirable, it’s just an idea).

But even if such an approach works and is adopted, that would still mean that for every service you’d have to download a separate blockchain. Of course, projects like Ethereum may seem like the “one stop shop” for cool blockchain-based applications, but fragmentation is already happening – there are alt-coins bundled with various services like file storage, DNS, etc. That will not be workable for end-users. And it’s certainly not an option for mobile, which is the dominant client now. If instead of downloading the entire chain, something like consistent hashing is used to distribute the content in small portions among clients, it might be workable. But how will trust work in that case, I don’t know. Maybe it’s possible, maybe not.

And yes, I know that you don’t necessarily have to install a wallet/client in order to make use of a given blockchain – you can just have a cloud-based wallet. Which is fairly convenient, but that gets me to my nitpicking from a few paragraphs above and to may second concern – this effectively turns a distributed system into a decentralized one – a limited number of cloud providers hold most of the data (just as a limited number of miners hold most of the processing power). And then, even though the underlying technology allows for a distributed deployment, we’ll end-up again with simply decentralized or even de-facto cenetralized, if mergers and acquisitions lead us there (and they probably will). And in order to be able to access our wallets/accounts from multiple devices, we’d use a convenient cloud service where we’d login with our username and password (because the private key is just too technical and hard for regular users). And that seems to defeat the whole idea.

Not only that, but there is an inevitable centralization of decisions (who decides on the size of the block, who has commit rights to the client repository) as well as a hidden centralization of power – how much GPU power does the Chinese mining “farms” control and can they influence the network significantly? And will the average user ever know that or care (as they don’t care that Google is centralized). I think that overall, distributed technologies will follow the power law, and the majority of data/processing power/decision power will be controller by a minority of actors. And so our distributed utopia will not happen in its purest form we dream of.

My third concern is incentive. Distributed technologies that have been successful so far have a pretty narrow set of incentives. The internet was promoted by large public institutions, including government agencies and big universitives. Bittorrent was successful mainly because it allowed free movies and songs with 2 clicks of the mouse. And Bitcoin was successful because it offered financial benefits. I’m oversimplifying of course, but “government effort”, “free & easy” and “source of more money” seem to have been the successful incentives. On the other side of the fence there are dozens of failed distributed technologies. I’ve tried many of them – alternative search engines, alternative file storage, alternative ride-sharings, alternative social networks, alternative “internets” even. None have gained traction. Because they are not easier to use than their free competitors and you can’t make money out of them (and no government bothers promoting them).

Will blockchain-based services have sufficient incentives to drive customers? Will centralized competitors just easily crush the distributed alternatives by being cheaper, more-user friendly, having sales departments that can target more than hardcore geeks who have no problem syncing their blockchain via the command line? The utopian slogans seem very cool to idealists and futurists, but don’t sell. “Free from centralized control, full control over your data” – we’d have to go through a long process of cultural change before these things make sense to more than a handful of people.

Speaking of services, often examples include “the sharing economy”, where one stranger offers a service to another stranger. Blockchain technology seems like a good fit here indeed – the services are by nature distributed, why should the technology be centralized? Here comes my fourth concern – identity. While for the cryptocurrencies it’s actually beneficial to be anonymous, for most of the real-world services (i.e. the industries that ought to be revolutionized) this is not an option. You can’t just go in the car of publicKey=5389BC989A342…. “But there are already distributed reputation systems”, you may say. Yes, and they are based on technical, not real-world identities. That doesn’t build trust. I don’t trust that publicKey=5389BC989A342… is the same person that got the high reputation. There may be five people behind that private key. The private key may have been stolen (e.g. in a cloud-provider breach).

The values of companies like Uber and AirBNB is that they serve as trust brokers. They verify and vouch for their drivers and hosts (and passengers and guests). They verify their identity through government-issued documents, skype calls, selfies, compare pictures to documents, get access to government databases, credit records, etc. Can a fully distributed service do that? No. You’d need a centralized provider to do it. And how would the blockchain make any difference then? Well, I may not be entirely correct here. I’ve actually been thinking quite a lot about decentralized identity. E.g. a way to predictably generate a private key based on, say biometrics+password+government-issued-documents, and use the corresponding public key as your identifier, which is then fed into reputation schemes and ultimately – real-world services. But we’re not there yet.

And that is part of my fifth concern – the technology itself. We are not there yet. There are bugs, there are thefts and leaks. There are hard-forks. There isn’t sufficient understanding of the technology (I confess I don’t fully grasp all the implementation details, and they are always the key). Often the technology is advertised as “just working”, but it isn’t. The other day I read an article (lost the link) that clarifies a common misconception about smart contracts – they cannot interact with the outside world – they can’t call APIs (e.g. stock market prices, bank APIs), they can’t push or fetch data from anywhere but the blockchain. That mandates the need, again, for a centralized service that pushes the relevant information before smart contracts can pick it up. I’m pretty sure that all cool-sounding applications are not possible without extensive research. And even if/when they are, writing distributed code is hard. Debugging a smart contract is hard. Yes, hard is cool, but that doesn’t drive economic value.

I have mostly been referring to public blockchains so far. Private blockchains may have their practical application, but there’s one catch – they are not exactly the cool distributed technology that the Bitcoin uses. They may be called “blockchains” because they…chain blocks, but they usually centralize trust. For example the Hyperledger project uses PKI, with all its benefits and risks. In these cases, a centralized authority issues the identity “tokens”, and then nodes communicate and form a shared ledger. That’s a bit easier problem to solve, and the nodes would usually be on actual servers in real datacenters, and not on your uncle’s Windows XP.

That said, hash chaining has been around for quite a long time. I did research on the matter because of a side-project of mine and it seems providing a tamper-proof/tamper-evident log/database on semi-trusted machines has been discussed in many computer science papers since the 90s. That alone is not “the magic blockchain” that will solve all of our problems, no matter what gossip protocols you sprinkle ontop. I’m not saying that’s bad, on the contrary – any variation and combinations of the building blocks of the blockchain (the hash chain, the consensus algorithm, the proof-of-work (or stake), possibly smart contracts), has potential for making useful products.

I know I sound like the a naysayer here, but I hope I’ve pointed out particular issues, rather than aimlessly ranting at the hype (though that’s tempting as well). I’m confident that blockchain-like technologies will have their practical applications, and we will see some successful, widely-adopted services and solutions based on that, just as pointed out in this detailed report. But I’m not convinced it will be revolutionizing.

I hope I’m proven wrong, though, because watching a revolutionizing technology closely and even being part of it would be quite cool.

The post Concerns About The Blockchain Technology appeared first on Bozho's tech blog.

OpenSUSE Leap 42.3 released

Post Syndicated from corbet original https://lwn.net/Articles/728881/rss

OpenSUSE
Leap 42.3
is now available. “After basing openSUSE Leap on SLE
(SUSE Linux Enterprise) and adding more source code to Leap 42.2 from SLE
12, Leap 42.3 adds even more packages from SLE 12 SP 3 and synchronizes
several common packages. The shared codebase allows for openSUSE Leap 42.3
to receive enhanced maintenance and bug fixes from both the openSUSE
community and SUSE engineers.
” There is quite a bit of new stuff in
this release; see this
page
for some details.