Tag Archives: speeding

Legal Blackmail: Zero Cases Brought Against Alleged Pirates in Sweden

Post Syndicated from Andy original https://torrentfreak.com/legal-blackmail-zero-cases-brought-against-alleged-pirates-in-sweden-180525/

While several countries in Europe have wilted under sustained pressure from copyright trolls for more than ten years, Sweden managed to avoid their controversial attacks until fairly recently.

With Germany a decade-old pit of misery, with many hundreds of thousands of letters – by now probably millions – sent out to Internet users demanding cash, Sweden avoided the ranks of its European partners until two years ago

In September 2016 it was revealed that an organization calling itself Spridningskollen (Distribution Check) headed up by law firm Gothia Law, would begin targeting the public.

Its spokesperson described its letters as “speeding tickets” for pirates, in that they would only target the guilty. But there was a huge backlash and just a couple of months later Spridningskollen headed for the hills, without a single collection letter being sent out.

That was the calm before the storm.

In February 2017, Danish law firm Njord Law was found to be at the center of a new troll operation targeting the subscribers of several ISPs, including Telia, Tele2 and Bredbandsbolaget. Court documents revealed that thousands of IP addresses had been harvested by the law firm’s partners who were determined to link them with real-life people.

Indeed, in a single batch, Njord Law was granted permission from the court to obtain the identities of citizens behind 25,000 IP addresses, from whom it hoped to obtain cash settlements of around US$550. But it didn’t stop there.

Time and again the trolls headed back to court in an effort to reach more people although until now the true scale of their operations has been open to question. However, a new investigation carried out by SVT has revealed that the promised copyright troll invasion of Sweden is well underway with a huge level of momentum.

Data collated by the publication reveals that since 2017, the personal details behind more than 50,000 IP addresses have been handed over by Swedish Internet service providers to law firms representing copyright trolls and their partners. By the end of this year, Njord Law alone will have sent out 35,000 letters to Swede’s whose IP addresses have been flagged as allegedly infringing copyright.

Even if one is extremely conservative with the figures, the levels of cash involved are significant. Taking a settlement amount of just $300 per letter, very quickly the copyright trolls are looking at $15,000,000 in revenues. On the perimeter, assuming $550 will make a supposed lawsuit go away, we’re looking at a potential $27,500,000 in takings.

But of course, this dragnet approach doesn’t have the desired effect on all recipients.

In 2017, Njord Law said that only 60% of its letters received any kind of response, meaning that even fewer would be settling with the company. So what happens when the public ignores the threatening letters?

“Yes, we will [go to court],” said lawyer Jeppe Brogaard Clausen last year.

“We wish to resolve matters as much as possible through education and dialogue without the assistance of the court though. It is very expensive both for the rights holders and for plaintiffs if we go to court.”

But despite the tough-talking, SVT’s investigation has turned up an interesting fact. The nuclear option, of taking people to court and winning a case when they refuse to pay, has never happened.

After trawling records held by the Patent and Market Court and all those held by the District Courts dating back five years, SVT did not find a single case of a troll taking a citizen to court and winning a case. Furthermore, no law firm contacted by the publication could show that such a thing had happened.

“In Sweden, we have not yet taken someone to court, but we are planning to file for the right in 2018,” Emelie Svensson, lawyer at Njord Law, told SVT.

While a case may yet reach the courts, when it does it is guaranteed to be a cut-and-dried one. Letter recipients can often say things to damage their case, even when they’re only getting a letter due to their name being on the Internet bill. These are the people who find themselves under the most pressure to pay, whether they’re guilty or not.

“There is a risk of what is known in English as ‘legal blackmailing’,” says Mårten Schultz, professor of civil law at Stockholm University.

“With [the copyright holders’] legal and economic muscles, small citizens are scared into paying claims that they do not legally have to pay.”

It’s a position shared by Marianne Levine, Professor of Intellectual Property Law at Stockholm University.

“One can only show that an IP address appears in some context, but there is no point in the evidence. Namely, that it is the subscriber who also downloaded illegitimate material,” she told SVT.

Njord Law, on the other hand, sees things differently.

“In Sweden, we have no legal case saying that you are not responsible for your IP address,” Emelie Svensson says.

Whether Njord Law will carry through with its threats will remain to be seen but there can be little doubt that while significant numbers of people keep paying up, this practice will continue and escalate. The trolls have come too far to give up now.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

Solving Complex Ordering Challenges with Amazon SQS FIFO Queues

Post Syndicated from Christie Gifrin original https://aws.amazon.com/blogs/compute/solving-complex-ordering-challenges-with-amazon-sqs-fifo-queues/

Contributed by Shea Lutton, AWS Cloud Infrastructure Architect

Amazon Simple Queue Service (Amazon SQS) is a fully managed queuing service that helps decouple applications, distributed systems, and microservices to increase fault tolerance. SQS queues come in two distinct types:

  • Standard SQS queues are able to scale to enormous throughput with at-least-once delivery.
  • FIFO queues are designed to guarantee that messages are processed exactly once in the exact order that they are received and have a default rate of 300 transactions per second.

As customers explore SQS FIFO queues, they often have questions about how the behavior works when messages arrive and are consumed. This post walks through some common situations to identify the exact behavior that you can expect. It also covers the behavior of message groups in depth and explains why message groups are key to understanding how FIFO queues work.

The simple case

Suppose that you run a major auction platform where people buy and sell a wide range of products. Your platform requires that transactions from buyers and sellers get processed in exactly the order received. Here’s how a FIFO queue helps you keep all your transactions in one straight flow.

A seller currently is holding an auction for a laptop, and three different bids are received for the same price. Ties are awarded to the first bidder at that price so it is important to track which arrived first. Your auction platform receives the three bids and sends them to a FIFO queue before they are processed.

Now observe how messages leave the queue. When your consumer asks for a batch of up to 10 messages, SQS starts filling the batch with the oldest message (bid A1). It keeps filling until either the batch is full or the queue is empty. In this case, the batch contains the three messages and the queue is now empty. After a batch has left the queue, SQS considers that batch of messages to be “in-flight” until the consumer either deletes them or the batch’s visibility timer expires.

 

When you have a single consumer, this is easy to envision. The consumer gets a batch of messages (now in-flight), does its processing, and deletes the messages. That consumer is then ready to ask for the next batch of messages.

The critical thing to keep in mind is that SQS won’t release the next batch of messages until the first batch has been deleted. By adding more messages to the queue, you can see more interesting behaviors. Imagine that a burst of 11 bids is sent to your FIFO queue, with two bids for Auction A arriving last.

The FIFO queue now has at least two batches of messages in it. When your single consumer requests the first batch of 10 messages, it receives a batch starting with B1 and ending with A1. Later, after the first batch has been deleted, the consumer can get the second batch of messages containing the final A2 message from the queue.

Adding complexity with multiple message groups

A new challenge arises. Your auction platform is getting busier and your dev team added a number of new features. The combination of increased messages and extra processing time for the new features means that a single consumer is too slow. The solution is to scale to have more consumers and process messages in parallel.

To work in parallel, your team realized that only the messages related to a single auction must be kept in order. All transactions for Auction A need to be kept in order and so do all transactions for Auction B. But the two auctions are independent and it does not matter which auctions transactions are processed first.

FIFO can handle that case with a feature called message groups. Each transaction related to Auction A is placed by your producer into message group A, and so on. In the diagram below, Auction A and Auction B each received three bid transactions, with bid B1 arriving first. The FIFO queue always keeps transactions within a message group in the order in which they arrived.

How is this any different than earlier examples? The consumer now gets the messages ordered by message groups, all the B group messages followed by all the A group messages. Multiple message groups create the possibility of using multiple consumers, which I explain in a moment. If FIFO can’t fill up a batch of messages with a single message group, FIFO can place more than one message group in a batch of messages. But whenever possible, the queue gives you a full batch of messages from the same group.

The order of messages leaving a FIFO queue is governed by three rules:

  1. Return the oldest message where no other message in the same message group is currently in-flight.
  2. Return as many messages from the same message group as possible.
  3. If a message batch is still not full, go back to rule 1.

To see this behavior, add a second consumer and insert many more messages into the queue. For simplicity, the delete message action has been omitted in these diagrams but it is assumed that all messages in a batch are processed successfully by the consumer and the batch is properly deleted immediately after.

In this example, there are 11 Group A and 11 Group B transactions arriving in interleaved order and a second consumer has been added. Consumer 1 asks for a group of 10 messages and receives 10 Group A messages. Consumer 2 then asks for 10 messages but SQS knows that Group A is in flight, so it releases 10 Group B messages. The two consumers are now processing two batches of messages in parallel, speeding up throughput and then deleting their batches. When Consumer 1 requests the next batch of messages, it receives the remaining two messages, one from Group A and one from Group B.

Consider this nuanced detail from the example above. What would happen if Consumer 1 was on a faster server and processed its first batch of messages before Consumer 2 could mark its messages for deletion? See if you can predict the behavior before looking at the answer.

If Consumer 2 has not deleted its Group B messages yet when Consumer 1 asks for the next batch, then the FIFO queue considers Group B to still be in flight. It does not release any more Group B messages. Consumer 1 gets only the remaining Group A message. Later, after Consumer 2 has deleted its first batch, the remaining Group B message is released.

Conclusion

I hope this post answered your questions about how Amazon SQS FIFO queues work and why message groups are helpful. If you’re interested in exploring SQS FIFO queues further, here are a few ideas to get you started:

New DDoS Reflection-Attack Variant

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2018/03/new_ddos_reflec.html

This is worrisome:

DDoS vandals have long intensified their attacks by sending a small number of specially designed data packets to publicly available services. The services then unwittingly respond by sending a much larger number of unwanted packets to a target. The best known vectors for these DDoS amplification attacks are poorly secured domain name system resolution servers, which magnify volumes by as much as 50 fold, and network time protocol, which increases volumes by about 58 times.

On Tuesday, researchers reported attackers are abusing a previously obscure method that delivers attacks 51,000 times their original size, making it by far the biggest amplification method ever used in the wild. The vector this time is memcached, a database caching system for speeding up websites and networks. Over the past week, attackers have started abusing it to deliver DDoSes with volumes of 500 gigabits per second and bigger, DDoS mitigation service Arbor Networks reported in a blog post.

Cloudflare blog post. BoingBoing post.

EDITED TO ADD (3/9): Brian Krebs covered this.

All-In on Unlimited Backup

Post Syndicated from Gleb Budman original https://www.backblaze.com/blog/all-in-on-unlimited-backup/

chips on computer with cloud backup

The cloud backup industry has seen its share of tumultuousness. BitCasa, Dell DataSafe, Xdrive, and a dozen others have closed up shop. Mozy, Amazon, and Microsoft offered, but later canceled, their unlimited offerings. Recently, CrashPlan for Home customers were notified that their service was being end-of-lifed. Then today we’ve heard from Carbonite customers who are frustrated by this morning’s announcement of a price increase from Carbonite.

We believe that the fundamental goal of a cloud backup is having peace-of-mind: knowing your data — all of it — is safe. For over 10 years Backblaze has been providing that peace-of-mind by offering completely unlimited cloud backup to our customers. And we continue to be committed to that. Knowing that your cloud backup vendor is not going to disappear or fundamentally change their service is an essential element in achieving that peace-of-mind.

Committed to Unlimited Backup

When Mozy discontinued their unlimited backup on Jan 31, 2011, a lot of people asked, “Does this mean Backblaze will discontinue theirs as well?” At that time I wrote the blog post Backblaze is committed to unlimited backup. That was seven years ago. Since then we’ve continued to make Backblaze cloud backup better: dramatically speeding up backups and restores, offering the unique and very popular Restore Return Refund program, enabling direct access and sharing of any file in your backup, and more. We also introduced Backblaze Groups to enable businesses and families to manage backups — all at no additional cost.

How That’s Possible

I’d like to answer the question of “How have you been able to do this when others haven’t?

First, commitment. It’s not impossible to offer unlimited cloud backup, but it’s not easy. The Backblaze team has been committed to unlimited as a core tenet.

Second, we have pursued the technical, business, and cultural steps required to make it happen. We’ve designed our own servers, written our cloud storage software, run our own operations, and been continually focused on every place we could optimize a penny out of the cost of storage. We’ve built a culture at Backblaze that cares deeply about that.

Ensuring Peace-of-Mind

Price increases and plan changes happen in our industry, but Backblaze has consistently been the low price leader, and continues to stand by the foundational element of our service — truly unlimited backup storage. Carbonite just announced a price increase from $60 to $72/year, and while that’s not an astronomical increase, it’s important to keep in mind the service that they are providing at that rate. The basic Carbonite plan provides a service that doesn’t back up videos or external hard drives by default. We think that’s dangerous. No one wants to discover that their videos weren’t backed up after their computer dies, or have to worry about the safety and durability of their data. That is why we have continued to build on our foundation of unlimited, as well as making our service faster and more accessible. All of these serve the goal of ensuring peace-of-mind for our customers.

3 Months Free For You & A Friend

As part of our commitment to unlimited, refer your friends to receive three months of Backblaze service through March 15, 2018. When you Refer-a-Friend with your personal referral link, and they subscribe, both of you will receive three months of service added to your account. See promotion details on our Refer-a-Friend page.

Want A Reminder When Your Carbonite Subscription Runs Out?

If you’re considering switching from Carbonite, we’d love to be your new backup provider. Enter your email and the date you’d like to be reminded in the form below and you’ll get a friendly reminder email from us to start a new backup plan with Backblaze. Or, you could start a free trial today.

We think you’ll be glad you switched, and you’ll have a chance to experience some of that Backblaze peace-of-mind for your data.

Please Send Me a Reminder When I Need a New Backup Provider



 

The post All-In on Unlimited Backup appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Rightscorp: Revenue From Piracy Settlements Down 48% in 2017

Post Syndicated from Andy original https://torrentfreak.com/rightscorp-revenue-from-piracy-settlements-down-48-in-2017-171125/

For the past several years, anti-piracy outfit Rightscorp has been trying to turn piracy into profit. The company monitors BitTorrent networks, captures IP addresses, then attempts to force ISPs to forward cash settlement demands to its subscribers.

Unlike other companies operating in the same area, Rightscorp has adopted a “speeding fine” type model, where it asks for $20 to $30 to make a supposed lawsuit go away, instead of the many hundreds demanded by its rivals. To date, this has resulted in the company closing more than 230,000 cases of infringement.

But despite the high numbers, the company doesn’t seem to be able to make it pay. Rightscorp’s latest set of financial results covering the three months ended September 30, 2017, show how bad things have got on the settlement front.

During the period in question, Rightscorp generated copyright settlement revenues of $45,848, an average of just $15,282 per month. That represents a decrease of 67% when compared to the $139,834 generated during the same period in 2016.

When looking at settlement revenues year to date, Rightscorp generated $184,362 in 2017, a decrease of 48% when compared to $354,160 generated during the same nine-month period in 2016.

But as bleak as these figures are, things get much worse. Out of these top-line revenues, Rightscorp has to deal with a whole bunch of costs before it can put anything into its own pockets. For example, in exchange for the right to pursue pirates, Rightscorp agrees to pay around 50% of everything it generates from settlements back to copyright holders.

So, for the past three months when it collected $45,848 from BitTorrent users, it must pay out $22,924 to copyright holders. Last year, in the same period, it paid them $69,143. For the year to date (nine months ended September 30, 2017), the company paid $92,181 to copyright holders, that’s versus $174,878 for the same period last year.

Whichever way you slice it, Rightscorp settlement model appears to be failing. With revenues from settlements down by almost half thus far this year, one has to question where this is all going, especially with BitTorrent piracy volumes continuing to fall in favor of other less traceable methods such as streaming.

However, Rightscorp does have a trick up its sleeve that is helping to keep the company afloat. As previously reported, the company has amassed a lot of intelligence on pirate activity which clearly has some value to copyright holders.

That data is currently being utilized by both BMG and the RIAA, who are using it as evidence in copyright liability lawsuits filed against ISPs Cox and Grande Communications, where each stand accused of failing to disconnect repeat infringers.

This selling of ‘pirate’ data is listed by Rightscorp in its financial reports as “consulting services” and thus far at least, it’s proving to be a crucial source of income.

“During the three months ended September 30, 2017, we generated revenues of $76,666 from consulting services rendered under service arrangements with prominent trade organizations,” Rightscorp reports.

“Under the agreements, the Company is providing certain data and consultation regarding copyright infringements on such organizations’ respective properties. During the three months ended September 30, 2016, we had no consulting services revenue.”

Year to date, the numbers begin to add up. In the nine months ended September 30, 2017, Rightscorp generated revenues of $224,998 from this facet of their business, that’s versus zero revenue in 2016.

It’s clear that without this “consulting” revenue, Rightscorp would be in an even worse situation than it is today. In fact, it appears that these services, provided to the likes of the RIAA, are now preventing the company from falling into the abyss. All that being said, there’s no guarantee that won’t happen anyway.

To the nine months ended September 30, 2017, Rightscorp recorded a net loss of $1,448,899, which is even more than the $1,380,698 it lost during the same period last year. As a result, the company had just $3,147 left in cash at the end of September. That crisis was eased by issuing 2.5 million shares to an investor for a purchase price just $50,000. But to keep going, Rightscorp will need more money – much more.

“Management believes that the Company will need an additional $250,000 to $500,000 in 2017 to fund operations based on our current operating plans,” it reports, noting that there is “substantial doubt” whether Rightscorp can continue as a going concern.

But despite all the bad news, Rightscorp manages to survive and at least in the short-term, the piracy data it has amassed holds value, beyond basic cash settlement letters. The question is, for how long?

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN discounts, offers and coupons

Event-Driven Computing with Amazon SNS and AWS Compute, Storage, Database, and Networking Services

Post Syndicated from Christie Gifrin original https://aws.amazon.com/blogs/compute/event-driven-computing-with-amazon-sns-compute-storage-database-and-networking-services/

Contributed by Otavio Ferreira, Manager, Software Development, AWS Messaging

Like other developers around the world, you may be tackling increasingly complex business problems. A key success factor, in that case, is the ability to break down a large project scope into smaller, more manageable components. A service-oriented architecture guides you toward designing systems as a collection of loosely coupled, independently scaled, and highly reusable services. Microservices take this even further. To improve performance and scalability, they promote fine-grained interfaces and lightweight protocols.

However, the communication among isolated microservices can be challenging. Services are often deployed onto independent servers and don’t share any compute or storage resources. Also, you should avoid hard dependencies among microservices, to preserve maintainability and reusability.

If you apply the pub/sub design pattern, you can effortlessly decouple and independently scale out your microservices and serverless architectures. A pub/sub messaging service, such as Amazon SNS, promotes event-driven computing that statically decouples event publishers from subscribers, while dynamically allowing for the exchange of messages between them. An event-driven architecture also introduces the responsiveness needed to deal with complex problems, which are often unpredictable and asynchronous.

What is event-driven computing?

Given the context of microservices, event-driven computing is a model in which subscriber services automatically perform work in response to events triggered by publisher services. This paradigm can be applied to automate workflows while decoupling the services that collectively and independently work to fulfil these workflows. Amazon SNS is an event-driven computing hub, in the AWS Cloud, that has native integration with several AWS publisher and subscriber services.

Which AWS services publish events to SNS natively?

Several AWS services have been integrated as SNS publishers and, therefore, can natively trigger event-driven computing for a variety of use cases. In this post, I specifically cover AWS compute, storage, database, and networking services, as depicted below.

Compute services

  • Auto Scaling: Helps you ensure that you have the correct number of Amazon EC2 instances available to handle the load for your application. You can configure Auto Scaling lifecycle hooks to trigger events, as Auto Scaling resizes your EC2 cluster.As an example, you may want to warm up the local cache store on newly launched EC2 instances, and also download log files from other EC2 instances that are about to be terminated. To make this happen, set an SNS topic as your Auto Scaling group’s notification target, then subscribe two Lambda functions to this SNS topic. The first function is responsible for handling scale-out events (to warm up cache upon provisioning), whereas the second is in charge of handling scale-in events (to download logs upon termination).

  • AWS Elastic Beanstalk: An easy-to-use service for deploying and scaling web applications and web services developed in a number of programming languages. You can configure event notifications for your Elastic Beanstalk environment so that notable events can be automatically published to an SNS topic, then pushed to topic subscribers.As an example, you may use this event-driven architecture to coordinate your continuous integration pipeline (such as Jenkins CI). That way, whenever an environment is created, Elastic Beanstalk publishes this event to an SNS topic, which triggers a subscribing Lambda function, which then kicks off a CI job against your newly created Elastic Beanstalk environment.

  • Elastic Load Balancing: Automatically distributes incoming application traffic across Amazon EC2 instances, containers, or other resources identified by IP addresses.You can configure CloudWatch alarms on Elastic Load Balancing metrics, to automate the handling of events derived from Classic Load Balancers. As an example, you may leverage this event-driven design to automate latency profiling in an Amazon ECS cluster behind a Classic Load Balancer. In this example, whenever your ECS cluster breaches your load balancer latency threshold, an event is posted by CloudWatch to an SNS topic, which then triggers a subscribing Lambda function. This function runs a task on your ECS cluster to trigger a latency profiling tool, hosted on the cluster itself. This can enhance your latency troubleshooting exercise by making it timely.

Storage services

  • Amazon S3: Object storage built to store and retrieve any amount of data.You can enable S3 event notifications, and automatically get them posted to SNS topics, to automate a variety of workflows. For instance, imagine that you have an S3 bucket to store incoming resumes from candidates, and a fleet of EC2 instances to encode these resumes from their original format (such as Word or text) into a portable format (such as PDF).In this example, whenever new files are uploaded to your input bucket, S3 publishes these events to an SNS topic, which in turn pushes these messages into subscribing SQS queues. Then, encoding workers running on EC2 instances poll these messages from the SQS queues; retrieve the original files from the input S3 bucket; encode them into PDF; and finally store them in an output S3 bucket.

  • Amazon EFS: Provides simple and scalable file storage, for use with Amazon EC2 instances, in the AWS Cloud.You can configure CloudWatch alarms on EFS metrics, to automate the management of your EFS systems. For example, consider a highly parallelized genomics analysis application that runs against an EFS system. By default, this file system is instantiated on the “General Purpose” performance mode. Although this performance mode allows for lower latency, it might eventually impose a scaling bottleneck. Therefore, you may leverage an event-driven design to handle it automatically.Basically, as soon as the EFS metric “Percent I/O Limit” breaches 95%, CloudWatch could post this event to an SNS topic, which in turn would push this message into a subscribing Lambda function. This function automatically creates a new file system, this time on the “Max I/O” performance mode, then switches the genomics analysis application to this new file system. As a result, your application starts experiencing higher I/O throughput rates.

  • Amazon Glacier: A secure, durable, and low-cost cloud storage service for data archiving and long-term backup.You can set a notification configuration on an Amazon Glacier vault so that when a job completes, a message is published to an SNS topic. Retrieving an archive from Amazon Glacier is a two-step asynchronous operation, in which you first initiate a job, and then download the output after the job completes. Therefore, SNS helps you eliminate polling your Amazon Glacier vault to check whether your job has been completed, or not. As usual, you may subscribe SQS queues, Lambda functions, and HTTP endpoints to your SNS topic, to be notified when your Amazon Glacier job is done.

  • AWS Snowball: A petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data.You can leverage Snowball notifications to automate workflows related to importing data into and exporting data from AWS. More specifically, whenever your Snowball job status changes, Snowball can publish this event to an SNS topic, which in turn can broadcast the event to all its subscribers.As an example, imagine a Geographic Information System (GIS) that distributes high-resolution satellite images to users via Web browser. In this example, the GIS vendor could capture up to 80 TB of satellite images; create a Snowball job to import these files from an on-premises system to an S3 bucket; and provide an SNS topic ARN to be notified upon job status changes in Snowball. After Snowball changes the job status from “Importing” to “Completed”, Snowball publishes this event to the specified SNS topic, which delivers this message to a subscribing Lambda function, which finally creates a CloudFront web distribution for the target S3 bucket, to serve the images to end users.

Database services

  • Amazon RDS: Makes it easy to set up, operate, and scale a relational database in the cloud.RDS leverages SNS to broadcast notifications when RDS events occur. As usual, these notifications can be delivered via any protocol supported by SNS, including SQS queues, Lambda functions, and HTTP endpoints.As an example, imagine that you own a social network website that has experienced organic growth, and needs to scale its compute and database resources on demand. In this case, you could provide an SNS topic to listen to RDS DB instance events. When the “Low Storage” event is published to the topic, SNS pushes this event to a subscribing Lambda function, which in turn leverages the RDS API to increase the storage capacity allocated to your DB instance. The provisioning itself takes place within the specified DB maintenance window.

  • Amazon ElastiCache: A web service that makes it easy to deploy, operate, and scale an in-memory data store or cache in the cloud.ElastiCache can publish messages using Amazon SNS when significant events happen on your cache cluster. This feature can be used to refresh the list of servers on client machines connected to individual cache node endpoints of a cache cluster. For instance, an ecommerce website fetches product details from a cache cluster, with the goal of offloading a relational database and speeding up page load times. Ideally, you want to make sure that each web server always has an updated list of cache servers to which to connect.To automate this node discovery process, you can get your ElastiCache cluster to publish events to an SNS topic. Thus, when ElastiCache event “AddCacheNodeComplete” is published, your topic then pushes this event to all subscribing HTTP endpoints that serve your ecommerce website, so that these HTTP servers can update their list of cache nodes.

  • Amazon Redshift: A fully managed data warehouse that makes it simple to analyze data using standard SQL and BI (Business Intelligence) tools.Amazon Redshift uses SNS to broadcast relevant events so that data warehouse workflows can be automated. As an example, imagine a news website that sends clickstream data to a Kinesis Firehose stream, which then loads the data into Amazon Redshift, so that popular news and reading preferences might be surfaced on a BI tool. At some point though, this Amazon Redshift cluster might need to be resized, and the cluster enters a ready-only mode. Hence, this Amazon Redshift event is published to an SNS topic, which delivers this event to a subscribing Lambda function, which finally deletes the corresponding Kinesis Firehose delivery stream, so that clickstream data uploads can be put on hold.At a later point, after Amazon Redshift publishes the event that the maintenance window has been closed, SNS notifies a subscribing Lambda function accordingly, so that this function can re-create the Kinesis Firehose delivery stream, and resume clickstream data uploads to Amazon Redshift.

  • AWS DMS: Helps you migrate databases to AWS quickly and securely. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database.DMS also uses SNS to provide notifications when DMS events occur, which can automate database migration workflows. As an example, you might create data replication tasks to migrate an on-premises MS SQL database, composed of multiple tables, to MySQL. Thus, if replication tasks fail due to incompatible data encoding in the source tables, these events can be published to an SNS topic, which can push these messages into a subscribing SQS queue. Then, encoders running on EC2 can poll these messages from the SQS queue, encode the source tables into a compatible character set, and restart the corresponding replication tasks in DMS. This is an event-driven approach to a self-healing database migration process.

Networking services

  • Amazon Route 53: A highly available and scalable cloud-based DNS (Domain Name System). Route 53 health checks monitor the health and performance of your web applications, web servers, and other resources.You can set CloudWatch alarms and get automated Amazon SNS notifications when the status of your Route 53 health check changes. As an example, imagine an online payment gateway that reports the health of its platform to merchants worldwide, via a status page. This page is hosted on EC2 and fetches platform health data from DynamoDB. In this case, you could configure a CloudWatch alarm for your Route 53 health check, so that when the alarm threshold is breached, and the payment gateway is no longer considered healthy, then CloudWatch publishes this event to an SNS topic, which pushes this message to a subscribing Lambda function, which finally updates the DynamoDB table that populates the status page. This event-driven approach avoids any kind of manual update to the status page visited by merchants.

  • AWS Direct Connect (AWS DX): Makes it easy to establish a dedicated network connection from your premises to AWS, which can reduce your network costs, increase bandwidth throughput, and provide a more consistent network experience than Internet-based connections.You can monitor physical DX connections using CloudWatch alarms, and send SNS messages when alarms change their status. As an example, when a DX connection state shifts to 0 (zero), indicating that the connection is down, this event can be published to an SNS topic, which can fan out this message to impacted servers through HTTP endpoints, so that they might reroute their traffic through a different connection instead. This is an event-driven approach to connectivity resilience.

More event-driven computing on AWS

In addition to SNS, event-driven computing is also addressed by Amazon CloudWatch Events, which delivers a near real-time stream of system events that describe changes in AWS resources. With CloudWatch Events, you can route each event type to one or more targets, including:

Many AWS services publish events to CloudWatch. As an example, you can get CloudWatch Events to capture events on your ETL (Extract, Transform, Load) jobs running on AWS Glue and push failed ones to an SQS queue, so that you can retry them later.

Conclusion

Amazon SNS is a pub/sub messaging service that can be used as an event-driven computing hub to AWS customers worldwide. By capturing events natively triggered by AWS services, such as EC2, S3 and RDS, you can automate and optimize all kinds of workflows, namely scaling, testing, encoding, profiling, broadcasting, discovery, failover, and much more. Business use cases presented in this post ranged from recruiting websites, to scientific research, geographic systems, social networks, retail websites, and news portals.

Start now by visiting Amazon SNS in the AWS Management Console, or by trying the AWS 10-Minute Tutorial, Send Fan-out Event Notifications with Amazon SNS and Amazon SQS.

 

Six Strikes Piracy Scheme May Be Dead But Those Warnings Keep on Coming

Post Syndicated from Andy original https://torrentfreak.com/six-strikes-piracy-scheme-may-be-dead-but-those-warnings-keep-on-coming-171001/

After at least 15 years of Internet pirates being monitored by copyright holders, one might think that the message would’ve sunk in by now. For many, it definitely hasn’t.

Bottom line: when people use P2P networks and protocols (such as BitTorrent) to share files including movies and music, copyright holders are often right there, taking notes about what is going on, perhaps in preparation for further action.

That can take a couple of forms, including suing users or, more probably, firing off a warning notice to their Internet service providers. Those notices are a little like a speeding ticket, telling the subscriber off for sharing copyrighted material but letting them off the hook if they promise to be good in future.

In 2013, the warning notice process in the US was formalized into what was known as the Copyright Alert System, a program through which most Internet users could receive at least six piracy warning notices without having any serious action taken against them. In January 2017, without having made much visible progress, it was shut down.

In some corners of the web there are still users under the impression that since the “six strikes” scheme has been shut down, all of a sudden US Internet users can forget about receiving a warning notice. In reality, the complete opposite is true.

While it’s impossible to put figures on how many notices get sent out (ISPs are reluctant to share the data), monitoring of various piracy-focused sites and forums indicates that plenty of notices are still being sent to ISPs, who are cheerfully sending them on to subscribers.

Also, over the past couple of months, there appears to have been an uptick in subscribers seeking advice after receiving warnings. Many report basic notices but there seems to be a bit of a trend of Internet connections being suspended or otherwise interrupted, apparently as a result of an infringement notice being received.

“So, over the weekend my internet got interrupted by my ISP (internet service provider) stating that someone on my network has violated some copyright laws. I had to complete a survey and they brought back the internet to me,” one subscriber wrote a few weeks ago. He added that his (unnamed) ISP advised him that seven warnings would get his account disconnected.

Another user, who named his ISP as Comcast, reported receiving a notice after downloading a game using BitTorrent. He was warned that the alleged infringement “may result in the suspension or termination of your Service account” but what remains unclear is how many warnings people can receive before this happens.

For example, a separate report from another Comcast user stated that one night of careless torrenting led to his mother receiving 40 copyright infringement notices the next day. He didn’t state which company the notices came from but 40 is clearly a lot in such a short space of time. That being said and as far as the report went, it didn’t lead to a suspension.

Of course, it’s possible that Comcast doesn’t take action if a single company sends many notices relating to the same content in a small time frame (Rightscorp is known to do this) but the risk is still there. Verizon, it seems, can suspend accounts quite easily.

“So lately I’ve been getting more and more annoyed with pirating because I get blasted with a webpage telling me my internet is disconnected and that I need to delete the file to reconnect, with the latest one having me actually call Verizon to reconnect,” a subscriber to the service reported earlier this month.

A few days ago, a Time Warner Cable customer reported having to take action after receiving his third warning notice from the ISP.

“So I’ve gotten three notices and after the third one I just went online to my computer and TWC had this page up that told me to stop downloading illegally and I had to click an ‘acknowledge’ button at the bottom of the page to be able to continue to use my internet,” he said.

Also posting this week, another subscriber of an unnamed ISP revealed he’d been disconnected twice in the past year. His comments raise a few questions that keep on coming up in these conversations.

“The first time [I was disconnected] was about a year ago and the next was a few weeks ago. When it happened I was downloading some fairly new movies so I was wondering if they monitor these new movie releases since they are more popular. Also are they monitoring what I am doing since I have been caught?” he asked.

While there is plenty of evidence to suggest that old content is also monitored, there’s little doubt that the fresher the content, the more likely it is to be monitored by copyright holders. If people are downloading a brand new movie, they should expect it to be monitored by someone, somewhere.

The second point, about whether risk increases after being caught already, is an interesting one, for a number of reasons.

Following the BMG v Cox Communication case, there is now a big emphasis on ISPs’ responsibility towards dealing with subscribers who are alleged to be repeat infringers. Anti-piracy outfit Rightscorp was deeply involved in that case and the company has a patent for detecting repeat infringers.

It’s becoming clear that the company actively targets such people in order to assist copyright holders (which now includes the RIAA) in strategic litigation against ISPs, such as Grande Communications, who are claimed to be going soft on repeat infringers.

Overall, however, there’s no evidence that “getting caught” once increases the chances of being caught again, but subscribers should be aware that the Cox case changed the position on the ground. If anecdotal evidence is anything to go by, it now seems that ISPs are tightening the leash on suspected pirates and are more likely to suspend or disconnect them in the face of repeated complaints.

The final question asked by the subscriber who was disconnected twice is a common one among people receiving notices.

“What can I do to continue what we all love doing?” he asked.

Time and time again, on sites like Reddit and other platforms attracting sharers, the response is the same.

“Get a paid VPN. I’m amazed you kept torrenting without protection after having your internet shut off, especially when downloading recent movies,” one such response reads.

Nevertheless, this still fails to help some people fully understand the notices they receive, leaving them worried about what might happen after receiving one. However, the answer is nearly always straightforward.

If the notice says “stop sharing content X”, then recipients should do so, period. And, if the notice doesn’t mention specific legal action, then it’s almost certain that no action is underway. They are called warning notices for a reason.

Also, notice recipients should consider the part where their ISP assures them that their details haven’t been shared with third parties. That is the truth and will remain that way unless subscribers keep ignoring notices. Then there’s a slim chance that a rightsholder will step in to make a noise via a lawyer. At that point, people shouldn’t say they haven’t been warned.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Backblaze Cloud Backup 5.0: The Rapid Access Release

Post Syndicated from Yev original https://www.backblaze.com/blog/cloud-backup-5-0-rapid-access/

Announcing Backblaze Cloud Backup 5.0: the Rapid Access Release. We’ve been at the backup game for a long time now, and we continue to focus on providing the best unlimited backup service on the planet. A lot of the features in this release have come from listening to our customers about how they want to use their data. “Rapid Access” quickly became the theme because, well, we’re all acquiring more and more data and want to access it in a myriad of ways.

This release brings a lot of new functionality to Backblaze Computer Backup: faster backups, accelerated file browsing, image preview, individual file download (without creating a “restore”), and file sharing. To top it all off, we’ve refreshed the user interface on our client app. We hope you like it!

Speeding Things Up

New code + new hardware + elbow grease = things are going to move much faster.

Faster Backups

We’ve doubled the number of threads available for backup on both Mac and PC . This gives our service the ability to intelligently detect the right settings for you (based on your computer, capacity, and bandwidth). As always, you can manually set the number of threads — keep in mind that if you have a slow internet connection, adding threads might have the opposite effect and slow you down. On its default settings, our client app will now automatically evaluate what’s best given your environment. We’ve internally tested our service backing up at over 100 Mbps, which means if you have a fast-enough internet connection, you could back up 50 GB in just one hour.

Faster Browsing

We’ve introduced a number of enhancements that increase file browsing speed by 3x. Hidden files are no longer displayed by default, but you can still show them with one click on the restore page. This gives the restore interface a cleaner look, and helps you navigate backup history if you need to roll back time.

Faster Restore Preparation

We take pride in providing a variety of ways for consumers to get their data back. When something has happened to your computer, getting your files back quickly is critical. Both web download restores and Restore by Mail will now be much faster. In some cases up to 10x faster!

Preview — Access — Share

Our system has received a number of enhancements — all intended to give you more access to your data.

Image Preview

If you have a lot of photos, this one’s for you. When you go to the restore page you’ll now be able to click on each individual file that we have backed up, and if it’s an image you’ll see a preview of that file. We hope this helps people figure out which pictures they want to download (this especially helps people with a lot of photos named something along the lines of: 2017-04-20-9783-41241.jpg). Now you can just click on the picture to preview it.

Access

Once you’ve clicked on a file (30MB and smaller), you’ll be able to individually download that file directly in your browser. You’ll no longer need to wait for a single-file restore to be built and zipped up; you’ll be able to download it quickly and easily. This was a highly requested feature and we’re stoked to get it implemented.

Share

We’re leveraging Backblaze B2 Cloud Storage and giving folks the ability to publicly share their files. In order to use this feature, you’ll need to enable Backblaze B2 on your account (if you haven’t already, there’s a simple wizard that will pop up the first time you try to share a file). Files can be shared anywhere in the world via URL. All B2 accounts have 10GB/month of storage and 1GB/day of downloads (equivalent to sharing an iPhone photo 1,000 times per month) for free. You can increase those limits in your B2 Settings. Keep in mind that any file you share will be accessible to anybody with the link. Learn more about File Sharing.

For now, we’ve limited the Preview/Access/Share functionality to files 30MB and smaller, but larger files will be supported in the coming weeks!

Other Goodies

In addition to adding 2FV via ToTP, we’ve also been hard at work on the client. In version 5.0 we’ve touched up the user interface to make it a bit more lively, and we’ve also made the client IPv6 compatible.

Backblaze 5.0 Available: August 10, 2017

We will slowly be auto-updating all users in the coming weeks. To update now:

This version is now the default download on www.backblaze.com.

We hope you enjoy Backblaze Cloud Backup v5.0!

The post Backblaze Cloud Backup 5.0: The Rapid Access Release appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Launch: Amazon ElastiCache Launches Enhanced Redis Backup and Restore with Cluster Resizing

Post Syndicated from Tara Walker original https://aws.amazon.com/blogs/aws/launch-amazon-elasticache-launches-enhanced-redis-backup-and-restore-with-cluster-resizing/

Most of us equate in-memory caching with improved performance and lower cost at scale when designing applications or building solutions. Now if there was only a service that would continually make it simpler to deploy and utilize in-memory cache in the cloud while increasing the ability to scale.

Okay no more joking around, the cloud service that provides this great functionality is, of course, Amazon ElastiCache. Amazon ElastiCache is an AWS managed service that provides a performant in-memory data store or cache in the cloud while offering a straightforward way to create, scale, and manage a distributed environment for low-latency, secure, access of I/O intensive or compute heavy data. Additionally, ElastiCache reduces the overhead of managing infrastructure for your in-memory data structure server or cache by detecting and replacing failed nodes while providing enhanced visibility into key performance metrics of the caching system nodes via Amazon CloudWatch. This exciting service is now launching support for Enhanced Redis Backup and Restore with Cluster Resizing.

For those of you familiar with Amazon ElastiCache, you are likely aware that ElastiCache currently supports two in-memory key-value engines:

  • Memcached: an open source, high-performing, distributed memory object caching system developed in 2003 with the initial goal of speeding up dynamic web applications by alleviating database load
  • Redis: an open source in-memory data structure store launched in 2009 developed as a broker for caching, messaging, and databases with built-in replication, atomic operation support, various levels of on-disk persistence, and high availability via Redis Cluster.

In October of 2016, support was added for Redis Cluster with Redis 3.2.4. This allowed ElastiCache Redis users to, not only take advantage of Redis clusters, but also gave users the ability to:

  • Create cluster-level backups.
  • Produce snapshots of each of the cluster’s shards contained within backups.
  • Scale their workloads with 3.5TiB of data across up to 15 shards.

You can read more about using Redis with ElastiCache and the related features by reviewing the product page for Amazon ElastiCache for Redis.

With the launch of the Enhanced Backup and Restore with Cluster Resizing feature, ElastiCache is providing even deeper support for Redis with a clear-cut migration path to a managed Redis Cluster experience. There are several benefits of this enhancement for ElastiCache and Redis users alike, such as:

  • Ability to restore backup into a Redis Cluster with a different number of shards and slot distribution
  • Deliver the capability for users to resize Redis workloads
  • Allow Redis database file (RDB) snapshots as input for creating a sharded Redis Cluster
  • Offer option to use snapshot(s) of Redis on EC2 implementations (both Redis Cluster and single-shard Redis) as data input for sharded Redis Cluster creation

To accomplish these tasks, ElastiCache will parse the Redis key space across the backup’s individual snapshots, and redistribute the keys in the new Cluster according to the requested number of shards and hash slots. You would simply take your RDB snapshots and store them on S3, then provide ElastiCache with the desired number of shards and the snapshot file. ElastiCache handles the heavy lifting of restoring the Redis data store into a Redis cluster.

I am sure that you all may be thinking; Is it really that easy to leverage the Enhanced Redis Backup and Restore with Cluster Resizing feature in ElastiCache? Well, there is no time like the present to find out. Let’s take a trip to the AWS Management Console, and put this newly launched enhancement in action by restoring an external RDB snapshot to a new cluster using ElastiCache.

My first stop in the AWS Management console is to the Amazon S3 console. I have some Redis .rdb snapshot files I received from some of my peers here at AWS in order to test the restore of an external Redis snapshot to ElastiCache. I will need to put these files into Amazon S3 so that I can access the snapshots as input for my ElastiCache Redis cluster.

In the S3 console, I will go to my S3 bucket, aws-blog-tew-posts, that I created for testing and development purposes. I’ll upload the .rdb snapshot files that were provided to me into this S3 bucket.

 

It is important to note that the name of your S3 bucket must conform to DNS standards. To be DNS-compliant, the name must be at least three characters, must contain only lowercase letters, numbers, and/or dashes, and it must start and end with a lowercase letter or number. While this may be obvious, I will also note that the bucket name cannot be in an IP address format. You can learn more about the S3 Bucket Restrictions at the link provided here.

With my .rdb files successfully uploaded into my aws-blog-tew-posts bucket, I need to take note of the S3 path to these backup files. For these files, the path would be aws-blog-tew-posts/dump_1.rdb or aws-blog-tew-posts/dump_10.rdb. If you have placed your files into a folder, the folder name would need to be included in this path, i.e. thebucketname/thefoldername/thefilename.

For ElastiCache to access these files, I need to ensure that the service has read permissions for each of the files. To provide access, I will update the permissions for each of .rdb files by assigning the Grantee as the canonical id for my region and grant the user Open/Download permissions. The canonical id for all regions, outside of China (Beijing) and AWS GovCloud (US), is as follows:

540804c33a284a299d2547575ce1010f2312ef3da9b3a053c8bc45bf233e4353

After I click the Save button, I am all set to use these files as input for an ElastiCache Redis cluster.

The next step is to go to the ElastiCache console. Here I will create a new ElastiCache Redis cluster and seed this new cluster with data from one of the RDB snapshots located in the files in my S3 bucket. I’ll choose the dump_1.rdb snapshot file to use as my data input to seed this new cluster. Since I want to explore the ElastiCache Redis capabilities added on this past October with 3.2.4 support of Redis Cluster, as well as, discuss the new Backup and Restore with Cluster Resizing enhancements, I’ll create a new Redis Cluster and ensure I have cluster mode enabled. At this point, I should note that you cannot restore from a backup created using a Redis (cluster mode enabled) cluster to a Redis (cluster mode disabled) cluster.

First, I will click the Get Started Now button from the ElastiCache console dashboard or the Create button based upon your console view.

In the Create your Amazon ElastiCache cluster dialog window, I’ll select Redis for my caching and make sure I click the checkbox for Cluster Mode enabled (Scale Out). The name of my new cluster will be, tew-rediscluster and I since I am enabling a Cluster mode, my ElastiCache Redis Engine version is 3.2.4. For this cluster, I will keep the default Redis port of 6379.

The key benefit of the ElastiCache enhanced Redis Backup and Restore feature is the cluster resizing capability that allows me to build a new cluster with a different number of shards than was originally used for the backup file. To build the new Redis Cluster, I am using only one RDB snapshot file, dump_1.rdb which is a small Redis instance backup with only one shard. However, in the creation of my new tew-rediscluster, I have opted for 3 shards with 2 replicas per shard.

In addition, I have the ability to specify a node type for my new cluster that is a different size than my original instance from the RDB snapshot. As I mentioned, the dump_1.rdb is a backup of a Redis instance that is significantly smaller than the size of the chosen node type for my tew-rediscluster shown below.

There are other options and data input needed in order to complete the creation of my ElastiCache Redis cluster that I will not show in this blog post. However, if you want to go through each of the steps necessary for creating an ElastiCache Redis cluster you can find more information in the AWS ElastiCache Getting Started documentation for Launch a Cluster.

Once I have provided all the information needed to create my ElastiCache Redis cluster, I will need to tell ElastiCache how to seed the cluster with the .rdb file by providing the file location from my S3 bucket. In the Import Data to Cluster section of the create dialog, I will enter the S3 path to my dump_1.rdb in the Seed RDB file S3 location textbox. Remember, the nomenclature for the S3 file path is Bucket/Folder/ObjectName so I will enter aws-blog-tew-posts/dump_1.rdb as the path to the RDB file in S3. All that is left now is to click the Create button.

 

That’s it! ElastiCache goes to work to creating the new Redis cluster. After a short time period, the ElastiCache console shows my new Amazon ElastiCache Redis cluster as available and I have successfully created this cluster with data restored from an external RDB snapshot file.

 

I just demonstrated how you have the capability to create an ElastiCache Redis cluster using an external RDB snapshot, but of course, you can create backups and restore from backups from your existing ElastiCache Redis clusters as well. To dig deeper into information about this newly launched feature, visit Restoring From a Backup with Cluster Resizing in the Amazon ElastiCache User Guide.

To learn more about making your applications more performant with Amazon ElastiCache, visit the AWS Amazon ElastiCache page for product details, resources, and customer testimonials.

– Tara

 

 

 

 

 

 

systemd for Administrators, Part VIII

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/the-new-configuration-files.html

Another episode of my
ongoing
series
on
systemd
for
Administrators:

The New Configuration Files

One of the formidable new features of systemd is
that it comes with a complete set of modular early-boot services that are
written in simple, fast, parallelizable and robust C, replacing the
shell “novels” the various distributions featured before. Our little
Project Zero Shell[1] has been a full success. We currently
cover pretty much everything most desktop and embedded
distributions should need, plus a big part of the server needs:

  • Checking and mounting of all file systems
  • Updating and enabling quota on all file systems
  • Setting the host name
  • Configuring the loopback network device
  • Loading the SELinux policy and relabelling /run and /dev as necessary on boot
  • Registering additional binary formats in the kernel, such as Java, Mono and WINE binaries
  • Setting the system locale
  • Setting up the console font and keyboard map
  • Creating, removing and cleaning up of temporary and volatile files and directories
  • Applying mount options from /etc/fstab to pre-mounted API VFS
  • Applying sysctl kernel settings
  • Collecting and replaying readahead information
  • Updating utmp boot and shutdown records
  • Loading and saving the random seed
  • Statically loading specific kernel modules
  • Setting up encrypted hard disks and partitions
  • Spawning automatic gettys on serial kernel consoles
  • Maintenance of Plymouth
  • Machine ID maintenance
  • Setting of the UTC distance for the system clock

On a standard Fedora 15 install, only a few legacy and storage
services still require shell scripts during early boot. If you don’t
need those, you can easily disable them end enjoy your shell-free boot
(like I do every day). The shell-less boot systemd offers you is a
unique feature on Linux.

Many of these small components are configured via configuration
files in /etc. Some of these are fairly standardized among
distributions and hence supporting them in the C implementations was
easy and obvious. Examples include: /etc/fstab,
/etc/crypttab or /etc/sysctl.conf. However, for
others no standardized file or directory existed which forced us to add
#ifdef orgies to our sources to deal with the different
places the distributions we want to support store these things. All
these configuration files have in common that they are dead-simple and
there is simply no good reason for distributions to distuingish
themselves with them: they all do the very same thing, just
a bit differently.

To improve the situation and benefit from the unifying force that
systemd is we thus decided to read the per-distribution configuration
files only as fallbacks — and to introduce new configuration
files as primary source of configuration wherever applicable. Of
course, where possible these standardized configuration files should
not be new inventions but rather just standardizations of the best
distribution-specific configuration files previously used. Here’s a
little overview over these new common configuration files systemd
supports on all distributions:

  • /etc/hostname:
    the host name for the system. One of the most basic and trivial
    system settings. Nonetheless previously all distributions used
    different files for this. Fedora used /etc/sysconfig/network,
    OpenSUSE /etc/HOSTNAME. We chose to standardize on the
    Debian configuration file /etc/hostname.
  • /etc/vconsole.conf:
    configuration of the default keyboard mapping and console font.
  • /etc/locale.conf:
    configuration of the system-wide locale.
  • /etc/modules-load.d/*.conf:
    a drop-in directory for kernel modules to statically load at
    boot (for the very few that still need this).
  • /etc/sysctl.d/*.conf:
    a drop-in directory for kernel sysctl parameters, extending what you
    can already do with /etc/sysctl.conf.
  • /etc/tmpfiles.d/*.conf:
    a drop-in directory for configuration of runtime files that need to be
    removed/created/cleaned up at boot and during uptime.
  • /etc/binfmt.d/*.conf:
    a drop-in directory for registration of additional binary formats for
    systems like Java, Mono and WINE.
  • /etc/os-release:
    a standardization of the various distribution ID files like
    /etc/fedora-release and similar. Really every distribution
    introduced their own file here; writing a simple tool that just prints
    out the name of the local distribution usually means including a
    database of release files to check. The LSB tried to standardize
    something like this with the lsb_release
    tool, but quite frankly the idea of employing a shell script in this
    is not the best choice the LSB folks ever made. To rectify this we
    just decided to generalize this, so that everybody can use the same
    file here.
  • /etc/machine-id:
    a machine ID file, superseding D-Bus’ machine ID file. This file is
    guaranteed to be existing and valid on a systemd system, covering also
    stateless boots. By moving this out of the D-Bus logic it is hopefully
    interesting for a lot of additional uses as a unique and stable
    machine identifier.
  • /etc/machine-info:
    a new information file encoding meta data about a host, like a pretty
    host name and an icon name, replacing stuff like
    /etc/favicon.png and suchlike. This is maintained by systemd-hostnamed.

It is our definite intention to convince you to use these new
configuration files in your configuration tools: if your
configuration frontend writes these files instead of the old ones, it
automatically becomes more portable between Linux distributions, and
you are helping standardizing Linux. This makes things simpler to
understand and more obvious for users and administrators. Of course,
right now, only systemd-based distributions read these files, but that
already covers all important distributions in one way or another, except for one. And it’s a bit of a
chicken-and-egg problem: a standard becomes a standard by being
used. In order to gently push everybody to standardize on these files
we also want to make clear that sooner or later we plan to drop the
fallback support for the old configuration files from
systemd. That means adoption of this new scheme can happen slowly and piece
by piece. But the final goal of only having one set of configuration
files must be clear.

Many of these configuration files are relevant not only for
configuration tools but also (and sometimes even primarily) in
upstream projects. For example, we invite projects like Mono, Java, or
WINE to install a drop-in file in /etc/binfmt.d/ from their
upstream build systems. Per-distribution downstream support for binary
formats would then no longer be necessary and your platform would work
the same on all distributions. Something similar applies to all
software which need creation/cleaning of certain runtime files and
directories at boot, for example beneath the /run hierarchy
(i.e. /var/run as it used to be known). These
projects should just drop in configuration files in
/etc/tmpfiles.d, also from the upstream build systems. This
also helps speeding up the boot process, as separate per-project SysV
shell scripts which implement trivial things like registering a binary
format or removing/creating temporary/volatile files at boot are no
longer necessary. Or another example, where upstream support would be
fantastic: projects like X11 could probably benefit from reading the
default keyboard mapping for its displays from
/etc/vconsole.conf.

Of course, I have no doubt that not everybody is happy with our
choice of names (and formats) for these configuration files. In the
end we had to pick something, and from all the choices these appeared
to be the most convincing. The file formats are as simple as they can
be, and usually easily written and read even from shell scripts. That
said, /etc/bikeshed.conf could of course also have been a
fantastic configuration file name!

So, help us standardizing Linux! Use the new configuration files!
Adopt them upstream, adopt them downstream, adopt them all across the
distributions!

Oh, and in case you are wondering: yes, all of these files were
discussed in one way or another with various folks from the various
distributions. And there has even been some push towards supporting
some of these files even outside of systemd systems.

Footnotes

[1] Our slogan: “The only shell that should get started
during boot is gnome-shell!
” — Yes, the slogan needs a bit of
work, but you get the idea.

systemd Status Update

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/systemd-update.html

It has been a while since my original
announcement of systemd
. Here’s a little status update, on what
happened since then. For simplicity’s sake I’ll just list here what we
worked on in a bulleted list, with no particular order and without
trying to cover this comprehensively:

  • systemd has been accepted as Feature for Fedora 14, and as it
    looks right now everything worked out nicely and we’ll ship F14 with
    systemd as init system.
  • We added a number of additional unit types: .timer for
    cron-style timer-based activation of services, .swap exposes
    swap files and partitions the same way we handle mount points, and
    .path can be used to activate units dependending on the
    existance/creation of files or fill status of spool directories.
  • We hooked systemd up to SELinux: systemd is now capabale of
    properly labelling directories, sockets and FIFOs it creates according
    to the SELinux policy for the services we maintain.
  • We hooked systemd up to the Linux auditing subsystem: as first
    init system at all systemd now generates auditing records for all
    services it starts/stops, including their failure status.
  • We hooked systemd up to TCP wrappers, for all socket connections
    it accepts.
  • We hooked systemd up to PAM, so that optionally, when systemd runs
    a service as a different user it initializes the usual PAM session
    setup and teardown hooks.
  • We hooked systemd up to D-Bus, so that D-Bus passes activation
    requests to systemd and systemd becomes the central point for all
    kinds of activation, thus greatly extending the control of the
    execution environment of bus activated services, and making them
    accessible through the same utilities as SysV services. Also, this
    enables us to do race-free parallelized start-up for D-Bus services
    and their clients, thus speeding up things even further.
  • systemd is now able to handle various Debian and OpenSUSE-specific
    extensions to the classic SysV init script formats natively, on top of
    the Fedora extensions we already parse.
  • The D-Bus coverage of the systemd interface is now complete,
    allowing both introspection of runtime data and of parsed
    configuration data. It’s fun now to introspect systemd with gdbus
    or d-feet.
  • We added a systemd
    PAM module
    , which assigns the processes of each user session to
    its own cgroup in the systemd cgroup tree. This also enables reliable
    killing of all processes associated with a session when the user logs
    out. This also manages a secure per-user /var/run-style directory
    which is supposed to be used for sockets and similar files that shall
    be cleaned up when the user logs out.
  • There’s a new tool systemd-cgls,
    which plots a pretty process tree based on the systemd cgroup
    hierarchy. It’s really pretty. Try it!
  • We now have our own cgroup hierarchy beneath
    /cgroup/systemd (though is will move to /sys/fs/
    before the F14 release).
  • We have pretty code that automatically spawns a getty on a serial
    port when the kernel console is redirected to a serial TTY.
  • systemctl got beefed up substantially (it can even draw
    dependency graphs now, via dot!), and the SysV compatiblity
    tools were extended to more completely and correctly support what was
    historically provided by SysV. For example, we’ll now warn the user
    when systemd service files have changed but systemd was not asked to
    reload its configuration. Also, you can now use systemd’s native
    client tools to reboot or shut-down an Upstart or sysvinit system, to
    facilitate upgrades.
  • We provide a reference
    implementation
    for the socket activation and other APIs for nicer
    interaction with systemd.
  • We have a pretty complete set of documentation
    now, some
    of it
    even extending to areas not directly related to systemd
    itself.
  • Quite a number of upstream packages now ship with systemd service
    files out-of-the-box now, that work across all distributions that have
    adopted systemd. It is our intention to unify the boot and service
    management between distributions with systemd, and this shows fruits
    already. Furthermore a number of upstream packages now ship our
    patches for socket-based activation.
  • Even more options that control the process execution environment
    or the sockets we create are now supported.
  • Earlier today I began my series of blog stories on systemd
    for administrators
    .
  • We reimplemented almost all boot-up and shutdown scripts of the
    standard Fedora install in much smaller, simpler and faster C
    utilities, or in systemd itself. Most of this will not be enabled in
    F14 however, even though it is shipped with systemd upstream. With
    this enabled the entire Linux system gains a completely new feeling as
    the number of shells we spawn approaches zero, and the PID of the
    first user terminal is way < 500 now, and the early boot-up is
    fully parallelized. We looked at the boot scripts of Fedora, OpenSUSE
    and Debian and distilled from this a list of functionality that makes
    up the early boot process and reimplemented this in C, if possible
    following the bahaviour of one of the existing implementations from
    these three distributions. This turned out to be much less effort than
    anticipated, and we are actually quite excited about this. Look
    forward to the fruits of this work in F15, when we might be able to
    present you a shell-less boot at least for standard desktop/laptop
    systems.
  • We spent some time reinvestigating the current syslog logic, and
    came up with an elegant and simple scheme to provide /dev/log
    compatible logging right from the time systemd is first initialized
    right until the time the kernel halts the machine. Through the wonders
    of socket based activation we first connect the /dev/log
    socket with a minimal bridge to the kernel log buffer (kmsg)
    and then, as soon as the real syslog is started up as part of the
    later bootup phase, we dynamically replace this minimal bridge by the
    real syslog daemon — without losing a single log message. Since one
    of the first things the real syslog daemon does is flushing the kernel
    log buffer into log files, all logged messages will sooner or later be
    stored on disk, regardless whether they have been generated during
    early boot, late boot or system runtime. On top of that if the syslog
    daemon terminates or is shut down during runtime, the bridge becomes
    active again and log output is written to kmsg again. The same applies
    when the system goes down. This provides a simple an robust way how we
    can ensure that no logs will ever be lost again, and logging is
    available from the beginning of boot-up to the end of
    shut-down. Plymouth will most likely adopt a similar scheme for initrd
    logging, thus ensuring that everything ever logged on the system will
    properly end up in the log files, whether it comes from the kernel,
    from the initrd, from early-boot, from runtime or shutdown. And if
    syslogd is not around, dmesg will provide you with access to
    the log messages. While this bridge is part of systemd upstream, we’ll
    most likely enable this bridge in Fedora only starting with F15. Also
    note that embedded systems that have no interest in shipping a full
    syslogd solution can simply use this syslog bridge during the entire
    runtime, and thus making the kernel log buffer the centralized log
    storage, with all the advantages this offers: zero disk IO at runtime,
    access to serial and netconsole logging, and remote debug access to
    the kernel log buffer.
  • We now install autofs units for many “API” kernel virtual file
    systems by default, such as binfmt_misc or
    hugetlbfs. That means that the file system access is readily
    available, client code no longer has to manually load the respective
    kernel modules, as they are autoloaded on first access of the file
    system. This has many advantages: it is not only faster to set up
    during boot, but also simpler for applications, as they can just
    assume the functionality is available. On top of that permission
    problems for the initialization go away, since manual module loading
    requires root privileges.
  • Many smaller fixes and enhancements, all across the board, which
    if mentioned here would make this blog story another blog
    novel. Suffice to say, we did a lot of polishing to ready systemd for
    F14.

All in all, systemd is progressing nicely, and the features we have
been working on in the last months are without exception features not
existing in any other of the init systems available on Linux and our
feature set already was far ahead of what the older init
implementations provide. And we have quite a bit planned for the
future. So, stay tuned!

Also note that I’ll speak about systemd at LinuxKongress
2010
in Nuremberg, Germany. Later this year I’ll also be speaking
at the Linux
Plumbers Conference
in Boston, MA. Make sure to drop by if you
want to learn about systemd or discuss exiciting new ideas or features
with us.

systemd for Administrators, Part 1

Post Syndicated from Lennart Poettering original http://0pointer.net/blog/projects/systemd-for-admins-1.html

As many of you know, systemd is the new
Fedora init system, starting with F14, and it is also on its way to being adopted in
a number of other distributions as well (for example, OpenSUSE). For administrators
systemd provides a variety of new features and changes and enhances the
administrative process substantially. This blog story is the first part of a
series of articles I plan to post roughly every week for the next months. In
every post I will try to explain one new feature of systemd. Many of these features
are small and simple, so these stories should be interesting to a broader audience.
However, from time to time we’ll dive a little bit deeper into the great new
features systemd provides you with.

Verifying Bootup

Traditionally, when booting up a Linux system, you see a lot of
little messages passing by on your screen. As we work on speeding up
and parallelizing the boot process these messages are becoming visible
for a shorter and shorter time only and be less and less readable —
if they are shown at all, given we use graphical boot splash
technology like Plymouth these days. Nonetheless the information of
the boot screens was and still is very relevant, because it shows you
for each service that is being started as part of bootup, wether it
managed to start up successfully or failed (with those green or red
[ OK ] or [ FAILED ] indicators). To improve the
situation for machines that boot up fast and parallelized and to make
this information more nicely available during runtime, we added a
feature to systemd that tracks and remembers for each service whether
it started up successfully, whether it exited with a non-zero exit
code, whether it timed out, or whether it terminated abnormally (by
segfaulting or similar), both during start-up and runtime. By simply
typing systemctl in your shell you can query the state of all
services, both systemd native and SysV/LSB services:

[[email protected]] ~# systemctl
UNIT                                          LOAD   ACTIVE       SUB          JOB             DESCRIPTION
dev-hugepages.automount                       loaded active       running                      Huge Pages File System Automount Point
dev-mqueue.automount                          loaded active       running                      POSIX Message Queue File System Automount Point
proc-sys-fs-binfmt_misc.automount             loaded active       waiting                      Arbitrary Executable File Formats File System Automount Point
sys-kernel-debug.automount                    loaded active       waiting                      Debug File System Automount Point
sys-kernel-security.automount                 loaded active       waiting                      Security File System Automount Point
sys-devices-pc...0000:02:00.0-net-eth0.device loaded active       plugged                      82573L Gigabit Ethernet Controller
[...]
sys-devices-virtual-tty-tty9.device           loaded active       plugged                      /sys/devices/virtual/tty/tty9
-.mount                                       loaded active       mounted                      /
boot.mount                                    loaded active       mounted                      /boot
dev-hugepages.mount                           loaded active       mounted                      Huge Pages File System
dev-mqueue.mount                              loaded active       mounted                      POSIX Message Queue File System
home.mount                                    loaded active       mounted                      /home
proc-sys-fs-binfmt_misc.mount                 loaded active       mounted                      Arbitrary Executable File Formats File System
abrtd.service                                 loaded active       running                      ABRT Automated Bug Reporting Tool
accounts-daemon.service                       loaded active       running                      Accounts Service
acpid.service                                 loaded active       running                      ACPI Event Daemon
atd.service                                   loaded active       running                      Execution Queue Daemon
auditd.service                                loaded active       running                      Security Auditing Service
avahi-daemon.service                          loaded active       running                      Avahi mDNS/DNS-SD Stack
bluetooth.service                             loaded active       running                      Bluetooth Manager
console-kit-daemon.service                    loaded active       running                      Console Manager
cpuspeed.service                              loaded active       exited                       LSB: processor frequency scaling support
crond.service                                 loaded active       running                      Command Scheduler
cups.service                                  loaded active       running                      CUPS Printing Service
dbus.service                                  loaded active       running                      D-Bus System Message Bus
[email protected]                            loaded active       running                      Getty on tty2
[email protected]                            loaded active       running                      Getty on tty3
[email protected]                            loaded active       running                      Getty on tty4
[email protected]                            loaded active       running                      Getty on tty5
[email protected]ice                            loaded active       running                      Getty on tty6
haldaemon.service                             loaded active       running                      Hardware Manager
[email protected]                            loaded active       running                      sda shock protection daemon
irqbalance.service                            loaded active       running                      LSB: start and stop irqbalance daemon
iscsi.service                                 loaded active       exited                       LSB: Starts and stops login and scanning of iSCSI devices.
iscsid.service                                loaded active       exited                       LSB: Starts and stops login iSCSI daemon.
livesys-late.service                          loaded active       exited                       LSB: Late init script for live image.
livesys.service                               loaded active       exited                       LSB: Init script for live image.
lvm2-monitor.service                          loaded active       exited                       LSB: Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
mdmonitor.service                             loaded active       running                      LSB: Start and stop the MD software RAID monitor
modem-manager.service                         loaded active       running                      Modem Manager
netfs.service                                 loaded active       exited                       LSB: Mount and unmount network filesystems.
NetworkManager.service                        loaded active       running                      Network Manager
ntpd.service                                  loaded maintenance  maintenance                  Network Time Service
polkitd.service                               loaded active       running                      Policy Manager
prefdm.service                                loaded active       running                      Display Manager
rc-local.service                              loaded active       exited                       /etc/rc.local Compatibility
rpcbind.service                               loaded active       running                      RPC Portmapper Service
rsyslog.service                               loaded active       running                      System Logging Service
rtkit-daemon.service                          loaded active       running                      RealtimeKit Scheduling Policy Service
sendmail.service                              loaded active       running                      LSB: start and stop sendmail
[email protected]:22-172.31.0.4:36368.service  loaded active       running                      SSH Per-Connection Server
sysinit.service                               loaded active       running                      System Initialization
systemd-logger.service                        loaded active       running                      systemd Logging Daemon
udev-post.service                             loaded active       exited                       LSB: Moves the generated persistent udev rules to /etc/udev/rules.d
udisks.service                                loaded active       running                      Disk Manager
upowerd.service                               loaded active       running                      Power Manager
wpa_supplicant.service                        loaded active       running                      Wi-Fi Security Service
avahi-daemon.socket                           loaded active       listening                    Avahi mDNS/DNS-SD Stack Activation Socket
cups.socket                                   loaded active       listening                    CUPS Printing Service Sockets
dbus.socket                                   loaded active       running                      dbus.socket
rpcbind.socket                                loaded active       listening                    RPC Portmapper Socket
sshd.socket                                   loaded active       listening                    sshd.socket
systemd-initctl.socket                        loaded active       listening                    systemd /dev/initctl Compatibility Socket
systemd-logger.socket                         loaded active       running                      systemd Logging Socket
systemd-shutdownd.socket                      loaded active       listening                    systemd Delayed Shutdown Socket
dev-disk-by\x1...x1db22a\x1d870f1adf2732.swap loaded active       active                       /dev/disk/by-uuid/fd626ef7-34a4-4958-b22a-870f1adf2732
basic.target                                  loaded active       active                       Basic System
bluetooth.target                              loaded active       active                       Bluetooth
dbus.target                                   loaded active       active                       D-Bus
getty.target                                  loaded active       active                       Login Prompts
graphical.target                              loaded active       active                       Graphical Interface
local-fs.target                               loaded active       active                       Local File Systems
multi-user.target                             loaded active       active                       Multi-User
network.target                                loaded active       active                       Network
remote-fs.target                              loaded active       active                       Remote File Systems
sockets.target                                loaded active       active                       Sockets
swap.target                                   loaded active       active                       Swap
sysinit.target                                loaded active       active                       System Initialization

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
JOB    = Pending job for the unit.

221 units listed. Pass --all to see inactive units, too.
[[email protected]] ~#

(I have shortened the output above a little, and removed a few lines not relevant for this blog post.)

Look at the ACTIVE column, which shows you the high-level state of
a service (or in fact of any kind of unit systemd maintains, which can
be more than just services, but we’ll have a look on this in a later
blog posting), whether it is active (i.e. running),
inactive (i.e. not running) or in any other state. If you look
closely you’ll see one item in the list that is marked maintenance
and highlighted in red. This informs you about a service that failed
to run or otherwise encountered a problem. In this case this is
ntpd. Now, let’s find out what actually
happened to ntpd, with the systemctl status
command:

[[email protected]] ~# systemctl status ntpd.service
ntpd.service - Network Time Service
	  Loaded: loaded (/etc/systemd/system/ntpd.service)
	  Active: maintenance
	    Main: 953 (code=exited, status=255)
	  CGroup: name=systemd:/systemd-1/ntpd.service
[[email protected]] ~#

This shows us that NTP terminated during runtime (when it ran as
PID 953), and tells us exactly the error condition: the process exited
with an exit status of 255.

In a later systemd version, we plan to hook this up to ABRT, as soon as
this enhancement request is fixed
. Then, if systemctl
status
shows you information about a service that crashed it will
direct you right-away to the appropriate crash dump in ABRT.

Summary: use systemctl and systemctl
status
as modern, more complete replacements for the traditional
boot-up status messages of SysV services. systemctl status
not only captures in more detail the error condition but also shows
runtime errors in addition to start-up errors.

That’s it for this week, make sure to come back next week, for the
next posting about systemd for administrators!