Safely validating usernames with Amazon Cognito

Post Syndicated from Larry Ogrodnek original

Guest post by AWS Community Hero Larry Ogrodnek. Larry is an independent consultant focused on cloud architecture, DevOps, serverless, and general software development on AWS. He’s always ready to talk about AWS over coffee, and enjoys development and helping other developers.

How are users identified in your system? Username? Email? Is it important that they’re unique?

You may be surprised to learn, as I was, that in Amazon Cognito both usernames and emails are treated as case-sensitive. So “JohnSmith” is different than “johnsmith,” who is different than “jOhNSmiTh.” The same goes for email addresses—it’s even specified in the SMTP RFC that users “smith” and “Smith” may have different mailboxes. That is crazy!

I recently added custom signup validation for Amazon Cognito. In this post, let me walk you through the implementation.

The problem with uniqueness

Determining uniqueness is even more difficult than just dealing with case insensitivity. Like many of you, I’ve received emails based on Internationalized Domain Name homograph attacks. A site is registered for “” but with Cyrillic character “a,” attempting to impersonate a legitimate site and collect information. This same type of attack is possible for user name registration. If I don’t check for this, someone may be able to impersonate another user on my site.

Do you have reservations?

Does my application have user-generated messages or content? Besides dealing with uniqueness, I may want to reserve certain names. For example, if I have user-editable information at or, what if someone registers “signup” or “faq” or “support”? What about “billing”?

It’s possible that a malicious user could impersonate my site and use it as part of an attack. Similar attacks are also possible if users have any kind of inbox or messaging. In addition to reserving usernames, I should also separate out user content to its own domain to avoid confusion. I remember GitHub reacting to something similar when it moved user pages from to in 2013.

James Bennet wrote about these issues in great detail in his excellent post, Let’s talk about usernames. He describes the types of validation performed in his django-registration application.

Integrating with Amazon Cognito

Okay, so now that you know a little bit more about this issue, how do I handle this with Amazon Cognito?

Well, I’m in luck, because Amazon Cognito lets me customize much of my authentication workflow with AWS Lambda triggers.

To add username or email validation, I can implement a pre-sign-up Lambda trigger, which lets me perform custom validation and accept or deny the registration request.

It’s important to note that I can’t modify the request. To perform any kind of case or name standardization (for example, forcing lower case), I have to do that on the client. I can only validate that it was done in my Lambda function. It would be handy if this was something available in the future.

To declare a sign-up as invalid, all I have to do is return an error from the Lambda function. In Python, this is as simple as raising an exception. If my validation passes, I just return the event, which already includes the fields that I need for a generic success response. Optionally, I can auto-verify some fields.

To enforce that my frontend is sending usernames standardized as lowercase, all I need is the following code:

def run(event, context):
  user = event[‘userName’]
  if not user.isLower():
    raise Exception(“Username must be lowercase”)
  return event

Adding unique constraints and reservations

I’ve extracted the validation checks from django-registration into a Python module named username-validator to make it easier to perform these types of uniqueness checks in Lambda:

pip install username-validator

In addition to detecting confusing homoglyphs, it also includes a standard set of reserved names like “www”, “admin”, “root”, “security”, “robots.txt”, and so on. You can provide your own additions for application-specific reservations, as well as perform individual checks.

To add this additional validation and some custom reservations, I update the function as follows:

from username_validator import UsernameValidator


validator = UsernameValidator(additional_names=MY_RESERVED)

def run(event, context):
  user = event['userName']

  if not user.islower():
    raise Exception("Username must be lowercase")


  return event

Now, if I attach that Lambda function to the Amazon Cognito user pool as a pre–sign-up trigger and try to sign up for “aws”, I get a 400 error. I also get some text that I could include in the signup form: Other attributes, including email (if used) are available under event[‘request’][‘userAttributes’]}. For example:

{ "request": {
    "userAttributes": {"name": "larry", "email": "[email protected]" }

What’s next?

I can validate other attributes in the same way. Or, I can add other custom validation by adding additional checks and raising an exception, with a custom message if it fails.

In this post, I covered why it’s important to think about identity and uniqueness, and demonstrated how to add additional validations to user signups in Amazon Cognito.

Now you know more about controlling signup validation with a custom Lambda function. I encourage you to check out the other user pool workflow customizations that are possible with Lambda triggers.

[$] Tracking pages from get_user_pages()

Post Syndicated from corbet original

As has been recently discussed here,
developers for the filesystem and memory-management subsystems have been
grappling for years with the problems posed by the get_user_pages()
mechanism. This function maps memory into the kernel’s address space for
direct access by the kernel or peripheral devices, but that kind of access
can create confusion in the filesystem layers, which may not be expecting
that memory to be written to at any given time. A new patch
from Jérôme Glisse tries to chip away at a piece of the problem,
but a complete solution is not yet in view.

From Poll to Push: Transform APIs using Amazon API Gateway REST APIs and WebSockets

Post Syndicated from Chris Munns original

This post is courtesy of Adam Westrich – AWS Principal Solutions Architect and Ronan Prenty – Cloud Support Engineer

Want to deploy a web application and give a large number of users controlled access to data analytics? Or maybe you have a retail site that is fulfilling purchase orders, or an app that enables users to connect data from potentially long-running third-party data sources. Similar use cases exist across every imaginable industry and entail sending long-running requests to perform a subsequent action. For example, a healthcare company might build a custom web portal for physicians to connect and retrieve key metrics from patient visits.

This post is aimed at optimizing the delivery of information without needing to poll an endpoint. First, we outline the current challenges with the consumer polling pattern and alternative approaches to solve these information delivery challenges. We then show you how to build and deploy a solution in your own AWS environment.

Here is a glimpse of the sample app that you can deploy in your own AWS environment:

What’s the problem with polling?

Many customers need implement the delivery of long-running activities (such as a query to a data warehouse or data lake, or retail order fulfillment). They may have developed a polling solution that looks similar to the following:

  1. POST sends a request.
  2. GET returns an empty response.
  3. Another… GET returns an empty response.
  4. Yet another… GET returns an empty response.
  5. Finally, GET returns the data for which you were looking.

The challenges of traditional polling methods

  • Unnecessary chattiness and cost due to polling for result sets—Every time your frontend polls an API, it’s adding costs by leveraging infrastructure to compute the result. Empty polls are just wasteful!
  • Hampered mobile battery life—One of the top contributors of apps that eat away at battery life is excessive polling. Make sure that your app isn’t on your users’ Top App Battery Usage list-of-shame that could result in deletion.
  • Delayed data arrival due to polling schedule—Some approaches to polling include an incremental backoff to limit the number of empty polls. This sometimes results in a delay between data readiness and data arrival.

But what about long polling?

  • User request deadlocks can hamper application performance—Long synchronous user responses can lead to unexpected user wait times or UI deadlocks, which can affect mobile devices especially.
  • Memory leaks and consumption could bring your app down—Keeping long-running tasks queries open may overburden your backend and create failure scenarios, which may bring down your app.
  • HTTP default timeouts across browsers may result in inconsistent client experience—These timeouts vary across browsers, and can lead to an inconsistent experience across your end users. Depending on the size and complexity of the requests, processing can last longer than many of these timeouts and take minutes to return results.

Instead, create an event-driven architecture and move your APIs from poll to push.

Asynchronous push model

To create optimal UX experiences, frontend developers often strive to create progressive and reactive user experiences. Users can interact with frontend objects (for example, push buttons) with little lag when sending requests and receiving data. But frontend developers also want users to receive timely data, without sacrificing additional user actions or performing unnecessary processing.

The birth of microservices and the cloud over the past several years has enabled frontend developers and backend service designers to think about these problems in an asynchronous manner. This enables the virtually unlimited resources of the cloud to choreograph data processing. It also enables clients to benefit from progressive and reactive user experiences.

This is a fresh alternative to the synchronous design pattern, which often relies on client consumers to act as the conductor for all user requests. The following diagram compares the flow of communication between patterns.Comparison of communication patterns

Asynchronous orchestration can be easily choreographed using the workflow definition, monitoring, and tracking console with AWS Step Functions state machines. Break up services into functions with AWS Lambda and track executions within a state diagram, like the following:

With this approach, consumers send a request, which initiates downstream distributed processing. The subsequent steps are invoked according to the state machine workflow and each execution can be monitored.

Okay, but how does the consumer get the result of what they’re looking for?

Multiple approaches to this problem

There are different approaches for consumers to retrieve their resulting data. In order to create the optimal solution, there are several questions that a service owner may need to ask.

Is there known trust with the client (such as another microservice)?

If the answer is Yes, one approach is to use Amazon SNS. That way, consumers can subscribe to topics and have data delivered using email, SMS, or even HTTP/HTTPS as an event subscriber (that is, webhooks).

With webhooks, the consumer creates an endpoint where the service provider can issue a callback using a small amount of consumer-side resources. The consumer is waiting for an incoming request to facilitate downstream processing.

  1. Send a request.
  2. Subscribe the consumer to the topic.
  3. Open the endpoint.
  4. SNS sends the POST request to the endpoint.

If the trust answer is No, then clients may not be able to open a lightweight HTTP webhook after sending the initial request. In that case, you must open the webhook. Consider an alternative framework.

Are you restricted to only using a REST protocol?

Some applications can only run HTTP requests. These scenarios include the age of the technology or browser for end users, and potential security requirements that may block other protocols.

In these scenarios, customers may use a GET method as a best practice, but we still advise avoiding polling. In these scenarios, some design questions may be: Does data readiness happen at a predefined time or duration interval? Or can the user experience tolerate time between data readiness and data arrival?

If the answer to both of these is Yes, then consider trying to send GET calls to your RESTful API one time. For example, if a job averages 10 minutes, make your GET call at 10 minutes after the submission. Sounds simple, right? Much simpler than polling.

Should I use GraphQL, a WebSocket API, or another framework?

Each framework has tradeoffs.

If you want a more flexible query schema, you may gravitate to GraphQL, which follows a “data-driven UI” approach. If data drives the UI, then GraphQL may be the best solution for your use case.

AWS AppSync is a serverless GraphQL engine that supports the heavy lifting of these constructs. It offers functionality such as AWS service integration, offline data synchronization, and conflict resolution. With GraphQL, there’s a construct called Subscriptions where clients can subscribe to event-based subscriptions upon a data change.

Amazon API Gateway makes it easy for developers to deploy secure APIs at scale. With the recent introduction of API Gateway WebSocket APIs, web, mobile clients, and backend services can communicate over an established WebSocket connection. This also allows clients to be more reactive to data updates and only do work one time after an update has been received over the WebSocket connection.

The typical frontend design approach is to create a UI component that is updated when the results of the given procedure are complete. This is beneficial over a complete webpage refresh and gains in the customer’s user experience.

Because many companies have elected to use the REST framework for creating API-driven tightly bound service contracts, a RESTful interface can be used to validate and receive the request. It can provide a status endpoint used to deliver status. Also, it provides additional flexibility in delivering the result to the variety of clients, along with the WebSocket API.

Poll-to-push solution with API Gateway

Imagine a scenario where you want to be updated as soon as the data is created. Instead of the traditional polling methods described earlier, use an API Gateway WebSocket API. That pushes new data to the client as it’s created, so that it can be rendered on the client UI.

Alternatively, a WebSocket server can be deployed on Amazon EC2. With this approach, your server is always running to accept and maintain new connections. In addition, you manage the scaling of the instance manually at times of high demand.

By using an API Gateway WebSocket API in front of Lambda, you don’t need a machine to stay always on, eating away your project budget. API Gateway handles connections and invokes Lambda whenever there’s a new event. Scaling is handled on the service side. To update our connected clients from the backend, we can use the API Gateway callback URL. The AWS SDKs make communication from the backend easy. As an example, see the Boto3 sample of post_to_connection:

import boto3 
#Use a layer or deployment package to 
#include the latest boto3 version.
apiManagement = boto3.client('apigatewaymanagementapi', region_name={{api_region}},
response = apiManagement.post_to_connection(Data={{message}},ConnectionId={{connectionId}})

Solution example

To create this more optimized solution, create a simple web application that enables a user to make a request against a large dataset. It returns the results in a flat file over WebSocket. In this case, we’re querying, via Amazon Athena, a data lake on S3 populated with AWS Twitter sentiment (Twitter: @awscloud or #awsreinvent). However, this solution could apply to any data store, data mart, or data warehouse environment, or the long-running return of data for a response.

For the frontend architecture, create this web application using a JavaScript framework (such as JQuery). Use API Gateway to accept a REST API for the data and then open a WebSocket connection on the client to facilitate the return of results:poll to push solution architecture

  1. The client sends a REST request to API Gateway. This invokes a Lambda function that starts the Step Functions state machine execution. The function also returns a task token for the open connection activity worker. The function then returns the execution ARN and task token to the client.
  2. Using the data returned by the REST request, the client connects to the WebSocket API and sends the task token to the WebSocket connection.
  3. The WebSocket notifies the Step Functions state machine that the client is connected. The Lambda function completes the OpenConnection task through validating the task token and sending a success message.
  4. After RunAthenaQuery and OpenConnection are successful, the state machine updates the connected client over the WebSocket API that their long-running job is complete. It uses the REST API call post_to_connection.
  5. The client receives the update over their WebSocket connection using the IssueCallback Lambda function, with the callback URL from the API Gateway WebSocket API.

In this example application, the data response is an S3 presigned URL composed of results from Athena. You can then configure your frontend to use that S3 link to download to the client.

Why not just open the WebSocket API for the request?

While this approach can work, we advise against it for this use case. Break the required interfaces for this use case into three processes:

  1. Submit a request.
  2. Get the status of the request (to detect failure).
  3. Deliver the results.

For the submit request interface, RESTful APIs have strong controls to ensure that user requests for data are validated and given guardrails for the intended query purpose. This helps prevent rogue requests and unforeseen impacts to the analytics environment, especially when exposed to a large number of users.

With this example solution, you’re restricting the data requests to specific countries in the frontend JavaScript. Using the RESTful POST method for API requests enables you to validate data as query string parameters, such as the following:


API Gateway models and mapping templates can also be used to validate or transform request payloads at the API layer before they hit the backend. Use models to ensure that the structure or contents of the client payload are the same as you expect. Use mapping templates to transform the payload sent by clients to another format, to be processed by the backend.

This REST validation framework can also be used to detect header information on WebSocket browser compatibility (external site). While using WebSocket has many advantages, not all browsers support it, especially older browsers. Therefore, a REST API request layer can pass this browser metadata and determine whether a WebSocket API can be opened.

Because a REST interface is already created for submitting the request, you can easily add another GET method if the client must query the status of the Step Functions state machine. That might be the case if a health check in the request is taking longer than expected. You can also add another GET method as an alternative access method for REST-only compatible clients.

If low-latency request and retrieval are an important characteristic of your API and there aren’t any browser-compatibility risks, use a WebSocket API with JSON model selection expressions to protect your backend with a schema.

In the spirit of picking the best tool for the job, use a REST API for the request layer and a WebSocket API to listen for the result.

This solution, although secure, is an example for how a non-polling solution can work on AWS. At scale, it may require refactoring due to the cross-talk at high concurrency that may result in client resubmissions.

To discover, deploy, and extend this solution into your own AWS environment, follow the PollToPush instructions in the AWS Serverless Application Repository.


When application consumers poll for long-running tasks, it can be a wasteful, detrimental, and costly use of resources. This post outlined multiple ways to refactor the polling method. Use API Gateway to host a RESTful interface, Step Functions to orchestrate your workflow, Lambda to perform backend processing, and an API Gateway WebSocket API to push results to your clients.

Ubuntu 19.04 (Disco Dingo) released

Post Syndicated from jake original

Ubuntu 19.04, code named “Disco Dingo”, has been released, along with the following flavors: Ubuntu Budgie, Kubuntu, Lubuntu, Ubuntu Kylin, Ubuntu MATE,
Ubuntu Studio, and Xubuntu.
The Ubuntu kernel has been updated to the 5.0 based Linux kernel,
our default toolchain has moved to gcc 8.3 with glibc 2.29, and we’ve
also updated to openssl 1.1.1b and gnutls 3.6.5 with TLS1.3 support.

Ubuntu Desktop 19.04 introduces GNOME 3.32 with increased performance,
smoother startup animations, quicker icon load times and reduced CPU+GPU
load. Fractional scaling for HiDPI screens is now available in Xorg
and Wayland.

Ubuntu Server 19.04 integrates recent innovations from key open
infrastructure projects like OpenStack Stein, Kubernetes, and Ceph with
advanced life-cycle management for multi-cloud and on-prem operations,
from bare metal, VMware and OpenStack to every major public cloud.” More information can be found in the release notes.

Build a SatNOGS ground station with a Raspberry Pi 3B+ | HackSpace magazine #18

Post Syndicated from Alex Bate original

The big feature on outer space in issue 18 of HackSpace magazine, available from today, shows you how to build your own satellite and launch it into orbit.

No, we’re not kidding, this is an actual thing you can do.

And to track the satellite you’ve launched, or another satellite you’re interested in, here’s how to build your own SatNOGS ground station with a Raspberry Pi 3B+.

Building a Raspberry Pi ground station

Once you’ve built and launched your small satellite, you’ll want to listen to all the glorious telemetry and data it‘s sending back as it hurtles around the Earth. Or perhaps you aspire to have a satellite up there, but in the meantime you want to listen to some other objects? What you need is a ground station, but a single ground station has one slight flaw. Most of the time a satellite will not be overhead of a single ground station; in fact, it may only pass over a ground station once every few days, massively reducing the amount of information or data we can receive. So we need a network of ground stations. The SatNOGS network solves this by creating a global network of stations that can work together to increase coverage.

SatNOGS is an open-source project that has numerous designs for satellite ground stations, but whichever design you pick, you can join the network that links them all via the web.

A station owner can use the website to browse for future passes of a satellite, and then click a button to schedule for their station to turn on, tune to frequency, and record the pass, sometimes even rotating the antenna on the station to track the satellite. Not only can a station owner schedule an observation on their own station, but they can schedule observations on any station on the global network.

As we can see from this map of data being collected of a recent SSTV broadcast from the ISS (sends single-frame images transmitted via audio from the ISS), the SatNOGS network has near-global coverage, rivalling most professional institutions in the world.

Simple setup

The simplest form of a SatNOGS station is one that doesn’t move or track and is made from a static antenna, a Raspberry Pi, and a cheap software-defined radio (SDR) dongle. The SDR dongle has become ubiquitous in maker circles as it is an affordable entry item into the world of receiving signals via SDR. Looking at our ingredients in the image below, let’s explore them a little more before we get started.

While a permanent station may do better connected by Ethernet cable, using the Raspberry Pi’s built-in wireless LAN functionality means we can run this simply with only a power cable. While many have used the cheapest Realtek SDR dongles with success, some people have found the slightly more refined versions can be more stable – a current recommendation is the RTL-SDR V3, which has a better casing for thermal dissipation, and slightly upgraded components. The RTL-SDR V3 is available here.

The classic antenna recommended for a static SatNOGS setup shown above is a ‘turnstile’ antenna; commercial models are available, such as the Wimo TA-1, but people have designed and built lots of different static antennas for different frequencies and with small budgets – check out the tutorial Make a Slim Jim antenna on page 112 (in HackSpace issue 18, links below).

In order to set up a ground station, one of the first tasks we need to do is set up an account on Registering on the site then gives us a dashboard where we can begin to set up a station. Click to add a station — we then need to supply it with some basic details as per the image below: a name for the station, a location in latitude and longitude (Google is your friend here!), and the elevation of the station above sea-level.

You need to decide what frequency your station is going to cover; the most common ranges are UHF and VHF, which would require different antennas, but either range has a huge number of objects you can schedule to observe. Many people opt for VHF, as this includes the frequency range for a lot of the different transmissions from the ISS, so we are going to choose VHF as well. You also need to add a minimum elevation value — this is the minimum angle that a satellite must be in terms of height for your station to see it — if you aren’t sure, either ask for help on the forums, or leave it for now at the default 10 degrees.

Having filled in the boxes to create the station (leave the ‘this is in testing’ box ticked for now), you should now see a ground station entry has been made on your account, as above. You will see (even though it isn’t set up yet) a list populating underneath the entry with ‘Pass Predictions’, which are things you could schedule to observe once you are up and running. Before we leave the website, we need to make a note of the number assigned to the ground station, and also our own personal API key — which we can find in our dashboard by clicking the API key button. These two pieces of information are what will ultimately connect our ground station hardware to the website account.

The next task is to sort out the Raspberry Pi. You can find the current custom SatNOGS image here.

Flash this to your microSD card as you would for a regular Raspberry Pi setup — the free app Etcher, for example, is a simple tool that allows you to flash an image to a card.

Once done, boot the Raspberry Pi, and you can either SSH into the Pi, or connect a keyboard and monitor and interact with the setup that way. The first things we need to do are not SatNOGS-specific, but are the usual things we do when setting up a Raspberry Pi. We need to set up a different password by running the sudo raspi‐config command. Once you’ve set a password and expanded the file system, it’s also useful to set the time zone to UTC, as this is used throughout the SatNOGS network. If you want to run this test station wirelessly, then you need to configure your network connection at this point. If you are connecting via an Ethernet cable, then you don’t need to do anything else. Apply the changes and reboot (then see ‘Final setup’ box above in HackSpace issue 18, links below).

Now, if we go back to our dashboard on the SatNOGS website (perhaps wait a few minutes and click Refresh), we should see that the station is now online, as above. We should see an orange spot on the network map showing our proud station in testing. Being in testing means that only you can schedule observations on the station, but when you are ready, you can change settings to take it out of testing and then it is fully on the network.

On the hunt

Power down one last time and connect the RTL-SDR dongle and the antenna, then reboot — you are now ready to hunt satellites! Scheduling observations is as simple as selecting passes from the list and clicking Schedule. There may be drop-down choices for different transmitters to listen for on the same satellite, and other choices, but essentially you click Calculate to create the observation and then Schedule for the job to be created and sent to the queue for your station. There are hundreds of satellites to try to observe, so don’t worry if you don’t understand what any of them are — in the pass predictions list, if you click the name of a satellite you will get a pop-up with information about it. For a more detailed walkthrough of scheduling an observation on the SatNOGS network, check out this blog post.

After the time of the pass, return to the observation page and, hopefully, you should see some signals. Don’t worry if your first few observations aren’t successful: try at least a dozen observations before making any changes, as there are many possible reasons for a signal not getting picked up; indeed, the satellite may not even have been transmitting. If you have received a signal, you should ‘vet’ the observation as good; this is particularly important if you have scheduled on someone else’s station – etiquette says we should check and vet our own observations. Check out the Slim Jim antenna (see page 112 of HackSpace magazine issue 18, links below) for a link to a successful observation you can listen to.

Happy satellite hunting!

Finally, it’s a great idea to join the Libre Space Foundation community forum (or IRC), as it hosts the SatNOGS community channels, and there is a wealth of expertise and help available there from a very welcoming community. If you build a station, go and share your achievement on the forum — everyone will be pleased to see it.

Get HackSpace magazine issue 18 — out today

HackSpace magazine issue 18 is out today, and available online, or from many high-street retailers such as WHSmith and Sainsbury’s in the UK, and Barnes & Nobel in the US.

You can also download issue 18 for free, today as a PDF, so there really is no reason not to give HackSpace a spin.

The post Build a SatNOGS ground station with a Raspberry Pi 3B+ | HackSpace magazine #18 appeared first on Raspberry Pi.

OpenSSH 8.0 released

Post Syndicated from jake original

OpenSSH 8.0 has been released with a bunch new features and some bug fixes, including one for a security problem:
This release contains mitigation for a weakness in the scp(1) tool
and protocol (CVE-2019-6111): when copying files from a remote system
to a local directory, scp(1) did not verify that the filenames that
the server sent matched those requested by the client. This could
allow a hostile server to create or clobber unexpected local files
with attacker-controlled content.

This release adds client-side checking that the filenames sent from
the server match the command-line request,

The scp protocol is outdated, inflexible and not readily fixed. We
recommend the use of more modern protocols like sftp and rsync for
file transfer instead.”

Security updates for Thursday

Post Syndicated from jake original

Security updates have been issued by CentOS (polkit), Gentoo (dovecot, libseccomp, and patch), openSUSE (aubio, blktrace, flac, lxc, lxcfs, pspp, SDL, sqlite3, and xen), Red Hat (java-1.8.0-openjdk, java-11-openjdk, and rh-maven35-jackson-databind), Scientific Linux (java-1.8.0-openjdk), Slackware (libpng), SUSE (python, python3, sqlite3, and xerces-c), and Ubuntu (ntfs-3g).


Post Syndicated from nellyo original

LUMIERE VOD е база данни с европейски филми (след 1996 г.), предлагани по заявка (VOD).

Базата е създадена с информационна цел и е предназначена главно за професионалистите в областта на аудиовизуалната индустрия: автори, продуценти,  регулатори.

LUMIERE VOD се управлява   от Европейската аудиовизуална обсерватория. Директорията в момента е в бета версия и включва около 250 VOD каталога. Броят на следените каталози и честотата на актуализациите постепенно ще се увеличават, заявяват от Обсерваторията.


Евро-избори 2019: има проблем със заявленията за гласуване в чужбина

Post Syndicated from Боян Юруков original

Проблемът не е в сайта на ЦИК или в самия формуляр. Не и този път. По-скоро е в броя събрани заявления. Изглежда тази година българите в чужбина не подават толкова заявления, колкото при предишни избори.

Една причина може да е информираността, че е нужно, но най-вероятно става дума за мотивация и очакване, че секциите ще бъдат отворени така или иначе. Доколкото може да се очаква, че интересът към евро-вотът ще е по-нисък от изборите за НС, това, което се вижда е притеснително.

Сега сме в 10-тия ден, откакто започнаха да се събират заявления и има малко над 2000 събрани. За същия период на парламентарните избори през 2017-та имаше над 5000 заявления само от страни от ЕС. Приблизително толкова имаше и за президентските избори година по-рано. Изключвам от тези числа заявленията подадени извън Европейския съюз.

Това означава, че в сегашната кампания заявленията се събират два пъти по-бавно. В същото време имаме доста по-малко време. Докато на изборите през 2016-та и 2017-та имаше 25-26 дни за събиране на заявления, сега са 22 и вече сме ги преповили.

При предишния вот показах нагледно как се развива кампанията по събиране на заявления. За целта използвах данните от, които се събират в реално време. Обнових графиките, за да сравня Всичко подадено в рамките на ЕС за този и предишния вот.

На горната графика виждате общата активност. Днес е 10-тия ден от червената линия и показва само заявленията от сутринта. Миналата година най-много са били подавани в последните дни от кампанията, както се надявам да стане и сега. Вече обаче имаме лошо начало.

Тук съм направил разбивка на трите страни с най-много българи в Европа – Германия, Великобритания и Испания.

Най-вече за Германия няма логика предвид, че има шанс да се отворят секции извън консулствата. Явно обаче тази информация не е известна на много.

В следващите две графики съм сравнил разбивката по държави между 2017 и 2019-та. Тази за вота през 2017-та се различава от тази, която пуснах през февруари 2017, защото тук съм махнал Турция и щатите. Двете графики долу са с един и същи мащаб за по-лесно сравнение.

Всичко това показва, че трябва да работим доста повече за информираност и мотивация на българите в чужбина за този вот.

2019 Репортери без граници

Post Syndicated from nellyo original

Репортери без граници публикуваха годишната си оценка за свободата на медиите

2019 World Press Freedom Index | RSF

Внесеният от Пеевски закон за прозрачността на медийната собственост не е убедил никого, че сега вече – с приемането на закона –  положението рязко се е подобрило. Според този индекс България остава на 111 място. Убийството на Виктория Маринова е споменато, без да се конкретизира причината според разследващите органи.

Оценката за България:

Ако сте очаквали повече свобода на медиите във връзка с председателството на Съвета на ЕС през 2018 г.,  очакването не се оправдава.

Корупцията и тайните споразумения между медиите, политиците и олигарсите са широко разпространени в България – и най-известното въплъщение на този факт е Делян Пеевски.


New DNS Hijacking Attacks

Post Syndicated from Bruce Schneier original

DNS hijacking isn’t new, but this seems to be an attack of uprecidented scale:

Researchers at Cisco’s Talos security division on Wednesday revealed that a hacker group it’s calling Sea Turtle carried out a broad campaign of espionage via DNS hijacking, hitting 40 different organizations. In the process, they went so far as to compromise multiple country-code top-level domains — the suffixes like or .ru that end a foreign web address — putting all the traffic of every domain in multiple countries at risk.

The hackers’ victims include telecoms, internet service providers, and domain registrars responsible for implementing the domain name system. But the majority of the victims and the ultimate targets, Cisco believes, were a collection of mostly governmental organizations, including ministries of foreign affairs, intelligence agencies, military targets, and energy-related groups, all based in the Middle East and North Africa. By corrupting the internet’s directory system, hackers were able to silently use “man in the middle” attacks to intercept all internet data from email to web traffic sent to those victim organizations.


Cisco Talos said it couldn’t determine the nationality of the Sea Turtle hackers, and declined to name the specific targets of their spying operations. But it did provide a list of the countries where victims were located: Albania, Armenia, Cyprus, Egypt, Iraq, Jordan, Lebanon, Libya, Syria, Turkey, and the United Arab Emirates. Cisco’s Craig Williams confirmed that Armenia’s .am top-level domain was one of the “handful” that were compromised, but wouldn’t say which of the other countries’ top-level domains were similarly hijacked.

Another news article.

Everything You Need to Know About the OSS Licensing War, Part 3.

Post Syndicated from Blogs on Grafana Labs Blog original

In Parts One and Two of this blog, we looked back at the ongoing open source licensing wars, focusing on the evolving situation between Elastic N.V. and AWS. In this final installment, we’ll offer some opinions on the situation, as well as share our own views on how we’re reacting at Grafana Labs.

So Who’s Right?

As faithful readers of the previous two installments hopefully realize by now: It’s complicated. Both parties have blood on their hands.

The knock against commercial open source companies is that they are trying to have their cake and eat it too. They want to stop the public cloud vendors from offering their software as a service and making money off their work. But preventing someone from using your software for any purpose violates a core freedom of open source.

Without those freedoms, many successful open source companies would not have been able to garner the adoption, mindshare, and support they enjoy today. With these new restrictions, they’re turning their back on open source, kicking the ladder out from under them and onto the community.

The knock against the public cloud companies is that they have been “unfairly” and “unsustainably” monetizing open source projects and companies; they’re making billions largely off the code others are creating. Plus, due to scale and channel advantages, the public cloud can stifle companies trying to offer their own software as a service.

Reversion to a Monoculture

But without contributing back in any notable way (either through people or dollars), the public clouds definitely haven’t lived up to the open source ideal. What’s happening isn’t a sustainable cycle.

I’ve mentioned how Linux and Red Hat both drove massive value and innovation for the whole infrastructure ecosystem. We’re talking about trillions of dollars of value, across many companies. It was truly a rising tide that lifted all boats.

I don’t feel the same way currently about the rise of AWS or Amazon. To me it feels more like a tax on our infrastructure, and the reversion to the kind of monoculture that the open source community fought so hard against.

Or the Beginning of a Change?

In Part Two, I pondered whether the new breed of open source companies could monetize fast enough to command their eye-popping valuations, and whether they’d be able to capture enough of the value that their open source projects create.

Arguably being the “center of mass” for their communities is the most valuable asset that companies like Elastic N.V. and MongoDB Inc. have.

In releasing the Open Distro for Elasticsearch, AWS forced another, perhaps more interesting question to commercial open source companies: What happens if you get the balance of value creation and value capture wrong? Does it leave your community vulnerable to getting stolen right out from under you?

What Does This Mean for Elastic N.V.?

I’ve written about my admiration for Elastic N.V. before. I think they’ve built an incredible business.

Overall, AWS’s move could be a positive thing for the Elasticsearch community, but it could really hurt Elastic N.V. the company. All of a sudden, much of its commercial differentiators are freely available in a permissive Apache2-licensed fork. That’s a huge potential hit to their monetization plans.

But running an open source project isn’t as simple as throwing some code over the wall and putting up a web page, as AWS has done. It’s about things like shepherding and supporting the community, encouraging committers, and having momentum on a compelling vision. AWS does not have a good track record at any of this. Yet.

What Does This Mean for Grafana Labs?

We’ve watched these developments with great interest and had a lot of internal discussions and debates. We are still fine-tuning our strategy but are ready to double down on a few important details.

Firstly, we will continue to partner with cloud vendors. Grafana Labs is all about meeting our customers where they are; we have an inclusive “big tent” philosophy which is about more than connecting different data sources – it’s about connecting different communities and being a trusted advisor to our customers. That’s why we’ve entered into commercial agreements with Microsoft and Google – to provide a first-class experience for our common users who use Grafana with Azure Monitor or Stackdriver. We hope to replicate this arrangement with AWS.

In addition, while the overwhelming majority of the code we write is open source, we will continue to offer an Enterprise version that is commercially licensed and clearly not open source. But, we will not play games with licensing. Our open source software will be licensed under a standard OSI license, and our commercial software will not be open source. We will be very deliberate about what we monetize and whom we are targeting with our commercial products.

Finally, we will be watching the situation very carefully, and we will be very transparent with our plans if and when they change. We don’t take the community we’ve fostered for granted. Far from it, we consider it to be the foundation that everything we do depends on.

Maybe this is all wishful thinking, and I’m in denial because I run one of these new commercial open source companies. I hope not, because otherwise I’m just poking the 800-pound gorilla in the room.

[$] Business models and open source

Post Syndicated from jake original

One of the more lively sessions that was held at the 2019 Legal and
Licensing Workshop (LLW) was Heather Meeker’s talk on
open-source business models and alternative licensing. As a lawyer in
private practice, Meeker worked on
a number of the alternative licenses that were drafted and
presented over the last year or so. But she is also part of a venture
capital (VC) firm that is exclusively investing in companies focused on
open source, so she
has experience in thinking about what kinds of models actually work for
those types of businesses.

Rousseau-inspired Raspberry Pi Zero LED piano visualiser

Post Syndicated from Alex Bate original

Unlock your inner Rousseau with this gorgeous Raspberry Pi Zero LED piano visualiser.

Piano LED Visualizer

Inspired by Rousseau videos I tried to build my own Piano Visualizer. It is made with Raspberry Pi and WS2812B LED strip. Screen and buttons: Waveshare LCD TFT 1,44” 128x128px.

Pianist Rousseau

Fans of the popular YouTube pianist Rousseau would be forgiven for thinking the thumbnail above is of one of his videos. It’s actually of a Raspberry Pi build by Aleksander Evening, who posted this project on Reddit last week as an homage to Rousseau, who is one of his favourite YouTubers.

Building an LED piano visualiser

After connecting the LED strip to the Raspberry Pi Zero W, and setting up the Pi as a Bluetooth MIDI host, Aleksander was almost good to go. There was just one thing standing in his way…

He wanted to use the Synthesia software for visualisations, and, unmodified, this software doesn’t support the MIDI files Aleksander planned to incorporate. Luckily, he found the workaround:

As of today Synthesia doesn’t support MIDI via Bluetooth, it should be added in next update. There is official workaround: you have to replace dll file. You also have to enable light support in Synthesia. In Visualizer settings you have to change “input” to RPI Bluetooth. After that when learning new song next-to-play keys will be illuminated in corresponding colors, blue for left hand and green for right hand.


Homemade Rousseau

The final piece is a gorgeous mix of LEDs, sound, and animation — worthy of the project’s inspiration.

Find more information, including parts, links to the code, and build instructions, on Aleksander’s GitHub repo. And as always, if you build your own, or if you’ve created a Raspberry Pi project in honour of your favourite musician, artist, or YouTuber, we’d love to see it in the comments below.

And now, a little something from Rousseau:

Ludovico Einaudi – Nuvole Bianche

Sheet music: Click the 🔔bell to join the notification squad! ♫ Listen on Spotify: ♫ MIDI: ♫ Facebook: ♫ Instagram: ♫ Twitter: ♫ Buy me a coffee: Hope you enjoy my performance of Nuvole Bianche by Ludovico Einaudi.

The post Rousseau-inspired Raspberry Pi Zero LED piano visualiser appeared first on Raspberry Pi.

A "Department of Cybersecurity"

Post Syndicated from Bruce Schneier original

Presidential candidate John Delaney has announced a plan to create a Department of Cybersecurity.

I have long been in favor of a new federal agency to deal with Internet — and especially Internet-of-Things — security. The devil is in the details, of course, and it’s really easy to get this wrong, I outline a strawman proposal in Click Here to Kill Everybody; I call it the “National Cyber Office” and model it on the Office of the Director of National Intelligence. But regardless of what you think of this idea, I’m glad that at least someone is talking about it.

Slashdot thread. News story.

EDITED TO ADD: Yes, this post is perilously close to presidential politics. Any comment that opines on the qualifications of this, or any other, presidential candidate will be deleted.

How Bloomberg Tracks Hundreds of Billions of Data Points Daily with MetricTank and Grafana

Post Syndicated from Blogs on Grafana Labs Blog original

Bloomberg is best known as a media company with its news destination site, its award-winning magazine Bloomberg Businessweek, and its daily 24-7 social media program, Tic Toc, on Twitter.

But the main product for the 38-year-old company is actually Bloomberg Terminal, a software system that aggregates real-time market data and delivers financial news to more than 325,000 subscribers around the world. The enterprise premium service handles about 120 billion (that’s billion) pieces of data from financial markets daily, 2 million stories from its news division and affiliates, and a messaging network (think “Instant Bloomberg”) that fields 1 billion messages.

“With all this, people seem to notice when it doesn’t work,” said Stig Sorenson, Head of Production Visibility Group at Bloomberg. “We have had a few outages that were high profile, so about three years ago we decided to embark on a journey where we took telemetry a bit more seriously.”

Since 2015, Bloomberg’s central telemetry team has been growing steadily – and the same could be said for its influence within the company. Today, “we’re storing 5 million data points a second and running over 2,500 rules on our metrics stream,” said Software Developer Sean Hanson during his talk at GrafanaCon 2019 in Los Angeles. “We also do a bit more on the log side with about 100 terabytes of raw log data a day and a lot of legacy log rules.”

Their most impressive feat, however, was rallying 5,500 engineers around streamlining their monitoring systems. “It’s a hard problem when you have a lot of independent users and a lot of teams,” Hanson admitted. “For a lot of users, monitoring is not a priority … So we try to give them as much as we can without them actually having to do something.”

Here’s how the telemetry team embarked on their “safari of stability” by solving three key problems within their infrastructure.

1. Centralizing Data

Historically Bloomberg isolated software so that outages only affected a small subset of customers. Teams would get an alert about a sub-failure and evaluate their individual products, utilizing their own data sources and their own telemetry stack.

However, working in silos led to issues when there was either an unforeseen single point of failure or when the alerts would snowball into multiple failures.

With teams working in isolation, “the outage would linger until it got bad enough that someone from the outside, either on our environment support team or a high-level manager, would be like, ‘Hey, I think you might all be working on the same problem independently,’” said Hanson. “Then the teams would piece together all of the individual data to track down a root cause.”

The telemetry team’s first step to stem this problem was to deploy agents to as many machines as possible to collect system metrics – file system, operating system, etc. – as well as each machine’s process tables. The telemetry team also worked with key infrastructure teams to gain insight into system frameworks within individual services, queues, or databases.

The goal was to centralize the data and provide a broader picture of the operating infrastructure at any given time. The Head of Engineering now has high-level system health dashboards in place to monitor outages. “Once we provided all of these displays, we were able to narrow down the pieces of data that could help triage outages as they happened or prevent them if we could alert on them,” said Hanson.

The Grafana dashboards also became valued assets throughout the organization, from high-level execs such as Sorenson who want a monitoring overview, to developers who want drill-down links on all the panels, to programmers who extract insights through a query API for more complex analysis.

“We have one place that users go to, to configure everything for their metrics, logs, alerts, Grafana folders, and distributed traces” said Hanson.

More importantly, the team automated processes to implement SRE best practices moving forward. Firm-wide rules around CPU, memory, file system storage, and service frameworks “take effect as soon as users create a new service or spin up a new machine,” said Hanson. Plus, because they are hooked into the machine-building process now, “even the machine creation process can publish its own metrics and report failures.”

2. Unifying Alerts

Prior to the formation of the telemetry team, Bloomberg had various systems that created different notifications.

Now, not only are alerts centralized. A link is served with every alert that shows correlation – what happened around the same time on the same set of machines or for the same basic rule – so teams can quickly detect whether it is an isolated incident or a problem with their software.

“All of the alerts we generate have a similar look and feel and a base set of information that we require which include our remediation plan,” said Hanson. “Every alert that comes out should have an action or a call to action available to you right at the top.”

In the case of tags, the team enforces tag key registration, not tag values, to ensure that when users try to register PIDs or timestamps as tag keys, they are alerted that they are off-base.

“We really wanted it to be easy to do the right thing, hard to do the wrong thing, but still possible to do something non-standard that we decide is sane,” added Hanson. “We designed our APIs to try to facilitate this.”

Recently the team has taken the initiative to meet with cross-functional teams “to talk about use cases and guide people to pre-built solutions where they exist or teach them how to use existing tools to build on top of,” said Hanson. “If there’s a really good use case or if we see it a lot, we just build it into our system so that people don’t have to even think about it. They just get it.”

3. Simplifying Queries

Bloomberg’s dashboards live in one “massive” Grafana instance that provides templates and uses the same query language and API as Graphite.

As users adapted to using the metrics, “we had growing dimensionality where users really want to drill down a lot into their data,” said Hanson. “So they want to keep adding more tags or labels, and some of the frameworks like Kubernetes cause a lot of transient time series to come around.”

In other words, more time series means more RAM. So this is where MetricTank came into play.

With MetricTank’s pattern-based pruning rules in the index and pattern-based retention rules, “we came up with a Goldilocks approach where users could pick their favorite flavor from three,” said Hanson. “If they want aggregate data for 10 years of trend analysis, they can pick longer lived. Or if they want advanced drilldowns, they can do that, but they don’t get the data for as long … We let the users pick, we apply it as a tag, and MetricTank does the rest.”

One snag the team hit with MetricTank came after releasing their query API to users for programmatic access. Problems came up for “[users] making a lot of queries sequentially or for users using the tag-based auto-complete in Grafana,” said Hanson. “When you get two-, three-, five-second lag on auto-complete, it’s pretty frustrating and noticeable. The more beta users that came onto the query API, the slower it got, until we had a daily report that took two days to run.”

Working closely with the Grafana team, the Bloomberg team implemented speculative querying which issues redundant queries to other replicas when slow peers were detected. This reduced the run time for daily reports down to four hours. “We also implemented native functions in MetricTank which prevented proxying through Graphite Web,” explained Hanson. “After that, we are now down to an hour and a half for our daily report. So there’s really not much more we can optimize there without actually trying to optimize the report itself.”

So What’s Next?

Improving dashboard discoverability is the next item on the Bloomberg Telemetry punch list.

Bloomberg currently has more than 3,500 dashboards in more than 500 folders, and there are many generic dashboards that prove to be popular and copied internally. But while imitation is the best form of flattery, it’s a poor form of organization.

Dashboards get copied repeatedly with names that are barely distinguishable from the next to the point that it’s hard to organically surface the original dashboards within the system. “People were only finding them by being linked through tickets,” said Hanson.

While folders and permission settings help limit access and editing rights to key dashboards, it didn’t solve the issue of rampant dashboard copies appearing in the system. So Bloomberg again reached out to Grafana Labs for a solution.

Together, the teams are working to enhance the auto-complete function. “This would allow us to search for keywords, descriptions, or even maybe metric names or tag labels inside the queries, which would be great,” said Hanson.

The goal is for Bloomberg’s telemetry team to “score” dashboards based on popularity with a custom weighting system. They are hoping to develop functions such as tagging dashboards “official” vs. “experimental” so users know which ones are more reliable compared to others.

Another big project: Meta tags, a seamless and cost-effective way to add metadata, is also in the works.

In creating a sustainable monitoring infrastructure, “starting with some good open source technology gets you a big step up,” said Hanson. “But since you didn’t build it, it might not work for you right off the bat. So we can’t be afraid to jump in and improve the product for yourself and the wider community.”

After all, “investing in telemetry pays dividends,” said Hanson, “which is my obligatory financial joke I save for the very end.”

For more from GrafanaCon 2019, check out all the talks on YouTube.

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.