Security updates have been issued by Debian (opencv and wireshark), Fedora (corosync and pcs), Oracle (firefox, kernel, libvncserver, and libvorbis), Slackware (gd), SUSE (kernel), and Ubuntu (apache2).
Security updates have been issued by CentOS (libvorbis and thunderbird), Debian (pjproject), Fedora (compat-openssl10, java-1.8.0-openjdk-aarch32, libid3tag, python-pip, python3, and python3-docs), Gentoo (ZendFramework), Oracle (thunderbird), Red Hat (ansible, gcc, glibc, golang, kernel, kernel-alt, kernel-rt, krb5, kubernetes, libvncserver, libvorbis, ntp, openssh, openssl, pcs, policycoreutils, qemu-kvm, and xdg-user-dirs), SUSE (openssl and openssl1), and Ubuntu (python-crypto, ubuntu-release-upgrader, and wayland).
Security updates have been issued by Arch Linux (openssl and zziplib), Debian (ldap-account-manager, ming, python-crypto, sam2p, sdl-image1.2, and squirrelmail), Fedora (bchunk, koji, libidn, librelp, nodejs, and php), Gentoo (curl, dhcp, libvirt, mailx, poppler, qemu, and spice-vdagent), Mageia (389-ds-base, aubio, cfitsio, libvncserver, nmap, and ntp), openSUSE (GraphicsMagick, ImageMagick, spice-gtk, and wireshark), Oracle (kubernetes), Slackware (patch), and SUSE (apache2 and openssl).
Security updates have been issued by Arch Linux (drupal), Debian (openjdk-7), Fedora (exempi, gd, and tomcat), SUSE (python-paramiko), and Ubuntu (kernel, libvncserver, linux, linux-aws, linux-kvm, linux-raspi2, linux-snapdragon, linux-lts-trusty, and linux-raspi2).
Security updates have been issued by Debian (dovecot, irssi, libevt, libvncserver, mercurial, mosquitto, openssl, python-django, remctl, rubygems, and zsh), Fedora (acpica-tools, dovecot, firefox, ImageMagick, mariadb, mosquitto, openssl, python-paramiko, rubygem-rmagick, and thunderbird), Mageia (flash-player-plugin and squirrelmail), Slackware (php), and Ubuntu (dovecot).
Security updates have been issued by Debian (memcached, openssl, openssl1.0, php5, thunderbird, and xerces-c), Fedora (python-notebook, slf4j, and unboundid-ldapsdk), Mageia (kernel, libvirt, mailman, and net-snmp), openSUSE (aubio, cacti, cacti-spine, firefox, krb5, LibVNCServer, links, memcached, and tomcat), Slackware (ruby), SUSE (kernel and python-paramiko), and Ubuntu (intel-microcode).
Security updates have been issued by CentOS (slf4j), Debian (firefox-esr, mupdf, net-snmp, and samba), Fedora (apache-commons-compress, calibre, chromium, glpi, kernel, libvncserver, libvorbis, mozjs52, ntp, slurm, sqlite, and wireshark), openSUSE (librelp), SUSE (librelp, LibVNCServer, and qemu), and Ubuntu (firefox and zsh).
Security updates have been issued by Arch Linux (intel-ucode), Debian (gifsicle), Fedora (awstats and kernel), Gentoo (icoutils, pysaml2, and tigervnc), Mageia (dokuwiki and poppler), Oracle (kernel), SUSE (glibc, kernel, microcode_ctl, tiff, and ucode-intel), and Ubuntu (intel-microcode).
Post Syndicated from Helen Lynn original https://www.raspberrypi.org/blog/a-hedgehog-cam-or-two/
Here we are, hauling ourselves out of the Christmas and New Year holidays and into January proper. It’s dawning on me that I have to go back to work, even though it’s still very cold and gloomy in northern Europe, and even though my duvet is lovely and warm. I found myself envying beings that hibernate, and thinking about beings that hibernate, and searching for things to do with hedgehogs. And, well, the long and the short of it is, today’s blog post is a short meditation on the hedgehog cam.
Someone called Barker has installed a Raspberry Pi–based hedgehog cam in a location with a distant view of a famous Alp, and as well as providing live views by visible and infrared light for the dedicated and the insomniac, they also make a sped-up version of the previous night’s activity available. With hedgehogs usually being in hibernation during January, you mightn’t see them in any current feed — but don’t worry! You’re guaranteed a few hedgehogs on Barker’s website, because they have also thrown in some lovely GIFs of hoggy (and foxy) divas that their camera captured in the past.
Build your own hedgehog cam
For pointers on how to replicate this kind of setup, you could do worse than turn to Andrew Wedgbury’s hedgehog cam write-up. Andrew’s Twitter feed reveals that he’s a Cambridge local, and there are hints that he was behind RealVNC’s hoggy mascot for Pi Wars 2017.
Another day at the office: testing our #PiWars mascot using a @Raspberry_Pi 3, #VNC Connect and @4tronix_uk Picon Zero. Name suggestions? https://t.co/iYY3xAX9Bk
Our infrared bird box and time-lapse camera resources will also set you well on the way towards your own custom wildlife camera. For a kit that wraps everything up in a weatherproof enclosure made with love, time, and serious amounts of design and testing, take a look at Naturebytes’ wildlife cam kit.
Or, if you’re thinking that a robot mascot is more dependable than real animals for the fluffiness you need in order to start your January with something like productivity and with your soul intact, you might like to put your own spin on our robot buggy.
While we’re on the subject of getting to grips with the new year, do take a look at yesterday’s blog post, in which we suggest a New Year’s project that’s different from the usual resolutions. However you tackle 2018, we wish you an excellent year of creative computing.
Post Syndicated from Roy Hasson original https://aws.amazon.com/blogs/big-data/genomic-analysis-with-hail-on-amazon-emr-and-amazon-athena/
Genomics analysis has taken off in recent years as organizations continue to adopt the cloud for its elasticity, durability, and cost. With the AWS Cloud, customers have a number of performant options to choose from. These options include AWS Batch in conjunction with AWS Lambda and AWS Step Functions; AWS Glue, a serverless extract, transform, and load (ETL) service; and of course, the AWS big data and machine learning workhorse Amazon EMR.
For this task, we use Hail, an open source framework for exploring and analyzing genomic data that uses the Apache Spark framework. In this post, we use Amazon EMR to run Hail. We walk through the setup, configuration, and data processing. Finally, we generate an Apache Parquet–formatted variant dataset and explore it using Amazon Athena.
Because Hail is still under active development, you must compile it before you can start using it. To help simplify the process, you can launch the following AWS CloudFormation template that creates an EMR cluster, compiles Hail, and installs a Jupyter Notebook so that you’re ready to go with Hail.
There are a few things to note about the AWS CloudFormation template. You must provide a password for the Jupyter Notebook. Also, you must provide a virtual private cloud (VPC) to launch Amazon EMR in, and make sure that you select a subnet from within that VPC. Next, update the cluster resources to fit your needs. Lastly, the HailBuildOutputS3Path parameter should be an Amazon S3 bucket/prefix, where you should save the compiled Hail binaries for later use. Leave the Hail and Spark versions as is, unless you’re comfortable experimenting with more recent versions.
When you’ve completed these steps, the following files are saved locally on the cluster to be used when running the Apache Spark Python API (PySpark) shell.
The files are also copied to the Amazon S3 location defined by the AWS CloudFormation template so that you can include them when running jobs using the Amazon EMR Step API.
Collecting genome data
To get started with Hail, use the 1000 Genome Project dataset available on AWS. The data that you will use is located at s3://1000genomes/release/20130502/.
For Hail to process these files in an efficient manner, they need to be block compressed. In many cases, files that use gzip compression are compressed in blocks, so you don’t need to recompress—you can just rename the file extension to “.bgz” from “.gz” . Hail can process .gz files, but it’s much slower and not recommended. The simple way to accomplish this is to copy the data files from the public S3 bucket to your own and rename them.
The following is the Bash command line to copy the first five genome Variant Call Format (VCF) files and rename them appropriately using the AWS CLI.
Now that you have some data files containing variants in the Variant Call Format, you need to get the sample annotations that go along with them. These annotations provide more information about each sample, such as the population they are a part of.
Open your browser, and navigate to http://www.internationalgenome.org/data-portal/data-collection/phase-3. In the Samples box, choose Download the list. Then copy the file to the S3 bucket location where you saved the VCF files from the previous step.
Exploring the genomic data
In this section, you use the data collected in the previous section to explore genome variations interactively using a Jupyter Notebook. You then create a simple ETL job to convert these variations into Parquet format. Finally, you query it using Amazon Athena.
Let’s open the Jupyter notebook. To start, sign in to the AWS Management Console, and open the AWS CloudFormation console. Choose the stack that you created, and then choose the Output tab. There you see the JupyterURL. Open this URL in your browser.
Go ahead and download the Jupyter Notebook that is provided to your local machine. Log in to Jupyter with the password that you provided during stack creation. Choose Upload on the right side, and then choose the notebook from your local machine.
After the notebook is uploaded, choose it from the list on the left to open it.
For information about the Hail API, visit the Python API Documentation page.
Select the first cell, update the S3 bucket location to point to the bucket where you saved the compiled Hail libraries, and then choose Run. This code imports the Hail modules that you compiled at the beginning. When the cell is executing, you will see In [*]. When the process is complete, the asterisk (*) is replaced by a number, for example, In .
Next, run the subsequent two cells, which imports the Hail module into PySpark and initiates the Hail context.
The next cell imports a single VCF file from the bucket where you saved your data in the previous section. If you change the Amazon S3 path to not include a file name, it imports all the VCF files in that directory. Depending on your cluster size, it might take a few minutes.
Remember that in the previous section, you also copied an annotation file. Now you use it to annotate the VCF files that you’ve loaded with Hail. Execute the next cell—as a shortcut, you can select the cell and press Shift+Enter.
The import_table API takes a path to the annotation file in TSV (tab-separated values) format and a parameter named impute that attempts to infer the schema of the file, as shown in the output below the cell.
At this point, you can interactively explore the data. For example, you can count the number of samples you have and group them by population.
You can also calculate the standard quality control (QC) metrics on your variants and samples.
What if you want to query this data outside of Hail and Spark, for example, using Amazon Athena? To start, you need to change the column names to lowercase because Athena currently supports only lowercase names. To do that, use the two functions provided in the notebook and call them on your virtual dedicated server (VDS), as shown in the following image. Note that you’re only changing the case of the variants and samples schemas. If you’ve further augmented your VDS, you might need to modify the lowercase functions to do the same for those schemas.
In the current version of Hail, the sample annotations are not stored in the exported Parquet VDS, so you need to save them separately. As noted by the Hail maintainers, in future versions, the data represented by the VDS Parquet output will change, and it is recommended that you also export the variant annotations. So let’s do that.
Note that both of these lines are similar in that they export a table representation of the sample and variant annotations, convert them to a Spark DataFrame, and finally save them to Amazon S3 in Parquet file format.
Finally, it is beneficial to save the VDS file back to Amazon S3 so that next time you need to analyze your data, you can load it without having to start from the raw VCF. Note that when Hail saves your data, it requires a path and a file name.
After you run these cells, expect it to take some time as it writes out the data.
Discovering table metadata
Before you can query your data, you need to tell Athena the schema of your data. You have a couple of options. The first is to use AWS Glue to crawl the S3 bucket, infer the schema from the data, and create the appropriate table. Before proceeding, you might need to migrate your Athena database to use the AWS Glue Data Catalog.
Creating tables in AWS Glue
To use the AWS Glue crawler, open the AWS Glue console and choose Crawlers in the left navigation pane.
Then choose Add crawler to create a new crawler.
Next, give your crawler a name and assign the appropriate IAM role. Leave Amazon S3 as the data source, and select the S3 bucket where you saved the data and the sample annotations. When you set the crawler’s Include path, be sure to include the entire path, for example: s3://output_bucket/hail_data/sample_annotations/
Under the Exclusion Paths, type _SUCCESS, so that you don’t crawl that particular file.
Continue forward with the default settings until you are asked if you want to add another source. Choose Yes, and add the Amazon S3 path to the variant annotation bucket s3://your_output_bucket/hail_data/sample_annotations/ so that it can build your variant annotation table. Give it an existing database name, or create a new one.
Provide a table prefix and choose Next. Then choose Finish. At this point, assuming that the data is finished writing, you can go ahead and run the crawler. When it finishes, you have two new tables in the database you created that should look something like the following:
You can explore the schema of these tables by choosing their name and then choosing Edit Schema on the right side of the table view; for example:
Creating tables in Amazon Athena
If you cannot or do not want to use AWS Glue crawlers, you can add the tables via the Athena console by typing the following statements:
For the sample annotations
And for variant annotations:
Querying annotations with Athena
In the Amazon Athena console, choose the database in which your tables were created. In this case, it looks something like the following:
To verify that you have data, choose the three dots on the right, and then choose Preview table.
Indeed, you can see some data.
You can further explore the sample and variant annotations along with the calculated QC metrics that you calculated previously using Hail.
To summarize, this post demonstrated the ease in which you can install, configure, and use Hail, an open source highly scalable framework for exploring and analyzing genomics data on Amazon EMR. We demonstrated setting up a Jupyter Notebook to make our exploration easy. We also used the power of Hail to calculate quality control metrics for variants and samples. We exported them to Amazon S3 and allowed a broader range of users and analysts to explore them on-demand in a serverless environment using Amazon Athena.
If you found this post useful, be sure to check out Interactive Analysis of Genomic Datasets Using Amazon Athena and Building High-Throughput Genomics Batch Workflows on AWS: Introduction (Part 1 of 4).
About the Author
Roy Hasson is a Global Business Development Manager for AWS Analytics. He works with customers around the globe to design solutions to meet their data processing, analytics and business intelligence needs. Roy is big Manchester United fan cheering his team on and hanging out with his family.
RDPY is an RDP Security Tool in Twisted Python with RDP Man in the Middle proxy support which can record sessions and Honeypot functionality.
RDPY is a pure Python implementation of the Microsoft RDP (Remote Desktop Protocol) protocol (client and server side). RDPY is built over the event driven network engine Twisted. RDPY support standard RDP security layer, RDP over SSL and NLA authentication (through ntlmv2 authentication protocol).
RDPY RDP Security Tool Features
RDPY provides the following RDP and VNC binaries:
- RDP Man In The Middle proxy which record session
- RDP Honeypot
- RDP Screenshoter
- RDP Client
- VNC Client
- VNC Screenshoter
- RSS Player
RDPY is fully implemented in python, except the bitmap decompression algorithm which is implemented in C for performance purposes.
Security updates have been issued by Debian (extplorer and libraw), Fedora (mingw-libsoup, python-tablib, ruby, and subversion), Mageia (avidemux, clamav, nasm, php-pear-CAS, and shutter), Oracle (xmlsec1), Red Hat (openssl tomcat), Scientific Linux (authconfig, bash, curl, evince, firefox, freeradius, gdm gnome-session, ghostscript, git, glibc, gnutls, groovy, GStreamer, gtk-vnc, httpd, java-1.7.0-openjdk, kernel, libreoffice, libsoup, libtasn1, log4j, mariadb, mercurial, NetworkManager, openldap, openssh, pidgin, pki-core, postgresql, python, qemu-kvm, samba, spice, subversion, tcpdump, tigervnc fltk, tomcat, X.org, and xmlsec1), SUSE (git), and Ubuntu (augeas, cvs, and texlive-base).
Security updates have been issued by Mageia (atril, mpg123, perl-SOAP-Lite, and virtualbox), openSUSE (kernel and libzypp, zypper), Oracle (authconfig, bash, curl, gdm and gnome-session, ghostscript, git, glibc, gnutls, gtk-vnc, kernel, libreoffice, libtasn1, mariadb, openldap, openssh, pidgin, postgresql, python, qemu-kvm, samba, tcpdump, tigervnc and fltk, and tomcat), Red Hat (kernel, kernel-rt, openstack-neutron, and qemu-kvm), and SUSE (puppet and tcmu-runner).
Security updates have been issued by Debian (varnish), Fedora (gcc, gcc-python-plugin, libtool, mingw-c-ares, and php-PHPMailer), Red Hat (bash, curl, evince, freeradius, gdm and gnome-session, ghostscript, git, glibc, golang, GStreamer, gtk-vnc, kernel, kernel-rt, libtasn1, mariadb, openldap, openssh, pidgin, postgresql, python, qemu-kvm, qemu-kvm-rhev, samba, tigervnc and fltk, tomcat, and X.org X11 libraries), Slackware (gnupg), and Ubuntu (apache2, lxc, and webkit2gtk).
Security updates have been issued by CentOS (kernel), Debian (graphicsmagick, imagemagick, kde4libs, and puppet), Fedora (FlightCrew, kernel, libvncserver, and wordpress), Gentoo (adobe-flash, smb4k, teeworlds, and xen), Mageia (kernel, kernel-linus, kernel-tmb, and perl-CGI-Emulate-PSGI), openSUSE (GraphicsMagick and rpcbind), Oracle (kernel), Red Hat (kernel and kernel-rt), and Scientific Linux (kernel).
Security updates have been issued by Debian (botan1.10, mysql-5.5, and rtmpdump), Fedora (collectd, firefox, java-1.8.0-openjdk, libdwarf, nss-softokn, nss-util, and tigervnc), Red Hat (httpd24-httpd and python27), and SUSE (kernel).
Security updates have been issued by Debian (feh, freetype, and radare2), Fedora (kernel and libsndfile), openSUSE (audiofile, dracut, gstreamer, gstreamer-plugins-bad, jasper, libpng15, proftpd, and tigervnc), Oracle (qemu-kvm), Red Hat (kernel, libreoffice, and qemu-kvm-rhev), and SUSE (bind and tiff).
Using servomotors and shadow tracking, Acrophobia 1.0’s mission to give a Raspberry Pi a nervous disposition is a rolling success.
Acrophobia, a nervous machine with no human-serving goal, but with a single fear: of dropping the ball. Unlike any other ball balancing machine, Acrophobia has no interest in keeping the ball centered. She is just afraid to drop it, getting trapped in near-infinite loops of her own making.
How to give a Raspberry Pi Acrophobia
Controlling the MDF body and 3D printed wheels, the heart of Acrophobia contains a Raspberry Pi 2 and a Camera Module. The camera tracks a shadow across a square of semi-elastic synthetic cloth, moving the Turnigy S901D servomotors at each corner to keep it within a set perimeter.
The shadow is cast by a small ball, and the single goal of Acrophobia is to keep that ball from dropping off the edge.
Acrophobia, a nervous machine with no human-serving goal, but with a single fear: of dropping the ball.
Unlike any other ball-balancing machine, Acrophobia has no interest in keeping the ball centered. She is just afraid to drop it, getting trapped in near-infinite loops of her own making.
To set up the build, the Raspberry Pi is accessed via VNC viewer on an iPad. Once the Python code is executed, Acrophobia is stuck in its near-infinite nightmare loop.
This video for Acrophobia 1.0 has only recently been uploaded to Vimeo, but the beta recording has been available for some time. You can see the initial iteration, created by George Adamopoulos, Dafni Papadopoulou, Maria Papacharisi and Filippos Pappas for the National Technical University of Athens School of Architecture Undergraduate course here, and compare the two. The beta video includes the details of the original Arduino/webcam setup that was eventually replaced by the Raspberry Pi and Camera Module.
I recently saw a similar build to this, again using a Raspberry Pi, which used tablet computers as game controllers. Instead of relying on a camera to track the ball, two players worked together to keep the ball within the boundaries of the sheet.
Naturally, now that I need the video for a blog post, I can’t find it. But if you know what I’m talking about, share the link in the comments below.
And if you don’t, it’s time to get making, my merry band of Pi builders. Who can turn Acrophobia into an interactive game?