Tag Archives: javascript

New improved version of the node-netflowv9 module for Node.JS

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/new-improved-version-of-node-netflowv9.html

As I have mentioned before, I have implemented a NetFlowV9 compatible library for decode of Cisco NetFlow version 9 packets for Node.JS.
Now I am upgrading it to version 0.1 which has few updates:
  • bug fixes (including avoidance of an issue that happens with ASR9k and IOS XR 4.3)
  • now you can start the collector (with a second parameter of true) in a mode where you want to receive only one call back per packet instead of one callback per flow (the default mode). That could be useful if you want to count the lost packets (otherwise the relation netflow packet – callback is lost)
  • decrease of the code size
  • now the module compiles the templates dynamically into a function (using new Function). I like this approach very much, as it creates really fast functions (in contrast to eval, Function is always JIT processed) and it allows me to spare loops, function calls and memory copy. I like to do things like that with every data structure that allows it. Anyway, as an effect of this, the new module is about 3 times faster with all the live tests I was able to perform

Simple example for Node.JS sflow collector

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/simple-example-for-nodejs-sflow.html

Sometimes you can use the SFlow or Netflow to extra add intelligence to your network. The collectors available on internet are usually there just to collect and store data used for accounting or nice graphics. But the collectors are either not allowing you to execute your own code in case of certain rules/thresholds reached, or do not react in real time (in general, the protocols delays you too. You cannot expect NetFlow accounting to be used in real time at all, while SFlow has modes that are bit more fast to react, by design, it is still not considered to be real-time sampling/accounting).
Just imagine you have a simple goal – you want to automatically detect floods and notify the operators or you can even automatically apply filters.
If you have an algorithm that can distinguish the incorrect traffic from the normal traffic from NetFlow/SFlow sampling you may like to execute an operation immediately when that happens.
The modern DoS attacks and floods may be complex and hard to detect. But mainly it is hard to make the currently available NetFlow/SFlow collector software to do that for you and then trigger/execute external application.
However, it is very easy to program it yourself.
I am giving you a simple example that uses the node-sflow module to collect packet samples, measure how many of them match a certain destination ip address and if they are above certain pps thresholds to execute an external program (that is supposed to block that traffic). Then after a period of time it will execute another program (that is supposed to unblock the traffic).
This program is very small – about 120 lines of code and allows you to use complex configuration file where you can define a list of rules that can match optionally vlans and networks for the sampled packet and then count how many samples you have per destination for that rule. The rule list is executed until first match in the configured order within the array, so that allows you to create black and white lists and different thresholds per networks and vlans, or to have different rules per overlapped ip addresses as long as they belong to different vlans.
Keep in mind this is just an example software there just for your example, showing you how to use node-sflow and pcap modules together! It is not supposed to be used in production, unless you definitely know what you are doing!
The goal of this example it here just to show you how easy is to add extra logic within your network.
The code is available on git-hub here https://github.com/delian/sflow-collector/

RTMP Api for Node.JS to ease the implementation of RTMP servers and clients

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/rtmp-api-for-nodejs-to-ease.html

Hello to all,
As I mentioned before, I needed to implement a RTMP streaming server in Node.JS. All of the available modules for implementation of RTMP in Node’s NPM repository were incomplete, incorrect or unusable. Not only that but the librtmp used by libav tools like avconv and avplay was incorrect and incomplete.
The same with most of the implementation I’ve checked (covering perl, python, others). I’ve tried to fix few of them but at the end I had to write one on my own.
This is my library of RTMP related tools and API for Node.JS. It is named node-rtmpapi and is available in the npm repository. Also you can get it here – https://github.com/delian/node-rtmpapi
It works well for me, and it has been tested with MistServer, OrbanEncoders and librtmp (from libav).
That does not mean it will work for you, though 🙂
RTMP is quite badly documented protocol and extremely badly implemented.
During my tests I have seen issues like crash of libraries (including the Adobe’s original one) if the upper layer commands has been sent in unexpected order (although this is allowed by the RTMP protocol and the order of the upper layer commands is not documented at all). Also I have seen (within Adobe’s rtmp library) incorrect implementation of the setPeerBandwidth command.
Generally, each different RTMP implementation is on its own and the only way to make it work is to adjust and tune it according to the software you communicate with.
Therefore I separated my code in utils that allows me to write my own RTMP server relatively easy and to adjust it according to my needs.
The current library supports only TCP as a transport (although TLS and HTTP/HTTPS is easy to be implemented, I haven’t focused on it yet).
It provides separate code that implements streaming (readQueue), the chunk layer of the protocol (rtmpChunk), the upper layer messaging (assembling and disassembling of message over chunks, rtmpMessage), stream processing (rtmpStream) and basic server implementation without the business logic (rtmpServer).
Simplified documentation is provided at the git-hub repository.
The current library uses callbacks for each upper layer command it receives. I am planning to migrate the code to use node streams and to trigger events per command, instead of callbacks. This will extremely simplify the usage and the understanding of the library for a node programmer. However, this is the future and in order to preserve compatibility, I will probably name it something different (like node-streams-rtmpapi)

AMF0/3 encoding/decoding in Node.JS

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/amf03-encodingdecoding-in-nodejs.html

I am writing my own RTMP restreamer (RTMP is Adobe’s dying streaming protocol widely used with Flash) in Node.JS.
Although, there are quite of few RTMP modules, no one is complete, nor operates with Node.JS buffers, nor support fully ether AMF0 or AMF3 encoding and decoding.
So I had to write one on my own.
The first module is the AMF0/AMF3 utils that allow me to encode or decode AMF data. AMF is a binary encoding used in Adobe’s protocols, very similar to BER (used in ITU’s protocols) but supporting complex objects. In general the goal of AMF is to encode ActiveScript objects into binary. As ActiveScript is a language belonging to the JavaScript’s familly, basically the ActiveScript’s objects are javascript objects (with the exception of some simplified arrays).
My module is named node-amfutils and is now available in the public NPM repository as well as here https://github.com/delian/node-amfutils
It is not fully completed nor very well tested as I have very limited environment to do the tests. However, it works for me and provides the best AMF0 and AMF3 support currently available for Node.JS – 
  • It can encode/decode all the objects defined in both AMF0 and AMF3 (the other AMF modules in the npm repository supports partial AMF0 or partial AMF3)
  • It uses Node.JS buffers (it is not necessary to do string to buffer to string conversion, as you have to do with the other modules)

It is easy to use this module. You just have to do something like this:

var amfUtils = require('node-amfutils');
var buffer = amfUtils.amf0Encode([{ a: "xxx"},b: null]);

SFlow version 5 module for Node.JS

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/sflow-version-5-module-for-nodejs.html

Unfortunately, as with NetFlow Version 9, SFlow version 5 (and SFlow in general) has not been very well supported by the Node.JS community up to now.
I needed modern SFlow version 5 compatible module, so I had to write one on my own.
Please welcome the newest module in Node.JS’s NPM that can decode SFlow version 5 packets and be used in the development of simple and easy SFlow collectors! The module is named node-sflow and you can look at its code here https://github.com/delian/node-sflow
Please be careful, as in the next days I may change the object structure of the flow representation to simplify it! Any tests and experiments are welcome.
The sflow module is available in the public npm (npm install node-sflow) repository.
To use it you have to do:
var Collector = require('node-sflow');

Collector(function(flow) {
    console.log(flow);
}).listen(3000); 
In general SFlow is much more powerful protocol than NetFlow, even it its latest version (version 9). It can represent more complex counters, report about errors, drops, full packet headers (not only their properties), collect information from interfaces, flows, vlans, and combine them in a much more complex reports.

However, the SFlow support in the agents – the networking equipment is usually extremely simplified – far from the richness and complexity the SFlow protocol may provide. Most of the vendors just do packet sampling and send them over SFlow as raw packet/frame header with an associated unclear counter.

In case of you having the issue specified above, this module cannot help much. You will just get the raw packet header (usually Ethernet + IP header) as a Node.JS buffer and then you have to decode it on your own. I want to keep the node-sflow module simple and I don’t plan to decode raw packet headers there as this feature is not a feature of the SFlow itself.

If you need to decode the raw packet header I can suggest one easy solution for you. You can use the pcap module from the npm repository and decode the raw header with it:

var Collector = require('node-sflow');
var pcap = require('pcap');

Collector(function(flow) {
    if (flow && flow.flow.records && flow.flow.records.length>0) {
        flow.flow.records.forEach(function(n) {
            if (n.type == 'raw') {
                if (n.protocolText == 'ethernet') {
                    try {
                        var pkt = pcap.decode.ethernet(n.header, 0);
                        if (pkt.ethertype!=2048) return;
                        console.log('VLAN',pkt.vlan?pkt.vlan.id:'none','Packet',pkt.ip.protocol_name,pkt.ip.saddr,':',pkt.ip.tcp?pkt.ip.tcp.sport:pkt.ip.udp.sport,'->',pkt.ip.daddr,':',pkt.ip.tcp?pkt.ip.tcp.dport:pkt.ip.udp.dport)
                    } catch(e) { console.log(e); }
                }
            }
        });
    }
}).listen(3000);