Tag Archives: javascript

UNIFICli – a CLI tool to manage Ubiquiti’s Unifi Controller

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2017/04/unificli-cli-tool-to-manage-ubiquitis.html

As mentioned earlier, I made a nodejs library interface to the Ubiquiti Unifi Controller’s REST API which is available here – https://github.com/delian/node-unifiapi
Now I am introducing a small, demo, CLI interface, which uses that same library to remotely connect and configure Ubiquiti Unifi Controller (or Ubiquiti UC-CK Cloud Key).
This software is available on GitHub here – https://github.com/delian/unificli and its main goal for me is to be able to test the node-unifiapi library. The calls and parameters are almost 1:1 with the library and this small code provides great example how such tools could be built.
This tool is able to connect to a controller either via direct HTTPS connection or via WebRTC trough Ubiquiti’s Unifi Clould network (they name it SDN). And also you have a command you could use to connect to wireless access point via SSH over WebRTC.
The tool is not completed, neither have any goal. Feel free to fix bugs, extend it with features or provide suggestions. Any help with the development will be appreciated.
Commands can be executed via the cli too:

npm start connectSSH 00:01:02:03:04:05 -l unifikeylocation

node-unifiapi – NodeJS API to access Ubiquiti Unifi Controller API

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2017/03/node-unifiapi-nodejs-api-to-access.html

I have completed the initial rewrite of the PHP based UniFi-API-Browser to JavaScript for NodeJS.
The module and some auto generated documentation with some basic examples is available here:
The supported features includes all of the UniFi-API-Browser calls via the direct REST http calls or via WebRTC (using Ubiquiti Unifi Cloud) as well as SSH access to Unifi Access Points via WebRTC.

Private Properties with JavaScript (prototypes)

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2017/01/private-properties-with-javascript.html

Some are saying that JavaScript does not have private properties with prototypes and it is not like TypeScript or Python.
However, that is not fully correct, especially with this comparison.
Python doesn’t have anything private. All the scopes are globally addressable and all the variables, properties and methods are accessible globally (as long as you know the scope path, which could be easily traced from within the software itself).
TypeScript is just a pre-processor. It will do extra semantic checks during compilation, but no enforcement of anything on runtime. 
However, it is generally untrue that you have no private properties available in JavaScript.
You do have scope inheritance and you don’t have global access to any scope from outside. Therefore you can make private properties easily.
For example, if you have:
function F1() {
  var XXX = 0;
  return function () {
     return XXX;
  }
}
F1()() will respond with the value of XXX but there is no way to access XXX from outside.
So what about prototyping?
You can do this there too.
function F1() {
  var XXX = ‘yyy’; // Private property
  function F1() {
     // You constructor is here
     this.YYY = ‘xxx’; // public property
  }
  F1.prototype.method1 = function() {
    return XXX + this.YYY;
  }
  return new F1();
}
Then if you do:
x = F1();
or
x = new F1();  // Both works the same way
and check:
console.log(x.YYY); // output ‘xxx’
console.log(x.XXX); // output Error
console.log(x.method1()); // output ‘yyyxxx’
So everything works! You have prototyping with private and public properties.
The same way you can do private and public methods.
The only thing that will not work with this method is instanceof
x instanceof F1 will respond with false, because x is not an instance of the upper F1, but the inner F1

The current ES6 standard have classes, but they are essentially a wrapper on top of prototype. So this technique still apply. Additionally with Object.defineProperty you could apply extra protection on top of a property.

There is another approach to the problem. Without prototypes.


function F1() {
  if (!this instanceof F1) return; // Protect against global scope polution
  var XXX = ‘yyy’; // Private property
  this.YYY = ‘xxx’; // Public property
  function PrivateMethod() {
    return XXX + this.YYY;
  }
  this.method1 = function() {
    return PrivateMethod();
  }
}

With the shown above approach you have both private and public methods and properties, with workable instanceof

Passport-http Digest Authentication and Express 4+ bug fix

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2015/10/passport-http-digest-authentication-and.html

Passport is quite popular environment for implementing authentication under Node.JS and Express Framework.
Passport-http is a plug-in that provides Digest Authentication interfaces and is very popular for Passport.
However, since Express 4, the default approach to the Express routes is them to be relative.
Example – 
Express <=3 default application assumed that every file who extend Express with routes should specify the full path to the route in a similar to this approach:
app.js:
app.get(‘{fullpath}’,function)…
Or:
require(‘routes/users’)(app)
Where users.js do as well:
app.get(‘{fullpath’,function)…
The default app of Express 4+ uses relative routes, which is quite better as it allows full isolation between the modules of the application.
Example:
app.js:
app.use(‘/rest’,require(‘routes/rest’));
Where rest.js has:
router = require(‘express’).Router();
router.get(‘/data’, function)… // where data is relative url and the actual will be /rest/data
This simplifies the readability. Also it allows you to separate the authentication. You could have different authentication approach or configuration to each different module. And that works well with Passport.
For example:
app.js:
app.use(passport.initialize());
app.use(passport.session());
rest.js:
var DigestStrategy = require(‘passport-http’).DigestStrategy;
… here there should be a code for authentication function using Digest …
and then:
router.get(‘/data’,authentication,function) ….
This simplifies, makes it more readable and isolates very much the code necessary for authentication.
Personally, I write my own authentication functions in a separate module, then I include them in the express route module where I want to use them and it became even more simpler:
rest.js:
var auth = require(‘../libs/auth.js’);
Router.get(‘/data’, auth(‘admins’), function) …
I even could apply different permissions, roles like – if you have pre authenticated session, then the interface will not ask you for authentication (saved one RTT) but if you don’t it will ask you for digest authentication. Quite simple and quite readable.
However, all this does not work with Passport-http, because of a very small bug within.
The bug:
For security reasons, passport-http module verifies that the authentication URI from the customer request is the same as the URL requested authentication. However, the authentication URI (creds.uri) is always full path, but it is compared to req.url which is always relative path. The comparision has to be between creds.uri and req.baseUrl+req.url.
And this is my fix proposed to the author of passport-http, which I hope will be merged with the code.

EmbeddedJS – async reimplementation

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2015/09/embeddedjs-async-reimplementation.html

I like using Embedded JS together with RequireJS in very small projects. It is very small, lighting fast and extremely simple, powerful and extensive.
However, there are hundreds of implementations, most of them out of date, and particularly the implementation I like is not supported anymore. But that I mean not only that the author is not supporting it, but also it works unpredictably in some browsers because it rely on Sync XMLHttpRequest which is not allowed in the main thread anymore.
So I decided to rewrite that EJS implementation myself, in a way I could use it in async mode.
So allow me to introduce the new Async Embedded JS implementation, which is here at https://github.com/delian/embeddedjs
It supports Node, AMD (requirejs) and globals (as the original) and detect them automatically.
A little documentation can be found here https://github.com/delian/embeddedjs/blob/master/README.md
The new code is written in ES5 and uses the new Function method from ES6. That makes it working only in modern browsers (IE10+). But it makes it really really fast. By current estimates about twice faster than the original.
It is still work in progress (no avoidance of cached URLs for example), but works perfectly fine for me.
If you use it and hit a bug, please report it at the Issues page at Github

JavaScript private methods and variables

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2015/08/javascript-private-methods-and-variables.html

Often I am told that JavaScript has no real private methods and variables the same way as this is for Python or Perl.
Although, that is true for Python and Perl (all scopes and their content are publicly accessible and there is always a way to list and access the content) it is not true for JavaScript.
JavaScript have one of the most powerful scoping technique available for a main-stream programing language and provides you with the ability to hide and make private some content if needed.
Let me show you an example:
function MyClass() {
    var fullyPrivateVar = ‘xxxx’;
    this.notFullyPrivateVar = ‘yyyy’;

    function privateMethod() {
        return fullyPrivateVar;
    }

    this.publicMethodWithPrivateAccess = function() { return privateMethod() }
}

MyClass.prototype.publicMethodWithNoPrivateAccess = function() {
    return this.notFullyPrivateVar;
}

x = new MyClass();
console.log(x.publicMethodWithNoPrivateAccess());
yyyy
console.log(x.publicMethodWithPrivateAccess());
xxxx

I think the example is self explanatory, but let me say few words on it:
In JavaScript you can have hierarchy of scopes and you can define unlimited amount function in functions levels (and scope levels) in difference to other programming languages where you have limited / fixed scope levels.
So I have a privateMethod and fullyPrivateVar defined within MyClass. Because of it, only things defined in this scope will have access to them. Like publicMethodWithPrivateAccess that I assign with the constructor to the newly constructed object. However,  publicMethodWithNoPrivateAccess is defined outside of the scope and has no access to those values. But it has access to everything in the this scope. One could argue, that this approach takes more memory for the methods because I define a new method with every initialization of the class (the function assignment). But that is not true. The function in the function is having a static code (it is not part of eval+variable) and is compiled during the initial javascript processing of the JS virtual machine. So it will not take extra memory. Only during execution the function will be dynamically assigned with the private scope of the constructor, something you want anyway.

Node.JS module to access Cisco IOS XR XML interface

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2015/03/nodejs-module-to-access-cisco-ios-xr.html

Hello to all,
This is the early version of my module for Node.JS that allows configuring routers and retrieving information over Cisco IOS XR’s XML interface.
The module is in its early phases – it still does not read IOS XR schema files and therefore decode the data (in JSON) in a little ugly way (too much arrays). I am planning to fix it, so there may be changes in the responses.
Please see bellow the first version of the documentation I’ve set in the github:

Module for Cisco XML API interface IOS XR

This is a small module that implements interface to Cisco IOS XR XML Interface.
This module open an maintain TCP session to the router, sends requests and receive responses.

Installation

To install the module do something like that:
npm install node-ciscoxml

Usage

It is very easy to use this module. See the methods bellow:

Load the module

To load and use the module, you have to use a code similar to this:
var cxml = require('node-ciscoxml');
var c = cxml( { ...connect options.. });

Module init and connect options

host (default 127.0.0.1) – the hostname of the router where we’ll connect
port (default 38751) – the port of the router where XML API is listening
username (default guest) – the username used for authentication, if username is requested by the remote side
password (default guest) – the password used for authentication, if password is requested by the remote side
connectErrCnt (default 3) – how many times it will retry to connect in case of an error
autoConnect (default true) – should it try to auto connect to the remote side if a request is dispatched and there is no open session already
autoDisconnect (default 60000) – how much milliseconds we will wait for another request before the tcp session to the remote side is closed. If the value is 0, it will wait forever (or until the remote side disconnects). Bear in mind autoConnect set to false does not assume autoDisconnect set to 0/false as well.
userPromptRegex (default (Username|Login)) – the rule used to identify that the remote side requests for a username
passPromptRegex (default Password) – the rule used to identify that the remote side requests for a password
xmlPromptRegex (default XML>) – the rule used to identify successful login/connection
noDelay (default true) – disables the Nagle algorithm (true)
keepAlive (default 30000) – enabled or disables (value of 0) TCP keepalive for the socket
ssl (default false) – if it is set to true or an object, then SSL session will be opened. Node.js TLS module is used for that so if the ssl points to an object, the tls options are taken from it. Be careful – enabling SSL does not change the default port from 38751 to 38752. You have to set it explicitly!
Example:
var cxml = require('node-ciscoxml');
var c = cxml( {
    host: '10.10.1.1',
    port: 5000,
    username: 'xmlapi',
    password: 'xmlpass'
});

connect method

This method forces explicitly a connection. It could accept any options of the above.
Example:
var cxml = require('node-ciscoxml');
var c = cxml();
c.connect( {
    host: '10.10.1.1',
    port: 5000,
    username: 'xmlapi',
    password: 'xmlpass'
});
The connect method is not necessary to be used. If autoConnect is enabled (default) the module will automatically open and close tcp connections when needed.
Connect supports callback. Example:
var cxml = require('node-ciscoxml');
cxml().connect( {
    host: '10.10.1.1',
    port: 5000,
    username: 'xmlapi',
    password: 'xmlpass'
}, function(err) {
    if (!err)
        console.log('Successful connection');
});
The callback may be the only parameter as well. Example:
var cxml = require('node-ciscoxml');
cxml({
    host: '10.10.1.1',
    port: 5000,
    username: 'xmlapi',
    password: 'xmlpass'
}).connect(function(err) {
    if (!err)
        console.log('Successful connection');
});
Example with SSL:
var cxml = require('node-ciscoxml');
var fs = require('fs');
cxml({
    host: '10.10.1.1',
    port: 38752,
    username: 'xmlapi',
    password: 'xmlpass',
    ssl: {
          // These are necessary only if using the client certificate authentication
          key: fs.readFileSync('client-key.pem'),
          cert: fs.readFileSync('client-cert.pem'),
          // This is necessary only if the server uses the self-signed certificate
          ca: [ fs.readFileSync('server-cert.pem') ]
    }
}).connect(function(err) {
    if (!err)
        console.log('Successful connection');
});

disconnect method

This method explicitly disconnects a connection.

sendRaw method

.sendRaw(data,callback)
Parameters:
data – a string containing valid Cisco XML request to be sent
callback – function that will be called when a valid Cisco XML response is received
Example:
var cxml = require('node-ciscoxml');
var c = cxml({
    host: '10.10.1.1',
    port: 5000,
    username: 'xmlapi',
    password: 'xmlpass'
});

c.sendRaw('<Request><GetDataSpaceInfo/></Request>',function(err,data) {
    console.log('Received',err,data);
});

sendRawObj method

.sendRawObj(data,callback)
Parameters:
data – a javascript object that will be converted to a Cisco XML request
callback – function that will be called with valid Cisco XML response converted to javascript object
Example:
var cxml = require('node-ciscoxml');
var c = cxml({
    host: '10.10.1.1',
    port: 5000,
    username: 'xmlapi',
    password: 'xmlpass'
});

c.sendRawObj({ GetDataSpaceInfo: '' },function(err,data) {
    console.log('Received',err,data);
});

rootGetDataSpaceInfo method

.rootGetDataSpaceInfo(callback)
Equivalent to .sendRawObj for GetDataSpaceInfo command

getNext

Sends getNext request with a specific id, so we can retrieve the rest of the previous operation if it has been truncated.
id – the ID callback – the callback with the data (in js object format)
Keep in mind next response may be truncated as well, so you have to check for IteratorID all the time.
Example:
var cxml = require('node-ciscoxml');
var c = cxml({
    host: '10.10.1.1',
    port: 5000,
    username: 'xmlapi',
    password: 'xmlpass'
});

c.sendRawObj({ Get: { Configuration: {} } },function(err,data) {
    console.log('Received',err,data);
    if ((!err) && data && data.Response.$.IteratorID) {
        return c.getNext(data.Response.$.IteratorID,function(err,nextData) {
            // .. code to merge data with nextData
        });
    }
    // .. code
});

sendRequest method

This method is equivalent to sendRawObj but it can automatically detect the need and resupply GetNext requests so the response is absolutley full. Therefore this method should be the preferred method for sending requests that expect very large replies.
Example:
var cxml = require('node-ciscoxml');
var c = cxml({
    host: '10.10.1.1',
    port: 5000,
    username: 'xmlapi',
    password: 'xmlpass'
});

c.sendRequest({ GetDataSpaceInfo: '' },function(err,data) {
    console.log('Received',err,data);
});

requestPath method

This is a method equivalent to sendRequest but instead of an object, the request may be formatted in a simple path string. This metod is not very useful for complex requests. But its value is in the ability to simplify very much the simple requests. The response is in JavaScript object
Example:
var cxml = require('node-ciscoxml');
var c = cxml({
    host: '10.10.1.1',
    port: 5000,
    username: 'xmlapi',
    password: 'xmlpass'
});

c.requestPath('Get.Configuration.Hostname',function(err,data) {
    console.log('Received',err,data);
});

reqPathPath method

This is the same method as requestPath, but the response is not an object, but a path array. The method supports optional filter, which has to be a RegExp object and all paths and values will be tested against it Only those returning true will be included in the response array.
Example:
var cxml = require('node-ciscoxml');
var c = cxml({
    host: '10.10.1.1',
    port: 5000,
    username: 'xmlapi',
    password: 'xmlpass'
});

c.reqPathPath('Get.Configuration.Hostname',/Hostname/,function(err,data) {
    console.log('Received',data[0]);
    // The output should be something like
    // [ 'Response("MajorVersion"="1","MinorVersion"="0").Get.Configuration.Hostname("MajorVersion"="1","MinorVersion"="0")',
           'asr9k-router' ] 
});
This method could be very useful for getting simple responses and configurations.

getConfig method

This method requests the whole configuration of the remote device and return it as object
Example:
c.getConfig(function(err,config) {
    console.log(err,config);
});

cliConfig method

This method is quite simple, it executes a command(s) in CLI Configuration mode and return the response in JS Object. You have to know that any configuration change in IOS XR is not effective unless it is committed!
Example:
c.cliConfig('username testuser\ngroup operator\n',function(err,data) {
    console.log(err,data);
    c.commit();
});

cliExec method

Executes a command(s) in CLI Exec mode and return the response in JS Object.
c.cliExec('show interfaces',function(err,data) {
    console.log(err,data?data.Response.CLI[0].Exec[0]);
});

commit method

Commit the current configuration.
Example:
c.commit(function(err,data) {
    console.log(err,data);
});

lock method

Locks the configuration mode.
Example:
c.lock(function(err,data) {
    console.log(err,data);
});

unlock method

Unlocks the configuration mode.
Example:
c.unlock(function(err,data) {
    console.log(err,data);
});

Configure Cisco IOS XR for XML agent

To configure IOS XR for remote XML configuration you have to:
Ensure you have *mgbl*** package installed and activated! Without it you will have no xml agentcommands!
Enable the XML agent with a similar configuration:
xml agent
  vrf default
    ipv4 access-list SECUREACCESS
  !
  ipv6 enable
  session timeout 10
  iteration on size 100000
!
You can enable tty and/or ssl agents as well!
(Keep in mind – full filtering of the XML access has to be done by the control-plane management-plane command! The XML interface does not use VTYs!)
You have to ensure you have correctly configured aaa as the xml agent uses default method for both authentication and authorization and that cannot be changed (last verified with IOS XR 5.3).
You have to have both aaa authentication and authorization. If authorization is not set (aaa authorization default local or none), you may not be able to log in. And you shall ensure that both the authentication and authorization share the same source (tacacs+ or local).
The default agent port is 38751 for the default agent and 38752 for SSL.

Debugging

The module uses “debug” module to log its outputs. You can enable the debugging by having in your code something like:
require('debug').enable('ciscoxml');
Or setting DEBUG environment to ciscoxml before starting the Node.JS

ExtJS short highlight of a gridrow in case of an update

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2015/03/extjs-short-highlight-of-gridrow-in.html

In my ExtJS code I often update the values of records associated with Stores/Data Models associated with Grid Panel. 
Sometimes this an automated update received from the web server (an example you can see here) sometimes not.
However, as a user I like to be able to see if somewhere in the grid some data has been modified.
And to create a code, that highlight for a moment the row or a cell where a change happened is actually not so hard. 
Here is a small example how you can do that – you just have to bind to event ‘itemupdate‘ on a tableview. See the example bellow, that is supposed to be self explanatory:

Ext.ComponentQuery.query(‘#mytableview’).on(‘itemupdate’, function(record, index, node) {
    node.className = node.className + ‘ x-grid-item-alt’;
    setTimeout(function() {
        node.className = node.className.replace(/x-grid-item-alt/, ”);
    }, 500);
});

node.js module implementing EventEmitter interface using MongoDB tailable cursors as backend

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2015/03/nodejs-module-implementing-eventemitter.html

I’ve published in the npm a new module, that I’ve used privately for a long time which implements EventEmitter interface using MongoDB tailable cursors as backend.
This module could be used as a messaging bus between processes or even between node.js modules as it allows implementing EventEmitter wihout need of sharing the object instance in advance.
Please see the first version of the README.md bellow:

Module for creating event bus interface based on MongoDB tailable cursors

The idea behind this module is to create EventEmitter like interface, which uses MongoDB capped collections and tailable cursors as an internal messaging bus. This model has a lot of advantages, especially if you already use MongoDB in your project.
The advantages are:
You don’t have to exchange the event emitter object between different pages or even different processes (forked, clustered, living on separate machines). As long as you use the same mongoUrl and capped collection name, you can exchange information. This way you can even create applications that runs on a different hardware and they may exchanging events and data as if they are the same application! Also your events are stored in a collection and could be used as a transaction log latley (mongodb’s own transaction log is implemented with capped collections).
It simplifies an application development very much.

Installation

To install the module run the following command:
npm install node-mongotailableevents

Short

It is easy to use that module. Look at the following example:
var ev = require('node-mongotailableevents');

var e = ev( { ...options ... }, callback );

e.on('event',callback);

e.emit('event',data);

Initialization and options

The following options can be used with the module
  • mongoUrl (default mongodb://127.0.0.1/test) – the URL to the mongo database
  • mongoOptions (default none) – Specific options to be used for the connection to the mongo database
  • name (default tailedEvents) – the name of the capped collection that will be created if it does not exists
  • size (default 1000000) – the maximum size of the capped collection (when reached, the oldest records will be automatically removed)
  • max (default 1000) – the maximum size in amount of records for the capped collection
You can call and create a new event emitter instance without options:
var ev = require('node-mongotailableevents');
var e = ev();
Or you can call and create a event emitter instance with options:
var ev = require('node-mongotailableevents');
var e = ev({
   mongoUrl: 'mongodb://127.0.0.1/mydb',
   name: 'myEventCollection'
});
Or you can call and create a event emitter instance with options and callback, which will be called when the collection is created successfuly:
var ev = require('node-mongotailableevents');
ev({
   mongoUrl: 'mongodb://127.0.0.1/mydb',
   name: 'myEventCollection'
}, function(err, e) {
    console.log('EventEmitter',e);
});
Or you can call and create event emitter with just callback (and default options):
ev(function(err, e) {
    console.log('EventEmitter',e);
});

Usage

This module inherits EventEmitter, so you can use all of the EventEmitter methods. Example:
ev(function(err, e) {
    if (err) throw err;

    e.on('myevent',function(data) {
        console.log('We have received',data);
    });

    e.emit('myevent','my data');
});
The best feature is that you can exchange events between different pages or processes, without the need of exchange in advance of the eventEmitter object instance or without any complex configuration, as long as both pages processes uses the same mongodb database (but it could be a different replica servers) and the same “name” (the name of the capped collection). This way you can create massive clusters and messaging bus distributed among multiple machines without a need of any separate messaging system and its configuration.
Do a simple example – start two separate node processes with the following code, and see what the results are:
var ev = require('node-mongotailableevents');
ev(function(err, e) {
    if (err) throw err;

    e.on('myevent',function(data) {
        console.log('We have received',data);
    });

    setInterval(function() {
        e.emit('myevent','my data'+parseInt(Math.random()*1000000));
    },5000);
});
You shall see on both of the outputs both of the messages received.

node-netflowv9 node.js module for processing of netflowv9 has been updated to 0.2.5

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2015/03/node-netflowv9-nodejs-module-for.html

My node-netflowv9 library has been updated to version 0.2.5

There are few new things –

  • Almost all of the IETF netflow types are decoded now. Which means practically that we support IPFIX
  • Unknown NetFlow v9 type does not throw an error. It is decoded into property with name ‘unknown_type_XXX’ where XXX is the ID of the type
  • Unknown NetFlow v9 Option Template scope does not throw an error. It is decoded in ‘unknown_scope_XXX’ where XXX is the ID of the scope
  • The user can overwrite how different types of NetFlow are decoded and the user can define its own decoding for new types. The same for scopes. And this can happen “on fly” – at any time.
  • The library supports well multiple netflow collectors running at the same time
  • A lot of new options and models for using of the library has been introduced
Bellow is the updated README.md file, describing how to use the library:

Usage

The usage of the netflowv9 collector library is very very simple. You just have to do something like this:
var Collector = require('node-netflowv9');

Collector(function(flow) {
    console.log(flow);
}).listen(3000);
or you can use it as event provider:
Collector({port: 3000}).on('data',function(flow) {
    console.log(flow);
});
The flow will be presented in a format very similar to this:
{ header: 
  { version: 9,
     count: 25,
     uptime: 2452864139,
     seconds: 1401951592,
     sequence: 254138992,
     sourceId: 2081 },
  rinfo: 
  { address: '15.21.21.13',
     family: 'IPv4',
     port: 29471,
     size: 1452 },
  packet: Buffer <00 00 00 00 ....>
  flow: [
  { in_pkts: 3,
     in_bytes: 144,
     ipv4_src_addr: '15.23.23.37',
     ipv4_dst_addr: '16.16.19.165',
     input_snmp: 27,
     output_snmp: 16,
     last_switched: 2452753808,
     first_switched: 2452744429,
     l4_src_port: 61538,
     l4_dst_port: 62348,
     out_as: 0,
     in_as: 0,
     bgp_ipv4_next_hop: '16.16.1.1',
     src_mask: 32,
     dst_mask: 24,
     protocol: 17,
     tcp_flags: 0,
     src_tos: 0,
     direction: 1,
     fw_status: 64,
     flow_sampler_id: 2 } } ]
There will be one callback for each packet, which may contain more than one flow.
You can also access a NetFlow decode function directly. Do something like this:
var netflowPktDecoder = require('node-netflowv9').nfPktDecode;
....
console.log(netflowPktDecoder(buffer))
Currently we support netflow version 1, 5, 7 and 9.

Options

You can initialize the collector with either callback function only or a group of options within an object.
The following options are available during initialization:
port – defines the port where our collector will listen to.
Collector({ port: 5000, cb: function (flow) { console.log(flow) } })
If no port is provided, then the underlying socket will not be initialized (bind to a port) until you call listen method with a port as a parameter:
Collector(function (flow) { console.log(flow) }).listen(port)
cb – defines a callback function to be executed for every flow. If no call back function is provided, then the collector fires ‘data’ event for each received flow
Collector({ cb: function (flow) { console.log(flow) } }).listen(5000)
ipv4num – defines that we want to receive the IPv4 ip address as a number, instead of decoded in a readable dot format
Collector({ ipv4num: true, cb: function (flow) { console.log(flow) } }).listen(5000)
socketType – defines to what socket type we will bind to. Default is udp4. You can change it to udp6 is you like.
Collector({ socketType: 'udp6', cb: function (flow) { console.log(flow) } }).listen(5000)
nfTypes – defines your own decoders to NetFlow v9+ types
nfScope – defines your own decoders to NetFlow v9+ Option Template scopes

Define your own decoders for NetFlow v9+ types

NetFlow v9 could be extended with vendor specific types and many vendors define their own. There could be no netflow collector in the world that decodes all the specific vendor types. By default this library decodes in readable format all the types it recognises. All the unknown types are decoded as ‘unknown_type_XXX’ where XXX is the type ID. The data is provided as a HEX string. But you can extend the library yourself. You can even replace how current types are decoded. You can even do that on fly (you can dynamically change how the type is decoded in different periods of time).
To understand how to do that, you have to learn a bit about the internals of how this module works.
  • When a new flowset template is received from the NetFlow Agent, this netflow module generates and compile (with new Function()) a decoding function
  • When a netflow is received for a known flowset template (we have a compiled function for it) – the function is simply executed
This approach is quite simple and provides enormous performance. The function code is as small as possible and as well on first execution Node.JS compiles it with JIT and the result is really fast.
The function code is generated with templates that contains the javascript code to be add for each netflow type, identified by its ID.
Each template consist of an object of the following form:
{ name: 'property-name', compileRule: compileRuleObject }
compileRuleObject contains rules how that netflow type to be decoded, depending on its length. The reason for that is, that some of the netflow types are variable length. And you may have to execute different code to decode them depending on the length. The compileRuleObject format is simple:
{
   length: 'javascript code as a string that decode this value',
   ...
}
There is a special length property of 0. This code will be used, if there is no more specific decode defined for a length. For example:
{
   4: 'code used to decode this netflow type with length of 4',
   8: 'code used to decode this netflow type with length of 8',
   0: 'code used to decode ANY OTHER length'
}

decoding code

The decoding code must be a string that contains javascript code. This code will be concatenated to the function body before compilation. If that code contain errors or simply does not work as expected it could crash the collector. So be careful.
There are few variables you have to use:
$pos – this string is replaced with a number containing the current position of the netflow type within the binary buffer.
$len – this string is replaced with a number containing the length of the netflow type
$name – this string is replaced with a string containing the name property of the netflow type (defined by you above)
buf – is Node.JS Buffer object containing the Flow we want to decode
o – this is the object where the decoded flow is written to.
Everything else is pure javascript. It is good if you know the restrictions of the javascript and Node.JS capabilities of the Function() method, but not necessary to allow you to write simple decoding by yourself.
If you want to decode a string, of variable length, you could write a compileRuleObject of the form:
{
   0: 'o["$name"] = buf.toString("utf8",$pos,$pos+$len)'
}
The example above will say that for this netfow type, whatever length it has, we will decode the value as utf8 string.

Example

Lets assume you want to write you own code for decoding a NetFlow type, lets say 4444, which could be of variable length, and contains a integer number.
You can write a code like this:
Collector({
   port: 5000,
   nfTypes: {
      4444: {   // 4444 is the NetFlow Type ID which decoding we want to replace
         name: 'my_vendor_type4444', // This will be the property name, that will contain the decoded value, it will be also the value of the $name
         compileRule: {
             1: "o['$name']=buf.readUInt8($pos);", // This is how we decode type of length 1 to a number
             2: "o['$name']=buf.readUInt16BE($pos);", // This is how we decode type of length 2 to a number
             3: "o['$name']=buf.readUInt8($pos)*65536+buf.readUInt16BE($pos+1);", // This is how we decode type of length 3 to a number
             4: "o['$name']=buf.readUInt32BE($pos);", // This is how we decode type of length 4 to a number
             5: "o['$name']=buf.readUInt8($pos)*4294967296+buf.readUInt32BE($pos+1);", // This is how we decode type of length 5 to a number
             6: "o['$name']=buf.readUInt16BE($pos)*4294967296+buf.readUInt32BE($pos+2);", // This is how we decode type of length 6 to a number
             8: "o['$name']=buf.readUInt32BE($pos)*4294967296+buf.readUInt32BE($pos+4);", // This is how we decode type of length 8 to a number
             0: "o['$name']='Unsupported Length of $len'"
         }
      }
   },
   cb: function (flow) {
      console.log(flow)
   }
});
It looks to be a bit complex, but actually it is not. In most of the cases, you don’t have to define a compile rule for each different length. The following example defines a decoding for a netflow type 6789 that carry a string:
var colObj = Collector(function (flow) {
      console.log(flow)
});

colObj.listen(5000);

colObj.nfTypes[6789] = {
    name: 'vendor_string',
    compileRule: {
        0: 'o["$name"] = buf.toString("utf8",$pos,$pos+$len)'
    }
}
As you can see, we can also change the decoding on fly, by defining a property for that netflow type within the nfTypes property of the colObj (the Collector object). Next time when the NetFlow Agent send us a NetFlow Template definition containing this netflow type, the new rule will be used (the routers usually send temlpates from time to time, so even currently compiled templates are recompiled).
You could also overwrite the default property names where the decoded data is written. For example:
var colObj = Collector(function (flow) {
      console.log(flow)
});
colObj.listen(5000);

colObj.nfTypes[14].name = 'outputInterface';
colObj.nfTypes[10].name = 'inputInterface';

Logging / Debugging the module

You can use the debug module to turn on the logging, in order to debug how the library behave. The following example show you how:
require('debug').enable('NetFlowV9');
var Collector = require('node-netflowv9');
Collector(function(flow) {
    console.log(flow);
}).listen(5555);

Multiple collectors

The module allows you to define multiple collectors at the same time. For example:
var Collector = require('node-netflowv9');

Collector(function(flow) { // Collector 1 listening on port 5555
    console.log(flow);
}).listen(5555);

Collector(function(flow) { // Collector 2 listening on port 6666
    console.log(flow);
}).listen(6666);

NetFlowV9 Options Template

NetFlowV9 support Options template, where there could be an option Flow Set that contains data for a predefined fields within a certain scope. This module supports the Options Template and provides the output of it as it is any other flow. The only difference is that there is a property isOption set to true to remind to your code, that this data has come from an Option Template.
Currently the following nfScope are supported – system, interface, line_card, netflow_cache. You can overwrite the decoding of them, or add another the same way (and using absolutley the same format) as you overwrite nfTypes.

node-netflowv9 is updated to support netflow v1, v5, v7 and v9

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/11/node-netflowv9-is-updated-to-support.html

My netflow module for Node.JS has been updated. Now it support more NetFlow versions – like NetFlow ver 1, ver 5, ver 7 and ver 9. Also it has been modified to be able to be used as Event Generator (instead of doing callbacks). Now you can do as well (the old model is still supported):
Collector({port: 3000}).on('data',function(flow) {
    console.log(flow);
});
Additionally, the module now supports and decode option templates and option data flows for NetFlow v9.

Sencha ExtJS grid update in real time from the back-end

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/10/sencha-extjs-grid-update-in-real-time.html

Hello to all,
I love using Sencha ExtJS in some projects as it is the most complete JavaScript UI framework, even though it is kind of slow, not fast reacting and being cpu and memory expensive. ExtJS allows you to do very fast and lazy development of otherwise complex UI and especially if you use Sencha Architect you can minimize the UI development time focusing only on the important things of your code.
However, ExtJS has quite few draw backs – missing features or some things are over complex and hard to be kept in mind by inexperienced developer (like their Controller idea). 
Here I would like to show you a little example how you can implement a very simple real time update of Sencha Grids (tables) from the backend for an multi user application.
Why do you need this?
I often develop apps that has to be used by multiple persons at the same time and they share and modify the same data.
In such situation, a developer usually has to resolve all those conflicting cases where two users try to modify the same exact data. And Sencha ExtJS grids are not very helpful here. Sencha uses the concept of Store that interact with the data of the back-end (for example by using REST API) and then the Store is assigned to a visualization object like ComboBox or a Grid (Tables). If you modify a table (with the help of Cell Edit Plugin or Row Edit Plugin) that has autoSync property set to true, then any modification you do automatically generates a REST POST/PUT/DELETE query to inform the back end. It can never be easier for a developer, right? But all the data sent to the back end contains the whole modified row – all the properties. On a first sight, this is not an issue. But it is, if you have multiple users editing the same table at the same time. The problem happens because the Sencha Store caches the data. So if User1 modifies it – it is stored on the server. But if User2 modifies the same row but a different column, it will do that over the old data and can overwrite the User1 modification. The backend cannot know which property has been modified and which not and who of the two modifications has to be kept.
There are a lot of tricks a developer usually use to avoid this conflicts. Keeping a version of the modification with each data row in the server, which is received in GET by the UI clients. So when a modification happens, it is accepted only if the client sends the same version number as the one stored in the server, and then the version in the server increases. If another one modification is received with older cached data, it will not be accepted as it will have a different version number. Then the customer will receive an error, then the UI software may refresh its data and updates the versions and the content visualized to the user. 
This is quite popular model, but it is not very nice for the user. The problem is that with multiple users working with the application modifying the same data over the same time, the user will constantly be outdated and will constantly receive errors loosing all its modifications.
The only good solution for both users and the system in general is if in case of change we can update the data in real time in all UI applications. This does not avoid all the possibilities for conflict. But it is highly minimizing it, making the whole operation more pleasant for the end user.
This problem and the need of resolving it happens quite often. Google Spreadsheet and later Google Docs has introduced real time update between the UI data of all the users modifying the same document about 4 years ago.
Example
I like to show here that it is not really hard to update in real time the Stores of ExtJS applications.
It actually requires very little additional code.
Lets imaging we are using a UI developed in Sencha ExtJS with Stores communicating through REST with the backend. The backend for this example will be Node.JS and MongoDB.
Between the Node.JS and the Ext.JS UI there will be Socket.IO session that we will use to push the updates from the Node.JS to the ExtJS Store. I love Socket.IO because it provides a simple WebSockets interface with fallback to HTTP pooling model in case of WebSockets cannot be open (which happens a lot, if you are so unlucky to use a Microsoft security software for example – it blocks WebSockets). 
At the MongoDB we may use capped collections. I love capped collections – they are not only limited in size, but also they allow you to bind a triggers (make the collection tailable) that will receive any new insertion immediately when it happen.
So imagine your Node.JS express REST code looks something like this:
app.get(‘/rest/myrest’,restGetMyrest);
app.put(‘/rest/myrest/:id’,restPutMyrest);
app.post(‘/rest/myrest/:id’,restPostMyrest);
app.del(‘/rest/myrest/:id’,restDelMyrest);

function restGetMyrest(req,res) { // READ REST method
   db.collection(‘myrest’).find().toArray(function(err,q) { return res.send(200,q) })
}

function restPutMyrest(req,res) { // UPDATE REST method
  var id = ObjectID.createFromHexString(req.param(‘id’));
  db.collection(‘myrest’).findAndModify({ _id: id }, [[‘_id’:’asc’]], { $set: req.body }, { safe: true, ‘new’: true }, function(err,q) {
      if (err || (!q)) return res.send(500);
      db.collection(‘capDb’).insert({ method: ‘myrest’, op: ‘update’, data: q }, function() {});
      return res.send(200,q);
  })
}

function restPostMyrest(req,res) { // CREATE REST method
  var id = ObjectID.createFromHexString(req.param(‘id’));
  db.collection(‘myrest’).insert({ _id: id },req.body, { safe: true }, function(err,q) {
      if (err || (!q)) return res.send(500);
      setTimeout(function() {
         db.collection(‘capDb’).insert({ method: ‘myrest’, op: ‘create’, data: q[0] }, function() {});
      },250);
      return res.send(200,q);
  })
}

function restDelMyrest(req,res) { // DELETE REST method
  var id = ObjectID.createFromHexString(req.param(‘id’));
  db.collection(‘myrest’).remove({ _id: id }, { $set: req.body }, { safe: true }, function(err,q) {
      if (err || (!q)) return res.send(500);
      db.collection(‘capDb’).insert({ method: ‘myrest’, op: ‘delete’, data: { _id: id } }, function() {});
      return res.send(201,{});
  })
}
As you can see above – we have implemented a classical CRUD REST method named “myrest” retrieving and storing data in a mongodb collection named ‘myrest’. However, with all modification we also store that modification in a mongodb capped collection named “capDb”.
We use this capped collection (in bold) as an internal mechanism for communication within the NodeDB. You can use events instead, or you can directly send this message to the Socket.IO receiver. However, I like capped db, as they set a lot of advantages – there can be multiple Node.JS processes listening on a capped db and receiving the updates simultaneously. So it is easier to implement clusters that way, including notifying Node.JS processes distributed over different machines.
So now, may be in another file or anywhere else, you may have a simple Node.JS Socket.IO code looking like this:
var s = sIo.of(‘/updates’);
db.createCollection(“capDb”, { capped: true, size: 100000 }, function (err, col) {
   var stream = col.find({},{ tailable: true, awaitdata: true, numberOfRetries: -1 }).stream();
   stream.on(‘data’,function(doc) {
       s.emit(doc.op,doc);
   }
});
 
With this little code above we are basically broadcasting to everyone connected with Socket.IO to /updates the content of the last insertion in the tailable capDb. Also we are creating this collection, if it does not exists from before.
This is everything you need in Node.JS 🙂
Now we can get back to the Ext.JS code. Simply you need to have somewhere in your HTML application this code executed:
var socket = io.connect(‘/updates’);
socket.on(‘create’, function(msg) {
   var s = Ext.StoreMgr.get(msg.method);
   if ((!s)||(s.getCount()>s.pageSize||s.findRecord(‘id’,msg.data._id)) return;
   s.suspendAutoSync();
   s.add(msg.data);
   s.commitChanges();
   s.resumeAutoSync();
});
socket.on(‘update’, function(msg) {
   var s = Ext.StoreMgr.get(msg.method);
   var r;
   if ((!s)||(!(r=s.findRecord(‘id’,msg.data._id))) return;
   s.suspendAutoSync();
   for (var k in msg.data) if (r.get(k) != msg.data[k]) r.set(k,msg.data[k]);
   s.commitChanges();
   s.resumeAutoSync();
});
socket.on(‘delete’,function(msg) {
   var s = Ext.StoreMgr.get(msg.method);
   var r;
   if ((!s)||(!(r=s.findRecord(‘id’,msg.data._id))) return;
   s.suspendAutoSync();
   s.remove(r);
   s.commitChanges();
   s.resumeAutoSync();
});
This is all.
Basically what we do from end to end –
If the Node.JS receives any CRUD REST operation it updates the data in the MongoDB, but also for Create, Update, Delete it notify over Socket.IO all the listening web clients about this operation (in my example, I use tailable capped collection in MongoDB as a an internal messaging bus, but you can emit to the Socket.IO directly or use another messaging bus like EventEmitter).
Then the ExtJS receives the update over Socket.IO and assumes that the method property contains the name of the Store that has to be updated. Then we find the store, suspedAutoSync if it exists (otherwise we can get into update->autosync->rest->update loop), modify the content of the record (or the store) and resume AutoSync.
With this simple code you can broadcast all the modifications in your data between all the extjs users that are currently online, so they can see updates in real time in their grids.
A single REST method may be used by multiple stores. In such case, you have to modify your code with some association between the REST method name and all the related stores.
However, for this simple example, that is unnecessary.
Some other day, I may show you my “ExtJS WebSockets CRUD proxy” I made, where you have only one communication channel between the stores and the backend – Socket.IO. It is much faster and removes the need of having REST code at all in your server. 

Fun with JavaScript inheritance

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/fun-with-javascript-inheritance.html

One could say that JavaScript does not support native inheritance for OOP and could not be so wrong. JavaScript happened to be having one of the most powerful OOP I have ever seen 🙂
Something more, JavaScript does the inheritance trough a prototype chaining, which actually provides even more power than you can imagine.
See the following simple chaining example, that shows a chained inheritance (not the typical one-step prototype->object inheritance everyone knows):
~$ node
> a={}
{}
> b = Object.create(a)
{}
> c = Object.create(b)
{}
> a.test = 1
1
> a
{ test: 1 }
> b
{}
> c
{}
> c.test
1
> b.text=2
2
> b
{ text: 2 }
> b.test
1
> c.test
1
> c.text
2
> c
{}
So now you can see above how can we do a very easy but powerful chain of objects. Something more, that works for ANY object, including chaining BOTH the methods and the prototypes of the objects, so you can modify either the objects directly, or their prototypes and everything is inherited with a two-shadow priority model.
~$ node
> a=[1,2,3,4]
[ 1, 2, 3, 4 ]
> b=Object.create(a)
{}
> b
{}
> b[1]
2
> b.length
4
> b
{}
So now you can see that b is still (an empty) object but has behavior and properties like an array 🙂

Operator overloading with JavaScript

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/operator-overloading-with-javascript.html

JavaScript happens to have an extremely powerful OOP tool-set that is quite unknown. I am surprised every day by the power and flexibility of it.
One of the things I didn’t knew is possible at all with JavaScript was operator overloading.
And this is understandable – JavaScript doesn’t have strict variable types. Therefore any type of polymorphism is very hard to be implemented, as it is bound to the object/variable types.
However, you can easily do polymorphism with JavaScript and as it happens, you can easily do operator overloading too 🙂
How that works?
It is really easy – JavaScript retrieve the value of an Object using the valueOf method. And every object has valueOf with the exception of few primitive types (as in Java and C++). Therefore you can use that to do operator oveloading.
Let me show you an example:
~$ node
> a={}
{}
> a == 3
false
> a.valueOf = function() { return 3 }
[Function]
> a == 3
true
And the best is that you can do that to the prototype (or to the inherited object):
~$ node
> {} == 3
false
> Object.prototype.valueOf = function() { return 3 }
[Function]
> {} == 3
true

New improved version of the node-netflowv9 module for Node.JS

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/new-improved-version-of-node-netflowv9.html

As I have mentioned before, I have implemented a NetFlowV9 compatible library for decode of Cisco NetFlow version 9 packets for Node.JS.
Now I am upgrading it to version 0.1 which has few updates:
  • bug fixes (including avoidance of an issue that happens with ASR9k and IOS XR 4.3)
  • now you can start the collector (with a second parameter of true) in a mode where you want to receive only one call back per packet instead of one callback per flow (the default mode). That could be useful if you want to count the lost packets (otherwise the relation netflow packet – callback is lost)
  • decrease of the code size
  • now the module compiles the templates dynamically into a function (using new Function). I like this approach very much, as it creates really fast functions (in contrast to eval, Function is always JIT processed) and it allows me to spare loops, function calls and memory copy. I like to do things like that with every data structure that allows it. Anyway, as an effect of this, the new module is about 3 times faster with all the live tests I was able to perform

Simple example for Node.JS sflow collector

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/simple-example-for-nodejs-sflow.html

Sometimes you can use the SFlow or Netflow to extra add intelligence to your network. The collectors available on internet are usually there just to collect and store data used for accounting or nice graphics. But the collectors are either not allowing you to execute your own code in case of certain rules/thresholds reached, or do not react in real time (in general, the protocols delays you too. You cannot expect NetFlow accounting to be used in real time at all, while SFlow has modes that are bit more fast to react, by design, it is still not considered to be real-time sampling/accounting).
Just imagine you have a simple goal – you want to automatically detect floods and notify the operators or you can even automatically apply filters.
If you have an algorithm that can distinguish the incorrect traffic from the normal traffic from NetFlow/SFlow sampling you may like to execute an operation immediately when that happens.
The modern DoS attacks and floods may be complex and hard to detect. But mainly it is hard to make the currently available NetFlow/SFlow collector software to do that for you and then trigger/execute external application.
However, it is very easy to program it yourself.
I am giving you a simple example that uses the node-sflow module to collect packet samples, measure how many of them match a certain destination ip address and if they are above certain pps thresholds to execute an external program (that is supposed to block that traffic). Then after a period of time it will execute another program (that is supposed to unblock the traffic).
This program is very small – about 120 lines of code and allows you to use complex configuration file where you can define a list of rules that can match optionally vlans and networks for the sampled packet and then count how many samples you have per destination for that rule. The rule list is executed until first match in the configured order within the array, so that allows you to create black and white lists and different thresholds per networks and vlans, or to have different rules per overlapped ip addresses as long as they belong to different vlans.
Keep in mind this is just an example software there just for your example, showing you how to use node-sflow and pcap modules together! It is not supposed to be used in production, unless you definitely know what you are doing!
The goal of this example it here just to show you how easy is to add extra logic within your network.
The code is available on git-hub here https://github.com/delian/sflow-collector/

RTMP Api for Node.JS to ease the implementation of RTMP servers and clients

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/rtmp-api-for-nodejs-to-ease.html

Hello to all,
As I mentioned before, I needed to implement a RTMP streaming server in Node.JS. All of the available modules for implementation of RTMP in Node’s NPM repository were incomplete, incorrect or unusable. Not only that but the librtmp used by libav tools like avconv and avplay was incorrect and incomplete.
The same with most of the implementation I’ve checked (covering perl, python, others). I’ve tried to fix few of them but at the end I had to write one on my own.
This is my library of RTMP related tools and API for Node.JS. It is named node-rtmpapi and is available in the npm repository. Also you can get it here – https://github.com/delian/node-rtmpapi
It works well for me, and it has been tested with MistServer, OrbanEncoders and librtmp (from libav).
That does not mean it will work for you, though 🙂
RTMP is quite badly documented protocol and extremely badly implemented.
During my tests I have seen issues like crash of libraries (including the Adobe’s original one) if the upper layer commands has been sent in unexpected order (although this is allowed by the RTMP protocol and the order of the upper layer commands is not documented at all). Also I have seen (within Adobe’s rtmp library) incorrect implementation of the setPeerBandwidth command.
Generally, each different RTMP implementation is on its own and the only way to make it work is to adjust and tune it according to the software you communicate with.
Therefore I separated my code in utils that allows me to write my own RTMP server relatively easy and to adjust it according to my needs.
The current library supports only TCP as a transport (although TLS and HTTP/HTTPS is easy to be implemented, I haven’t focused on it yet).
It provides separate code that implements streaming (readQueue), the chunk layer of the protocol (rtmpChunk), the upper layer messaging (assembling and disassembling of message over chunks, rtmpMessage), stream processing (rtmpStream) and basic server implementation without the business logic (rtmpServer).
Simplified documentation is provided at the git-hub repository.
The current library uses callbacks for each upper layer command it receives. I am planning to migrate the code to use node streams and to trigger events per command, instead of callbacks. This will extremely simplify the usage and the understanding of the library for a node programmer. However, this is the future and in order to preserve compatibility, I will probably name it something different (like node-streams-rtmpapi)

AMF0/3 encoding/decoding in Node.JS

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/amf03-encodingdecoding-in-nodejs.html

I am writing my own RTMP restreamer (RTMP is Adobe’s dying streaming protocol widely used with Flash) in Node.JS.
Although, there are quite of few RTMP modules, no one is complete, nor operates with Node.JS buffers, nor support fully ether AMF0 or AMF3 encoding and decoding.
So I had to write one on my own.
The first module is the AMF0/AMF3 utils that allow me to encode or decode AMF data. AMF is a binary encoding used in Adobe’s protocols, very similar to BER (used in ITU’s protocols) but supporting complex objects. In general the goal of AMF is to encode ActiveScript objects into binary. As ActiveScript is a language belonging to the JavaScript’s familly, basically the ActiveScript’s objects are javascript objects (with the exception of some simplified arrays).
My module is named node-amfutils and is now available in the public NPM repository as well as here https://github.com/delian/node-amfutils
It is not fully completed nor very well tested as I have very limited environment to do the tests. However, it works for me and provides the best AMF0 and AMF3 support currently available for Node.JS – 
  • It can encode/decode all the objects defined in both AMF0 and AMF3 (the other AMF modules in the npm repository supports partial AMF0 or partial AMF3)
  • It uses Node.JS buffers (it is not necessary to do string to buffer to string conversion, as you have to do with the other modules)

It is easy to use this module. You just have to do something like this:

var amfUtils = require('node-amfutils');
var buffer = amfUtils.amf0Encode([{ a: "xxx"},b: null]);

SFlow version 5 module for Node.JS

Post Syndicated from Anonymous original http://deliantech.blogspot.com/2014/06/sflow-version-5-module-for-nodejs.html

Unfortunately, as with NetFlow Version 9, SFlow version 5 (and SFlow in general) has not been very well supported by the Node.JS community up to now.
I needed modern SFlow version 5 compatible module, so I had to write one on my own.
Please welcome the newest module in Node.JS’s NPM that can decode SFlow version 5 packets and be used in the development of simple and easy SFlow collectors! The module is named node-sflow and you can look at its code here https://github.com/delian/node-sflow
Please be careful, as in the next days I may change the object structure of the flow representation to simplify it! Any tests and experiments are welcome.
The sflow module is available in the public npm (npm install node-sflow) repository.
To use it you have to do:
var Collector = require('node-sflow');

Collector(function(flow) {
    console.log(flow);
}).listen(3000); 
In general SFlow is much more powerful protocol than NetFlow, even it its latest version (version 9). It can represent more complex counters, report about errors, drops, full packet headers (not only their properties), collect information from interfaces, flows, vlans, and combine them in a much more complex reports.

However, the SFlow support in the agents – the networking equipment is usually extremely simplified – far from the richness and complexity the SFlow protocol may provide. Most of the vendors just do packet sampling and send them over SFlow as raw packet/frame header with an associated unclear counter.

In case of you having the issue specified above, this module cannot help much. You will just get the raw packet header (usually Ethernet + IP header) as a Node.JS buffer and then you have to decode it on your own. I want to keep the node-sflow module simple and I don’t plan to decode raw packet headers there as this feature is not a feature of the SFlow itself.

If you need to decode the raw packet header I can suggest one easy solution for you. You can use the pcap module from the npm repository and decode the raw header with it:

var Collector = require('node-sflow');
var pcap = require('pcap');

Collector(function(flow) {
    if (flow && flow.flow.records && flow.flow.records.length>0) {
        flow.flow.records.forEach(function(n) {
            if (n.type == 'raw') {
                if (n.protocolText == 'ethernet') {
                    try {
                        var pkt = pcap.decode.ethernet(n.header, 0);
                        if (pkt.ethertype!=2048) return;
                        console.log('VLAN',pkt.vlan?pkt.vlan.id:'none','Packet',pkt.ip.protocol_name,pkt.ip.saddr,':',pkt.ip.tcp?pkt.ip.tcp.sport:pkt.ip.udp.sport,'->',pkt.ip.daddr,':',pkt.ip.tcp?pkt.ip.tcp.dport:pkt.ip.udp.dport)
                    } catch(e) { console.log(e); }
                }
            }
        });
    }
}).listen(3000);