Open Sourcing Mantis: A Platform For Building Cost-Effective, Realtime, Operations-Focused…

Post Syndicated from Netflix Technology Blog original https://medium.com/netflix-techblog/open-sourcing-mantis-a-platform-for-building-cost-effective-realtime-operations-focused-5b8ff387813a?source=rss----2615bd06b42e---4

Open Sourcing Mantis: A Platform For Building Cost-Effective, Realtime, Operations-Focused Applications

By Jeff Chao on behalf of the Mantis team

Today we’re excited to announce that we’re open sourcing Mantis, a platform that helps Netflix engineers better understand the behavior of their applications to ensure the highest quality experience for our members. We believe the challenges we face here at Netflix are not necessarily unique to Netflix which is why we’re sharing it with the broader community.

As a streaming microservices ecosystem, the Mantis platform provides engineers with capabilities to minimize the costs of observing and operating complex distributed systems without compromising on operational insights. Engineers have built cost-efficient applications on top of Mantis to quickly identify issues, trigger alerts, and apply remediations to minimize or completely avoid downtime to the Netflix service. Where other systems may take over ten minutes to process metrics accurately, Mantis reduces that from tens of minutes down to seconds, effectively reducing our Mean-Time-To-Detect. This is crucial because any amount of downtime is brutal and comes with an incredibly high impact to our members — every second counts during an outage.

As the company continues to grow our member base, and as those members use the Netflix service even more, having cost-efficient, rapid, and precise insights into the operational health of our systems is only growing in importance. For example, a five-minute outage today is equivalent to a two-hour outage at the time of our last Mantis blog post.

Mantis Makes It Easy to Answer New Questions

The traditional way of working with metrics and logs alone is not sufficient for large-scale and growing systems. Metrics and logs require that you to know what you want to answer ahead of time. Mantis on the other hand allows us to sidestep this drawback completely by giving us the ability to answer new questions without having to add new instrumentation. Instead of logs or metrics, Mantis enables a democratization of events where developers can tap into an event stream from any instrumented application on demand. By making consumption on-demand, you’re able to freely publish all of your data to Mantis.

Mantis is Cost-Effective in Answering Questions

Publishing 100% of your operational data so that you’re able to answer new questions in the future is traditionally cost prohibitive at scale. Mantis uses an on-demand, reactive model where you don’t pay the cost for these events until something is subscribed to their stream. To further reduce cost, Mantis reissues the same data for equivalent subscribers. In this way, Mantis is differentiated from other systems by allowing us to achieve streaming-based observability on events while empowering engineers with the tooling to reduce costs that would otherwise become detrimental to the business.

From the beginning, we’ve built Mantis with this exact guiding principle in mind: Let’s make sure we minimize the costs of observing and operating our systems without compromising on required and opportunistic insights.

Guiding Principles Behind Building Mantis

The following are the guiding principles behind building Mantis.

  1. We should have access to raw events. Applications that publish events into Mantis should be free to publish every single event. If we prematurely transform events at this stage, then we’re already at a disadvantage when it comes to getting insight since data in its original form is already lost.
  2. We should be able to access these events in realtime. Operational use cases are inherently time sensitive by nature. The traditional method of publishing, storing, and then aggregating events in batch is too slow. Instead, we should process and serve events one at a time as they arrive. This becomes increasingly important with scale as the impact becomes much larger in far less time.
  3. We should be able to ask new questions of this data without having to add new instrumentation to your applications. It’s not possible to know ahead of time every single possible failure mode our systems might encounter despite all the rigor built in to make these systems resilient. When these failures do inevitably occur, it’s important that we can derive new insights with this data. You should be able to publish as large of an event with as much context as you want. That way, when you think of a new questions to ask of your systems in the future, the data will be available for you to answer those questions.
  4. We should be able to do all of the above in a cost-effective way. As our business critical systems scale, we need to make sure the systems in support of these business critical systems don’t end up costing more than the business critical systems themselves.

With these guiding principles in mind, let’s take a look at how Mantis brings value to Netflix.

How Mantis Brings Value to Netflix

Mantis has been in production for over four years. Over this period several critical operational insight applications have been built on top of the Mantis platform.

A few noteworthy examples include:

Realtime monitoring of Netflix streaming health which examines all of Netflix’s streaming video traffic in realtime and accurately identifies negative impact on the viewing experience with fine-grained granularity. This system serves as an early warning indicator of the overall health of the Netflix service and will trigger and alert relevant teams within seconds.

Contextual Alerting which analyzes millions of interactions between dozens of Netflix microservices in realtime to identify anomalies and provide operators with rich and relevant context. The realtime nature of these Mantis-backed aggregations allows the Mean-Time-To-Detect to be cut down from tens of minutes to a few seconds. Given the scale of Netflix this makes a huge impact.

Raven which allows users to perform ad-hoc exploration of realtime data from hundreds of streaming sources using our Mantis Query Language (MQL).

Cassandra Health check which analyzes rich operational events in realtime to generate a holistic picture of the health of every Cassandra cluster at Netflix.

Alerting on Log data which detects application errors by processing data from thousands of Netflix servers in realtime.

Chaos Experimentation monitoring which tracks user experience during a Chaos exercise in realtime and triggers an abort of the chaos exercise in case of an adverse impact.

Realtime Personally Identifiable Information (PII) data detection samples data across all streaming sources to quickly identify transmission of sensitive data.

Try It Out Today

To learn more about Mantis, you can check out the main Mantis page. You can try out Mantis today by spinning up your first Mantis cluster locally using Docker or using the Mantis CLI to bootstrap a minimal cluster in AWS. You can also start contributing to Mantis by getting the code on Github or engaging with the community on the users or dev mailing list.

Acknowledgements

A lot of work has gone into making Mantis successful at Netflix. We’d like to thank all the contributors, in alphabetical order by first name, that have been involved with Mantis at various points of its existence:

Andrei Ushakov, Ben Christensen, Ben Schmaus, Chris Carey, Cody Rioux, Daniel Jacobson, Danny Yuan, Erik Meijer, Indrajit Roy Choudhury, Jeff Chao, Josh Evans, Justin Becker, Kathrin Probst, Kevin Lew, Neeraj Joshi, Nick Mahilani, Piyush Goyal, Prashanth Ramdas, Ram Vaithalingam, Ranjit Mavinkurve, Sangeeta Narayanan, Santosh Kalidindi, Seth Katz, Sharma Podila, Zhenzhong Xu.


Open Sourcing Mantis: A Platform For Building Cost-Effective, Realtime, Operations-Focused… was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

[$] “git request-pull” and confusing diffstats

Post Syndicated from corbet original https://lwn.net/Articles/802797/rss

When a kernel subsystem maintainer has a set of commits to send up the
chain toward the mainline, the git request-pull
command
is usually
the right tool for the job. But various maintainers have noticed over the
years that this command can sometimes generate confusing results when
confronted with anything but the simplest of histories. A brief
conversation on the linux-kernel mailing list delved into why this
situation comes about and what maintainers can do in response.

За хиляда пробити рубли

Post Syndicated from original https://bivol.bg/%D0%B7%D0%B0-%D1%85%D0%B8%D0%BB%D1%8F%D0%B4%D0%B0-%D0%BF%D1%80%D0%BE%D0%B1%D0%B8%D1%82%D0%B8-%D1%80%D1%83%D0%B1%D0%BB%D0%B8.html

понеделник 21 октомври 2019


Братчедът Ставри ми звъни оня ден. Всяка година по това време е все така. В зависимост от атмосферните условия де, но превали ли средата на октомври, и първо за мен се сеща. Преди се опитваше и други от родата и от махалата да търси, но бързо разбра, че няма да му мине номерът. Всеки си има извинение. Я жена му била болна и нямало кой да гледа децата, я точно в тоя ден бил на смяна, я собственото си лозе още не бил сколасал да обере, пък камо ли да помага на братчедa Ставри за неговото.

Аз обаче извинения нямам. Ни жена, ни деца, ни работа, пък камо ли лозе. Разчитам само на социални помощи и твърдо вярвам, че истината за добруването на този грешен свят е в съзерцанието, а не в инициативността. Злите езици твърдят, че съм бил чисто и просто един философстващ мързеливец, но аз ги подминавам с мълчание. Ами, че те и Буда не са го приемали на сериозно дълго време, ама после милиони са се вслушали в думите му. Не нашия Буда, оня, индийския, макар и нашия да събра много доверие и въодушевление с бръщолевения, не може да му се отрече.

Така че, нямайки какво да измисля, трябваше за пореден път да пристана на акъла на братчеда Ставри. Не, не мислете, че съм чак толкова слабохарактерен, но всяка година по гроздобер се случва така, че му вися с някой и друг лев, взет назаем, за да покрия неотложните си битийни нужди и няма как да го отсвиря, защото знам, че следващия път, когато опра до него, няма да ми се отвори парашута и ще стоя гладен. Вече не ми дават на вересия в нито един от магазините в селото. Търговци. По-скоро търгаши. Само за печалба мислят. Как да очакваш от тях да оценят една възвишена философия и да са съпричастни на житейското ми кредо?

Когато влязох в двора на братчеда Ставри и се насочих към голямата асма, под която той си беше устроил малка механичка, от която излизаше само в краен случай, не само се заковах на едно място от изненада, но дори изпуснах лозарската ножица.

–          Какво е това бе, братчед? – успях да смотолевя с неудържимо трепереща долна челюст.

–          Сядай, сядай, братчед! – подкани ме той с жест, а когато се наведох да взема ножиците добави. – Остави това! Днес няма да ни трябва.

–          Какви са тия пари? Откъде са? – послушно седнах, без да мога да откъсна поглед от тлъстите пачки, които братчедът Ставри размяташе разсеяно из дланите си, все едно жонглираше с тях.

–          Какво значение има? – изсумтя той, надигна се от мястото си, облегна се на кокалчетата на пръстите си и се наведе над мен. Чак усещах спиртосания му дъх. – Важното е, братчед, че можеш да получиш част от тях.

–          Какво трябва да направя? – нещо в поведението, най-вече в погледа му, ме накара наистина да се смутя. – Това са много пари.

–          Не всички са за теб – братчедът Ставри отново се отпусна назад. – Трябват ми и други съмишленици.

–          За какво? – продължавах да недоумявам, но той просто се усмихна, отвори кутия с хубави цигари и я протегна към мен. Запалих с треперещи ръце. Наистина бях изпаднал в неведение.

–          За да помогнем на най-достойния кандидат за кмет на общината да спечели изборите. – отвърна братчедът Ставри и заби изпитателен поглед в мен. Явно очакваше да реагирам по някакъв начин, но след като не го направих, продължи. – Е, той така или иначе ще спечели, но трябва да сме сигурни, защото се навъдиха разни самозванци. Тук били учили, там били направили успешна кариера, на трето място се били доказали… Измишльотини.

–          И кой е най-достойният кандидат? – попитах, без да схващам точно какво говори.

–          Нима има някакво значение, братчед? – усмихна се широко той, но лицето му за миг стана сериозно и сбърчи вежди. – Не ми ли вярваш? Това, което се иска от теб, е да вземеш тия пари, да задраскаш в изборната стаичка номера, който ще ти напиша и да убедиш колкото се може повече хора от селото да постъпят по същия начин.

–          Но… Това не е ли незаконно? – тръпки ме побиха. Никога не бях участвал в нищо извън закона, но, странно защо, не можех да отделя поглед от пачките, които братчедът Ставри все още премяташе из ръцете си.

–          Че кое е законно в тази държава!? – той дълго се смя на глас, преди да продължи – Само ти ли ще си балък и ще се съобразяваш с правилата? Погледни се само! Та ти нямаш пари дори за свястна храна.

–          Не знам… – почувствах се като аутсайдер, какъвто си и бях, но все още имах искрица съмнение в душата си.

–          Ти няма да караш никого насила – продължи брачедът Ставри, вече равно и спокойно. – Просто ще подпиташ този и онзи, пък който се съгласи, също ще бъде възнаграден подобаващо. Тук съм ти подготвил едно списъче с хора, които според мен биха били склонни да станат част от каузата ни.

–          Ами, ако откажа? – погледнах листчето, което ми побутна. Имаше поне двайсетина имена, което за малко село като нашето си е доста, съгласете се. – Или ако някой от тях откаже?

–          Няма проблем, братчед – по гласа му личеше, че определено има проблем. – Ще се наложи да помоля и теб, и всеки друг, който не е на моя страна, да си ми върне парите и да не пристъпва никога през прага ми с каквато и да е молба. Или сме задружни, или не. Това е. Когато сте поискали нещо от мен, аз съм откликвал. Очаквам същото от вас. Иначе ще съм много разочарован.

–          Колко? – главата ми се въртеше. Повече от всичко исках това да не се случва, а да ме беше викнал просто за гроздобера.

–          Хиляда за глас. Твоят и на останалите – братчедът Ставри извади една банкнота от пачката. – И по петстотин за всеки друг глас, които ми осигуриш.

–          Хиляда лева за глас!? – ахнах.

–          Не, братчед! – той пак се разсмя, ама тоя път май веселието му беше искрено. – Ние плащаме в рубли. Такава е политиката ни.

–          Колко е курса? – попитах машинално.

–          Не знам, провери – той побутна пачките към мен. – Във всеки случай сделката си я бива. Чу ли как товариш Путин заяви на американците, че ще срине долара? Някога да не си е държал на думата? Той е мъжкар. Бъди и ти мъжкар и ще бъдеш богато възнаграден. А сега тръгвай, че съм зает. Успех, братчед. И да не забравяш да отмяташ с чавка в списъка всеки, който е обещал да гласува и си е взел парсата. Номерът на нашия кандидат е отзад на листа.

Набутах пачките в пазвата си, защото джобовете на панталона ми отдавна бяха скъсани и побързах да се махна оттук. На дворната врата се сетих, че лозарските ми ножици все още са на пътеката, където ги изпуснах. Не се върнах.

BREIN, MPA, and ACE Shut Down Massive ‘Pirate CDN’

Post Syndicated from Andy original https://torrentfreak.com/brein-mpa-and-ace-shut-down-massive-pirate-cdn-191021/

Earlier this year, cyber-security company Group-IB shared an interesting report with TorrentFreak.

The company told us that “large monopolists” were supplying huge amounts of content to thousands of websites via dedicated ‘pirate’ Content Delivery Networks (CDNs).

Group-IB provided specific details on a CDN called ‘Moonwalk’ which reportedly began operating in 2013. According to the company, at the time the system carried 33,490 movies and TV shows, paying out $0.60 per 1000 views.

Group-IB complained that since most of Moonwalk’s servers were outside Russia, the Netherlands in particular, enforcement by local rightsholders was proving difficult. Several months later, it now transpires that Dutch anti-piracy group BREIN has stepped up in an effort to deal with the problem.

BREIN chief Tim Kuik informs TorrentFreak that on Friday, bailiffs acting on its behalf served ex parte court orders on five hosting providers requiring them to disconnect streaming servers and preserve evidence in relation to Moonwalk.

Three court orders targeted Dutch companies and two “ostensibly foreign companies” whose servers are located in the Netherlands. While the action is being headed up by BREIN, the anti-piracy group is working with both the Motion Picture Association and the global Alliance for Creativity and Entertainment.

BREIN describes Moonwalk as a “video load balancer” which provides both the back-end and also huge volumes of pirated content to around 80% of known Russian streaming sites.

“The top 50 of these websites entertain 395 million visits from 89.9 million unique visitors per month causing hundreds of millions of euros/dollars in losses,” BREIN says.

BREIN’s estimates of the amount of content being provided by Moonwalk exceed the figures provided by Group-IB earlier this year. Overall, the Dutch anti-piracy outfit says that the system was recently providing more than 26,000 movies and 10,000 TV shows. That’s around 2,500 additional pieces of video entertainment which suggests growth over recent months.

The ex parte court orders were obtained by BREIN following a joint investigation with ACE, which counts almost three dozen of the world’s leading content and broadcasting companies as members. It’s clear the orders were intended to cause the shutdown of Moonwalk while providing evidence on its operations and presumably, its operators.

“The fight against piracy is global and we are going after operators of these services and their hosting infrastructure as well as other intermediaries supporting these illegal services”, says BREIN chief Tim Kuik.

Jan Van Voorn, Executive Vice President and Chief of Global Content Protection at the Motion Picture Association, stressed that cooperating internationally is crucial to dealing with today’s piracy issues.

“Effectively fighting piracy today requires strong partnerships at global and local level,” he says.

“This action coordinated between BREIN, ACE and the MPA is a significant win and another step towards preserving a healthy and vibrant ecosystem in which the creative community can produce, distribute and protect their content so that audiences can enjoy them.”

What happens next in the investigation isn’t clear but a website associated with Moonwalk currently states that due to this action, the service is not only down, but down for good.

Gone forever?

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

A Path Towards Reasonable Autonomous Weapons Regulation

Post Syndicated from Ronald Arkin original https://spectrum.ieee.org/automaton/robotics/artificial-intelligence/a-path-towards-reasonable-autonomous-weapons-regulation

Editor’s Note: The debate on autonomous weapons systems has been escalating over the past several years as the underlying technologies evolve to the point where their deployment in a military context seems inevitable. IEEE Spectrum has published a variety of perspectives on this issue. In summary, while there is a compelling argument to be made that autonomous weapons are inherently unethical and should be banned, there is also a compelling argument to be made that autonomous weapons could potentially make conflicts less harmful, especially to non-combatants. Despite an increasing amount of international attention (including from the United Nations), progress towards consensus, much less regulatory action, has been slow. The following workshop paper on autonomous weapons systems policy is remarkable because it was authored by a group of experts with very different (and in some cases divergent) views on the issue. Even so, they were able to reach consensus on a roadmap that all agreed was worth considering. It’s collaborations like this that could be the best way to establish a reasonable path forward on such a contentious issue, and with the permission of the authors, we’re excited to be able to share this paper (originally posted on Georgia Tech’s Mobile Robot Lab website) with you in its entirety.

Test Ops Agile Design and Test

Post Syndicated from IEEE Spectrum Recent Content full text original https://spectrum.ieee.org/whitepaper/test-ops-agile-design-and-test

In the 1990s, agile software development profoundly transformed software development. Agile is far more than a process; it’s a new way to work. Today, a similar transformation is happening in test and measurement: TestOps. Learn about TestOps and how to accelerate your product development workflow.

photo

Security updates for Monday

Post Syndicated from ris original https://lwn.net/Articles/802776/rss

Security updates have been issued by Debian (aspell, graphite-web, imagemagick, mediawiki, milkytracker, nfs-utils, and openjdk-11), Fedora (kernel, kernel-headers, kernel-tools, mediawiki, and radare2), openSUSE (dhcp, libpcap, lighttpd, and tcpdump), Scientific Linux (java-1.8.0-openjdk), Slackware (python), SUSE (bluez, kernel, and python-xdg), and Ubuntu (aspell).

Top 10 Most Pirated Movies of The Week on BitTorrent – 10/21/19

Post Syndicated from Ernesto original https://torrentfreak.com/top-10-most-pirated-movies-of-the-week-on-bittorrent-10-21-19/

This week we have one newcomer in our chart.

The Lion King is the most downloaded movie.

The data for our weekly download chart is estimated by TorrentFreak, and is for informational and educational reference only. All the movies in the list are Web-DL/Webrip/HDRip/BDrip/DVDrip unless stated otherwise.

RSS feed for the articles of the recent weekly movie download charts.

This week’s most downloaded movies are:
Movie RankRank last weekMovie nameIMDb Rating / Trailer
Most downloaded movies via torrents
1(1)The Lion King7.1 / trailer
2(2)Fast & Furious Presents: Hobbs & Shaw6.7 / trailer
3(…)El Camino7.6 / trailer
4(3)Toy Story 48.1 / trailer
5(4)Dark Phoenix6.0 / trailer
6(5)Spider-Man: Far from Home7.8 / trailer
7(7)Joker (HDCam)8.1 / trailer
8(6)It: Chapter Two6.9 / trailer
9(8)Crawl6.4 / trailer
10(…)Stuber6.2 / trailer

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

Using Grab’s Trust Counter Service to Detect Fraud Successfully

Post Syndicated from Grab Tech original https://engineering.grab.com/using-grabs-trust-counter-service-to-detect-fraud-successfully

Background

Fraud is not a new phenomenon, but with the rise of the digital economy it has taken different and aggressive forms. Over the last decade, novel ways to exploit technology have appeared, and as a result, millions of people have been impacted and millions of dollars in revenue have been lost. According to ACFE survey, companies lost USD6.3 billion due to fraud. Organizations lose 5% of its revenue annually due to fraud.

In this blog, we take a closer look at how we developed an anti-fraud solution using the Counter service, which can be an indispensable tool in the highly complex world of fraud detection.

Anti-fraud solution using counters

At Grab, we detect fraud by deploying data science, analytics, and engineering tools to search for anomalous and suspicious transactions, or to identify high-risk individuals who are likely to commit fraud. Grab’s Trust Platform team provides a common anti-fraud solution across a variety of business verticals, such as transportation, payment, food, and safety. The team builds tools for managing data feeds, creates SDK for engineering integration, and builds rules engines and consoles for fraud detection.

One example of fraudulent behavior could be that of an individual who masquerades as both driver and passenger, and makes cashless payments to get promotions, for example, earn a one dollar rebate in the next transaction.In our system, we analyze real time booking and payment signals, compare it with the historical data of the driver and passenger pair, and create rules using the rule engine. We count the number of driver and passenger pairs at a given time frame. This counter is provided as an input to the rule.If the counter value exceeds a predefined threshold value, the rule evaluates it as a fraud transaction. We send this verdict back to the booking service.

The conventional method

Fraud detection is a job that requires cross-functional teams like data scientists, data analysts, data engineers, and backend engineers to work together. Usually data scientists or data analysts come up with an offline idea and apply it to real-time traffic. For example, a rule gets invented after brainstorming sessions by data scientists and data analysts. In the conventional method, the rule needs to be communicated to engineers.

Automated solution using the Counter service

To overcome the challenges in the conventional method, the Trust platform team decided to come out with the Counter service, a self-service platform, which provides management tools for users, and a computing engine for integrating with the backend services. This service provides an interface, such as a UI based rule editor and data feed, so that analysts can experiment and create rules without interacting with engineers. The platform team also decided to provide different data contracts, APIs, and SDKs to engineers so that the business verticals can use it quickly and easily.

The major engineering challenges faced in designing the Counter service

There are millions of transactions happening at Grab every day, which implies we needed to perform billions of fraud and safety detections. As seen from the example shared earlier, most predictions require a group of counters. In the above use case, we need to know how many counts of the cashless payment happened for a driver and passenger pair. Due to the scale of Grab’s business, the potential combinations of drivers and passengers could be exponential. However, this is only one use case. So imagine that there could be hundreds of counters for different use cases. Hence it’s important that we provide a platform for stakeholders to manage counters.

Some of the common challenges we faced were:

Scalability

As mentioned above, we could potentially have an exponential number of passengers and drivers in a single counter. So it’s a great challenge to store the counters in the database, read, and query them in real-time. When there are billions of counter keys across a long period of time, the Trust team had to find a scalable way to write and fetch keys effectively and meet the client’s SLA.

Self-serving

A counter is usually invented by data scientists or analysts and used by engineers. For example, every time a new type of counter is needed from data scientists, developers need to manually make code changes, such as adding a new stream, capturing related data sets for the counter, and storing it on the fraud service, then doing a deployment to make the counters ready. It usually takes two or more weeks for the whole iteration, and if there are any changes from the data analysts’ side, which happens often, the situation loops again. The team had to come up with a solution to prevent the long loop of manual tasks by coming out with a self-serving interface.

Manageable and extendable

Due to a lack of connection between real-time and offline data, data analysts and data scientists did not have a clear picture of what is written in the counters. That’s because the conventional counter data were stored in Redis database to satisfy the query SLA. They could not track the correctness of counter value, or its history. With the new solution, the stakeholders can get a real-time picture of what is stored in the counters using the data engineering tools.

The Machine Learning challenges solved by the Counter service

The Counter service plays an important role in our Machine Learning (ML) workflow.

Data Consistency Challenge/Issue

Most of the machine learning workflows need dedicated input data. However, when there is an anti-fraud model that is trained using offline data from the data lake, it is difficult to use the same model in real-time. This is because the model lacks the data contract and the consistency with the data source. In this case, the Counter service becomes a type of data source by providing the value of counters to file system.

ML featuring

Counters are important features for the ML models. Imagine there is a new invention of counters, which data scientists need to evaluate. We need to provide a historical data set for counters to work. The Counter service provides a counter replay feature, which allows data scientists to simulate the counters via historical payload.

In general, the Counter service is a bridge between online and offline datasets, data scientists, and engineers. There was technical debt with regards to data consistency and automation on the ML pipeline, and the Counter service closed this loop.

How we designed the Counter service

We followed the principle of asynchronized data ingestion, and synchronized transaction for designing the Counter service.

The diagram shows how the counters are generated and saved to database.

How the counters are generated and saved to the database

Counter creation workflow

  1. User opens the Counter Creation UI and creates a new key “fraud:counter:counter_name”.
  2. Configures required fields.
  3. The Counter service monitors the new counter-creation, puts a new counter into load script storage, and starts processing new counter events (see Counter Write below).

Counter write workflow

  1. The Counter service monitors multiple streams, assembles extra data from online data services (i.e. Common Data Service (CDS), passenger service, hydra service, etc), so that rich dataset would also be available for editors on each stream resource.
  2. The Counter Processor evaluates the user-configured expression and writes the evaluated values to the dedicated Grab-Stats stream using the GrabPlugin tool.

Counter read workflow

Counter read workflow

We use Grab-Stats as our storage service. Basically Grab-Stats runs above ScyllaDB, which is a distributed NoSQL data store. We use ScyllaDB because of its good performance on aggregation in memory to deal with the time series dataset. In comparison with in-memory storage like AWS elasticCache, it is 10 times cheaper and as reliable as AWS in terms of stability. The p99 of reading from ScyllaDB is less than 150ms which satisfies our SLA.

How we improved the Counter service performance

We used the multi-buckets strategy to improve the Counter service performance.

Background

There are different time windows when you perform a query. Some counters are time sensitive so that it needs to know what happened in the last 30 or 60 minutes. Some other counters focus on the long term and need to know the events in the last 30 or 90 days.

From a transactional database perspective, it’s not possible to serve small range as well as long term events at the same time. This is because the more the need for the accuracy of the data and the longer the time range, the more aggregations need to happen on database. Which means we would not be able to satisfy the SLA. Otherwise we will need to block other process which leads to the service downgrade.

Solution for improving the query

We resolved this problem by using different granularities of the tables. We pre-aggregated the signals into different time buckets, such as 15min, 1 hour, and 1 day.

When a request comes in, the time-range of the request will be divided by the buckets, and the results are conquered. For example, if there is a request for 9/10 23:15:20 to 9/12 17:20:18, the handler will query 15min buckets within the hour.  It will query for hourly buckets for the same day. And it will query the daily buckets for the rest of 2 days. This way, we avoid doing heavy aggregations, but still keep the accuracy in 15 minutes level in a scalable response time.

Counter service UI

We allowed data analysts and data scientists to onboard counters by themselves, from a dedicated web portal. After the counter is submitted, the Counter service takes care of the integration and parsing the logic at runtime.

Counter service UI

Backend integration

We provide SDK for quicker and better integration. The engineers only need to provide the counter identifier ID (which is shown in the UI) and the time duration in the query. Under the hood we provide a GRPC protocol to communicate across services. We divide the query time window to smaller granularities, fetching from different time series tables and then conquering the result. We are also providing a short TTL cache layer to take the uncommon traffic from client such as network retry or traffic throttle. Our QPS are designed to target 100K.

Monitoring the Counter service

The Counter service dashboard helps to track the human errors while editing the counters in real-time. The Counter service sends alerts to slack channel to notify users if there is any error.

Counter service dashboard

We setup Datadog for monitoring multiple system metrics. The figure below shows a portion of stream processing and counter writing. In the example below, the total stream QPS would reach 5k at peak hour, and the total counter saved to storage tier is about 4k. It will keep climbing without an upper limit, when more counters are onboarded.

Counter service dashboard with multiple metrics

The Counter service UI portal also helps users to fetch real-time counter results for verification purposes.

Counter service UI

Future plans

Here’s what we plan to do in the near future to improve the Counter service.

Close the ML workflow loop

As mentioned above, we plan to send the resource payload of the Counter service to the offline data lake, in order to complete the counter replay function for data scientists. We are working on the project called “time traveler”. As the name indicates, it is used not only for serving the online transactional data, but also supports historical data analytics, and provides more flexibility on counter inventions and experiments.

There are more automation steps we plan to do, such as adding a replay button on the web portal, and hooking up with the offline big data engine to trigger the analytics jobs. The performance metrics will be collected and displayed on the web portal. A single platform would be able to manage both the online and offline data.

Integration to Griffin

Griffin is our rule engine. Counters are sometimes an input to a particular rule, and one rule usually needs many counters to work together. We need to provide a better integration to Griffin on backend. We plan to minimize the current engineering effort when using counters on Griffin. A counter then becomes an automated input variable on Griffin, which can be configured on the web portal by any users.

Coming Soon: Augmented Reality Glasses for the Masses

Post Syndicated from Tekla S. Perry original https://spectrum.ieee.org/view-from-the-valley/consumer-electronics/portable-devices/when-will-we-see-ar-glasses-for-the-masses

A lot can change in seven years. Google Glassa wearable display with a camera and other tools that feed wearers information and allow them to capture photos and videos, began shipping to selected developers in 2013. It was released as a more open beta test in 2014. Then, in early 2015, Google withdrew the product. It has since reemerged, along with a variety of competitors, as a specialized product for use in industry—often for training or displaying diagrams or other information during specific tasks.

As a consumer product though, the technology stalled.

Until now, that is. Facebook last month confirmed that it’s building augmented reality (AR) glasses. Apple is rumored to be getting ready to release its own version of AR glasses next year.

But are AR glasses finally ready for prime time?

I asked Nandan Nayampally, vice president and general manager of ARM’s Immersive Experience Group, to consider whether the technology—and consumers—are ready for AR glasses. Here’s what he had to say.

Raspberry Pi retro gaming on Reddit

Post Syndicated from Alex Bate original https://www.raspberrypi.org/blog/raspberry-pi-retro-gaming-on-reddit/

Reddit was alive with the sound of retro gaming this weekend.

First out to bat is this lovely minimalist, wall-mounted design built by u/sturnus-vulgaris, who states:

I had planned on making a bar top arcade, but after I built the control panel, I kind of liked the simplicity. I mounted a frame of standard 2×4s cut with a miter saw. Might trim out in black eventually (I have several panels I already purchased), but I do like the look of wood.

Next up, a build with Lego bricks, because who doesn’t love Lego bricks?

Just completed my mini arcade cabinet that consists of approximately 1,000 [Lego bricks], a Raspberry Pi, a SNES style controller, Amazon Basics computer speakers, and a 3.5″ HDMI display.

u/RealMagicman03 shared the build here, so be sure to give them an upvote and leave a comment if, like us, you love Raspberry Pi projects that involve Lego bricks.

And lastly, this wonderful use of the Raspberry Pi Compute Module 3+, proving yet again how versatile the form factor can be.

CM3+Lite cartridge for GPi case. I made this cartridge for fun at first, and it works as all I expected. Now I can play more games l like on this lovely portable stuff. And CM3+ is as powerful as RPi3B+, I really like it.

Creator u/martinx72 goes into far more detail in their post, so be sure to check it out.

What other projects did you see this weekend? Share your links with us in the comments below.

The post Raspberry Pi retro gaming on Reddit appeared first on Raspberry Pi.

RapidVideo Shuts Down Following Legal Pressure from Warner Bros and Netflix

Post Syndicated from Ernesto original https://torrentfreak.com/rapidvideo-shuts-down-following-legal-pressure-from-warner-bros-and-netflix-191021/

RapidVideo is a popular file-hosting service that specializes in hosting videos.

Similar to other file-hosting services, it can be used for good and bad. The bad, in this case, is people uploading pirated videos. 

Whether the site’s operators want it or not, that’s what many of RapidVideo’s users are indeed doing. Two weeks ago this resulted in yet another scathing report from movie industry group MPA, which branded the site as a “notorious” piracy haven. 

Behind the scenes, the website’s operator faces mounting pressure as well. RapidVideo has been targeted by lawyers from the MPA and ACE, two of the most powerful anti-piracy forces, which are demanding far-reaching copyright enforcement measures from the site.

To back up their pressure, two MPA/ACE members, Warner Bros. Entertainment and Netflix, filed a lawsuit in Germany to stop the alleged copyright infringements the site enables. While this case remains ongoing, the site’s operator decided not to await the verdict and has shut the site down effective immediately.

The millions of users who regularly visit the site currently see nothing more than a 404 error.

RapidVideo not accessible

TorrentFreak spoke to “Alex Bytes,” the operator of RapidVideo, who informed us that the shutdown is permanent. The site’s operator already considered throwing the towel after the adoption of the new EU Copyright Directive earlier this year, which may make upload filters semi-mandatory for some sites.

“It was high time to quit, because of the upcoming law changes within the EU, due to Article 13/17, where it is a more challenging situation for service providers,” RapidVideo’s Alex tells us.

By shutting the service down, RapidVideo also hopes to get the lawsuit from Warner Bros. and Netflix off its back. In addition, Alex points out that advertising revenues were dropping significantly, so it was hardly worth continuing anyway.

According to RapidVideo’s operator, ACE and the MPA previously demanded far-reaching measures to prevent piracy. The rightsholders requested a thorough “take down, stay down” policy, that would go further than hash or filename filtering.

Instead, rightsholders wanted the site to implement a system similar to YouTube’s Content-ID where more advanced fingerprinting techniques are used to match file uploads to potentially infringing content.

This wasn’t an option for RapidVideo, likely because it would require substantial investments. The other option, shutting the entire site down, became more and more attractive instead, especially in light of the pending lawsuit.

“By shutting down, the lawyers have no more reason to fight in the court against me,” Alex tells us.

For now, however, the court case remains ongoing. TorrentFreak reached out to the lawyer of Warner Bros. and Netflix for a comment on RapidVideo’s decision and the future of their legal claims, but at the time of writing, we have yet to hear back.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and more. We also have VPN reviews, discounts, offers and coupons.

UAV-Based LiDAR Can Measure Shallow Water Depth

Post Syndicated from SBG Systems original https://spectrum.ieee.org/robotics/drones/uavbased-lidar-can-measure-shallow-water-depth

World’s first small-scale topographic and bathymetric scanning LiDAR

ASTRALiTe’s edge™ is the world’s first small-scale topographic and bathymetric scanning LiDAR that can detect small underwater objects, measure shallow water depth, and survey critical underwater infrastructure from a small UAV platform.

The edge™ can see beneath the water surface at depths from 0-5 meters and is completely self-contained with its own Inertial Navigation System with GNSS, battery, and onboard computer. It weighs about 5 kg and is designed for deployment on UAV systems for faster, safer, and more accurate bathymetric surveys. This patented 2-in-1 topographic and bathymetric LiDAR offers a centimeter-level depth resolution. There are numerous possible applications for this LiDAR, such as coastal mapping and surveying, infrastructure inspection, or even military logistics. 

Importance of geo-referencing and motion stabilization

“We needed a motion and navigation solution for our LiDAR. Our requirements included high accuracy along with low size, weight, and power” explains Andy Gisler, Director of Lidar Systems with ASTRALiTe. In addition, the system needed to be able to apply Post-Processing Kinematic (PPK) corrections to the LiDAR data to provide higher accuracy results to ASTRALiTe’s customers.

The LiDAR provides a comprehensive point cloud that needs to be motion-compensated and geo-referenced to be usable. Two methods can be used to reach the centimeter-level accuracy requested by surveyors. The first one is Real-Time Kinematic (RTK), which makes use of corrections obtained from a base station or a base station network in real-time thanks to a radio or a GSM link. The second one is used after the mission using a PPK software. This software will apply the same correction as RTK, but it will also re-compute all the inertial data and raw GNSS observables with a forward-backward-merge algorithm to correct all the trajectories, fill any loss of position, and greatly improve the overall accuracy.

ASTRALiTe chose SBG Systems’ dual antenna Ellipse2-D inertial navigation system which provides motion, RTK, and PPK. The weight of the INS/GNSS solution was especially important to ASTRALiTe as they were designing a system to be flown on most UAVs, where light payload capacities are required for UAV compatibility. The possibility to use two antennas was a key element to consider, as they required a robust heading even during slow-speed flights. In addition to this INS, they also use Qinertia, SBG Systems’ in-house post-processing software

This PPK software gives access to offline RTK corrections from more than 7,000 base stations located in 164 countries and is designed to help UAV integrators get the best of their GNSS or INS/GNSS solution.

About SBG Systems INS/GNSS

SBG Systems is an international company which develops Inertial Measurement Unit with embedded GNSS, from miniature to high accuracy ranges. Combined with cutting-edge calibration techniques and advanced embedded algorithms, SBG Systems manufactures inertial solutions for industrial & research projects such as unmanned vehicle control (land, marine, and aerial), antenna tracking, camera stabilization, and surveying applications.

The collective thoughts of the interwebz

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close