Tag Archives: VK

Netflix Expands Content Protection Team to Reduce Piracy

Post Syndicated from Ernesto original https://torrentfreak.com/netflix-expands-content-protection-team-to-reduce-piracy-171015/

There is little doubt that, in the United States and many other countries, Netflix has become the standard for watching movies on the Internet.

Despite the widespread availability, however, Netflix originals are widely pirated. Episodes from House of Cards, Narcos, and Orange is the New Black are downloaded and streamed millions of times through unauthorized platforms.

The streaming giant is obviously not happy with this situation and has ramped up its anti-piracy efforts in recent years. Since last year the company has sent out over a million takedown requests to Google alone and this volume continues to expand.

This growth coincides with an expansion of the company’s internal anti-piracy division. A new job posting shows that Netflix is expanding this team with a Copyright and Content Protection Coordinator. The ultimate goal is to reduce piracy to a fringe activity.

“The growing Global Copyright & Content Protection Group is looking to expand its team with the addition of a coordinator,” the job listing reads.

“He or she will be tasked with supporting the Netflix Global Copyright & Content Protection Group in its internal tactical take down efforts with the goal of reducing online piracy to a socially unacceptable fringe activity.”

Among other things, the new coordinator will evaluate new technological solutions to tackle piracy online.

More old-fashioned takedown efforts are also part of the job. This includes monitoring well-known content platforms, search engines and social network sites for pirated content.

“Day to day scanning of Facebook, YouTube, Twitter, Periscope, Google Search, Bing Search, VK, DailyMotion and all other platforms (including live platforms) used for piracy,” is listed as one of the main responsibilities.

Netflix’ Copyright and Content Protection Coordinator Job

The coordinator is further tasked with managing Facebook’s Rights Manager and YouTube’s Content-ID system, to prevent circumvention of these piracy filters. Experience with fingerprinting technologies and other anti-piracy tools will be helpful in this regard.

Netflix doesn’t do all the copyright enforcement on its own though. The company works together with other media giants in the recently launched “Alliance for Creativity and Entertainment” that is spearheaded by the MPAA.

In addition, the company also uses the takedown services of external anti-piracy outfits to target more traditional infringement sources, such as cyberlockers and piracy streaming sites. The coordinator has to keep an eye on these as well.

“Liaise with our vendors on manual takedown requests on linking sites and hosting sites and gathering data on pirate streaming sites, cyberlockers and usenet platforms.”

The above shows that Netflix is doing its best to prevent piracy from getting out of hand. It’s definitely taking the issue more seriously than a few years ago when the company didn’t have much original content.

The switch from being merely a distribution platform to becoming a major content producer and copyright holder has changed the stakes. Netflix hasn’t won the war on piracy, it’s just getting started.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

MPAA Reports Pirate Sites, Hosts and Ad-Networks to US Government

Post Syndicated from Ernesto original https://torrentfreak.com/mpaa-reports-pirate-sites-hosts-and-ad-networks-to-us-government-171004/

Responding to a request from the Office of the US Trade Representative (USTR), the MPAA has submitted an updated list of “notorious markets” that it says promote the illegal distribution of movies and TV-shows.

These annual submissions help to guide the U.S. Government’s position towards foreign countries when it comes to copyright enforcement.

What stands out in the MPAA’s latest overview is that it no longer includes offline markets, only sites and services that are available on the Internet. This suggests that online copyright infringement is seen as a priority.

The MPAA’s report includes more than two dozen alleged pirate sites in various categories. While this is not an exhaustive list, the movie industry specifically highlights some of the worst offenders in various categories.

“Content thieves take advantage of a wide constellation of easy-to-use online technologies, such as direct download and streaming, to create infringing sites and applications, often with the look and feel of legitimate content distributors, luring unsuspecting consumers into piracy,” the MPAA writes.

According to the MPAA, torrent sites remain popular, serving millions of torrents to tens of millions of users at any given time.

The Pirate Bay has traditionally been one of the main targets. Based on data from Alexa and SimilarWeb, the MPAA says that TPB has about 62 million unique visitors per month. The other torrent sites mentioned are 1337x.to, Rarbg.to, Rutracker.org, and Torrentz2.eu.

MPAA calls out torrent sites

The second highlighted category covers various linking and streaming sites. This includes the likes of Fmovies.is, Gostream.is, Primewire.ag, Kinogo.club, MeWatchSeries.to, Movie4k.tv and Repelis.tv.

Direct download sites and video hosting services also get a mention. Nowvideo.sx, Openload.co, Rapidgator.net, Uploaded.net and the Russian social network VK.com. Many of these services refuse to properly process takedown notices, the MPAA claims.

The last category is new and centers around piracy apps. These sites offer mobile applications that allow users to stream pirated content, such as IpPlayBox.tv, MoreTV, 3DBoBoVR, TVBrowser, and KuaiKa, which are particularly popular in Asia.

Aside from listing specific sites, the MPAA also draws the US Government’s attention to the streaming box problem. The report specifically mentions that Kodi-powered boxes are regularly abused for infringing purposes.

“An emerging global threat is streaming piracy which is enabled by piracy devices preloaded with software to illicitly stream movies and television programming and a burgeoning ecosystem of infringing add-ons,” the MPAA notes.

“The most popular software is an open source media player software, Kodi. Although Kodi is not itself unlawful, and does not host or link to unlicensed content, it can be easily configured to direct consumers toward unlicensed films and television shows.”

Pirate streaming boxes

There are more than 750 websites offering infringing devices, the Hollywood group notes, adding that the rapid growth of this problem is startling. Interestingly, the report mentions TVAddons.ag as a “piracy add-on repository,” noting that it’s currently offline. Whether the new TVAddons is also seen a problematic is unclear.

The MPAA also continues its trend of calling out third-party intermediaries, including hosting providers. These companies refuse to take pirate sites offline following complaints, even when the MPAA views them as blatantly violating the law.

“Hosting companies provide the essential infrastructure required to operate a website,” the MPAA writes. “Given the central role of hosting providers in the online ecosystem, it is very concerning that many refuse to take action upon being notified…”

The Hollywood group specifically mentions Private Layer and Netbrella as notorious markets. CDN provider CloudFlare is also named. As a US-based company, the latter can’t be included in the list. However, the MPAA explains that it is often used as an anonymization tool by sites and services that are mentioned in the report.

Another group of intermediaries that play a role in fueling piracy (mentioned for the first time) are advertising networks. The MPAA specifically calls out the Canadian company WWWPromoter, which works with sites such as Primewire.ag, Projectfreetv.at and 123movies.to

“The companies connecting advertisers to infringing websites and inadvertently contribute to the prevalence and prosperity of infringing sites by providing funding to the operators of these sites through advertising revenue,” the MPAA writes.

The MPAA’s full report is available here (pdf). The USTR will use this input above to make up its own list of notorious markets. This will help to identify current threats and call on foreign governments to take appropriate action.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Какво искат каталунците? (Част 2)

Post Syndicated from Йовко Ламбрев original https://yovko.net/what-catalans-want-2/

След 1939 г. Испания потъва в мрачния период на диктатурата на Франко. Каталуния отново е наказана да не говори своя език под смъртна заплаха. Докато живях в Барселона, лично се запознах със съвсем млади каталунци, загубили дядо или баба заради това. Рани, твърде скорошни, за да бъдат забравени или простени. Още по-малко, че прошка не е поискана.

Първите години след войната са кошмарно тежки за цяла Испания. Едва в края на 50-те и началото на 60-те започва икономическо и културно възстановяване на Каталуния. Езикът все още е забранен в медиите, но бива позволен в театрите, иначе въпреки забраната книгопечатането на литература на каталунски, макар и силно затруднено, не е прекъсвало. Както вероятно вече сте разбрали от първата част каталунците са смели и много упорити хора.

Каталунският език

Един от митовете за каталунския език е, че той е диалект на кастилския (испанския), което обаче изобщо не е вярно. Езикът има само някои прилики с кастилския, както има с френския и италианския, което е нормално, доколкото са в една езикова група и с близка география. Думите, значението и произношението са в огромна степен различни. Глаголите са различни и с различни корени, макар да има сходни. Има и капани – такива глаголи, които се изписват еднакво, но в различните езици имат напълно друго значение (напр. acostar на кастилски е лягам, докато на каталунски е да донеса нещо по-близо). Ако искам да кажа простичкото как се казвам – на кастилски ще е Me llamo Yovko или Mi nombre es Yovko, а на каталунски Em dic Yovko (произнася се ам дик Йовко) или ако спазим горния конструкт El meu nom és Yovko (обърнете внимание на членуването). Кастилският е еволюирал и се е опростил значително в доста отношения заради по-масовата му употреба, докато каталунският е останал по-близо до древността и корените, носи по-сложна граматика.

Живо доказателство, че каталунският е различен език, е, че говорещите кастилски не могат да говорят каталунски. Разбират по смисъл думите, които са сходни, но дотам. По същия начин испанците и италианците се разбират в прилична степен дори и всеки да говори на своя език – това не означава, че единият език е диалект на другия, нали?

В моят курс по каталунски със съпругата ми бяхме единствените, на които кастилският не им е майчин език или поне не основен. Нещо повече – нашият кастилски беше ужасно базов. И макар да учехме нов език (каталунски), чрез друг език, който не владеем добре (кастилския), ние завършихме сред отличниците, докато на доста курсисти с роден кастилски им беше трудно да достигнат дори средни резултати. Иначе казано познанията по кастилски не носеха никакво предимство.

Сега тук по-веселото е, че самият каталунски има поне три диалекта, без да броим валенсианския, който си е всъщност каталунски с някои дребни разлики. Но пък тук ще вземем да вбесим валенсианците, затова млъквам!

Така или иначе темата за езика е чувствителна за каталунците – те си го обичат много, изстрадали са възможността да го говорят и имат всички основания да го пазят като репер на своята културна идентичност. Това е тяхно право, достойно за уважение от всички ни!

Управлението на автономията

По време на своето управление Франко всъщност не възстановява монархията, едва малко преди да умре кротко в леглото си през 1975 г. е посочил наследника си – поредният Бурбон, внук на последния крал и дочакал да седне на престола на Кралство Испания под крилото на Франко – Хуан-Карлос.

Каталунци-конституционалисти оказват сериозна юридическа помощ при съставянето на днешната Испанска конституция от 1978 г., с която Каталуния възстановява своята политическа и културна автономия. Година по-късно, през 1979 г., е приет и Статутът на автономията (основният закон на областта).

Според този основен закон Каталуния е автономна област със самостоятелно политическо и юридическо самоуправление. Парламентът се избира през 4 години и излъчва президент (който е и министър-председател) и правителство.

Каталуния има собствена военизирана полиция (жандармерия) – Mossos d’Esquadra, която е под командването на каталунското правителство и не е подчинена на националните Guardia Civil (жандармерия) и Policía Nacional (полиция). Единствено ако бъде суспендирана автономията, Испания може да подчини Mossos-ите на националните сили за сигурност (чл.155 от Конституцията). Затова, когато ви разиграват сценки от селски вечеринки, че някакъв си прокурор, бил той и главен или гневен, се бил разпоредил каталунската полиция да се подчини на Guardia Civil, им кажете, както направиха каталунците – да си гледат работата – защото това не може да се случи с прокурорско разпореждане. Испанските закони бива да важат и за Испания, нали?

Националните сили за сигурност в Каталуния имат правомощия само да охраняват пристанища, летища, крайбрежие, национални граници, митници и да се грижат за имиграционния контрол и антитерористични операции.

Народната партия и Мариано Рахой

През 2006 г. е одобрен новият Статут на Каталуния чрез референдум, както е по правилата. Гласуван е също и с мнозинство от каталунския и от испанския парламент. Промените обаче са обжалвани пред Конституционния съд от кръгове около дясно-консервативната Народна партия (Partido Popular), която от 2004 г. се ръководи от Мариано Рахой, а от 2011 г. досега управлява Испания. През 2015 г. те всъщност загубиха изборите, но понеже не се сформира правителство, останаха на власт до следващите предсрочни избори през 2016 г., след които управляват в правителство на малцинството, тъй като социалистите от PSOE тихо съдействаха, отказвайки се да участват в гласуването, което позволи Pахой да прокара кабинета си през парламента с обикновено мнозинство. Реално подкрепата на изборите за Народната партия бе едва 33% (от 70% гласували) – иначе казано се ползва с доверието на едва 23% от имащите право на глас в Испания.

Partido Popular е консервативна християндемократическа партия, член на ЕНП (Европейската народна партия). Под ръководството на Рахой партията все повече залита към патриотични и националистически тези, а политическата му стратегия е основана на две базови теми – едната да противостои на административната и политическа еволюция на Каталуния (вкл. оспорвайки дефакто одобрения ѝ статут), а другата – да противостои на политическите договорки с баските. Чудно, нали?

Partido Popular и Рахой не са в състояние да генерират никакво модерно и ново политическо послание. Единственото им спасение е да концентрират влияние чрез десен популизъм и радикализиране на патриотични тези, защото на тази плоскост могат да пързалят гласоподавателите си, които са предимно сред по-възрастните, консервативните, религиозните и по-заможните испанци. В същия момент профилът на техните гласоподаватели е най-слаб откъм образование.

Partido Popular е затънала в корупционни скандали – точно утихне един и се случва следващ – черни партийни каси и странни парични потоци във всякакви посоки, очевидно за търговия с влияние, все излизат на яве, изгаря по някой бушон, но Мариано Рахой се крепи над водата. Интересен факт е, че цели петима последователни ковчежници на партията му са съдени, разследвани, обвинени или отстранени. И Европа, и ЕНП си мълчат и стискат широко затворени очи, щото нали, в името на стабилността, че иначе ако дойде Подемос на власт…

Всъщност, испанската политика в последните години се изразява горе-долу в това Partido Popular и PSOE да си подават топката. Това ще ви обясни защо испанците са склонни да търсят изход в Подемос и други нови партии, опитвайки се да избягат от пинг-понга между двете основни големи партии, потънали в корупционни скандали и безгранични сфери на задкулисно влияние.

Най-близкото приближение на Partido Popular у нас е ГЕРБ (те са и членове на ЕНП неслучайно), а на PSOE е БСП. И сега си представете ту да ви управлява ГЕРБ, ту БСП… познато ли ви е усещането? А присви ли ви душичката. Ами така и трябва! И испанците ги присвива от доста време насам!

Но да се върнем в Каталуния…

Възпалението на раната

През 2010 г. Конституционният съд, сезиран от хора на Partido Popular, отмени част от текстовете в Статута. Важно е да уточним, че в състава на този съд преобладават членове, които дължат постовете си на Partido Popular. В момента през 2017 г. това е още по-вярно. Конституционният съд на Испания, включително самият му Председател, е тежко зависим от партията на премиера Мариано Рахой! И да не си помислите сега, че искам да внушавам нещо – съвсем си е законно всичко. Това са едни почтени и достолепни хорица, в които нямаме никакво право да се усъмним. Поглеждате към нашия Конституционен съд или ВСС и… чувствате хармонията, нали? Хайде, опитайте се да прокарате нещо смислено през тях, да ви видя…

Та нищо че каталунските и испанските парламенти одобряват промените в Статута с нужното мнозинство, нищо че цяла Каталуния се е произнесла и е одобрила промените чрез законен Референдум. Излизат една шепа съдии и отменят 14 члена като противоконституционни и дават ограничителни тълкувания на други 27. Текстовете им са свързани предимно с автономната правосъдна система на Каталуния, някои важни детайли в преразпределянето на финансирането, статутът на каталунския език и определянето на Каталуния като нация.

След всичко това през 2010 г. каталунците истински се ядосаха! По улиците на Барселона излязоха между милион и милион и половина души. Сформира се гражданска организация, която се нарече ANC – Assemblea Nacional Catalana (Национално събрание на Каталуния), която си постави за цел да постигне независимост. А всяка година на 11 септември – националният празник на Каталуния – оттогава насам се организират масови демонстрации за независимост.

Още масло в огъня

Уточнихме вече, че драмата с каталунския език е чувствителна тема, след всички забранявания и преследвания и загинали заради езика си до съвсем скоро. Испанската конституция обаче се грижи за задължителността единствено на кастилския език (това, което сме свикнали да наричаме испански), а каталунците вписаха като задължителен и каталунския в границите на автономията си, но точно този текст бе сред отменените.

Не стига това, ами през 2012 г. министърът на образованието на Испания се изцепи, че неговата цел е “да се испанизират (“españolizar”) каталунските ученици” и вкара законопроект, който не само противоречи на каталунската юрисдикция, ами позволява каталунските деца да бъдат обучавани едноезично на испански, което от една страна е тъпо, когато детето ти може да излезе с два езика от училище, да го насилваш да излезе с един, а от друга – каталунците възприеха това като колониална политика, каквато тя недвусмислено беше.

Та испанската държава и управляващите от Partido Popular вместо да ходят на пръсти по тънкия лед на регионалната си политика, скачат шумно с кални обувки отгоре му.

На 23 януари 2013 г. каталунският парламент прие Декларация за суверенитет и право на самоопределение на Каталуния, която, разбира се, беше първо суспендирана от Конституционния съд, а после отменена частта ѝ за суверенитета. След още купчина юридически пречки все пак Правителството на Каталуния организира необвързващ референдум за независимост на 9 ноември 2014 и 81% от участвалите се произнесоха в полза на независима Каталуния. Активността обаче беше ниска (37-42% според зависи кой и как брои, защото Референдумът беше необвързващ и беше дадена възможност на 16 и 17-годишни да гласуват, както и на неиспански граждани, което иначе не би било възможно). Заради организирането на това допитване тогавашният президент Artur Mas, вицепрезидентът Joana Ortega и образователният министър Irene Rigau бяха обвинени и осъдени на около две години да не заемат обществени постове, както и на глоби – най-голямата за Мас, възлизаща на 36 500 евро. Има и нови обвинения за 5.2 милиона евро заради разходване на публични средства за същото допитване. Преди това обаче Мас разпусна правителството си и свика извънредни парламентарни избори на 27 септември 2015, които бяха спечелени от коалиция от партии, които подкрепят независимостта.

Кралят

Междувременно покрай тези събития Хуан Карлос абдикира в полза на сина си, Фелипе VI – нещо, което испанската конституция също не допуска, но беше променена скоростно за по-малко от седмица, което само показа на каталунците колко невъзможна е тяхната кауза в текущия политически контекст в Мадрид.

След лавината от скандали в кралското семейство на Хуан Карлос, свързани с извънбрачна връзка на краля и негов незаконен син, харчовете на двореца, особено в кризата, ловджийските му гафове, скандалите с корупционни схеми на едната му дъщеря и прането на пари и укриването на данъци от зет му, имиджът на монархията в Испания напоследък хич не е висок. Прехвърлянето на топката към Фелипе VI изглеждаше като спасителен ход в контекста на зачестилите демонстрации, искащи референдум за република, и доколкото младият крал изглежда умерен и по-рационален, за разлика от баща си – женен е за простосмъртна съпруга (била е журналистка преди да се омъжи за него), говори свободно каталунски, освен испански. Дори се бяха появили надежди, че с перфектния си каталунски може да спечели сърцата на всички като поеме ролята на медиатор и спаси ситуацията в Каталуния, но до този момент не се забелязва такова негово желание и едва ли някой още мисли, че това е възможно, доколкото той вече избра обичайната позиция на кралска надменност към проблемите на простосмъртните.

Подготовката на процеса за независимост

Предсрочните избори от 2015 г. имаха допълнителна цел. Основните партии, подкрепящи независимостта, участваха с обща гражданска (непартийна) листа, към която впоследствие се присъедини и една по-малка партия. Така управляващата коалиция в Каталуния има мнозинство в локалния парламент, с което прокара няколко закона от ключово значение за евентуална бъдеща независима република – например за Каталунска данъчна администрация, за въпросния референдум и т.н.

Реално юристите от двете страни спорят каква част от това законодателство е ОК, но предвид сложността на юрисдикциите на автономните области в Испания отговорът не е еднозначен. Тук за Рахой работи простичката теза – абе, не може локалното законодателство да има превес над националното и тези закони са “незаконни” – но всъщност не е така, зависи от много неща. Каталуния не е област Стара Загора, а автономия със собствен основен закон и локално законодателство. То не може да противоречи на националното, но може да бъде много различно от него в много посоки. И не подценявайте юридическата култура и опит на каталунците, моля – обърнете се назад и вижте натрупванията им…

Всъщност популярна теза, която испанските медии и Народната партия на Рахой непрекъснато повтарят, е, че всичко, което се случва в Каталуния, е “незаконно”. Това е непрецизно и популистко обобщение. Доказателство е, че дори приятелски настроеният към Рахой и партията му Конституционен съд на Испания не твърди такова нещо. Няма твърдение, че референдумът е незаконен или противоконституционен, а е само суспендиран от Конституционния съд, докато той прегледа законосъобразността му и се произнесе.

Не четете само El País – това е все едно да се информирате само от “24 часа”.

Вярно, редно е да признаем, че и каталунците използват всички процедурни хватки в своя полза. Законите бяха гласувани в последния момент, за да оставят в цайтнот тромавия Конституционният съд. Но реално това не е нарушение. Войната на нерви се води с всички средства и от двете страни. Особено когато няма желание за диалог.

Данъците

Популярна теза е, че каталунците искат повече пари за себе си и това е проява на егоизъм от тяхна страна. От друга те са богат, индустриален район, който осигурява солидна част за националната икономика – 20% от БВП на Испания и 25% от износа, а е само един от седемнайсетте района. Богатите райони подкрепят бедните региони при преразпределение на данъците, но проблемът е в математиката и кой как пресмята.

Според каталунските икономисти фискалният дефицит на региона надвишава 8% от БВП, което според всички международни стандарти е твърде голяма стойност и спъва развитието на икономиката. Те спорят, че реално стойността е по-голяма, защото има разминаване между разпределения дял (на хартия) за Каталуния от националните финанси, които се връщат най-вече под формата на инфраструктура, и това, което реално Каталуния получава.

Испанската държава не е съгласна. И това е нещо, което се решава на масата на преговорите, с експертни оценки и експертни спорове. Народната партия и Мариано Рахой обаче с години отказват да дискутират каталунските теми – така това се превърна в ключов аргумент на индепендистите.

И не е случайно, че прогресивните испанци твърдят, че основният двигател на процеса на независимост на Каталуния е правителството в Мадрид.

Политически диалог ли?

И за среднограмотен човек е ясно, че ако беше проведен някакъв политически диалог, всичко можеше да се размине. Но двете страни си говорят през медиите и с декларации. Испания пропиля 7 пълни години, през които можеше да потуши напрежението. Пропиля ги генерално и пълноценно, отказвайки всяко предложение да диалог. Лично Рахой се грижеше да аргументира всеки отказ.

В пространно интервю в края на август президентът на Каталуния потвърди, че дори и в последния момент, ако испанската държава се реши на диалог, той ще откликне.

Уви, Мариано Рахой е от друга планета и думите “преговори” и “политически диалог” очевидно са му чужди. Всъщност това отговаря напълно на неговия сценарий – конфронтация и радикализация. Рискува да счупи миноритарното си управление, но това е единствената стратегия, която празната му откъм идеи глава може да роди.

На 15 септември 2017 г. отново Президентът на Каталуния, Вицепрезидентът, Председателката на Каталунския парламент и кметът на Барселона заедно изпратиха писмо до Рахой и краля с предложение за диалог.

Същият ден Рахой каза само, че неговото правителство ще направи всичко възможно да осуети референдума, неговият говорител пък, че в Мадрид не са получили писмото, но в последния момент можели да го тълкуват само като заплаха, а кралят… той, както обикновено, запази царствено мълчание.

Всичко това не е от вчера

Друга весела теза е, че каталунците едва ли не вчера им е хрумнала идеята за независимост. По повода ще остава само тази картинка – отляво е вестник Guardian от края на 1918 година, а отдясно статия в същия вестник отпреди няколко дни. Открийте разликите 🙂

guardian

Републиката

Нещо, което някак остава под килима, но е редно да отчетем, е фактът, че каталунците в мнозинството си са прорепубликански настроени. Това обяснява антипатията на краля и монархистите към тях, но всеки обсъждан дотук референдум не поставя под никакво съмнение, че евентуалната независима Каталуния ще бъде република.

В Испания също се чуват гласове за референдум за ново държавно устройство и това кара определени консервативни и влиятелни кръгове да потръпват при мисълта това да се случи.

Демократичността на испанската конституция

Каталунците често критикуват демократичността на испанската конституция по принцип, макар двама от бащите ѝ да са каталунци. Истината е, че имат основания. Четирима от седмината “бащи” на испанската конституция са били част от фашисткия апарат, включително един от тях е Министърът на пропагандата на Франко. Представете си дали е възможно съвременната германска конституция да е писана от Гьобелс?

Армията е оказала силно влияние в процеса на създаване на конституцията, за да опази своя интерес, и макар одобрена на референдум с 88%, съмненията, че зад този резултат стои пряката или косвена заплаха на бившите военни на Франко, са напълно основателни.

Обобщение

Всъщност, макар и тлеещ отдавна, проблемът не беше нерешим. Каталунците са сговорчиви и работливи хора, които в мнозинството си искат да бъдат оставени на мира да си вадят хляба, да правят музика, книги и изкуство и да се веселят на многобройните си фестивали. Те са адски толерантни и широкоскроени хора, с модерни възгледи за себе си, бъдещето и Европа.

Каквото и да четете в испанските медии, в мнозинството си каталунците нямат нищо против испанците. Това, което им тежи, не са съседите, а испанската държава. Те точно така наричат държавата си – испанската държава – за да акцентират на административния апарат, а не на нацията, и… за да намекнат, че не е тяхната държава…

А тя не е тяхна, защото в общия национален парламент те имат скромно присъствие, обусловено от тежестта на региона върху картата. Не биха могли да прокарат нищо през националния парламент без подкрепата на основните испански партии, които рядко изобщо обръщат внимание на регионите. Локалното им законодателство е под терора на Конституциония съд, който особено откакто Рахой и Partido Popular са на власт, действа по поръчка.

Испания отказва всякакъв диалог с каталунските представители, въпреки че те са легитимно избрани и овластени от хората. Прави го и защото се страхува, че ако изгуби Каталуния, ще последват баските, а после може би Галисия. Баските също от години чакат обещанията на Мадрид да се реализират и все повече губят търпение, но и за това няма да прочетете много в испанските, нашите или европейските медии…

Каталуния е разделена

Истината е, че въпреки всичко Каталуния е разделена. Важно е да правим разлика между това, че 70-80% от каталунците са с нагласа да гласуват на този Референдум, и това как точно ще гласуват.

Ако не се беше стигнало до тази ескалация в последните дни, реално по-малко от половината каталунци щяха да гласуват за отделяне в неделя и всичко щеше да утихне поне за някакъв период от време. Рахой обаче изпрати жандармерия и полиция в нечуван обем, арестува каталунски политици, претърси медии и печатници, конфискува бюлетини, урни и плакати, обвини предварително стотици кметове, че съдействат на организацията по референдума, заплаши да спре националната каталунска телевизия, блокира каталунски сайтове и заплаши да спре целият top level domain на Каталуния .cat, докато междувременно е насъскал прокуратурата да рови за някоя мръсна риза на текущия каталунски президент от времето, когато е бил кмет, чрез прокурорско разпореждане се опита да вземе контрола над каталунската жандармерия, което е незаконно и противоречи едновременно на испанската конституция и на каталунския статут.

Всичко това преди референдумът да се е случил и преди да е обявен за незаконен от Конституционния съд – иначе казано, дори да допуснем хипотезата, че референдумът е престъпление – то още не е се е случило, за да има виновни за него!

След всичко това никой вече не знае как ще гласуват каталунците, защото ескалацията и радикализацията и от двете страни е факт и играта на нерви вече не е безопасна.

Европа мълчи, защото основните европейски партии са обречени заради своите “приятелски” зависимости. Иска им се това да си остане вътрешна работа на Испания и нещата да се оправят някак от само себе си. Отдавна трябваше да бъде предложено посредничество в този спор, да бъде уговорен Рахой да отстъпи нещичко и да изглади нещата. Но ЕНП няма този кураж. А европейците за пореден път виждат една куха бюрокрация, която не работи, скатава се и прибира дъждобрана точно когато завали проливен дъжд (ако изобщо е имало дъждобран). Затова не се чудете, когато хората залитат насам и натам, търсейки изход – кой в популизма, кой в национализма, кой в крайнолеви и дори понякога утопични концепции.

Искат гласът им да се чува и да има значение!

Всъщност каталунците искат едно нещо – да гласуват и гласът им да има значение – и това не може и не бива да противоречи на никоя конституция! Още по-малко в Европа! Днес. Правото на глас и самоопределение е основно човешко право и е наднационално!

Събудете се, хора! Какви легенди са наблъскали в главите ви, ако ви е нужна причина или повод, за да признаете правото на някого да изрази позиция – особено пък когато това са няколко милиона души? Наистина ли сте затрили чувствителността си към свободата, това което сте – а сте свободни хора – когато не са ви нужни причини и правила, за да изразите волята си. Правилата са за да ви гарантират това право, а не за да ви го отнемат. Правилата идват после – първо е свободната воля!

Дали каталунците ще се отделят или не е второстепеннен въпрос. По-важният е да могат да решат това свободно! А ние, останалите, няма да сме европейци и не заслужаваме да се наричаме свободни хора, ако не защитим това им право – звучно и категорично!

Visca Catalunya!

How to Configure an LDAPS Endpoint for Simple AD

Post Syndicated from Cameron Worrell original https://aws.amazon.com/blogs/security/how-to-configure-an-ldaps-endpoint-for-simple-ad/

Simple AD, which is powered by Samba  4, supports basic Active Directory (AD) authentication features such as users, groups, and the ability to join domains. Simple AD also includes an integrated Lightweight Directory Access Protocol (LDAP) server. LDAP is a standard application protocol for the access and management of directory information. You can use the BIND operation from Simple AD to authenticate LDAP client sessions. This makes LDAP a common choice for centralized authentication and authorization for services such as Secure Shell (SSH), client-based virtual private networks (VPNs), and many other applications. Authentication, the process of confirming the identity of a principal, typically involves the transmission of highly sensitive information such as user names and passwords. To protect this information in transit over untrusted networks, companies often require encryption as part of their information security strategy.

In this blog post, we show you how to configure an LDAPS (LDAP over SSL/TLS) encrypted endpoint for Simple AD so that you can extend Simple AD over untrusted networks. Our solution uses Elastic Load Balancing (ELB) to send decrypted LDAP traffic to HAProxy running on Amazon EC2, which then sends the traffic to Simple AD. ELB offers integrated certificate management, SSL/TLS termination, and the ability to use a scalable EC2 backend to process decrypted traffic. ELB also tightly integrates with Amazon Route 53, enabling you to use a custom domain for the LDAPS endpoint. The solution needs the intermediate HAProxy layer because ELB can direct traffic only to EC2 instances. To simplify testing and deployment, we have provided an AWS CloudFormation template to provision the ELB and HAProxy layers.

This post assumes that you have an understanding of concepts such as Amazon Virtual Private Cloud (VPC) and its components, including subnets, routing, Internet and network address translation (NAT) gateways, DNS, and security groups. You should also be familiar with launching EC2 instances and logging in to them with SSH. If needed, you should familiarize yourself with these concepts and review the solution overview and prerequisites in the next section before proceeding with the deployment.

Note: This solution is intended for use by clients requiring an LDAPS endpoint only. If your requirements extend beyond this, you should consider accessing the Simple AD servers directly or by using AWS Directory Service for Microsoft AD.

Solution overview

The following diagram and description illustrates and explains the Simple AD LDAPS environment. The CloudFormation template creates the items designated by the bracket (internal ELB load balancer and two HAProxy nodes configured in an Auto Scaling group).

Diagram of the the Simple AD LDAPS environment

Here is how the solution works, as shown in the preceding numbered diagram:

  1. The LDAP client sends an LDAPS request to ELB on TCP port 636.
  2. ELB terminates the SSL/TLS session and decrypts the traffic using a certificate. ELB sends the decrypted LDAP traffic to the EC2 instances running HAProxy on TCP port 389.
  3. The HAProxy servers forward the LDAP request to the Simple AD servers listening on TCP port 389 in a fixed Auto Scaling group configuration.
  4. The Simple AD servers send an LDAP response through the HAProxy layer to ELB. ELB encrypts the response and sends it to the client.

Note: Amazon VPC prevents a third party from intercepting traffic within the VPC. Because of this, the VPC protects the decrypted traffic between ELB and HAProxy and between HAProxy and Simple AD. The ELB encryption provides an additional layer of security for client connections and protects traffic coming from hosts outside the VPC.

Prerequisites

  1. Our approach requires an Amazon VPC with two public and two private subnets. The previous diagram illustrates the environment’s VPC requirements. If you do not yet have these components in place, follow these guidelines for setting up a sample environment:
    1. Identify a region that supports Simple AD, ELB, and NAT gateways. The NAT gateways are used with an Internet gateway to allow the HAProxy instances to access the internet to perform their required configuration. You also need to identify the two Availability Zones in that region for use by Simple AD. You will supply these Availability Zones as parameters to the CloudFormation template later in this process.
    2. Create or choose an Amazon VPC in the region you chose. In order to use Route 53 to resolve the LDAPS endpoint, make sure you enable DNS support within your VPC. Create an Internet gateway and attach it to the VPC, which will be used by the NAT gateways to access the internet.
    3. Create a route table with a default route to the Internet gateway. Create two NAT gateways, one per Availability Zone in your public subnets to provide additional resiliency across the Availability Zones. Together, the routing table, the NAT gateways, and the Internet gateway enable the HAProxy instances to access the internet.
    4. Create two private routing tables, one per Availability Zone. Create two private subnets, one per Availability Zone. The dual routing tables and subnets allow for a higher level of redundancy. Add each subnet to the routing table in the same Availability Zone. Add a default route in each routing table to the NAT gateway in the same Availability Zone. The Simple AD servers use subnets that you create.
    5. The LDAP service requires a DNS domain that resolves within your VPC and from your LDAP clients. If you do not have an existing DNS domain, follow the steps to create a private hosted zone and associate it with your VPC. To avoid encryption protocol errors, you must ensure that the DNS domain name is consistent across your Route 53 zone and in the SSL/TLS certificate (see Step 2 in the “Solution deployment” section).
  2. Make sure you have completed the Simple AD Prerequisites.
  3. We will use a self-signed certificate for ELB to perform SSL/TLS decryption. You can use a certificate issued by your preferred certificate authority or a certificate issued by AWS Certificate Manager (ACM).
    Note: To prevent unauthorized connections directly to your Simple AD servers, you can modify the Simple AD security group on port 389 to block traffic from locations outside of the Simple AD VPC. You can find the security group in the EC2 console by creating a search filter for your Simple AD directory ID. It is also important to allow the Simple AD servers to communicate with each other as shown on Simple AD Prerequisites.

Solution deployment

This solution includes five main parts:

  1. Create a Simple AD directory.
  2. Create a certificate.
  3. Create the ELB and HAProxy layers by using the supplied CloudFormation template.
  4. Create a Route 53 record.
  5. Test LDAPS access using an Amazon Linux client.

1. Create a Simple AD directory

With the prerequisites completed, you will create a Simple AD directory in your private VPC subnets:

  1. In the Directory Service console navigation pane, choose Directories and then choose Set up directory.
  2. Choose Simple AD.
    Screenshot of choosing "Simple AD"
  3. Provide the following information:
    • Directory DNS – The fully qualified domain name (FQDN) of the directory, such as corp.example.com. You will use the FQDN as part of the testing procedure.
    • NetBIOS name – The short name for the directory, such as CORP.
    • Administrator password – The password for the directory administrator. The directory creation process creates an administrator account with the user name Administrator and this password. Do not lose this password because it is nonrecoverable. You also need this password for testing LDAPS access in a later step.
    • Description – An optional description for the directory.
    • Directory Size – The size of the directory.
      Screenshot of the directory details to provide
  4. Provide the following information in the VPC Details section, and then choose Next Step:
    • VPC – Specify the VPC in which to install the directory.
    • Subnets – Choose two private subnets for the directory servers. The two subnets must be in different Availability Zones. Make a note of the VPC and subnet IDs for use as CloudFormation input parameters. In the following example, the Availability Zones are us-east-1a and us-east-1c.
      Screenshot of the VPC details to provide
  5. Review the directory information and make any necessary changes. When the information is correct, choose Create Simple AD.

It takes several minutes to create the directory. From the AWS Directory Service console , refresh the screen periodically and wait until the directory Status value changes to Active before continuing. Choose your Simple AD directory and note the two IP addresses in the DNS address section. You will enter them when you run the CloudFormation template later.

Note: Full administration of your Simple AD implementation is out of scope for this blog post. See the documentation to add users, groups, or instances to your directory. Also see the previous blog post, How to Manage Identities in Simple AD Directories.

2. Create a certificate

In the previous step, you created the Simple AD directory. Next, you will generate a self-signed SSL/TLS certificate using OpenSSL. You will use the certificate with ELB to secure the LDAPS endpoint. OpenSSL is a standard, open source library that supports a wide range of cryptographic functions, including the creation and signing of x509 certificates. You then import the certificate into ACM that is integrated with ELB.

  1. You must have a system with OpenSSL installed to complete this step. If you do not have OpenSSL, you can install it on Amazon Linux by running the command, sudo yum install openssl. If you do not have access to an Amazon Linux instance you can create one with SSH access enabled to proceed with this step. Run the command, openssl version, at the command line to see if you already have OpenSSL installed.
    [[email protected] ~]$ openssl version
    OpenSSL 1.0.1k-fips 8 Jan 2015

  2. Create a private key using the command, openssl genrsa command.
    [[email protected] tmp]$ openssl genrsa 2048 > privatekey.pem
    Generating RSA private key, 2048 bit long modulus
    ......................................................................................................................................................................+++
    ..........................+++
    e is 65537 (0x10001)

  3. Generate a certificate signing request (CSR) using the openssl req command. Provide the requested information for each field. The Common Name is the FQDN for your LDAPS endpoint (for example, ldap.corp.example.com). The Common Name must use the domain name you will later register in Route 53. You will encounter certificate errors if the names do not match.
    [[email protected] tmp]$ openssl req -new -key privatekey.pem -out server.csr
    You are about to be asked to enter information that will be incorporated into your certificate request.

  4. Use the openssl x509 command to sign the certificate. The following example uses the private key from the previous step (privatekey.pem) and the signing request (server.csr) to create a public certificate named server.crt that is valid for 365 days. This certificate must be updated within 365 days to avoid disruption of LDAPS functionality.
    [[email protected] tmp]$ openssl x509 -req -sha256 -days 365 -in server.csr -signkey privatekey.pem -out server.crt
    Signature ok
    subject=/C=XX/L=Default City/O=Default Company Ltd/CN=ldap.corp.example.com
    Getting Private key

  5. You should see three files: privatekey.pem, server.crt, and server.csr.
    [[email protected] tmp]$ ls
    privatekey.pem server.crt server.csr

    Restrict access to the private key.

    [[email protected] tmp]$ chmod 600 privatekey.pem

    Keep the private key and public certificate for later use. You can discard the signing request because you are using a self-signed certificate and not using a Certificate Authority. Always store the private key in a secure location and avoid adding it to your source code.

  6. In the ACM console, choose Import a certificate.
  7. Using your favorite Linux text editor, paste the contents of your server.crt file in the Certificate body box.
  8. Using your favorite Linux text editor, paste the contents of your privatekey.pem file in the Certificate private key box. For a self-signed certificate, you can leave the Certificate chain box blank.
  9. Choose Review and import. Confirm the information and choose Import.

3. Create the ELB and HAProxy layers by using the supplied CloudFormation template

Now that you have created your Simple AD directory and SSL/TLS certificate, you are ready to use the CloudFormation template to create the ELB and HAProxy layers.

  1. Load the supplied CloudFormation template to deploy an internal ELB and two HAProxy EC2 instances into a fixed Auto Scaling group. After you load the template, provide the following input parameters. Note: You can find the parameters relating to your Simple AD from the directory details page by choosing your Simple AD in the Directory Service console.
Input parameter Input parameter description
HAProxyInstanceSize The EC2 instance size for HAProxy servers. The default size is t2.micro and can scale up for large Simple AD environments.
MyKeyPair The SSH key pair for EC2 instances. If you do not have an existing key pair, you must create one.
VPCId The target VPC for this solution. Must be in the VPC where you deployed Simple AD and is available in your Simple AD directory details page.
SubnetId1 The Simple AD primary subnet. This information is available in your Simple AD directory details page.
SubnetId2 The Simple AD secondary subnet. This information is available in your Simple AD directory details page.
MyTrustedNetwork Trusted network Classless Inter-Domain Routing (CIDR) to allow connections to the LDAPS endpoint. For example, use the VPC CIDR to allow clients in the VPC to connect.
SimpleADPriIP The primary Simple AD Server IP. This information is available in your Simple AD directory details page.
SimpleADSecIP The secondary Simple AD Server IP. This information is available in your Simple AD directory details page.
LDAPSCertificateARN The Amazon Resource Name (ARN) for the SSL certificate. This information is available in the ACM console.
  1. Enter the input parameters and choose Next.
  2. On the Options page, accept the defaults and choose Next.
  3. On the Review page, confirm the details and choose Create. The stack will be created in approximately 5 minutes.

4. Create a Route 53 record

The next step is to create a Route 53 record in your private hosted zone so that clients can resolve your LDAPS endpoint.

  1. If you do not have an existing DNS domain for use with LDAP, create a private hosted zone and associate it with your VPC. The hosted zone name should be consistent with your Simple AD (for example, corp.example.com).
  2. When the CloudFormation stack is in CREATE_COMPLETE status, locate the value of the LDAPSURL on the Outputs tab of the stack. Copy this value for use in the next step.
  3. On the Route 53 console, choose Hosted Zones and then choose the zone you used for the Common Name box for your self-signed certificate. Choose Create Record Set and enter the following information:
    1. Name – The label of the record (such as ldap).
    2. Type – Leave as A – IPv4 address.
    3. Alias – Choose Yes.
    4. Alias Target – Paste the value of the LDAPSURL on the Outputs tab of the stack.
  4. Leave the defaults for Routing Policy and Evaluate Target Health, and choose Create.
    Screenshot of finishing the creation of the Route 53 record

5. Test LDAPS access using an Amazon Linux client

At this point, you have configured your LDAPS endpoint and now you can test it from an Amazon Linux client.

  1. Create an Amazon Linux instance with SSH access enabled to test the solution. Launch the instance into one of the public subnets in your VPC. Make sure the IP assigned to the instance is in the trusted IP range you specified in the CloudFormation parameter MyTrustedNetwork in Step 3.b.
  2. SSH into the instance and complete the following steps to verify access.
    1. Install the openldap-clients package and any required dependencies:
      sudo yum install -y openldap-clients.
    2. Add the server.crt file to the /etc/openldap/certs/ directory so that the LDAPS client will trust your SSL/TLS certificate. You can copy the file using Secure Copy (SCP) or create it using a text editor.
    3. Edit the /etc/openldap/ldap.conf file and define the environment variables BASE, URI, and TLS_CACERT.
      • The value for BASE should match the configuration of the Simple AD directory name.
      • The value for URI should match your DNS alias.
      • The value for TLS_CACERT is the path to your public certificate.

Here is an example of the contents of the file.

BASE dc=corp,dc=example,dc=com
URI ldaps://ldap.corp.example.com
TLS_CACERT /etc/openldap/certs/server.crt

To test the solution, query the directory through the LDAPS endpoint, as shown in the following command. Replace corp.example.com with your domain name and use the Administrator password that you configured with the Simple AD directory

$ ldapsearch -D "[email protected]corp.example.com" -W sAMAccountName=Administrator

You should see a response similar to the following response, which provides the directory information in LDAP Data Interchange Format (LDIF) for the administrator distinguished name (DN) from your Simple AD LDAP server.

# extended LDIF
#
# LDAPv3
# base <dc=corp,dc=example,dc=com> (default) with scope subtree
# filter: sAMAccountName=Administrator
# requesting: ALL
#

# Administrator, Users, corp.example.com
dn: CN=Administrator,CN=Users,DC=corp,DC=example,DC=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: user
description: Built-in account for administering the computer/domain
instanceType: 4
whenCreated: 20170721123204.0Z
uSNCreated: 3223
name: Administrator
objectGUID:: l3h0HIiKO0a/ShL4yVK/vw==
userAccountControl: 512
…

You can now use the LDAPS endpoint for directory operations and authentication within your environment. If you would like to learn more about how to interact with your LDAPS endpoint within a Linux environment, here are a few resources to get started:

Troubleshooting

If you receive an error such as the following error when issuing the ldapsearch command, there are a few things you can do to help identify issues.

ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
  • You might be able to obtain additional error details by adding the -d1 debug flag to the ldapsearch command in the previous section.
    $ ldapsearch -D "[email protected]" -W sAMAccountName=Administrator –d1

  • Verify that the parameters in ldap.conf match your configured LDAPS URI endpoint and that all parameters can be resolved by DNS. You can use the following dig command, substituting your configured endpoint DNS name.
    $ dig ldap.corp.example.com

  • Confirm that the client instance from which you are connecting is in the CIDR range of the CloudFormation parameter, MyTrustedNetwork.
  • Confirm that the path to your public SSL/TLS certificate configured in ldap.conf as TLS_CAERT is correct. You configured this in Step 5.b.3. You can check your SSL/TLS connection with the command, substituting your configured endpoint DNS name for the string after –connect.
    $ echo -n | openssl s_client -connect ldap.corp.example.com:636

  • Verify that your HAProxy instances have the status InService in the EC2 console: Choose Load Balancers under Load Balancing in the navigation pane, highlight your LDAPS load balancer, and then choose the Instances

Conclusion

You can use ELB and HAProxy to provide an LDAPS endpoint for Simple AD and transport sensitive authentication information over untrusted networks. You can explore using LDAPS to authenticate SSH users or integrate with other software solutions that support LDAP authentication. This solution’s CloudFormation template is available on GitHub.

If you have comments about this post, submit them in the “Comments” section below. If you have questions about or issues implementing this solution, start a new thread on the Directory Service forum.

– Cameron and Jeff

MPAA Revenue Stabilizes, Chris Dodd Earns $3.5 Million

Post Syndicated from Ernesto original https://torrentfreak.com/mpaa-revenue-stabilizes-chris-dodd-earns-3-5-million170813/

Protecting the interests of Hollywood, the MPAA has been heavily involved in numerous anti-piracy efforts around the world in recent years.

Through its involvement in the shutdowns of Popcorn Time, YIFY, isoHunt, Hotfile, Megaupload and several other platforms, the MPAA has worked hard to target piracy around the globe.

Perhaps just as importantly, the group lobbies lawmakers globally while managing anti-piracy campaigns both in and outside the US, including the Creative Content UK program.

All this work doesn’t come for free, obviously, so the MPAA relies on six major movie studios for financial support. After its revenues plummeted a few years ago, they have steadily recovered and according to its latest tax filing, the MPAA’s total income is now over $72 million.

The IRS filing, covering the fiscal year 2015, reveals that the movie studios contributed $65 million, the same as a year earlier. Overall revenue has stabilized as well, after a few years of modest growth.

Going over the numbers, we see that salaries make up a large chunk of the expenses. Former Senator Chris Dodd, the MPAA’s Chairman and CEO, is the highest paid employee with a total income of more than $3.5 million, including a $250,000 bonus.

It was recently announced that Dodd will leave the MPAA next month. He will be replaced by Charles Rivkin, another political heavyweight. Rivkin previously served as Assistant Secretary of State for Economic and Business Affairs in the Obama administration.

In addition to Dodd, there are two other employees who made over a million in 2015, Global General Counsel Steve Fabrizio and Diane Strahan, the MPAA’s Chief Operating Officer.

Looking at some of the other expenses we see that the MPAA’s lobbying budget remained stable at $4.2 million. Another $4.4 million went to various grants, while legal costs totaled $7.2 million that year.

More than two million dollars worth of legal expenses were paid to the US law firm Jenner & Block, which represented the movie studios in various court cases. In addition, the MPAA paid more than $800,000 to the UK law firm Wiggin, which assisted the group in local site-blocking efforts.

Finally, it’s worth looking at the various gifts and grants the MPAA hands out. As reported last year, the group handsomely contributes to various research projects. This includes a recurring million dollar grant for Carnegie Mellon’s ‘Initiative for Digital Entertainment Analytics’ (IDEA), which researches various piracy related topics.

IDEA co-director Rahul Telang previously informed us that the gift is used to hire researchers and pay for research materials. It is not tied to a particular project.

We also see $70,000+ in donations for both the Democratic and Republican Attorneys General associations. The purpose of the grants is listed as “general support.” Interestingly, just recently over a dozen Attorneys General released a public service announcement warning the public to stay away from pirate sites.

These type of donations and grants are nothing new and are a regular part of business across many industries. Still, they are worth keeping in mind.

It will be interesting to see which direction the MPAA takes in the years to come. Under Chris Dodd it has booked a few notable successes, but there is still a long way to go before the piracy situation is somewhat under control.



MPAA’s full form 990 was published in Guidestar recently and a copy is available here (pdf).

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Under the Hood of Server-Side Encryption for Amazon Kinesis Streams

Post Syndicated from Damian Wylie original https://aws.amazon.com/blogs/big-data/under-the-hood-of-server-side-encryption-for-amazon-kinesis-streams/

Customers are using Amazon Kinesis Streams to ingest, process, and deliver data in real time from millions of devices or applications. Use cases for Kinesis Streams vary, but a few common ones include IoT data ingestion and analytics, log processing, clickstream analytics, and enterprise data bus architectures.

Within milliseconds of data arrival, applications (KCL, Apache Spark, AWS Lambda, Amazon Kinesis Analytics) attached to a stream are continuously mining value or delivering data to downstream destinations. Customers are then scaling their streams elastically to match demand. They pay incrementally for the resources that they need, while taking advantage of a fully managed, serverless streaming data service that allows them to focus on adding value closer to their customers.

These benefits are great; however, AWS learned that many customers could not take advantage of Kinesis Streams unless their data-at-rest within a stream was encrypted. Many customers did not want to manage encryption on their own, so they asked for a fully managed, automatic, server-side encryption mechanism leveraging centralized AWS Key Management Service (AWS KMS) customer master keys (CMK).

Motivated by this feedback, AWS added another fully managed, low cost aspect to Kinesis Streams by delivering server-side encryption via KMS managed encryption keys (SSE-KMS) in the following regions:

  • US East (N. Virginia)
  • US West (Oregon)
  • US West (N. California)
  • EU (Ireland)
  • Asia Pacific (Singapore)
  • Asia Pacific (Tokyo)

In this post, I cover the mechanics of the Kinesis Streams server-side encryption feature. I also share a few best practices and considerations so that you can get started quickly.

Understanding the mechanics

The following section walks you through how Kinesis Streams uses CMKs to encrypt a message in the PutRecord or PutRecords path before it is propagated to the Kinesis Streams storage layer, and then decrypt it in the GetRecords path after it has been retrieved from the storage layer.

When server-side encryption is enabled—which takes just a few clicks in the console—the partition key and payload for every incoming record is encrypted automatically as it’s flowing into Kinesis Streams, using the selected CMK. When data is at rest within a stream, it’s encrypted.

When records are retrieved through a GetRecords request from the encrypted stream, they are decrypted automatically as they are flowing out of the service. That means your Kinesis Streams producers and consumers do not need to be aware of encryption. You have a fully managed data encryption feature at your fingertips, which can be enabled within seconds.

AWS also makes it easy to audit the application of server-side encryption. You can use the AWS Management Console for instant stream-level verification; the responses from PutRecord, PutRecords, and getRecords; or AWS CloudTrail.

Calling PutRecord or PutRecords

When server-side encryption is enabled for a particular stream, Kinesis Streams and KMS perform the following actions when your applications call PutRecord or PutRecords on a stream with server-side encryption enabled. The Amazon Kinesis Producer Library (KPL) uses PutRecords.

 

  1. Data is sent from a customer’s producer (client) to a Kinesis stream using TLS via HTTPS. Data in transit to a stream is encrypted by default.
  2. After data is received, it is momentarily stored in RAM within a front-end proxy layer.
  3. Kinesis Streams authenticates the producer, then impersonates the producer to request input keying material from KMS.
  4. KMS creates key material, encrypts it by using CMK, and sends both the plaintext and encrypted key material to the service, encrypted with TLS.
  5. The client uses the plaintext key material to derive data encryption keys (data keys) that are unique per-record.
  6. The client encrypts the payload and partition key using the data key in RAM within the front-end proxy layer and removes the plaintext data key from memory.
  7. The client appends the encrypted key material to the encrypted data.
  8. The plaintext key material is securely cached in memory within the front-end layer for reuse, until it expires after 5 minutes.
  9. The client delivers the encrypted message to a back-end store where it is stored at rest and fetchable by an authorized consumer through a GetRecords The Amazon Kinesis Client Library (KCL) calls GetRecords to retrieve records from a stream.

Calling getRecords

Kinesis Streams and KMS perform the following actions when your applications call GetRecords on a server-side encrypted stream.

 

  1. When a GeRecords call is made, the front-end proxy layer retrieves the encrypted record from its back-end store.
  2. The consumer (client) makes a request to KMS using a token generated by the customer’s request. KMS authorizes it.
  3. The client requests that KMS decrypt the encrypted key material.
  4. KMS decrypts the encrypted key material and sends the plaintext key material to the client.
  5. Kinesis Streams derives the per-record data keys from the decrypted key material.
  6. If the calling application is authorized, the client decrypts the payload and removes the plaintext data key from memory.
  7. The client delivers the payload over TLS and HTTPS to the consumer, requesting the records. Data in transit to a consumer is encrypted by default.

Verifying server-side encryption

Auditors or administrators often ask for proof that server-side encryption was or is enabled. Here are a few ways to do this.

To check if encryption is enabled now for your streams:

  • Use the AWS Management Console or the DescribeStream API operation. You can also see what CMK is being used for encryption.
  • See encryption in action by looking at responses from PutRecord, PutRecords, or GetRecords When encryption is enabled, the encryptionType parameter is set to “KMS”. If encryption is not enabled, encryptionType is not included in the response.

Sample PutRecord response

{
    "SequenceNumber": "49573959617140871741560010162505906306417380215064887298",
    "ShardId": "shardId-000000000000",
    "EncryptionType": "KMS"
}

Sample GetRecords response

{
    "Records": [
        {
            "Data": "aGVsbG8gd29ybGQ=", 
            "PartitionKey": "test", 
            "ApproximateArrivalTimestamp": 1498292565.825, 
            "EncryptionType": "KMS", 
            "SequenceNumber": "495735762417140871741560010162505906306417380215064887298"
        }, 
        {
            "Data": "ZnJvZG8gbGl2ZXMK", 
            "PartitionKey": "3d0d9301-3c30-4c48-a9a8-e485b2982b28", 
            "ApproximateArrivalTimestamp": 1498292801.747, 
            "EncryptionType": "KMS", 
            "SequenceNumber": "49573959617140871741560010162507115232237011062036103170"
        }
    ], 
    "NextShardIterator": "AAAAAAAAAAEvFypHZDx/4bJVAS34puwdiNcwssKqbh/XhRK7HSYRq3RS+YXJnVKJ8j0gQUt94bONdqQYHk9X9JHgefMUDKzDzndy5WbZWO4CS3hRdMdrbmJ/9KoR4lOfZvqTLt6JWQjDqXv0IaKs06/LHYcEA3oPcyQLOTJHdJl2EzplCTZnn/U295ovxvqF9g9DY8y2nVoMkdFLmdcEMVXjhCDKiRIt", 
    "MillisBehindLatest": 0
}

To check if encryption was enabled, use CloudTrail, which logs the StartStreamEncryption() and StopStreamEncryption() API calls made against a particular stream.

Getting started

It’s very easy to enable, disable, or modify server-side encryption for a particular stream.

  1. In the Kinesis Streams console, select a stream and choose Details.
  2. Select a CMK and select Enabled.
  3. Choose Save.

You can enable encryption only for a live stream, not upon stream creation.  Follow the same process to disable a stream. To use a different CMK, select it and choose Save.

Each of these tasks can also be accomplished using the StartStreamEncryption and StopStreamEncryption API operations.

Considerations

There are a few considerations you should be aware of when using server-side encryption for Kinesis Streams:

  • Permissions
  • Costs
  • Performance

Permissions

One benefit of using the “(Default) aws/kinesis” AWS managed key is that every producer and consumer with permissions to call PutRecord, PutRecords, or GetRecords inherits the right permissions over the “(Default) aws/kinesis” key automatically.

However, this is not necessarily the same case for a CMK. Kinesis Streams producers and consumers do not need to be aware of encryption. However, if you enable encryption using a custom master key but a producer or consumer doesn’t have IAM permissions to use it, PutRecord, PutRecords, or GetRecords requests fail.

This is a great security feature. On the other hand, it can effectively lead to data loss if you inadvertently apply a custom master key that restricts producers and consumers from interacting from the Kinesis stream. Take precautions when applying a custom master key. For more information about the minimum IAM permissions required for producers and consumers interacting with an encrypted stream, see Using Server-Side Encryption.

Costs

When you apply server-side encryption, you are subject to KMS API usage and key costs. Unlike custom KMS master keys, the “(Default) aws/kinesis” CMK is offered free of charge. However, you still need to pay for the API usage costs that Kinesis Streams incurs on your behalf.

API usage costs apply for every CMK, including custom ones. Kinesis Streams calls KMS approximately every 5 minutes when it is rotating the data key. In a 30-day month, the total cost of KMS API calls initiated by a Kinesis stream should be less than a few dollars.

Performance

During testing, AWS discovered that there was a slight increase (typically 0.2 millisecond or less per record) with put and get record latencies due to the additional overhead of encryption.

If you have questions or suggestions, please comment below.

Protesters Physically Block HQ of Russian Web Blocking Watchdog

Post Syndicated from Andy original https://torrentfreak.com/protesters-physically-block-hq-of-russian-web-blocking-watchdog-170701/

Hardly a week goes by without the Russian web-blocking juggernaut rolling on to new targets. Whether they’re pirate websites, anonymity and proxy services, or sites that the government feels are inappropriate, web blocks are now a regular occurance in the region.

With thousands of domains and IP addresses blocked, the situation is serious. Just recently, however, blocks have been more problematic than usual. Telecoms watchdog Roskomnadzor, which oversees blocking, claims that innocent services are rarely hit. But critics say that overbroad IP address blockades are affecting the innocent.

Earlier this month there were reports that citizens across the country couldn’t access some of the country’s largest sites, including Google.ru, Yandex.ru, local Facebook variant vKontakte, and even the Telegram messaging app.

There have been various explanations for the problems, but the situation with Google appears to have stemmed from a redirect to an unauthorized gambling site. The problem was later resolved, and Google was removed from the register of banned sites, but critics say it should never have been included in the first place.

These and other developments have proven too much for some pro-freedom activists. This week they traveled to Roskomnadzor’s headquarters in St. Petersburg to give the blocking watchdog a small taste of its own medicine.

Activists from the “Open Russia” and “Civil Petersburg” movements positioned themselves outside the entrance to the telecom watchdog’s offices and built up their own barricade constructed from boxes. Each carried a label with the text “Blocked Citizens of Russia.”

Blockading the blockaders in Russia

“Freedom of information, like freedom of expression, are the basic values of our society. Those who try to attack them, must themselves be ‘blocked’ from society,” said Open Russia coordinator Andrei Pivovarov.

Rather like Internet blockades, the image above shows Open Russia’s blockade only partially doing its job by covering just three-quarters of Roskomnadzor’s entrance.

Whether that was deliberate or not is unknown but the video embedded below clearly shows staff walking around its perimeter. The protestors were probably just being considerate, but there are suggestions that staff might have been using VPNs or Tor.

Moving forward, new advice from Roskomnadzor to ISPs is that they should think beyond IP address and domain name blocking and consider using Deep Packet Inspection. This would help ensure blocks are carried out more accurately, the watchdog says.

There’s even a suggestion that rather than doing their own website filtering, Internet service providers could buy a “ready cleaned” Internet feed from an approved supplier instead. This would remove the need for additional filtering at their end, it’s argued, but it sounds like more problems waiting to happen.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

AWS Hot Startups – May 2017

Post Syndicated from Tina Barr original https://aws.amazon.com/blogs/aws/aws-hot-startups-may-2017/

April showers bring May startups! This month we have three hot startups for you to check out. Keep reading to find out what they’re up to, and how they’re using AWS to do it.

Today’s post features the following startups:

  • Lobster – an AI-powered platform connecting creative social media users to professionals.
  • Visii – helping consumers find the perfect product using visual search.
  • Tiqets – a curated marketplace for culture and entertainment.

Lobster (London, England)

Every day, social media users generate billions of authentic images and videos to rival typical stock photography. Powered by Artificial Intelligence, Lobster enables brands, agencies, and the press to license visual content directly from social media users so they can find that piece of content that perfectly fits their brand or story. Lobster does the work of sorting through major social networks (Instagram, Flickr, Facebook, Vk, YouTube, and Vimeo) and cloud storage providers (Dropbox, Google Photos, and Verizon) to find media, saving brands and agencies time and energy. Using filters like gender, color, age, and geolocation can help customers find the unique content they’re looking for, while Lobster’s AI and visual recognition finds images instantly. Lobster also runs photo challenges to help customers discover the perfect image to fit their needs.

Lobster is an excellent platform for creative people to get their work discovered while also protecting their content. Users are treated as copyright holders and earn 75% of the final price of every sale. The platform is easy to use: new users simply sign in with an existing social media or cloud account and can start showcasing their artistic talent right away. Lobster allows users to connect to any number of photo storage sources so they’re able to choose which items to share and which to keep private. Once users have selected their favorite photos and videos to share, they can sit back and watch as their work is picked to become the signature for a new campaign or featured on a cool website – and start earning money for their work.

Lobster is using a variety of AWS services to keep everything running smoothly. The company uses Amazon S3 to store photography that was previously ordered by customers. When a customer purchases content, the respective piece of content must be available at any given moment, independent from the original source. Lobster is also using Amazon EC2 for its application servers and Elastic Load Balancing to monitor the state of each server.

To learn more about Lobster, check them out here!

Visii (London, England)

In today’s vast web, a growing number of products are being sold online and searching for something specific can be difficult. Visii was created to cater to businesses and help them extract value from an asset they already have – their images. Their SaaS platform allows clients to leverage an intelligent visual search on their websites and apps to help consumers find the perfect product for them. With Visii, consumers can choose an image and immediately discover more based on their tastes and preferences. Whether it’s clothing, artwork, or home decor, Visii will make recommendations to get consumers to search visually and subsequently help businesses increase their conversion rates.

There are multiple ways for businesses to integrate Visii on their website or app. Many of Visii’s clients choose to build against their API, but Visii also work closely with many clients to figure out the most effective way to do this for each unique case. This has led Visii to help build innovative user interfaces and figure out the best integration points to get consumers to search visually. Businesses can also integrate Visii on their website with a widget – they just need to provide a list of links to their products and Visii does the rest.

Visii runs their entire infrastructure on AWS. Their APIs and pipeline all sit in auto-scaling groups, with ELBs in front of them, sending things across into Amazon Simple Queue Service and Amazon Aurora. Recently, Visii moved from Amazon RDS to Aurora and noted that the process was incredibly quick and easy. Because they make heavy use of machine learning, it is crucial that their pipeline only runs when required and that they maximize the efficiency of their uptime.

To see how companies are using Visii, check out Style Picker and Saatchi Art.

Tiqets (Amsterdam, Netherlands)

Tiqets is making the ticket-buying experience faster and easier for travelers around the world.  Founded in 2013, Tiqets is one of the leading curated marketplaces for admission tickets to museums, zoos, and attractions. Their mission is to help travelers get the most out of their trips by helping them find and experience a city’s culture and entertainment. Tiqets partners directly with vendors to adapt to a customer’s specific needs, and is now active in over 30 cities in the US, Europe, and the Middle East.

With Tiqets, travelers can book tickets either ahead of time or at their destination for a wide range of attractions. The Tiqets app provides real-time availability and delivers tickets straight to customer’s phones via email, direct download, or in the app. Customers save time skipping long lines (a perk of the app!), save trees (don’t need to physically print tickets), and most importantly, they can make the most out of their leisure time. For each attraction featured on Tiqets, there is a lot of helpful information including best modes of transportation, hours, commonly asked questions, and reviews from other customers.

The Tiqets platform consists of the consumer-facing website, the internal and external-facing APIs, and the partner self-service portals. For the app hosting and infrastructure, Tiqets uses AWS services such as Elastic Load Balancing, Amazon EC2, Amazon RDS, Amazon CloudFront, Amazon Route 53, and Amazon ElastiCache. Through the infrastructure orchestration of their AWS configuration, they can easily set up separate development or test environments while staying close to the production environment as well.

Tiqets is hiring! Be sure to check out their jobs page if you are interested in joining the Tiqets team.

Thanks for reading and don’t forget to check out April’s Hot Startups if you missed it.

-Tina Barr

 

 

Crash Course Computer Science with Carrie Anne Philbin

Post Syndicated from Alex Bate original https://www.raspberrypi.org/blog/crash-course-carrie-anne-philbin/

Get your teeth into the history of computer science with our Director of Education, Carrie Anne Philbin, and the team at YouTube’s incredible Crash Course channel.

Crash Course Computer Science Preview

Starting February 22nd, Carrie Anne Philbin will be hosting Crash Course Computer Science! In this series, we’re going to trace the origins of our modern computers, take a closer look at the ideas that gave us our current hardware and software, discuss how and why our smart devices just keep getting smarter, and even look towards the future!

The brainchild of Hank and John Green (the latter of whom is responsible for books such as The Fault in Our Stars and all of my resultant heartbroken tears), Crash Course is an educational YouTube channel specialising in courses for school-age tuition support.

As part of the YouTube Orginal Channel Initiative, and with their partners PBS Digital Studios, the team has completed courses in subjects such as physics, hosted by Dr. Shini Somara, astronomy with Phil Plait, and sociology with Nicole Sweeney.

Raspberry Pi Carrie Anne Philbin Crash Course

Oh, and they’ve recently released a new series on computer science with Carrie Anne Philbin , whom you may know as Raspberry Pi’s Director of Education and the host of YouTube’s Geek Gurl Diaries.

Computer Science with Carrie Anne

Covering topics such as RAM, Boolean logic, CPU design , and binary, the course is currently up to episode twelve of its run. Episodes are released every Tuesday, and there are lots more to come.

Crash Course Carrie Anne Philbin Raspberry Pi

Following the fast-paced, visual style of the Crash Course brand, Carrie Anne takes her viewers on a journey from early computing with Lovelace and Babbage through to the modern-day electronics that power our favourite gadgets such as tablets, mobile phones, and small single-board microcomputers…

The response so far

A few members of the Raspberry Pi team recently attended VidCon Europe in Amsterdam to learn more about making video content for our community – and also so I could exist in the same space as the Holy Trinity, albeit briefly.

At VidCon, Carrie Anne took part in an engaging and successful Women in Science panel with Sally Le Page, Viviane Lalande, Hana Shoib, Maddie Moate, and fellow Crash Course presenter Dr. Shini Somara. I could see that Crash Course Computer Science was going down well from the number of people who approached Carrie Anne to thank her for the course, from those who were learning for the first time to people who were rediscovering the subject.

Crash Course Carrie Anne Philbin Raspberry Pi

Take part in the conversation

Join in the conversation! Head over to YouTube, watch Crash Course Computer Science, and join the discussion in the comments.

Crash Course Carrie Anne Philbin Raspberry Pi

You can also follow Crash Course on Twitter for release updates, and subscribe on YouTube to get notifications of new content.

Oh, and who can spot the sneaky Raspberry Pi in the video introduction?

“Cheers!”

Crash Course Computer Science Outtakes

In which Carrie Anne presents a new sing-a-long format and faces her greatest challenge yet – signing off an episode. Want to find Crash Course elsewhere on the internet? Facebook – http://www.facebook.com/YouTubeCrashCourse Twitter – http://www.twitter.com/TheCrashCourse Tumblr – http://thecrashcourse.tumblr.com Support Crash Course on Patreon: http://patreon.com/crashcourse CC Kids: http://www.youtube.com/crashcoursekids Produced in collaboration with PBS Digital Studios: http://youtube.com/pbsdigitalstudios The Latest from PBS Digital Studios: https://www.youtube.com/playlist?list=PL1mtdjDVOoOqJzeaJAV15Tq0tZ1vKj7ZV We’ve got merch!

The post Crash Course Computer Science with Carrie Anne Philbin appeared first on Raspberry Pi.

Russia Wants To Hold Social Networks Liable For Internet Piracy

Post Syndicated from Andy original https://torrentfreak.com/russia-wants-hold-social-networks-liable-internet-piracy-170402/

When file-sharing was in its infancy, most infringement took place via P2P software such as Kazaa or LimeWire. With their built-in search and download features they were an all-in-one solution, ripe for a full on legal attack.

In more recent times the web has played a much more important role in the distribution of copyright-infringing material, via torrent or streaming sites, for example. However, the rise of social media presents a new threat, with huge numbers of people now accessing copyrighted content via Facebook, Twitter, and other platforms.

The social media problem is considered to be particularly problematic in Russia, with users sharing full movies, TV shows and music, via platforms such as vKontakte, Russia’s Facebook. Such sites claim to be fully compliant with copyright law and do make efforts to reduce infringement with licensing deals and content recognition software. Nevertheless, if the Russian government has its way, the noose could tighten significantly in the future.

Social networking platforms currently enjoy the status of ‘information intermediary’ in Russia, a standing that puts them on a par with Internet service providers who can not generally be held liable for the infringing acts of their subscribers.

However, the Ministry of Culture believes that since much of the copyright infringement in Russia is now carried out via social networks, it will soon be necessary to strip them of their intermediary status. That would have the effect of rendering them jointly liable for infringement alongside their errant subscribers.

According to news outlet Izvestia, the relevant bill has already been drafted and, after gaining approval from the Ministry of Culture board, will be presented for public comment.

The basic premise is that when imposing liability on a social platform, courts must consider several factors. They include whether a platform should have been aware that content is infringing, whether any preventative measures were taken to mitigate infringement (filtering), whether timely steps to stop infringement were taken once the platform was made aware (takedowns), and whether or not profit was generated from illegal use (advertising).

“Despite the excuses, the technical ability to [prevent infringement] exists. Of course, this will require a lot of money, but if you want to use content you have to pay for it,” a content producer told Izvestia.

“Today, virtually all Internet traffic is comprised of audiovisual content, all supplied by content creators and often not paid for. Measures should be taken – such as those in the Ministry of Culture bill, and many others – to drive illegal content into the ‘ghetto’. Piracy is theft, and it must be fought.”

But while content producers and distributors believe there are simple solutions, others view the situation as more complex.

In common with complaints voiced by critics in the US and Europe, there are concerns that huge burdens will be placed on platform providers if they are required to conduct a full legal analysis of every file uploaded by their users. There are also worries that non-infringing content (such as public domain material) could get caught up in filtering systems.

Quite how these plans will play out is unclear, but it seems likely that social networks will put up a fight to ensure that whatever responsibilities are imposed on them allow room for development and innovation.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Encrypt and Decrypt Amazon Kinesis Records Using AWS KMS

Post Syndicated from Temitayo Olajide original https://aws.amazon.com/blogs/big-data/encrypt-and-decrypt-amazon-kinesis-records-using-aws-kms/

Customers with strict compliance or data security requirements often require data to be encrypted at all times, including at rest or in transit within the AWS cloud. This post shows you how to build a real-time streaming application using Kinesis in which your records are encrypted while at rest or in transit.

Amazon Kinesis overview

The Amazon Kinesis platform enables you to build custom applications that analyze or process streaming data for specialized needs. Amazon Kinesis can continuously capture and store terabytes of data per hour from hundreds of thousands of sources such as website clickstreams, financial transactions, social media feeds, IT logs, and transaction tracking events.

Through the use of HTTPS, Amazon Kinesis Streams encrypts data in-flight between clients which protects against someone eavesdropping on records being transferred. However, the records encrypted by HTTPS are decrypted once the data enters the service. This data is stored at rest for 24 hours (configurable up to 168 hours) to ensure that your applications have enough headroom to process, replay, or catch up if they fall behind.

Walkthrough

In this post you build encryption and decryption into sample Kinesis producer and consumer applications using the Amazon Kinesis Producer Library (KPL), the Amazon Kinesis Consumer Library (KCL), AWS KMS, and the aws-encryption-sdk. The methods and the techniques used in this post to encrypt and decrypt Kinesis records can be easily replicated into your architecture. Some constraints:

  • AWS charges for the use of KMS API requests for encryption and decryption, for more information see AWS KMS Pricing.
  • You cannot use Amazon Kinesis Analytics to query Amazon Kinesis Streams with records encrypted by clients in this sample application.
  • If your application requires low latency processing, note that there will be a slight hit in latency.

The following diagram shows the architecture of the solution.

Encrypting the records at the producer

Before you call the PutRecord or PutRecords API, you will encrypt the string record by calling KinesisEncryptionUtils.toEncryptedString.

In this example, we used a sample stock sales ticker object:

example {"tickerSymbol": "AMZN", "salesPrice": "900", "orderId": "300", "timestamp": "2017-01-30 02:41:38"}. 

The method (KinesisEncryptionUtils.toEncryptedString) call takes four parameters:

  • amazonaws.encryptionsdk.AwsCrypto
  • stock sales ticker object
  • amazonaws.encryptionsdk.kms.KmsMasterKeyProvider
  • util.Map of an encryption context

A ciphertext is returned back to the main caller which is then also checked for size by calling KinesisEncryptionUtils.calculateSizeOfObject. Encryption increases the size of an object. To prevent the object from being throttled, the size of the payload (one or more records) is validated to ensure it is not greater than 1MB. In this example encrypted records sizes with payload exceeding 1MB are logged as warning. If the size is less than the limit, then either addUserRecord or PutRecord and PutRecords are called if you are using the KPL or the Kinesis Streams API respectively 

Example: Encrypting records with KPL

//Encrypting the records
String encryptedString = KinesisEncryptionUtils.toEncryptedString(crypto, ticker, prov,context);
log.info("Size of encrypted object is : "+ KinesisEncryptionUtils.calculateSizeOfObject(encryptedString));
//check if size of record is greater than 1MB
if(KinesisEncryptionUtils.calculateSizeOfObject(encryptedString) >1024000)
   log.warn("Record added is greater than 1MB and may be throttled");
//UTF-8 encoding of encrypted record
ByteBuffer data = KinesisEncryptionUtils.toEncryptedByteStream(encryptedString);
//Adding the encrypted record to stream
ListenableFuture<UserRecordResult> f = producer.addUserRecord(streamName, randomPartitionKey(), data);
Futures.addCallback(f, callback);

In the above code, the example sales ticker record is passed to the KinesisEncryptionUtils.toEncryptedString and an encrypted record is returned. The encryptedRecord value is also passed to KinesisEncryptionUtils.calculateSizeOfObject and the size of the encrypted payload is returned and checked to see if it is less than 1MB. If it is, the payload is then UTF-8 encoded (KinesisEncryptionUtils.toEncryptedByteStream), then sent to the stream for processing.

Example: Encrypting the records with Streams PutRecord

//Encrypting the records
String encryptedString = KinesisEncryptionUtils.toEncryptedString(crypto, ticker, prov, context);
log.info("Size of encrypted object is : " + KinesisEncryptionUtils.calculateSizeOfObject(encryptedString));
//check if size of record is greater than 1MB
if (KinesisEncryptionUtils.calculateSizeOfObject(encryptedString) > 1024000)
    log.warn("Record added is greater than 1MB and may be throttled");
//UTF-8 encoding of encryptyed record
ByteBuffer data = KinesisEncryptionUtils.toEncryptedByteStream(encryptedString);
putRecordRequest.setData(data);
putRecordRequest.setPartitionKey(randomPartitionKey());
//putting the record into the stream
kinesis.putRecord(putRecordRequest);

Verifying that records are encrypted

After the call to KinesisEncryptionUtils.toEncryptedString, you can print out the encrypted string record just before UTF-8 encoding. An example of what is printed to standard output when running this sample application is shown below.

[main] INFO kinesisencryption.streams.EncryptedProducerWithStreams - String Record is TickerSalesObject{tickerSymbol='FB', salesPrice='184.285409142', orderId='2a0358f1-9f8a-4bbe-86b3-c2929047e15d', timeStamp='2017-01-30 02:41:38'} and Encrypted Record String is AYADeMf6zmVg9JvIkGNv5M39rhUAbgACAAdLaW5lc2lzAARjYXJzABVhd3MtY3J5cHRvLXB1YmxpYy1rZXkAREFpUkpCaG1UOFQ3UTZQZ253dm9FSU9iUDZPdE1xTHdBZ1JjNlZxN2doMDZ3QlBEWUZndWdJSEFKaXNvT0ZPUGsrdz09AAEAB2F3cy1rbXMAS2Fybjphd3M6a21zOnVzLWVhc3QtMTo1NzM5MDY1ODEwMDI6a2V5LzM3ZGM5MGRjLTNmMWMtNGE3Ny1hNTFkLWE2NTNiMTczZmNkYgCnAQEBAHgbPoaYTiF/oIMp49yPBkZmVVotylZpUqwkkzJJicLjLQAAAH4wfAYJKoZIhvcNAQcGoG8wbQIBADBoBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDCCYBk+hfB3tOGVx7QIBEIA7FqaEcOWpic+gKNeT+dUe4yttB9dsZSFPAUTlz2L2zlyLXSLMh1otRH24SO485ov+TCTtRCgiA8a9rYQCAAAAAAwAABAArlGWPO8BavNSJIpJOtJekRUhOwbM+WM1NBVXB/////8AAAABXNZnRND3J7u8EZx3AAAAkfSxVPMYUv0Ovrd4AIUTmMcaiR0Z+IcJNAXqAhvMmDKpsJaQG76Q6pYExarolwT+6i87UOi6TGvAiPnH74GbkEniWe66rAF6mOra2JkffK6pBdhh95mEOGLaVPBqs2jswUTfdcBJQl9NEb7wx9XpFX8fNDF56Vly7u6f8OQ7lY6fNrOupe5QBFnLvwehhtogd72NTQ/yEbDDoPKUZN3IlWIEAGYwZAIwISFw+zdghALtarsHSIgPMs7By7/Yuda2r3hqSmqlCyCXy7HMFIQxHcEILjiLp76NAjB1D8r8TC1Zdzsfiypi5X8FvnK/6EpUyFoOOp3y4nEuLo8M2V/dsW5nh4u2/m1oMbw=

You can also verify that the record stayed encrypted in Streams by printing out the UTF-8 decoded received record immediately after the getRecords API call. An example of the print output when running the sample application is shown below.

[Thread-2] INFO kinesisencryption.utils.KinesisEncryptionUtils - Verifying object received from stream is encrypted. -Encrypted UTF-8 decoded : AYADeBJz/kt7Fm3L1lvS8Wy8jhAAbgACAAdLaW5lc2lzAARjYXJzABVhd3MtY3J5cHRvLXB1YmxpYy1rZXkAREFrM2N4K2s1ODJuOGVlNWF3TVJ1dk1UUHZQc2FHeGoxQisxb09kNWtDUExHYjJDS0lMZW5LSnlYakRmdFR4dzQyUT09AAEAB2F3cy1rbXMAS2Fybjphd3M6a21zOnVzLWVhc3QtMTo1NzM5MDY1ODEwMDI6a2V5LzM3ZGM5MGRjLTNmMWMtNGE3Ny1hNTFkLWE2NTNiMTczZmNkYgCnAQEBAHgbPoaYTiF/oIMp49yPBkZmVVotylZpUqwkkzJJicLjLQAAAH4wfAYJKoZIhvcNAQcGoG8wbQIBADBoBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDAGI3oWLlIJ2p6kffQIBEIA7JVUOTsLtEyNK8vS4GIS9iyTejuB2xhIpRXfG8o0lUfHawcrCbNbNH8XLm/8RW5JbgXo10EpOs8dSjkICAAAAAAwAABAAy64r24sGVKWN4C1gXCwJYHvZkLpJJj16SZlhpv////8AAAABg2pPFchIiaM7D9VuAAAAkwh10ul5sZQ08KsgkFszOOvFoQu95CiY7cK8H+tBloVOZglMqhhhvoIIZLr9hmI8/lQvRXzGDdo7Xkp0FAT5Jpztt8Hq/ZuLfZtNYIWOw594jShqpZt6uXMdMnpb/38R3e5zLK5vrYkM6NS4WPMFrHsOKN5tn0CDForgojRcdpmCJ8+cWLNltb2S+EJiWiyWS+ibw2vJ/RFm6WZO6nD+MXn3vyMAZzBlAjAuIUTYL1cbQ3ENxDIeXHJAWQguNPqxq4HgaCmCEI9/rn/GAKSc2nT9ln3UsVq/2dgCMQC7yNJ3DCTnppavfxTbcVS+rXaDDpZZx/ZsluMqXAFM5/FFvKRqr0dVML28tGunxmU=

Decrypting the records at the consumer

After you receive the records into your consumer as a list, you can get the data as a ByteBuffer by calling record.getData. You then decode and decrypt the byteBuffer by calling the KinesisEncryptionUtils.decryptByteStream. This method takes five parameters:

  • amazonaws.encryptionsdk.AwsCrypto
  • record ByteBuffer
  • amazonaws.encryptionsdk.kms.KmsMasterKeyProvider
  • key arn string
  • java.util.Map of your encryption context

A string representation of the ticker sales object is returned back to the caller for further processing. In this example, this representation is just printed to standard output.

[Thread-2] INFO kinesisencryption.streams.DecryptShardConsumerThread - Decrypted Text Result is TickerSalesObject{tickerSymbol='AMZN', salesPrice='304.958313333', orderId='50defaf0-1c37-4e84-85d7-bc15597355eb', timeStamp='2017-01-30 02:41:38'}

Example: Decrypting records with the KCL and Streams API

ByteBuffer buffer = record.getData();
//Decrypting the encrypted record data
String decryptedResult = KinesisEncryptionUtils.decryptByteStream(crypto,buffer,prov,this.getKeyArn(), this.getContext());
log.info("Decrypted Text Result is " + decryptedResult);

With the above code, records in the Kinesis Streams are decrypted using the same key ARN and encryption context that was previously used to encrypt it at the producer side.

Maven dependencies

To use the implementation I’ve outlined in this post, you need to use a few maven dependencies outlined below in the pom.xml together with the Bouncy Castle libraries. Bouncy Castle provides a cryptography API for Java.

 <dependency>
        <groupId>org.bouncycastle</groupId>
        <artifactId>bcprov-ext-jdk15on</artifactId>
        <version>1.54</version>
    </dependency>
<dependency>
   <groupId>com.amazonaws</groupId>
   <artifactId>aws-encryption-sdk-java</artifactId>
   <version>0.0.1</version>
</dependency>

Summary

You may incorporate above sample code snippets or use it as a guide in your application code to just start encrypting and decrypting your records to and from an Amazon Kinesis Stream.

A complete producer and consumer example application and a more detailed step-by-step example of developing an Amazon Kinesis producer and consumer application on AWS with encrypted records is available at the kinesisencryption github repository.

If you have questions or suggestions, please comment below.


About the Author

Temitayo Olajide is a Cloud Support Engineer with Amazon Web Services. He works with customers to provide architectural solutions, support and guidance to implementing high velocity streaming data applications in the cloud. In his spare time, he plays ping-pong and hangs out with family and friends

 

 


Related

Secure Amazon EMR with Encryption

 

 

The command-line, for cybersec

Post Syndicated from Robert Graham original http://blog.erratasec.com/2017/01/the-command-line-for-cybersec.html

On Twitter I made the mistake of asking people about command-line basics for cybersec professionals. A got a lot of useful responses, which I summarize in this long (5k words) post. It’s mostly driven by the tools I use, with a bit of input from the tweets I got in response to my query.

bash

By command-line this document really means bash.

There are many types of command-line shells. Windows has two, ‘cmd.exe’ and ‘PowerShell’. Unix started with the Bourne shell ‘sh’, and there have been many variations of this over the years, ‘csh’, ‘ksh’, ‘zsh’, ‘tcsh’, etc. When GNU rewrote Unix user-mode software independently, they called their shell “Bourne Again Shell” or “bash” (queue “JSON Bourne” shell jokes here).

Bash is the default shell for Linux and macOS. It’s also available on Windows, as part of their special “Windows Subsystem for Linux”. The windows version of ‘bash’ has become my most used shell.

For Linux IoT devices, BusyBox is the most popular shell. It’s easy to clear, as it includes feature-reduced versions of popular commands.

man

‘Man’ is the command you should not run if you want help for a command.

Man pages are designed to drive away newbies. They are only useful if you already mostly an expert with the command you desire help on. Man pages list all possible features of a program, but do not highlight examples of the most common features, or the most common way to use the commands.

Take ‘sed’ as an example. It’s used most commonly to do a search-and-replace in files, like so:

$ sed ‘s/rob/dave/’ foo.txt

This usage is so common that many non-geeks know of it. Yet, if you type ‘man sed’ to figure out how to do a search and replace, you’ll get nearly incomprehensible gibberish, and no example of this most common usage.

I point this out because most guides on using the shell recommend ‘man’ pages to get help. This is wrong, it’ll just endlessly frustrate you. Instead, google the commands you need help on, or better yet, search StackExchange for answers.

You might try asking questions, like on Twitter or forum sites, but this requires a strategy. If you ask a basic question, self-important dickholes will respond by telling you to “rtfm” or “read the fucking manual”. A better strategy is to exploit their dickhole nature, such as saying “too bad command xxx cannot do yyy”. Helpful people will gladly explain why you are wrong, carefully explaining how xxx does yyy.

If you must use ‘man’, use the ‘apropos’ command to find the right man page. Sometimes multiple things in the system have the same or similar names, leading you to the wrong page.

apt-get install yum

Using the command-line means accessing that huge open-source ecosystem. Most of the things in this guide do no already exist on the system. You have to either compile them from source, or install via a package-manager. Linux distros ship with a small footprint, but have a massive database of precompiled software “packages” in the cloud somewhere. Use the “package manager” to install the software from the cloud.

On Debian-derived systems (like Ubuntu, Kali, Raspbian), type “apt-get install masscan” to install “masscan” (as an example). Use “apt-cache search scan” to find a bunch of scanners you might want to install.

On RedHat systems, use “yum” instead. On BSD, use the “ports” system, which you can also get working for macOS.

If no pre-compiled package exists for a program, then you’ll have to download the source code and compile it. There’s about an 80% chance this will work easy, following the instructions. There is a 20% chance you’ll experience “dependency hell”, for example, needing to install two mutually incompatible versions of Python.

Bash is a scripting language

Don’t forget that shells are really scripting languages. The bit that executes a single command is just a degenerate use of the scripting language. For example, you can do a traditional for loop like:

$ for i in $(seq 1 9); do echo $i; done

In this way, ‘bash’ is no different than any other scripting language, like Perl, Python, NodeJS, PHP CLI, etc. That’s why a lot of stuff on the system actually exists as short ‘bash’ programs, aka. shell scripts.

Few want to write bash scripts, but you are expected to be able to read them, either to tweek existing scripts on the system, or to read StackExchange help.

File system commands

The macOS “Finder” or Windows “File Explorer” are just graphical shells that help you find files, open, and save them. The first commands you learn are for the same functionality on the command-line: pwd, cd, ls, touch, rm, rmdir, mkdir, chmod, chown, find, ln, mount.

The command “rm –rf /” removes everything starting from the root directory. This will also follow mounted server directories, deleting files on the server. I point this out to give an appreciation of the raw power you have over the system from the command-line, and how easy you can disrupt things.

Of particular interest is the “mount” command. Desktop versions of Linux typically mount USB flash drives automatically, but on servers, you need to do it manually, e.g.:

$ mkdir ~/foobar
$ mount /dev/sdb ~/foobar

You’ll also use the ‘mount’ command to connect to file servers, using the “cifs” package if they are Windows file servers:

# apt-get install cifs-utils
# mkdir /mnt/vids
# mount -t cifs -o username=robert,password=foobar123  //192.168.1.11/videos /mnt/vids

Linux system commands

The next commands you’ll learn are about syadmin the Linux system: ps, top, who, history, last, df, du, kill, killall, lsof, lsmod, uname, id, shutdown, and so on.

The first thing hackers do when hacking into a system is run “uname” (to figure out what version of the OS is running) and “id” (to figure out which account they’ve acquired, like “root” or some other user).

The Linux system command I use most is “dmesg” (or ‘tail –f /var/log/dmesg’) which shows you the raw system messages. For example, when I plug in USB drives to a server, I look in ‘dmesg’ to find out which device was added so that I can mount it. I don’t know if this is the best way, it’s just the way I do it (servers don’t automount USB drives like desktops do).

Networking commands

The permanent state of the network (what gets configured on the next bootup) is configured in text files somewhere. But there are a wealth of commands you’ll use to view the current state of networking, make temporary changes, and diagnose problems.

The ‘ifconfig’ command has long been used to view the current TCP/IP configuration and make temporary changes. Learning how TCP/IP works means playing a lot with ‘ifconfig’. Use “ifconfig –a” for even more verbose information.

Use the “route” command to see if you are sending packets to the right router.

Use ‘arp’ command to make sure you can reach the local router.

Use ‘traceroute’ to make sure packets are following the correct route to their destination. You should learn the nifty trick it’s based on (TTLs). You should also play with the TCP, UDP, and ICMP options.

Use ‘ping’ to see if you can reach the target across the Internet. Usefully measures the latency in milliseconds, and congestion (via packet loss). For example, ping NetFlix throughout the day, and notice how the ping latency increases substantially during “prime time” viewing hours.

Use ‘dig’ to make sure DNS resolution is working right. (Some use ‘nslookup’ instead). Dig is useful because it’s the raw universal DNS tool – every time they add some new standard feature to DNS, they add that feature into ‘dig’ as well.

The ‘netstat –tualn’ command views the current TCP/IP connections and which ports are listening. I forget what the various options “tualn” mean, only it’s the output I always want to see, rather than the raw “netstat” command by itself.

You’ll want to use ‘ethtool –k’ to turn off checksum and segmentation offloading. These are features that break packet-captures sometimes.

There is this new fangled ‘ip’ system for Linux networking, replacing many of the above commands, but as an old timer, I haven’t looked into that.

Some other tools for diagnosing local network issues are ‘tcpdump’, ‘nmap’, and ‘netcat’. These are described in more detail below.

ssh

In general, you’ll remotely log into a system in order to use the command-line. We use ‘ssh’ for that. It uses a protocol similar to SSL in order to encrypt the connection. There are two ways to use ‘ssh’ to login, with a password or with a client-side certificate.

When using SSH with a password, you type “ssh [email protected]”. The remote system will then prompt you for a password for that account.

When using client-side certificates, use “ssh-keygen” to generate a key, then either copy the public-key of the client to the server manually, or use “ssh-copy-id” to copy it using the password method above.

How this works is basic application of public-key cryptography. When logging in with a password, you get a copy of the server’s public-key the first time you login, and if it ever changes, you get a nasty warning that somebody may be attempting a man in the middle attack.

$ ssh [email protected]
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

When using client-side certificates, the server trusts your public-key. This is similar to how client-side certificates work in SSL VPNs.

You can use SSH for things other than loging into a remote shell. You can script ‘ssh’ to run commands remotely on a system in a local shell script. You can use ‘scp’ (SSH copy) to transfer files to and from a remote system. You can do tricks with SSH to create tunnels, which is popular way to bypass the restrictive rules of your local firewall nazi.

openssl

This is your general cryptography toolkit, doing everything from simple encryption, to public-key certificate signing, to establishing SSL connections.

It is extraordinarily user hostile, with terrible inconsistency among options. You can only figure out how to do things by looking up examples on the net, such as on StackExchange. There are competing SSL libraries with their own command-line tools, like GnuTLS and Mozilla NSS that you might find easier to use.

The fundamental use of the ‘openssl’ tool is to create public-keys, “certificate requests”, and creating self-signed certificates. All the web-site certificates I’ve ever obtained has been using the openssl command-line tool to create CSRs.

You should practice using the ‘openssl’ tool to encrypt files, sign files, and to check signatures.

You can use openssl just like PGP for encrypted emails/messages, but following the “S/MIME” standard rather than PGP standard. You might consider learning the ‘pgp’ command-line tools, or the open-source ‘gpg’ or ‘gpg2’ tools as well.

You should learn how to use the “openssl s_client” feature to establish SSL connections, as well as the “openssl s_server” feature to create an SSL proxy for a server that doesn’t otherwise support SSL.

Learning all the ways of using the ‘openssl’ tool to do useful things will go a long way in teaching somebody about crypto and cybersecurity. I can imagine an entire class consisting of nothing but learning ‘openssl’.

netcat (nc, socat, cyptocat, ncat)

A lot of Internet protocols are based on text. That means you can create a raw TCP connection to the service and interact with them using your keyboard. The classic tool for doing this is known as “netcat”, abbreviated “nc”. For example, connect to Google’s web server at port and type the HTTP HEAD command followed by a blank line (hit [return] twice):

$ nc www.google.com 80
HEAD / HTTP/1.0

HTTP/1.0 200 OK
Date: Tue, 17 Jan 2017 01:53:28 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP=”This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info.”
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: NID=95=o7GT1uJCWTPhaPAefs4CcqF7h7Yd7HEqPdAJncZfWfDSnNfliWuSj3XfS5GJXGt67-QJ9nc8xFsydZKufBHLj-K242C3_Vak9Uz1TmtZwT-1zVVBhP8limZI55uXHuPrejAxyTxSCgR6MQ; expires=Wed, 19-Jul-2017 01:53:28 GMT; path=/; domain=.google.com; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding

Another classic example is to connect to port 25 on a mail server to send email, spoofing the “MAIL FROM” address.

There are several versions of ‘netcat’ that work over SSL as well. My favorite is ‘ncat’, which comes with ‘nmap’, as it’s actively maintained. In theory, “openssl s_client” should also work this way.

nmap

At some point, you’ll need to port scan. The standard program for this is ‘nmap’, and it’s the best. The classic way of using it is something like:

# nmap –A scanme.nmap.org

The ‘-A’ option means to enable all the interesting features like OS detection, version detection, and basic scripts on the most common ports that a server might have open. It takes awhile to run. The “scanme.nmap.org” is a good site to practice on.

Nmap is more than just a port scanner. It has a rich scripting system for probing more deeply into a system than just a port, and to gather more information useful for attacks. The scripting system essentially contains some attacks, such as password guessing.

Scanning the Internet, finding services identified by ‘nmap’ scripts, and interacting with them with tools like ‘ncat’ will teach you a lot about how the Internet works.

BTW, if ‘nmap’ is too slow, using ‘masscan’ instead. It’s a lot faster, though has much more limited functionality.

Packet sniffing with tcpdump and tshark

All Internet traffic consists of packets going between IP addresses. You can capture those packets and view them using “packet sniffers”. The most important packet-sniffer is “Wireshark”, a GUI. For the command-line, there is ‘tcpdump’ and ‘tshark’.

You can run tcpdump on the command-line to watch packets go in/out of the local computer. This performs a quick “decode” of packets as they are captured. It’ll reverse-lookup IP addresses into DNS names, which means its buffers can overflow, dropping new packets while it’s waiting for DNS name responses for previous packets (which can be disabled with -n):

# tcpdump –p –i eth0

A common task is to create a round-robin set of files, saving the last 100 files of 1-gig each. Older files are overwritten. Thus, when an attack happens, you can stop capture, and go backward in times and view the contents of the network traffic using something like Wireshark:

# tcpdump –p -i eth0 -s65535 –C 1000 –W 100 –w cap

Instead of capturing everything, you’ll often set “BPF” filters to narrow down to traffic from a specific target, or a specific port.

The above examples use the –p option to capture traffic destined to the local computer. Sometimes you may want to look at all traffic going to other machines on the local network. You’ll need to figure out how to tap into wires, or setup “monitor” ports on switches for this to work.

A more advanced command-line program is ‘tshark’. It can apply much more complex filters. It can also be used to extract the values of specific fields and dump them to a text files.

Base64/hexdump/xxd/od

These are some rather trivial commands, but you should know them.

The ‘base64’ command encodes binary data in text. The text can then be passed around, such as in email messages. Base64 encoding is often automatic in the output from programs like openssl and PGP.

In many cases, you’ll need to view a hex dump of some binary data. There are many programs to do this, such as hexdump, xxd, od, and more.

grep

Grep searches for a pattern within a file. More important, it searches for a regular expression (regex) in a file. The fu of Unix is that a lot of stuff is stored in text files, and use grep for regex patterns in order to extra stuff stored in those files.

The power of this tool really depends on your mastery of regexes. You should master enough that you can understand StackExhange posts that explain almost what you want to do, and then tweek them to make them work.

Grep, by default, shows only the matching lines. In many cases, you only want the part that matches. To do that, use the –o option. (This is not available on all versions of grep).

You’ll probably want the better, “extended” regular expressions, so use the –E option.

You’ll often want “case-insensitive” options (matching both upper and lower case), so use the –i option.

For example, to extract all MAC address from a text file, you might do something like the following. This extracts all strings that are twelve hex digits.

$ grep –Eio ‘[0-9A-F]{12}’ foo.txt

Text processing

Grep is just the first of the various “text processing filters”. Other useful ones include ‘sed’, ‘cut’, ‘sort’, and ‘uniq’.

You’ll be an expert as piping output of one to the input of the next. You’ll use “sort | uniq” as god (Dennis Ritchie) intended and not the heresy of “sort –u”.

You might want to master ‘awk’. It’s a new programming language, but once you master it, it’ll be easier than other mechanisms.

You’ll end up using ‘wc’ (word-count) a lot. All it does is count the number of lines, words, characters in a file, but you’ll find yourself wanting to do this a lot.

csvkit and jq

You get data in CSV format and JSON format a lot. The tools ‘csvkit’ and ‘jq’ respectively help you deal with those tools, to convert these files into other formats, sticking the data in databases, and so forth.

It’ll be easier using these tools that understand these text formats to extract data than trying to write ‘awk’ command or ‘grep’ regexes.

strings

Most files are binary with a few readable ASCII strings. You use the program ‘strings’ to extract those strings.

This one simple trick sounds stupid, but it’s more powerful than you’d think. For example, I knew that a program probably contained a hard-coded password. I then blindly grabbed all the strings in the program’s binary file and sent them to a password cracker to see if they could decrypt something. And indeed, one of the 100,000 strings in the file worked, thus finding the hard-coded password.

tail -f

So ‘tail’ is just a standard Linux tool for looking at the end of files. If you want to keep checking the end of a live file that’s constantly growing, then use “tail –f”. It’ll sit there waiting for something new to be added to the end of the file, then print it out. I do this a lot, so I thought it’d be worth mentioning.

tar –xvfz, gzip, xz, 7z

In prehistorical times (like the 1980s), Unix was backed up to tape drives. The tar command could be used to combine a bunch of files into a single “archive” to be sent to the tape drive, hence “tape archive” or “tar”.

These days, a lot of stuff you download will be in tar format (ending in .tar). You’ll need to learn how to extract it:

$ tar –xvf something.tar

Nobody knows what the “xvf” options mean anymore, but these letters most be specified in that order. I’m joking here, but only a little: somebody did a survey once and found that virtually nobody know how to use ‘tar’ other than the canned formulas such as this.

Along with combining files into an archive you also need to compress them. In prehistoric Unix, the “compress” command would be used, which would replace a file with a compressed version ending in ‘.z’. This would found to be encumbered with patents, so everyone switched to ‘gzip’ instead, which replaces a file with a new one ending with ‘.gz’.

$ ls foo.txt*
foo.txt
$ gzip foo.txt
$ ls foo.txt*
foo.txt.gz

Combined with tar, you get files with either the “.tar.gz” extension, or simply “.tgz”. You can untar and uncompress at the same time:

$ tar –xvfz something .tar.gz

Gzip is always good enough, but nerds gonna nerd and want to compress with slightly better compression programs. They’ll have extensions like “.bz2”, “.7z”, “.xz”, and so on. There are a ton of them. Some of them are supported directly by the ‘tar’ program:

$ tar –xvfj something.tar.bz2

Then there is the “zip/unzip” program, which supports Windows .zip file format. To create compressed archives these days, I don’t bother with tar, but just use the ZIP format. For example, this will recursively descend a directory, adding all files to a ZIP file that can easily be extracted under Windows:

$ zip –r test.zip ./test/

dd

I should include this under the system tools at the top, but it’s interesting for a number of purposes. The usage is simply to copy one file to another, the in-file to the out-file.

$ dd if=foo.txt of=foo2.txt

But that’s not interesting. What interesting is using it to write to “devices”. The disk drives in your system also exist as raw devices under the /dev directory.

For example, if you want to create a boot USB drive for your Raspberry Pi:

# dd if=rpi-ubuntu.img of=/dev/sdb

Or, you might want to hard erase an entire hard drive by overwriting random data:

# dd if=/dev/urandom of=/dev/sdc

Or, you might want to image a drive on the system, for later forensics, without stumbling on things like open files.

# dd if=/dev/sda of=/media/Lexar/infected.img

The ‘dd’ program has some additional options, like block size and so forth, that you’ll want to pay attention to.

screen and tmux

You log in remotely and start some long running tool. Unfortunately, if you log out, all the processes you started will be killed. If you want it to keep running, then you need a tool to do this.

I use ‘screen’. Before I start a long running port scan, I run the “screen” command. Then, I type [ctrl-a][ctrl-d] to disconnect from that screen, leaving it running in the background.

Then later, I type “screen –r” to reconnect to it. If there are more than one screen sessions, using ‘-r’ by itself will list them all. Use “-r pid” to reattach to the proper one. If you can’t, then use “-D pid” or “-D –RR pid” to forced the other session to detached from whoever is using it.

Tmux is an alternative to screen that many use. It’s cool for also having lots of terminal screens open at once.

curl and wget

Sometimes you want to download files from websites without opening a browser. The ‘curl’ and ‘wget’ programs do that easily. Wget is the traditional way of doing this, but curl is a bit more flexible. I use curl for everything these days, except mirroring a website, in which case I just do “wget –m website”.

The thing that makes ‘curl’ so powerful is that it’s really designed as a tool for poking and prodding all the various features of HTTP. That it’s also useful for downloading files is a happy coincidence. When playing with a target website, curl will allow you do lots of complex things, which you can then script via bash. For example, hackers often write their cross-site scripting/forgeries in bash scripts using curl.

node/php/python/perl/ruby/lua

As mentioned above, bash is its own programming language. But it’s weird, and annoying. So sometimes you want a real programming language. Here are some useful ones.

Yes, PHP is a language that runs in a web server for creating web pages. But if you know the language well, it’s also a fine command-line language for doing stuff.

Yes, JavaScript is a language that runs in the web browser. But if you know it well, it’s also a great language for doing stuff, especially with the “nodejs” version.

Then there are other good command line languages, like the Python, Ruby, Lua, and the venerable Perl.

What makes all these great is the large library support. Somebody has already written a library that nearly does what you want that can be made to work with a little bit of extra code of your own.

My general impression is that Python and NodeJS have the largest libraries likely to have what you want, but you should pick whichever language you like best, whichever makes you most productive. For me, that’s NodeJS, because of the great Visual Code IDE/debugger.

iptables, iptables-save

I shouldn’t include this in the list. Iptables isn’t a command-line tool as such. The tool is the built-in firewalling/NAT features within the Linux kernel. Iptables is just the command to configure it.

Firewalling is an important part of cybersecurity. Everyone should have some experience playing with a Linux system doing basic firewalling tasks: basic rules, NATting, and transparent proxying for mitm attacks.

Use ‘iptables-save’ in order to persistently save your changes.

MySQL

Similar to ‘iptables’, ‘mysql’ isn’t a tool in its own right, but a way of accessing a database maintained by another process on the system.

Filters acting on text files only goes so far. Sometimes you need to dump it into a database, and make queries on that database.

There is also the offensive skill needed to learn how targets store things in a database, and how attackers get the data.

Hackers often publish raw SQL data they’ve stolen in their hacks (like the Ashley-Madisan dump). Being able to stick those dumps into your own database is quite useful. Hint: disable transaction logging while importing mass data.

If you don’t like SQL, you might consider NoSQL tools like Elasticsearch, MongoDB, and Redis that can similarly be useful for arranging and searching data. You’ll probably have to learn some JSON tools for formatting the data.

Reverse engineering tools

A cybersecurity specialty is “reverse engineering”. Some want to reverse engineer the target software being hacked, to understand vulnerabilities. This is needed for commercial software and device firmware where the source code is hidden. Others use these tools to analyze viruses/malware.

The ‘file’ command uses heuristics to discover the type of a file.

There’s a whole skillset for analyzing PDF and Microsoft Office documents. I play with pdf-parser. There’s a long list at this website:
https://zeltser.com/analyzing-malicious-documents/

There’s a whole skillset for analyzing executables. Binwalk is especially useful for analyzing firmware images.

Qemu is useful is a useful virtual-machine. It can emulate full systems, such as an IoT device based on the MIPS processor. Like some other tools mentioned here, it’s more a full subsystem than a simple command-line tool.

On a live system, you can use ‘strace’ to view what system calls a process is making. Use ‘lsof’ to view which files and network connections a process is making.

Password crackers

A common cybersecurity specialty is “password cracking”. There’s two kinds: online and offline password crackers.

Typical online password crackers are ‘hydra’ and ‘medusa’. They can take files containing common passwords and attempt to log on to various protocols remotely, like HTTP, SMB, FTP, Telnet, and so on. I used ‘hydra’ recently in order to find the default/backdoor passwords to many IoT devices I’ve bought recently in my test lab.

Online password crackers must open TCP connections to the target, and try to logon. This limits their speed. They also may be stymied by systems that lock accounts, or introduce delays, after too many bad password attempts.

Typical offline password crackers are ‘hashcat’ and ‘jtr’ (John the Ripper). They work off of stolen encrypted passwords. They can attempt billions of passwords-per-second, because there’s no network interaction, nothing slowing them down.

Understanding offline password crackers means getting an appreciation for the exponential difficulty of the problem. A sufficiently long and complex encrypted password is uncrackable. Instead of brute-force attempts at all possible combinations, we must use tricks, like mutating the top million most common passwords.

I use hashcat because of the great GPU support, but John is also a great program.

WiFi hacking

A common specialty in cybersecurity is WiFi hacking. The difficulty in WiFi hacking is getting the right WiFi hardware that supports the features (monitor mode, packet injection), then the right drivers installed in your operating system. That’s why I use Kali rather than some generic Linux distribution, because it’s got the right drivers installed.

The ‘aircrack-ng’ suite is the best for doing basic hacking, such as packet injection. When the parents are letting the iPad babysit their kid with a loud movie at the otherwise quite coffeeshop, use ‘aircrack-ng’ to deauth the kid.

The ‘reaver’ tool is useful for hacking into sites that leave WPS wide open and misconfigured.

Remote exploitation

A common specialty in cybersecurity is pentesting.

Nmap, curl, and netcat (described above) above are useful tools for this.

Some useful DNS tools are ‘dig’ (described above), dnsrecon/dnsenum/fierce that try to enumerate and guess as many names as possible within a domain. These tools all have unique features, but also have a lot of overlap.

Nikto is a basic tool for probing for common vulnerabilities, out-of-date software, and so on. It’s not really a vulnerability scanner like Nessus used by defenders, but more of a tool for attack.

SQLmap is a popular tool for probing for SQL injection weaknesses.

Then there is ‘msfconsole’. It has some attack features. This is humor – it has all the attack features. Metasploit is the most popular tool for running remote attacks against targets, exploiting vulnerabilities.

Text editor

Finally, there is the decision of text editor. I use ‘vi’ variants. Others like ‘nano’ and variants. There’s no wrong answer as to which editor to use, unless that answer is ‘emacs’.

Conclusion

Obviously, not every cybersecurity professional will be familiar with every tool in this list. If you don’t do reverse-engineering, then you won’t use reverse-engineering tools.

On the other hand, regardless of your specialty, you need to know basic crypto concepts, so you should know something like the ‘openssl’ tool. You need to know basic networking, so things like ‘nmap’ and ‘tcpdump’. You need to be comfortable processing large dumps of data, manipulating it with any tool available. You shouldn’t be frightened by a little sysadmin work.

The above list is therefore a useful starting point for cybersecurity professionals. Of course, those new to the industry won’t have much familiarity with them. But it’s fair to say that I’ve used everything listed above at least once in the last year, and the year before that, and the year before that. I spend a lot of time on StackExchange and Google searching the exact options I need, so I’m not an expert, but I am familiar with the basic use of all these things.

vKontakte Responds to US Notorious Pirate Market Allegations

Post Syndicated from Andy original https://torrentfreak.com/vkontakte-responds-us-notorious-pirate-market-allegations-161223/

vkEvery year the United States Trade Representative calls on foreign countries to take action against a broad range of websites which are allegedly involved in copyright infringement. Perhaps understandably, The Pirate Bay is one of the most obvious platforms to appear in the USTR’s review but there are others that don’t fit the same profile.

One of those is Russian social networking giant vKontakte (VK). Like all platforms with millions of users contributing content on a daily basis, VK has to deal with allegations of copyright infringement. Traditionally, the most vocal critics have hailed from the music industry but in recent times VK has made its peace with several distributors via licensing deals.

But despite these efforts, VK is apparently still falling short of the standards set by the United States government. In its yearly “Out-of-Cycle Review of Notorious Markets” report published this week, the United States Trade Representative laid out the case against VK.

“Nominated again this year, VK is one of the most popular sites in the world and continues to operate as an extremely popular social networking site in Russia and neighboring countries. VK reportedly facilitates the distribution of copyright-infringing files,” the USTR wrote.

Noting the commercial and cultural value of social networking platforms that go about their business without infringing rightsholders’ copyrights, the USTR said that VK’s recent steps to tackle piracy were “encouraging”.

VK has reached deals with major labels, taken steps taken to limit third-party applications dedicated to downloading infringing content, and experimented with fingerprinting technology, the USTR said. Even the RIAA said it was happy with VK, having taken the social network off its list in October.

Nevertheless, the mere fact that VK is on the USTR list again this cycle indicates that rightsholders outside the music industry are still complaining about the site. In its report, the USTR appeared to confirm it.

“Despite these positive signals, VK reportedly continues to be a hub of infringing activity and continues to be listed pending the institutionalization of appropriate measures to promote respect on its platform for IPR of all right holders, not just those with whom it has contracts, that are comparable to those measures used by other social media sites,” the USTR said.

Given that during the summer vKontakte felt it was already time for it to be removed from the United States’ blacklist, TorrentFreak caught up with the social network to gauge its reaction to this week’s apparent snub from the USTR.

“VKontakte continues its proactive work on licensing all audio and video content available on the social network,” a VK spokesperson said.

“In 2016, the Company signed an agreement with the world’s leading international music rights holders, including Universal Music, Sony Music, Warner Music, The Orchard, Merlin Network and Believe Digital, among others. The music section of VK’s mobile app for iOS has been available to users since August 2016, after a two-year absence.”

Addressing the USTR’s claims over other contentious content still present on the platform, VK said that more progress would be made during 2017.

“Next year, the Company will continue its focus on signing up the remaining rights holders not covered by the previous agreements,” the company said.

The measured tone from VK shows an unexpected level of patience, particularly after being mentioned in the same breath as The Pirate Bay yet again and despite significant efforts to appease rightsholders. Will it have done enough by this time next year? Only time will tell.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

US Government Targets Pirate Bay and Other ‘Piracy Havens’

Post Syndicated from Ernesto original https://torrentfreak.com/us-government-targets-pirate-bay-and-other-piracy-havens-161221/

ustrbIn its yearly “Out-of-Cycle Review of Notorious Markets”, the United States Trade Representative (USTR) has listed more than a dozen websites said to be involved in piracy and counterfeiting.

The overview is largely based on input from industry groups including the RIAA and MPAA, who submitted their recommendations a few weeks ago.

While the USTR admits that the list is not meant to reflect legal violations, the goal of the review is to motivate owners and foreign Governments to take appropriate action and reduce piracy.

“The United States encourages all responsible authorities to intensify efforts to combat piracy and counterfeiting, and to use the information contained in the Notorious Markets List to pursue legal actions where appropriate,” the USTR announced.

As in previous years, The Pirate Bay remains one of the primary offenders.

According to the USTR, the site continues to facilitate downloading of copyright-infringing material. The Government further highlights the site’s resilience and “symbolic importance” as one of the longest-running pirate sites

“Despite enforcement actions around the world and drawn-out legal battles against its operators, The Pirate Bay is of symbolic importance as one of the longest-running and most vocal torrent sites for admittedly illegal downloads of movies, television, music, and other copyrighted content,” the report reads.

ustrtpb

Other prominent torrent sites mentioned in the review are ExtraTorrent, Rutorrent, RARBG, and 1337x.to.

For the first time, USTR has also included a stream ripping site; YouTube-MP3.org. While this phenomenon has been around for a decade, the report includes a special “issue focus” mentioning it as an emerging threat.

“Stream ripping is an emerging trend in digital copyright infringement that is increasingly causing substantial economic harm to music creators and undermining legitimate services,” the USTR writes.

The mention follows a report from earlier this year, which also highlighted concerns about stream ripping. Soon after, several major music labels filed a lawsuit against YouTube-MP3 in a U.S. federal court.

A few newcomers aside, the review is mostly made up of familiar names, including 4shared, Putlocker, Nowvideo, Rapidgator and Uploaded, as well as several non-English language piracy portals and counterfeiting platforms.

In addition to individual sites and services, the USTR notes that some hosting services have also become piracy havens. The report specifically calls out the Swiss company Private layer for hosting the-watch-series.to, projectfree-tv, using a legal loophole.

The U.S has urged Switzerland to implement new legislation to make it easier to take action against pirate sites, but this hasn’t happened thus far.

The full list of the notorious online pirate sites and services that are highlighted in the report (pdf) are included below. The complete overview also contains various e-commerce and counterfeiting sites, including Alibaba’s Taobao.com.

– 4shared.com
– Beevideo.tv
– Bookfi and Libgen
– ExtraTorrent
– Gongchang.com
– Movshare group (allegedly operating Nowvideo.sx, Watchseriesfree.to, Videoweed.es, Novamov.com and others)
– MP3va.com
– Muaban.net
– Myegy.to
– Nanjing Imperiosus (domainerschoice.com)
– Pobieramy24.pl, Darkwarez.pl, Catshare.net and Fileshark.pl
– Private Layer hosted sites (including the-watch-series.to and projectfree-tv.to)
– Putlocker
– RARBG.to
– Rutracker.org and Rapidgator.org
– Taobao.com
– The Pirate Bay
– Uploaded.net
– Vibbo.com
– VK.com
– Youtube-MP3.org

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

RIAA: CloudFlare Shields Pirates and Frustrates Blocking Efforts

Post Syndicated from Ernesto original https://torrentfreak.com/riaa-cloudflare-shields-pirates-and-frustrates-blocking-efforts-161013/

cassetteFollowing in the footsteps of the MPAA, the RIAA has submitted its overview of “notorious markets” to the Office of the US Trade Representative (USTR).

These annual submissions help to guide the U.S. Government’s position toward foreign countries when it comes to copyright enforcement.

This year the RIAA’s report includes 47 alleged pirate sites in various categories. As in previous years, popular torrent sites such as The Pirate Bay and ExtraTorrent are prominently mentioned.

There’s also a strong focus on so-called “stream-ripping” sites. While these have been around for roughly a decade, the music industry sees them as a growing threat, which is also evidenced by the recent lawsuit against YouTube-MP3.

According to the music group, it is getting harder to target these sites, as they are increasingly taking precautions.

“It is exceedingly difficult to track, enforce against, and accurately associate various notorious websites,” RIAA writes, listing domain hopping, reverse proxy services and anonymous domain name registrations as the main factors.

Obstructing factors

riaaharder

The Pirate Bay is one of the prime examples of a site that has switched domain names in the past. Due to various enforcement efforts it burnt through more than a dozen domains with ease.

In addition, TPB and other pirate sites are increasingly using the popular CDN CloudFlare. Besides saving costs, it also acts as a reverse proxy and shields the true hosting location from public view.

This hasn’t gone unnoticed by the RIAA which repeatedly mentions CloudFlare in its report.

“BitTorrent sites, like many other pirate sites, are increasing (sic) turning to Cloudflare because routing their site through Cloudflare obfuscates the IP address of the actual hosting provider, masking the location of the site,” the RIAA writes.

Throughout the report the RIAA attempts to point out the hosting location of all pirate sites, but it often has to put down “obfuscated by Cloudflare” instead.

Obstructing factors

obfuscloud

Aside from making it harder to identify the hosting location, CloudFlare can also make it harder for ISPs to block websites.

Traditionally, some ISPs have blocked pirate sites by IP-address, but this is no longer an option since CloudFlare customers share IPs with other sites, which can lead to overblocking.

“The use of Cloudflare’s services can also act to frustrate site-blocking orders because multiple non-infringing sites may share a Cloudflare IP address with the infringing site,” the RIAA notes in its report.

While CloudFlare itself isn’t tagged as a notorious site, the fact that both the RIAA and MPAA are highlighting the service in their report is not without reason. The industry groups are likely to demand a more proactive anti-piracy policy from CloudFlare in the future.

Apart from all the doom and gloom, there is also a positive development. After being labeled as a notorious pirate site for years, the RIAA has taken social network VK.com off its list. This is the direct result of licensing agreements between the site and various major labels.

“Russia’s vKontakte has now reached licensing agreements with major record companies and has thus been removed from our list,” the RIAA writes.

Finally, it’s worth noting that MP3Skull is no longer on the list. As we suggested yesterday, the RIAA believes that the people behind the site switched their operation to Emp3world.ch. Curiously, this knowledge didn’t prevent them from seizing the domain name of a seemingly unrelated site.

The full list of RIAA’s “notorious” pirate sites can be found below, and the full report is available here (pdf).

Stream-Ripping Sites

– Youtube-mp3.org
– Mp3juices.cc
– Convert2mp3.net
– Aiomp3.com
– Clipconverter.cc
– Savefrom.net
– Youtube2mp3.cc
– Onlinevideoconverter.com

Search-and-Download Sites

– Emp3world.ch
– Audiocastle.biz
– Viperial2.com
– Im1music.info
– Albumkings.com
– Newalbumreleases.net

BitTorrent Indexing and Tracker Sites

– Thepiratebay.org
– Extratorrent.cc
– Bitsnoop.com
– Isohunt.to
– Torrentdownloads.me
– LimeTorrents.cc
– Rarbg.to
– 1337x.to

Cyberlockers

– 4shared.com
– Uploaded.net
– Zippyshare.com
– Rapidgator.net
– Dopefile.pk
– Chomikuj.pl
– Turbobit.net
– Hitfile.net
– 1fichier.com
– Bigfile.to
– Share-online.biz
– Ulozto.cz

Unlicensed Pay-for-Download Sites

– Mp3va.com
– Soundsbox.com
– Iomoio.com
– Soundike.com
– Payplay.fm
– Mp3million.com
– Megaboon.com
– Melodishop.com
– Melodysale.com
– Mp3caprice.com
– Ivave.com
– Mediasack.com
– Goldenmp3.ru

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

MPAA Reports Pirate Sites and Hosting Providers to U.S. Government

Post Syndicated from Ernesto original https://torrentfreak.com/mpaa-reports-pirate-sites-and-hosting-providers-to-u-s-government-161010/

mpaa-logoResponding to a request from the Office of the US Trade Representative (USTR), the MPAA has sent in its annual list of notorious markets.

In its latest submission the Hollywood group targets a wide variety of “rogue” sites and services which they claim are promoting the illegal distribution of movies and TV-shows, with declining incomes and lost jobs in the movie industry as a result.

“The criminals who profit from the most notorious markets throughout the world threaten the very heart of our industry and in so doing they threaten the livelihoods of the people who give it life,” the MPAA writes.

What’s new this year is that the MPAA calls out several hosting providers. These companies refuse to take pirate sites offline following complaints, even when the MPAA views them as blatantly violating the law.

“Hosting companies provide the essential infrastructure required to operate a website,” MPAA writes. “Given the central role of hosting providers in the online ecosystem, it is very concerning that many refuse to take action upon being notified.”

The Hollywood group specifically mentions Private Layer, Altushost and Netbrella, which are linked to various countries including the Netherlands, Panama, Sweden and Switzerland.

CDN provider CloudFlare is also named. As a US-based company it can’t be included in the list. However, MPAA explains that it is often used as an anonymization tool by sites and services that are mentioned in the report.

“An example of a CDN frequently exploited by notorious markets to avoid detection and enforcement is Cloudflare. CloudFlare is a CDN that also provides reverse proxy functionality. Reverse proxy functionality hides the real IP address of a web server.”

Stressing the importance of third-party services, the MPAA notes that domain name registrars can also be seen as possible “notorious markets.” As an example, the report mentions the Indian Public Domain Registry (PDR) which has repeatedly refused to take action against pirate sites.

At the heart of the MPAA’s report are as always the pirate sites themselves. This year they list 23 sites in separate categories, each with a suspected location, as defined by the movie industry group.

Torrent Sites

According to the MPAA, BitTorrent remains the most popular source of P2P piracy, despite the shutdowns of large sites such as KAT, Torrentz and YTS.

The Pirate Bay has traditionally been one of the main targets. Based on data from Alexa and SimilarWeb, the MPAA says that TPB has about 47 million unique visitors per month.

The MPAA writes that the site was hit by various enforcement actions in recent years. They also mistakenly suggest that the site is no longer the number one pirate site, but add that it gained traction after KAT and Torrentz were taken down.

“While it has never returned to its number one position, it has had a significant comeback after kat.cr and torrentz.eu went offline in 2016,” the MPAA writes.

ExtraTorrent is another prime target. The site offers millions of torrents and is affiliated with the Trust.Zone VPN, which they advertise on their site.

“Extratorrent.cc claims astonishing piracy statistics: offering almost three million free files with sharing optimized through over 64 million seeders and more than 39 million leechers.

“The homepage currently displays a message warning users to use a VPN when downloading torrents. Extratorrent.cc is affiliated with Trust.Zone,” MPAA adds.

The full list of reported torrent sites is as follows:

-1337x.to (Switzerland)
-Extratorrent.cc (Latvia)
-Rarbg.to (Bosnia and Herzegovina)
-Rutracker.org (Russia)
-ThePirateBay.org (Unknown)

Direct Download and Streaming Cyberlockers

The second category of pirate sites reported by the MPAA are cyberlockers. The movie industry group points out that these sites generate millions of dollars in revenue, citing a report from Netnames.

The “Movshare Group,” which allegedly operates Nowvideo.sx, Movshare.net, Novamov.com, Videoweed.es, Nowdownload.ch, Divxstage.to and several other pirate sites is a particularly large threat, they say.

As in previous submissions VKontakte, Russia’s equivalent of Facebook, is also listed as a notorious market.

-Allmyvideos.net (Netherlands)
-Nowvideo.sx and the “Movshare Group” (several locations)
-Openload.co (Netherlands)
-Rapidgator.net (Russia)
-Uploaded.net (Netherlands/Switzerland)
-VK.com (Russia)

Linking Websites

Finally, there are various linking websites, many of which focus on a foreign audience. These sites don’t host the infringing material, but only link to it. The full list of linking sites is as follows.

123movies.to (Unknown)
-Filmesonlinegratis.net (Brazil/Portugal)
-Kinogo.club (Netherlands)
-Movie4k.to (Russia)
-Newmovie-hd.com (Thailand)
-Pelis24.com (Spain/Mexico/Argentina/Venezuela/Peru/Chile)
-Primewire.ag (Switzerland)
-Projectfreetv.at (Romania)
-Putlocker.is (Switzerland/Vietnam)
-Repelis.tv (Mexico/Argentina/Spain/Peru/Venezuela)
-Watchseries.ac (France)

In its closing comments the Hollywood industry group calls on USTR and the U.S. government at large to help combat these threats, either directly or by encouraging foreign nations to take action.

“We strongly support efforts by the U.S. government to work with trading partners to protect and enforce intellectual property rights and, in so doing, protect U.S. jobs,” the MPAA concludes.

MPAA’s full submission is available here.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Russian Anti-Piracy Chief Arrested in ‘Fraud’ Investigation

Post Syndicated from Ernesto original https://torrentfreak.com/russian-anti-piracy-chief-arrested-fraud-investigation-161007/

copyright-brandedOne of Russia’s most prominent anti-piracy fighters has found himself at the center of a criminal fraud investigation.

Maxim Ryabyko, Director General of Association for the Protection of Copyright on the Internet (AZAPO), was reportedly arrested in Moscow earlier this week, together with a friend.

According to local news reports, the pair were carrying out a money handoff after offering to help drop a criminal investigation against a local Internet entrepreneur in return for 50 million rubles ($800,000).

Russia’s Ministry of Internal Affairs confirmed the arrests but didn’t mention any names. Interior Ministry spokeswoman Irina Volk said that two men were arrested in a cafe in the center of Moscow on Thursday, suspected of fraud. Both have since been released on bail.

A source in the Government, however, confirmed to the TASS news agency that AZAPO’s General Director was one of the arrested men.

While details are scarce, there are reports suggesting that the case is related to the prominent pirate book library Lib.rus.ec. Last week AZAPO announced that the site was being investigated in both Russia and Ecuador, while the operators are on the run.

AZAPO’s announcement specifically highlighted the possible involvement of the Russian service iMobilco, which reportedly charged for access to infringing books obtained through Lib.rus.ec, without compensating copyright holders.

In a separate article, AZAPO reprinted a news report from Gazeta which reported that Russian authorities carried out a search at the Moscow home of iMobilco’s founder, Nikolai Belousov.

Belousov is reportedly the person who was being offered a “deal” in exchange for 50 million rubles.

iMobilco’s founder told the news site Vedomosti that he is aware of Ryabyko’s arrest. He says that he was being extorted after AZAPO accused him of profiting from copyright infringements linked to Lib.rus.ec.

AZAPO’s General Director, meanwhile, denies all allegations including the arrest itself, which he says he first heard about in the media.

In recent months the anti-piracy group has been in the news several times, most notably for its efforts to block the prominent torrent site RuTracker, and limit copyright infringements on sites such as VKontakte.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

Russia Plans Social Media Piracy Crackdown

Post Syndicated from Andy original https://torrentfreak.com/russia-plans-social-media-piracy-crackdown-160810/

peopleDespite a reputation for not doing enough to thwart online piracy, Russian authorities have become unusually keen to make amends in recent years.

Site-blocking, for example, is now a common occurrence, with sites that infringe multiple times now being subjected to a permanent lifetime injunction, actioned by local ISPs.

But while users continue to flock to torrent sites and streaming portals, copyright holders and local authorities are concerned that social networking platforms are a potentially more serious threat.

In many cases, users are allowed to upload content at will, thereby creating huge libraries of infringing material, a serious headache for copyright holders.

To tackle this problem, authorities and entertainment industry groups are now in the process of drafting fresh legislation aimed at those social media platforms that allow users to upload content.

According to Izvestia, the Ministry of Culture and groups including the National Federation of the Music Industry (NFMI) and the Association of Producers of Cinema and Television (APKIT), believe that a change in the law will make it harder for social platforms to evade liability.

Under Article 1253.1 of the Civil Code, social media sites are considered “information brokers”, meaning that sites like vKontakte (Russia’s Facebook) can avoid being held liable for infringing content uploaded by their users.

Rightsholders would like that legislation to be removed or rewritten in a way that would provide them with more useful options to enforce their intellectual property rights.

Also under consideration are changes to the law that would further punish sites that have already been ordered to be blocked by the Moscow City Court. Currently, local ISPs currently put Internet blockades in place but rightsholders foresee a situation where the finances of infringing sites are put under pressure too.

On the table are proposals to ban those sites from carrying advertising. In the West, advertisers are working on voluntary schemes that aim to keep their funding away from ‘pirate’ sites but it appears that Russia is considering enshrining those principles into law.

Additionally, rightsholders are asking for sites that run on a subscription basis to be forbidden from accepting payments from their users. Again, voluntary agreements with companies such as Visa, MasterCard and PayPal are already in place in the United States and Europe, but legislation could compel Russian companies to comply.

Also continuing its path through the system is another bill designed to tackle the rise of so-called mirrors, sites that crop up after a site is blocked in order to facilitate access to the same content.

The draft bill, which also proposes an obligation to have search engines strip content from results and measures to tackle VPNs and proxies, has already been sent to the Ministry of Communications.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.

VKontakte CEO: Time to Remove Us From ‘Pirate’ Blacklists

Post Syndicated from Andy original https://torrentfreak.com/vkontakte-ceo-time-to-remove-us-from-pirate-blacklists-160724/

After years of being branded one of the world’s worst Internet piracy facilitators, last week social networking giant vKontakte took another important step towards making peace with rightsholders.

Parent company Mail.ru signed a licensing agreement with Universal Music and United Music Agency which will see music and video content appear legally on vKontakte, Classmates (Odnoklassniki) and My World, the three most-visited social networking sites in Russia.

With all copyright-related disputes now settled with Universal Music, the deal effectively transforms bitter conflict into cooperation, opening up opportunities for music sales development in a notoriously difficult region.

To find out more about the deal, this week TF caught up with VKontakte CEO Boris Dobrodeyev, who told us he’s optimistic for the future.

TF: Can you explain how the music licensing system with Universal Music will work?
 
BD: In accordance with the terms of the agreement, we cannot disclose the specific licensing provisions. However, we can say that the licensing agreements cover use of content on existing and planned new services on all of Mail.Ru Group’s social networks: VKontakte, Odnoklassniki and My World.

TF: What happens to the thousands/millions of ‘pirate’ tracks that are stored already on VKontakte? Do these effectively become legal or will Universal Music be supplying new content?
 
BD: The term “piracy” is not applicable to User Generated Content (UGC) services. Our position, which we have successfully defended in legal disputes, is that we do not distribute pirate content.

VKontakte’s content is user-generated, and so the rights holders’ requests were directed to them. From our side, we do everything that we can to protect the rights of the holders and remove content that violates their ownership rights.

Now that VKontake has signed the respective agreements with the major music companies, it is implementing substantial measures to identify the ownership of user content on the basis of the original files provided by the rights holders. VKontakte’s new services will be created using original content from the labels (including Warner, Sony and Universal).

Boris Dobrodeyev, VKontakte CEO

Boris Dobrodeyev

TF: Is VKontakte obliged to end all music piracy on its platform now, or just for the recoding labels it has struck a deal with?
 
BD: We would reiterate that the term “piracy” is not correct when talking about UGC services. Following significant efforts to license music content, the overwhelming majority of music by global artists on VK is completely legal.

With regard to music rights holders that have not yet signed an agreement with VK, at the very least they are able to use the existing procedures and technology in place to remove content and prevent it from being re-uploaded. It goes without saying that we also intend to sign corresponding agreements with these other rights holders in the near future.

We would add that many users and artists voluntarily upload their own music to VK in order to increase their popularity.

TF: Will fingerprinting technology or any other anti-piracy measures implemented?
 
BD: Yes, we intend to use a unique content identification system, which we developed in-house, and is similar to the technologies used in Audible Magic and Gracenote Content ID.

We will work together with the rights holders to continuously improve this technology in line with the development of the IT industry in general.

Moving forward

Of course, after years of copyright disputes vKontakte’s reputation in the United States has been somewhat sullied, largely due to rightsholders lobbying the United States Trade Representative to add the site to its Notorious Markets list. Dobrodeyev informs TF that it’s now time to move forward.

“We certainly hope that VKontakte will be removed from ‘piracy’ lists following the settlements and taking into account the enormous amount of work that the network has undertaken in this area,” he concludes.

Mail.ru and its subsidiaries now have licensing agreements in place with the three leading recording labels – Universal Music Group (UMG), Warner Music and Sony Music. Together they’ll hope to make inroads and indeed profit from a difficult and largely untapped Russian music market.

Source: TF, for the latest info on copyright, file-sharing, torrent sites and ANONYMOUS VPN services.