Tag Archives: Donald Trump

Don Jr.: I’ll bite

Post Syndicated from Robert Graham original http://blog.erratasec.com/2017/11/don-jr-ill-bite.html

So Don Jr. tweets the following, which is an excellent troll. So I thought I’d bite. The reason is I just got through debunk Democrat claims about NetNeutrality, so it seems like a good time to balance things out and debunk Trump nonsense.

The issue here is not which side is right. The issue here is whether you stand for truth, or whether you’ll seize any factoid that appears to support your side, regardless of the truthfulness of it. The ACLU obviously chose falsehoods, as I documented. In the following tweet, Don Jr. does the same.

It’s a preview of the hyperpartisan debates are you are likely to have across the dinner table tomorrow, which each side trying to outdo the other in the false-hoods they’ll claim.

What we see in this number is a steady trend of these statistics since the Great Recession, with no evidence in the graphs showing how Trump has influenced these numbers, one way or the other.

Stock markets at all time highs

This is true, but it’s obviously not due to Trump. The stock markers have been steadily rising since the Great Recession. Trump has done nothing substantive to change the market trajectory. Also, he hasn’t inspired the market to change it’s direction.
To be fair to Don Jr., we’ve all been crediting (or blaming) presidents for changes in the stock market despite the fact they have almost no influence over it. Presidents don’t run the economy, it’s an inappropriate conceit. The most influence they’ve had is in harming it.

Lowest jobless claims since 73

Again, let’s graph this:

As we can see, jobless claims have been on a smooth downward trajectory since the Great Recession. It’s difficult to see here how President Trump has influenced these numbers.

6 Trillion added to the economy

What he’s referring to is that assets have risen in value, like the stock market, homes, gold, and even Bitcoin.
But this is a well known fallacy known as Mercantilism, believing the “economy” is measured by the value of its assets. This was debunked by Adam Smith in his book “The Wealth of Nations“, where he showed instead the the “economy” is measured by how much it produces (GDP – Gross Domestic Product) and not assets.
GDP has grown at 3.0%, which is pretty good compared to the long term trend, and is better than Europe or Japan (though not as good as China). But Trump doesn’t deserve any credit for this — today’s rise in GDP is the result of stuff that happened years ago.
Assets have risen by $6 trillion, but that’s not a good thing. After all, when you sell your home for more money, the buyer has to pay more. So one person is better off and one is worse off, so the net effect is zero.
Actually, such asset price increase is a worrisome indicator — we are entering into bubble territory. It’s the result of a loose monetary policy, low interest rates and “quantitative easing” that was designed under the Obama administration to stimulate the economy. That’s why all assets are rising in value. Normally, a rise in one asset means a fall in another, like selling gold to pay for houses. But because of loose monetary policy, all assets are increasing in price. The amazing rise in Bitcoin over the last year is as much a result of this bubble growing in all assets as it is to an exuberant belief in Bitcoin.
When this bubble collapses, which may happen during Trump’s term, it’ll really be the Obama administration who is to blame. I mean, if Trump is willing to take credit for the asset price bubble now, I’m willing to give it to him, as long as he accepts the blame when it crashes.

1.5 million fewer people on food stamps

As you’d expect, I’m going to debunk this with a graph: the numbers have been falling since the great recession. Indeed, in the previous period under Obama, 1.9 fewer people got off food stamps, so Trump’s performance is slight ahead rather than behind Obama. Of course, neither president is really responsible.

Consumer confidence through the roof

Again we are going to graph this number:

Again we find nothing in the graph that suggests President Trump is responsible for any change — it’s been improving steadily since the Great Recession.

One thing to note is that, technically, it’s not “through the roof” — it still quite a bit below the roof set during the dot-com era.

Lowest Unemployment rate in 17 years

Again, let’s simply graph it over time and look for Trump’s contribution. as we can see, there doesn’t appear to be anything special Trump has done — unemployment has steadily been improving since the Great Recession.
But here’s the thing, the “unemployment rate” only measures those looking for work, not those who have given up. The number that concerns people more is the “labor force participation rate”. The Great Recession kicked a lot of workers out of the economy.
Mostly this is because Baby Boomer are now retiring an leaving the workforce, and some have chosen to retire early rather than look for another job. But there are still some other problems in our economy that cause this. President Trump has nothing particular in order to solve these problems.

Conclusion

As we see, Don Jr’s tweet is a troll. When we look at the graphs of these indicators going back to the Great Recession, we don’t see how President Trump has influenced anything. The improvements this year are in line with the improvements last year, which are in turn inline with the improvements in the previous year.
To be fair, all parties credit their President with improvements during their term. President Obama’s supporters did the same thing. But at least right now, with these numbers, we can see that there’s no merit to anything in Don Jr’s tweet.
The hyperpartisan rancor in this country is because neither side cares about the facts. We should care. We should care that these numbers suck, even if we are Republicans. Conversely, we should care that those NetNeutrality claims by Democrats suck, even if we are Democrats.

Skillz: editing a web page

Post Syndicated from Robert Graham original http://blog.erratasec.com/2017/02/skillz-editing-web-page.html

So one of the skillz you ought to have in cybersec is messing with web-pages client-side using Chrome’s Developer Tools. Web-servers give you a bunch of HTML and JavaScript code which, once it reaches your browser, is yours to change and play with. You can do a lot with web-sites that they don’t intend by changing that code.

Let me give you an example. It’s only an example — touching briefly on steps to give you an impression what’s going on. It’s not a ground up explanation of everything, which you may find off-putting. Click on the images to expand them so you can see fully what’s going on.

Today is the American holiday called “Presidents Day”. It’s actually not a federal holiday, but a holiday in all 50 states. Originally it was just Washington’s birthday (February 22), but some states choose to honor other presidents as well, hence “Presidents Day”.
Thus of us who donated to Donald Trump’s campaign (note: I donated to all candidates campaigns back in 2015) received an email today suggesting that to honor Presidents Day, we should “sign a card” for Trump. It’s a gross dis-honoring of the Presidents the day is supposed to commemorate, but whatever, it’s the 21st century.
Okay, let’s say we want to honor the current President with a bunch of 🖕🖕🖕🖕 in order to point out his crassness of exploiting this holiday, and clicked on the URL [*], and filled it in as such (with multiple skin tones for the middle finger, just so he knows its from all of us):
Okay, now we hit the submit button “Add My Name” in order to send this to his campaign. The only problem is, the web page rejects us, telling us “Please enter a valid name” (note, I’m changing font sizes in these screen shots so you can see the message):
This is obviously client side validation of the field. It’s at this point that we go into Developer Tools in order to turn it off. One way is to [right-click] on that button, and from the popup menu, select “Inspect”, which gets you this screen (yes, the original page is squashed to the left-hand side):
We can edit the HTML right there and add the “novalidate” flag, as shown below, then hit the “Add My Name” button again:
This doesn’t work. The scripts on the webpage aren’t honoring the HTML5 “novalidate” flag. Therefore, we’ll have to go edit those scripts. We do that by clicking on the Sources tab, then press [ctrl-shift-f] to open the ‘find’ window in the sources, and type “Please enter a valid name”, and you’ll find the JavaScript source file (validation.js) where the validation function is located:
If at this point you find all these windows bewildering, then yes, you are on the right track. We typed in the search there near the bottom next to the classic search icon 🔍. Then right below that we got the search results. We clicked on the search results, then up above popped up the source file (validation.js) among all the possible source files with the line selected that contains our search term. Remember: when you pull down a single HTML page, like the one from donaldtrump.com, it can pull in a zillion JavaScript files as well.
Unlike the HTML, we can’t change the JavaScript on the fly (at least, I don’t know how to). Instead, we have to run more JavaScript. Specifically, we need to run a script that registers a new validation function. If you look in the original source, it contains a function that validates the input by making sure it matches a regular expression:
  1. jQuery.validator.addMethod(“isname”, function(value, element) {
  2.     return this.optional(element) || (/^[a-zA-Z]+[ ]+(([‘,. -][a-zA-Z ])?[a-zA-Z]*)+.?$/.test(value.trim()));
  3. }, “Please enter a valid name”);
From the console, we are going to call the addMethod function ourselves to register a different validation function for isname, specifically a validation function that always returns true, meaning the input is valid. This will override the previously registered function. As the Founders of our country say, the solution to bad JavaScript is not to censor it, but to add more JavaScript.
  1. jQuery.validator.addMethod(“isname”, function () {
  2.     return true});
We just type that in the Console as shown below (in the bottom window where Search used to be) and hit [enter]. It gives us the response “undefined”, but that’s OK. (Note: in the screenshot I misspelled it as isName, it should instead be all lowercase isname).
Now we can close Developer Tools and press the “Add My Name” button, and we get the following response:
Darn, foiled again. But at least this time, our request went to the server. It was on the server side that the request was rejected. We successfully turned off client-side checking. Had the server accepted our Unicode emoji, we would’ve reached the next step, where it asks for donations. (By the way, the entire purpose of “sign this card” is to get users to donate, nothing else).

Conclusion

So we didn’t actually succeed at doing anything here, but I thought I’d write it up anyway. Editing the web-page client-side, or mucking around with JavaScript client-side, is a skill that every cybersec professional should have. Hopefully, this is an amusing enough example that people will follow the steps to see how this is done.

Тръмпутинизмът

Post Syndicated from Григор original http://www.gatchev.info/blog/?p=2018

Понякога ми се случва най-нагло да открадна отнякъде текст, който смятам за особено важен, и да го пусна тук. Има случаи, когато важността на написаното е над тази на законите за авторските права.

Такъв е според мен случаят и с тази статия на Джон Суини, журналист в BBC Panorama. Оригиналното ѝ заглавие е “Who are the figures that push Donald Trump and Vladimir Putin together?” („Кои са хората, които сближават Доналд Тръмп и Владимир Путин?“), и е публикувана на новинарския сайт на BBC. Тя е почти единственият материал, който не само отбелязва много точно приликата между двамата, но и обръща внимание на един друг човек – Александр Дугин – чието влияние не е за подценяване. Малко мислене върху тази статия дава много разбиране на много неща в съвременния свят.

(Може би е добре да седна да напиша някой запис и за истината и лъжата в съвременния свят… Ех, да имах времето да пиша всичко, което смятам за важно!)

—-

Въпросът дали руският лидер Владимир Путин има материал, с който да изнудва Доналд Тръмп, засега няма отговор. По-важното обаче е, че той е много настрани от истински важното – че двамата мислят твърде подобно.

Вярата на г-н Тръмп в американския традиционализъм и омразата му към сверки с истината повтаря музиката на Кремъл. Нацията, властта и непоносимостта към критики са новият (и много руски) световен ред.

Можете да наречете този начин на мислене Тръмпутинизъм.

Чуваемостта между Кремъл и Тръмп Тауър е силна, става все по-силна и е много, много добра новина за г-н Путин.

Както Тръмп заяви пред Майкъл Гоув в понеделник, нов договор за съкращение на ядрените оръжия може би ще предложи свързано с него преразглеждане на санкциите срещу Русия.

Оглушителната тишина в случая и странната липса на критичност на г-н Тръмп към г-н Путин, например заради руското хакерстване срещу американската демокрация, присвояването с военна сила на Крим и руската роля в продължаващата война в Донбас.

Странно е, че г-н Тръмп в туитовете си подкрепя руската теза, а не примерно тази на ЦРУ и другите разузнавателни служби на САЩ.

Че защо е нужно да критикуваш световен лидер, с който си предимно съгласен?

Покрай тръмпутинизма в общественото внимание се появиха трима други: Найджъл Фарадж, за когото ЕС е далеч по-голяма опасност за световния мир от Русия; неговият приятел Стив Банън, който към момента е главен стратег на Тръмп, и руският философ Александр Дугин.

С дългите си коси и класическа славянска външност, г-н Дугин бива описван като „мозъкът на Путин“ и „Путиновият Распутин“. Той има собствено про-кремълско шоу, което разпространява тезата за руско-православното превъзходство във всичко, в любопитна смес от риторика подобна на Гьобелсовата и религиозни химни.

Много хора смятат, че гласът на Дугин се чува в Кремъл.

Също така той е обект на западни санкции, заради разпалеността на изказването му в подкрепа на руската инвазия в Украйна, която до момента е коствала живота на около 10 000 души.

Господата Фарадж, Банън и Дугин са обединени от тезата, че най-страшната опасност за западната цивилизация е ислямският екстремизъм.

Г-н Банън беше оповестил възгледите си на среща на крайно десни близо до Ватикана през 2014 г.

Той твърди, че т.нар. Ислямска държава има акаунт в Туитър, чиято цел е „да превърне Съединените щати в река от кръв“.

„Вярвайте ми, това ще дойде и в Европа“, добави той. „В добавка към това според мен ние в момента сме в началните стадии на световна война срещу ислямския фашизъм.“

Под риск ли са демократичните ценности?

Опасността тук е, че когато подкрепяте начина, по който Кремъл се бори срещу „ислямския фашизъм“ примерно в Алепо, вие се присъединявате към това, което някои наричат „руски фашизъм“. Или поне към загърбването на демократичните ценности и на правилата за водене на война. А с това се превръщате във висококачествен рекрутьор на Ислямска държава.

Това е риск, на който г-н Дугин изглежда не придава значение. Интервюто ми с него в Москва не протече добре.

Александр Дугин, 20 декември, 9:35

Днес изритах кореспондентите на BBC. Не бях срещал толкова гнусни ****** от много време. Водеше ги Джон Суини. Както виждате по фамилията му, той е глобалист и мръсна свиня. Съчиняват лъжливи новини, че Русия била помогнала на Тръмп да стане президент. Доказателствата им: Путин бил работил преди за КГБ. Пълни кретени! Професионалното им ниво е нула. Чиста проба съветски пропагандисти. Не съветвам никого да се доближава до тях.

Още в началото на интервюто ни той категорично определи шансовете Русия да е хакерствала срещу американците като „кръгла нула“.

Попитах го доколко е привързан г-н Путин към демокрацията.

– Моля, внимавайте – беше отговорът му. – Не можете да ни учите на демокрация, защото се опитвате да наложите на всеки народ, всяка държава и всяко общество, вашата западна, американска или така наречена американска система от ценности, без да питате… И това е расизъм. Вие сте расисти.

Доста от критиците на г-н Путин бяха убити – поне 20 откакто той пое властта през 2000 г. Срещал съм се и съм се възхищавал на трима: Анна Политковская, Наташа Естемирова и Борис Немцов. Който беше застрелян току пред стените на Кремъл.

Попитах г-н Дугин какво говори смъртта му за руската демокрация.

– При вас пък, ако сте свързани с Уикилийкс, могат да ви убият – контрира ме той.

Тогава помолих г-н Дугин да ми покаже списък на американските журналисти, които са били убити покрай Барак Обама. Той отговори, че това е „абсолютно глупав разговор“, и напусна интервюто.

Малко по-късно пусна запис в блога си, пред 20 000 (по неговите думи) последователи, илюстриран със снимката ми. С обвинението, че „съчинявам лъжливи новини“ и че съм „пълен кретен“ и „глобалистка свиня“.

Това е езикът на новия световен ред.

Няколко дни по-късно гледах пресконференцията, на която г-н Тръмп отказа да отговори на въпрос от репортер на CNN, като го обвини, че те „съчиняват лъжливи новини“.

При тръмпутинизма ехото между Русия и Америка става все по-силно от ден на ден.

—-

И малко коментари от мен:

Мой познат, изключително интелигентен човек, преди време беше казал: „Следете кога в Русия ще се заговори за фашизъм в България. Това ще е заповедта за подготовка на руска агресия тук.“

(Ако изразът „руска агресия“ не ви харесва, запитайте се дали предизвиква у вас същите чувства като примерно „турска агресия“. Ако не, си задайте въпроса на коя държава сте патриот… или зомби.)

Затова ми е странно, когато Банън говори за „ислямски фашизъм“. Възгледът, че фашизмът може да е религиозен или пък десен, е популярен на само едно място на света – бившия соц-лагер. СССР имаше отчаяна нужда да прикрие приликата между националистическия социализъм и комунистическия социализъм. Затова насила нарече немския нацизъм „фашизъм“, умишлено обърквайки го с останалия в сянката му италиански… Така че да направи точно същите грешки американец навежда на размисли – откъде е попил този начин на мислене, и какви изводи следват от това.

За Дугин – в статията е казано достатъчно. Ще добавя само, че очевидно той отлично знае коя е най-гнусната форма на възможен пропагандизъм. Сравнете го обаче с оценките му на съветската система. Как да не си припомни човек нашичките „реформирани“ комунисти, които хем прехвалват комунизма при всяка възможност, хем „комунист“ е най-страшната обида в устата им.

Ако бях примерно възторжен почитател на християнството, щеше ли „християнин“ да е най-лютата обида, която мога да измисля?! Според мен това е възможно само за човек, напълно изгубил представа за разликата между добро и зло. Или дори понятието, че това са две различни неща… Което пък еднозначно идентифицира слугите на злото.

(Да, не вярвам в божества. Но доброто и злото според мен съществуват, въпреки че не могат да се пипнат. Точно както не можете да пипнете числата, но те съществуват – или пък не можете да пипнете нечия личност или спомен, но те безспорно съществуват… Но това е друга, много дълга тема.)

За Найджъл Фарадж не смятам да се разпростирам. Времето ще разкаже кой, какъв и защо за него по-добре от мен.

Колкото до Тръмп и Путин… В САЩ демокрацията все още е изключително силна. Нищо чудно да успеят да се отърват с една по-сериозна катастрофа в най-различни области. С малко повече късмет може дори да не изгубят водещата си роля в света. Но още отначало “Make America great again!” ми звучеше подозрително подобно на “Deutschland, Deutschland uber alles!”

Но гледката в какво се превръщат обикновените руснаци под Путиновата пропаганда е ужасяваща. Доскоро извикваше в паметта ми сцени от „Обикновен фашизъм“ – сега те бледнеят пред това, което виждам и научавам за там. Точно както великоруският нацизъм на Дугин слага Хитлеровия нацизъм в джоба си. И неговите химни звучат подозрително познато… Страх ме е руският народ да не се превърне в язва на човечеството, която да е много трудна за излекуване.

A Comment on the Trump Dossier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2017/01/a_comment_on_th_1.html

Imagine that you are someone in the CIA, concerned about the future of America. You have this Russian dossier on Donald Trump, which you have some evidence might be true. The smartest thing you can do is to leak it to the public. By doing so, you are eliminating any leverage Russia has over Trump and probably reducing the effectiveness of any other blackmail material any government might have on Trump. I believe you do this regardless of whether you ultimately believe the document’s findings or not, and regardless of whether you support or oppose Trump. It’s simple game-theory.

This document is particularly safe to release. Because it’s not a classified report of the CIA, leaking it is not a crime. And you release it now, before Trump becomes president, because doing so afterwards becomes much more dangerous.

MODERATION NOTE: Please keep comments focused on this particular point. More general comments, especially uncivil comments, will be deleted.

My Priorities for the Next Four Years

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2016/12/my_priorities_f.html

Like many, I was surprised and shocked by the election of Donald Trump as president. I believe his ideas, temperament, and inexperience represent a grave threat to our country and world. Suddenly, all the things I had planned to work on seemed trivial in comparison. Although Internet security and privacy are not the most important policy areas at risk, I believe he — and, more importantly, his cabinet, administration, and Congress — will have devastating effects in that area, both in the US and around the world.

The election was so close that I’ve come to see the result as a bad roll of the dice. A few minor tweaks here and there — a more enthusiastic Sanders endorsement, one fewer of Comey’s announcements, slightly less Russian involvement — and the country would be preparing for a Clinton presidency and discussing a very different social narrative. That alternative narrative would stress business as usual, and continue to obscure the deep social problems in our society. Those problems won’t go away on their own, and in this alternative future they would continue to fester under the surface, getting steadily worse. This election exposed those problems for everyone to see.

I spent the last month both coming to terms with this reality, and thinking about the future. Here is my new agenda for the next four years:

One, fight the fights. There will be more government surveillance and more corporate surveillance. I expect legislative and judicial battles along several lines: a renewed call from the FBI for backdoors into encryption, more leeway for government hacking without a warrant, no controls on corporate surveillance, and more secret government demands for that corporate data. I expect other countries to follow our lead. (The UK is already more extreme than us.) And if there’s a major terrorist attack under Trump’s watch, it’ll be open season on our liberties. We may lose a lot of these battles, but we need to lose as few as possible and as little of our existing liberties as possible.

Two, prepare for those fights. Much of the next four years will be reactive, but we can prepare somewhat. The more we can convince corporate America to delete their saved archives of surveillance data and to store only what they need for as long as they need it, the safer we’ll all be. We need to convince Internet giants like Google and Facebook to change their business models away from surveillance capitalism. It’s a hard sell, but maybe we can nibble around the edges. Similarly, we need to keep pushing the truism that privacy and security are not antagonistic, but rather are essential for each other.

Three, lay the groundwork for a better future. No matter how bad the next four years get, I don’t believe that a Trump administration will permanently end privacy, freedom, and liberty in the US. I don’t believe that it portends a radical change in our democracy. (Or if it does, we have bigger problems than a free and secure Internet.) It’s true that some of Trump’s institutional changes might take decades to undo. Even so, I am confident — optimistic even — that the US will eventually come around; and when that time comes, we need good ideas in place for people to come around to. This means proposals for non-surveillance-based Internet business models, research into effective law enforcement that preserves privacy, intelligent limits on how corporations can collect and exploit our data, and so on.

And four, continue to solve the actual problems. The serious security issues around cybercrime, cyber-espionage, cyberwar, the Internet of Things, algorithmic decision making, foreign interference in our elections, and so on aren’t going to disappear for four years while we’re busy fighting the excesses of Trump. We need to continue to work towards a more secure digital future. And to the extent that cybersecurity for our military networks and critical infrastructure allies with cybersecurity for everyone, we’ll probably have an ally in Trump.

Those are my four areas. Under a Clinton administration, my list would have looked much the same. Trump’s election just means the threats will be much greater, and the battles a lot harder to win. It’s more than I can possibly do on my own, and I am therefore substantially increasing my annual philanthropy to support organizations like EPIC, EFF, ACLU, and Access Now in continuing their work in these areas.

My agenda is necessarily focused entirely on my particular areas of concern. The risks of a Trump presidency are far more pernicious, but this is where I have expertise and influence.

Right now, we have a defeated majority. Many are scared, and many are motivated — and few of those are applying their motivation constructively. We need to harness that fear and energy to start fixing our society now, instead of waiting four or even eight years, at which point the problems would be worse and the solutions more extreme. I am choosing to proceed as if this were cowpox, not smallpox: fighting the more benign disease today will be much easier than subjecting ourselves to its more virulent form in the future. It’s going to be hard keeping the intensity up for the next four years, but we need to get to work. Let’s use Trump’s victory as the wake-up call and opportunity that it is.

That anti-Trump Recode article is terrible

Post Syndicated from Robert Graham original http://blog.erratasec.com/2016/12/that-anti-trump-recode-article-is.html

Trump’s a dangerous populist. However, the left-wing media’s anti-Trump fetishism is doing nothing to stop Trump. It’s no better than “fake news” — it gets passed around a lot on social-media, but is intellectually bankrupt, unlikely to change anybody’s mind. A good example is this op-ed on Re/Code [*] about Silicon Valley leaders visiting Trump.

The most important feature of that Re/code article is that it contains no criticism of Trump other than the fact that he’s a Republican. Half the country voted for Trump. Half the country voted Republican. It’s not just Trump that this piece imagines as being unreasonable, but half the country. It’s a fashionable bigotry among some of Silicon Valley’s leftist elite.

But CEOs live in a world where half their customers are Republican, where half their share holders are Republican. They cannot lightly take political positions that differ from their investors/customers. The Re/code piece claims CEOs said “we are duty-bound as American citizens to attend”. No, what they said was “we are duty-bound as officers of our corporations to attend”.

The word “officer”, as in “Chief Operating Officer”, isn’t an arbitrary title like “Senior Software Engineer” that has no real meaning. Instead, “officer” means “bound by duty”. It includes a lot of legal duties, for which they can go to jail if they don’t follow. It includes additional duties to shareholders, for which the board can fire them if they don’t follow.

Normal employees can have Twitter disclaimers saying “these are my personal opinions only, not that of my employer”. Officers of corporations cannot. They are the employer. They cannot champion political causes of their own that would impact their stock price. Sure, they can do minor things, like vote, or contribute quietly to campaigns, as long as they aren’t too public. They can also do political things that enhances stock price, such as opposing encryption backdoors. Tim Cook can announce he’s gay, because that enhances the brand image among Apple’s key demographic of millennials. It’s not something he could do if he were the CEO of John Deere Tractors.

Among the things the CEO’s cannot do is take a stance against Donald Trump. The Boeing thing is a good example. The Boeing’s CEO criticized Trump’s stance on free trade, and 30 minutes later Trump tweeted criticisms of a $4 billion contract with Boeing, causing an immediate billion drop in Boeing’s stock price.

This incident shows why the rest of us need to oppose Trump. Such vindictive politics is how democracies have failed. We cannot allow this to happen here. But the hands of CEOs are tied — they are duty bound to avoid such hits to their stock price.

On the flip, this is one of the few chances CEOs will be able to lobby Trump. If Trump has proven anything, it’s that he has no real positions on things. This would be a great time to change his mind on “encryption backdoors”, for example.

Trump is a dangerous populist who sews distrust in the institutions that give us a stable, prosperous country. Any institution, from the press, to the military, to the intelligence services, to the election system, is attacked, brought into disrepute, even if it supports him. Trump has a dubious relationship with the truth, such as his repeated insistence he won a landslide rather than by a slim margin. He has deep character flaws, such as his vindictive attacks against those who oppose him (Boeing is just one of many examples). Hamilton electors cite deep, patriotic principles for changing their votes, such as Trump’s foreign influences and demagoguery.

What I’m demonstrating here is that thinking persons have good reasons to oppose Trump that can be articulated without mentioning political issues that divide Democrats and Republicans. That the Re/code article is unable to do so makes it simply “hyper-partisan news”, the sort that stroke’s people’s prejudices and passions to get passed around a lot on social media, but which is unlikely to inform anybody or change any minds. In other words, it’s no better than “fake-news”.


Hacking and the 2016 Presidential Election

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2016/11/hacking_and_the.html

Was the 2016 presidential election hacked? It’s hard to tell. There were no obvious hacks on Election Day, but new reports have raised the question of whether voting machines were tampered with in three states that Donald Trump won this month: Wisconsin, Michigan and Pennsylvania.

The researchers behind these reports include voting rights lawyer John Bonifaz and J. Alex Halderman, the director of the University of Michigan Center for Computer Security and Society, both respected in the community. They have been talking with Hillary Clinton’s campaign, but their analysis is not yet public.

According to a report in New York magazine, the share of votes received by Clinton was significantly lower in precincts that used a particular type of voting machine: The magazine story suggested that Clinton had received 7 percent fewer votes in Wisconsin counties that used electronic machines, which could be hacked, than in counties that used paper ballots. That is exactly the sort of result we would expect to see if there had been some sort of voting machine hack. There are many different types of voting machines, and attacks against one type would not work against the others. So a voting anomaly correlated to machine type could be a red flag, although Trump did better across the entire Midwest than pre-election polls expected, and there are also some correlations between voting machine type and the demographics of the various precincts. Even Halderman wrote early Wednesday morning that “the most likely explanation is that the polls were systematically wrong, rather than that the election was hacked.”

What the allegations, and the ripples they’re causing on social media, really show is how fundamentally untrustworthy our hodgepodge election system is.

Accountability is a major problem for US elections. The candidates are the ones required to petition for recounts, and we throw the matter into the courts when we can’t figure it out. This all happens after an election, and because the battle lines have already been drawn, the process is intensely political. Unlike many other countries, we don’t have an independent body empowered to investigate these matters. There is no government agency empowered to verify these researchers’ claims, even if it would be merely to reassure voters that the election count was accurate.

Instead, we have a patchwork of voting systems: different rules, different machines, different standards. I’ve seen arguments that there is security in this setup ­ an attacker can’t broadly attack the entire country ­ but the downsides of this system are much more critical. National standards would significantly improve our voting process.

Further investigation of the claims raised by the researchers would help settle this particular question. Unfortunately, time is of the essence ­ underscoring another problem with how we conduct elections. For anything to happen, Clinton has to call for a recount and investigation. She has until Friday to do it in Wisconsin, until Monday in Pennsylvania and until next Wednesday in Michigan. I don’t expect the research team to have any better data before then. Without changes to the system, we’re telling future hackers that they can be successful as long as they’re able to hide their attacks for a few weeks until after the recount deadlines pass.

Computer forensics investigations are not easy, and they’re not quick. They require access to the machines. They involve analysis of Internet traffic. If we suspect a foreign country like Russia, the National Security Agency will analyze what they’ve intercepted from that country. This could easily take weeks, perhaps even months. And in the end, we might not even get a definitive answer. And even if we do end up with evidence that the voting machines were hacked, we don’t have rules about what to do next.

Although winning those three states would flip the election, I predict Clinton will do nothing (her campaign, after all, has reportedly been aware of the researchers’ work for nearly a week). Not because she does not believe the researchers ­- although she might not -­ but because she doesn’t want to throw the post-election process into turmoil by starting a highly politicized process whose eventual outcome will have little to do with computer forensics and a lot to do with which party has more power in the three states.

But we only have two years until the next national elections, and it’s time to start fixing things if we don’t want to be wondering the same things about hackers in 2018. The risks are real: Electronic voting machines that don’t use a paper ballot are vulnerable to hacking.

Clinton supporters are seizing on this story as their last lifeline of hope. I sympathize with them. When I wrote about vote-hacking the day after the election, I said: “Elections serve two purposes. First, and most obvious, they are how we choose a winner. But second, and equally important, they convince the loser ­- and all the supporters ­- that he or she lost.” If the election system fails to do the second, we risk undermining the legitimacy of our democratic process. Clinton’s supporters deserve to know whether this apparent statistical anomaly is the result of a hack against our election system or a spurious correlation. They deserve an election that is demonstrably fair and accurate. Our patchwork, ad hoc system means they may never feel confident in the outcome. And that will further erode the trust we have in our election systems.

This essay previously appeared in the Washington Post.

Edited to Add: Green Party candidate Jill Stein is calling for a recount in the three states. I have no idea if a recount includes forensic analysis to ensure that the machines were not hacked, but I doubt it. It would be funny if it wasn’t all so horrible.

Also, here’s an article from 538.com arguing that demographics explains all the discrepancies.

Cybersecurity Issues for the Next Administration

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2016/10/cybersecurity_i.html

On today’s Internet, too much power is concentrated in too few hands. In the early days of the Internet, individuals were empowered. Now governments and corporations hold the balance of power. If we are to leave a better Internet for the next generations, governments need to rebalance Internet power more towards the individual. This means several things.

First, less surveillance. Surveillance has become the business model of the Internet, and an aspect that is appealing to governments worldwide. While computers make it easier to collect data, and networks to aggregate it, governments should do more to ensure that any surveillance is exceptional, transparent, regulated and targeted. It’s a tall order; governments such as that of the US need to overcome their own mass-surveillance desires, and at the same time implement regulations to fetter the ability of Internet companies to do the same.

Second, less censorship. The early days of the Internet were free of censorship, but no more. Many countries censor their Internet for a variety of political and moral reasons, and many large social networking platforms do the same thing for business reasons. Turkey censors anti-government political speech; many countries censor pornography. Facebook has censored both nudity and videos of police brutality. Governments need to commit to the free flow of information, and to make it harder for others to censor.

Third, less propaganda. One of the side-effects of free speech is erroneous speech. This naturally corrects itself when everybody can speak, but an Internet with centralized power is one that invites propaganda. For example, both China and Russia actively use propagandists to influence public opinion on social media. The more governments can do to counter propaganda in all forms, the better we all are.

And fourth, less use control. Governments need to ensure that our Internet systems are open and not closed, that neither totalitarian governments nor large corporations can limit what we do on them. This includes limits on what apps you can run on your smartphone, or what you can do with the digital files you purchase or are collected by the digital devices you own. Controls inhibit innovation: technical, business, and social.

Solutions require both corporate regulation and international cooperation. They require Internet governance to remain in the hands of the global community of engineers, companies, civil society groups, and Internet users. They require governments to be agile in the face of an ever-evolving Internet. And they’ll result in more power and control to the individual and less to powerful institutions. That’s how we built an Internet that enshrined the best of our societies, and that’s how we’ll keep it that way for future generations.

This essay previously appeared on Time.com, in a section about issues for the next president. It was supposed to appear in the print magazine, but was preempted by Donald Trump coverage.

On a technicality

Post Syndicated from Eevee original https://eev.ee/blog/2016/07/22/on-a-technicality/

Apropos of nothing, I’d like to tell you a story. I’ve touched on this before, but this is the full version. It’s the story of hypothetical small-to-medium Internet community.

Stop me if you’ve heard this one

You create a little community for a thing you like. You give it a phpBB forum or something.

You want people to be nice, so you make a couple rules. No swearing. No spamming. Don’t use all caps.

You invite your friends, and they invite their friends, and all is well and good. There are a few squabbles now and then, but they get resolved without too much trouble, and everyone more or less gets along.

One day, a new person shows up, and starts linking to their website in almost every thread. Their website mostly consists of very mean-spirited articles written about several well-known and well-liked people in the group. When people ask them to stop, they lash out with harsh insults.

So you ban them.

There is immediate protest from a number of people, most of whom you strangely don’t recognize. The person didn’t break any of the rules — how dare you ban them? They never swore. They never used all caps. They never even spammed, because technically spam is unwanted and automated, and this was a real person linking their website which is related to the thing the community is about.

You can’t think of a good counter-argument for this, so you unban them. You also add a new rule, prohibiting linking to websites.

Now the majority of the community is affected, because they can’t link their own work any more. This won’t work. You repeal the previous rule, and instead make one that limits the number of website links to one per day.

The original jerk responds by linking their website once a day, and then making other posts that link to that first post they made. They continue to be abrasive towards everyone else, but they never swear, and you’re just not sure what to do about that.

A few other people start posting, seemingly just to make fun of the rest of you, but likewise never break any of your rules.

A preposterous arms race follows, with the rules becoming increasingly nitpicky as you try to distinguish overt antagonism from mundane and innocent behavior.

After a while, you notice that many of your friends no longer come around. And there seem to be a lot more jerks than there were before. You don’t understand why. Your rules are reasonable, and you enforced them fairly, right?

But it’s not really a swear word

I’ve noticed that people really like to write rules that sound objective. Seems like a good enough idea, right? Lets everyone know exactly what the line is.

The trick is that human behavior, and especially human language, are very… squishy. We gauge each other based on a lot of unspoken context: our prior relationship, how both of us seem to be feeling, whether or not we skipped lunch today. When the same comment or action can mean radically different things in different circumstances, it’s hard to draw a fine distinction between what’s acceptable behavior and what’s not.

And rules are written in human language, which makes them just as squishy. Who decides what “swearing” is? If all caps aren’t allowed, how about 90%? Who decides what’s a slur? What, precisely, constitutes harassment? These things sound straightforward and concrete, but they can still be nitpicked to death.

We give people the benefit of the doubt and assume they’ll try to respect what we clearly mean, but there’s nothing guaranteeing that.

Have you ever tried to politely decline a request or invitation, and been asked why not? Then the other party starts trying to weasel around your reason, and now you’re somehow part of a debate about what you want? I’ve seen it happen with mundane social interactions, with freelance workers, and of course, with small online communities.

This isn’t to say that hunting for technicalities is a sign of aggressive malice; it’s human nature. We want to do a thing, we’re told me can’t because of X, and so we see X as an obstacle to overcome. Language is subjective, so it’s the easiest avenue of attack.

Fixing this in rules is a hard problem. The obvious approach is to add increasingly specific details, though then you risk catching innocent behaviors, and you can end up stuck in an almost comical game of cat-and-mouse where you keep trying to find ways to edit your own rules so you’re allowed to punish someone you’ve already passed judgment on.

I think we forget that even real laws are somewhat subjective, often hinging on intent. There are entire separate crimes for homicide, depending on whether it was intentional or accidental or due to clear neglect. These things get decided by a judge or a jury and become case law, the somewhat murky extra rules that aren’t part of formal law but are binding nonetheless.

(In an awkward twist, a lot of communities — especially very large platforms! — don’t explain their reasoning for punishing any particular behavior. That somewhat protects them from being “but technically“-ed, but it also means there’s no case law, and no one else can quite be sure what’s expected behavior.)

That’s why I mostly now make quasirules like “don’t be a dick” or “keep your vitriol to your own blog“. The general expectation is still clear, and it’s obvious that I reserve the right to judge individual cases — which, in the case of a small community, is going to happen anyway. Let’s face it: small communities are monarchies, not democracies.

I do have another reason for this, which is based on another observation I’ve made of small communities. I’ve joined a few where I didn’t bother reading the rules, made some conversation, never bothered anyone, and then later discovered that I’d pretty clearly violated a rule. But no one ever pointed it out, and perhaps no one even noticed, because I wasn’t being a dick.

So I concluded that, for a smaller community, the people who need the rules are likely to be people who you don’t want around in the first place. And “don’t be a dick” covers that just as well.

Evaporative cooling

There are some nice people in the world. I mean nice people, the sort I couldn’t describe myself as. People who are friends with everyone, who are somehow never involved in any argument, who seem content to spend their time drawing pictures of bumblebees on flowers that make everyone happy.

Those people are great to have around. You want to hold onto them as much as you can.

But people only have so much tolerance for jerkiness, and really nice people often have less tolerance than the rest of us.

The trouble with not ejecting a jerk — whether their shenanigans are deliberate or incidental — is that you allow the average jerkiness of the community to rise slightly. The higher it goes, the more likely it is that those really nice people will come around less often, or stop coming around at all. That, in turn, makes the average jerkiness rise even more, which teaches the original jerk that their behavior is acceptable and makes your community more appealing to other jerks. Meanwhile, more people at the nice end of the scale are drifting away.

And this goes for a community of any size, though it may take more jerks to significantly affect a very large platform.

It’s still hard to give someone the boot, though, because it just feels like a really harsh thing to do to someone, especially for an abstract reason like “preserving the feel of the community”. And a jerk is more likely to make a fuss about being made to leave, which makes it feel like a huge issue — whereas nice people generally leave very quietly, and you may not even notice until several of them have been gone for a while.

There’s a human tendency to measure peace as though it were the inverse of volume: the louder people get, the less peaceful it is. We then try to optimize for the least arguing. I’m sure you’ve seen this happen before: someone in a group points out that the group is doing something destructive, that causes an argument, and then onlookers blame the person who pointed out the problem for causing the argument to happen. You can probably think of some pretty high-profile examples in some current events.

(You may relatedly enjoy the tale of the missing stair.)

Have you ever watched one of those TV shows where a dude comes in to berate restaurant owners for all the ridiculous things they’ve been doing? One of the most common defenses is: “well, no one complained“.

In the age of the Internet, where it seems like everyone is always complaining about something, it’s easy to forget that by and large people don’t complain. Sure, they might complain on their Twitter or to their friends or whatever, but chances are, they won’t complain to you. Consider: either you’re aware of the problem and have failed to solve it, or you’re clueless for not noticing. Either way, complaining won’t help anything; it’ll just cause conflict, making them that person who “caused” an argument by pointing out the obvious.

Gamification

Some people are aware of the technicality game on some level, and decide to play it — deliberately. Maybe to get their way; maybe just for fun.

These are people who think “it’d be a shame if something happened to it” is just the way people talk. Layered thick with multiple levels of irony, cloaked in jokes and misdirection, up to its eyeballs in plausible deniability, but crystal clear to the right audience.

It’s a game that offers them a massive advantage, because even if you both know you’re playing it, they have much more experience. Oh, and chances are they don’t even truly care about whether they’re banned or not, so they have nothing to lose — whereas you’re stuck with an existential crisis, questioning everything you believe about free speech and community management, while your nicest peers sneak out the back door.

I remember a time when someone in a community I helped run decided they didn’t like me. They started making subtle jabs, and eventually built up to saying the most biting and personal things they could think to say. Those things weren’t true, but they didn’t know that, and they phrased everything in such a way that their friends could rationalize them as not really trying to be cruel. And they had quite a lot of friends in the community, which put me in a pretty awkward position. How do I justify banning them, if a significant number of people are sure they’re innocent? Am I fucking crazy for seeing this glaring pattern when no one else does?

I did eventually ban them, but it contributed to a complete schism where most of the more grating people left to form their own clubhouse. Win/win?

Or let’s say, hypothetically, that some miscreant constructs a fake tweet screenshot. It’s shared by a high-profile person and spreads like wildfire.

Should either of them be punished? Which one, and why? The faker probably regarded it as a harmless joke; if not for the sharer, it would’ve remained one. Yet the sharer’s only crime was being popular. Did the sharer know it was fake? Was the sharer trying to inflict harm, draw attention to troubling behavior, or share something that made them laugh? Are the faker and the sharer the same person? If you can’t be sure either way, does it matter?

What if, instead of the thing you may be thinking about, the forgery depicted Donald Trump plagiarizing Barack Obama’s tweet congratulating Michelle Obama for her speech? Does that change any of the answers?

This is really difficult in extremely large groups, where you most want to avoid doling out arbitrary punishment, yet where people who play this game can inflict the most damage. The people who make and enforce the rules may not even be part of the group any more, and certainly can’t form an impression of every individual person in the group, so how can anything be enforced consistently? How do you account for intention, sarcasm, irony, self-deprecating humor? How do you explain this clearly without subjecting yourself to an endless deluge of technicalities? You could refuse to explain yourself at all, of course, but then you leave yourself open for people to offer their own explanations: you’re a tyrant who bans anyone who contradicts you, or you hated them for demographic reasons, or you’re just plain irrational and do zany cruel things to people around you on a whim.

I don’t have any good answers

I’m not sure there are any. Corralling people is a tricky problem. We still barely know how to do it in meatspace groups of half a dozen, let alone digital groups numbering in the hundreds of millions.

Our current approaches kinda suck, though.

Russians Hacking DNC Computers

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2016/06/russians_hackin.html

The Washington Post is reporting that Russian hackers penetrated the network of the Democratic National Committee and stole opposition research on Donald Trump. The evidence is from CrowdStrike:

The firm identified two separate hacker groups, both working for the Russian government, that had infiltrated the network, said Dmitri Alperovitch, CrowdStrike co-founder and chief technology officer. The firm had analyzed other breaches by both groups over the last two years.

One group, which CrowdStrike had dubbed Cozy Bear, had gained access last summer and was monitoring the DNC’s email and chat communications, Alperovitch said.

The other, which the firm had named Fancy Bear, broke into the network in late April and targeted the opposition research files. It was this breach that set off the alarm. The hackers stole two files, Henry said. And they had access to the computers of the entire research staff — an average of about several dozen on any given day.

This seems like standard political espionage to me. We certainly don’t want it to happen, but we shouldn’t be surprised when it does.

Slashdot thread.

EDITED TO ADD (6/16): From the Washington Post article, the Republicans were also hacked:

The intrusion into the DNC was one of several targeting American political organizations. The networks of presidential candidates Hillary Clinton and Donald Trump were also targeted by Russian spies, as were the computers of some Republican political action committees, U.S. officials said. But details on those cases were not available.

EDITED TO ADD (6/16): These leaks might be from this hack, or from another unrelated hack. They don’t seem to be related to the Russian government at all.

EDITED TO ADD (6/12): Another view.

Drumpf: this is not how German works

Post Syndicated from Robert Graham original http://blog.erratasec.com/2016/06/drumpf-this-is-not-how-german-works.html

In our willingness to believe any evil of Trump, some have claimed his original name was “Drumpf”. This isn’t true, this isn’t how the German language works. Trump has the power to short-circuit critical thinking in both his supporters and his enemies. The “Drumpf” meme is just one example.

There was no official pronunciation or spelling of German words/names until after Trump’s grandfather was born. As this The Guardian article describes, in the city (“Kallstadt”) where Trump’s grandfather was born, you’ll see many different spellings of the family name in the church’s records. like “Drumb, Tromb, Tromp, Trum, Trumpff, Dromb” and Trump. A person might spell their name different ways on different documents, and the names of children might be spelled different than their parent’s. It makes German genealogy tough sometimes.

During that time, different areas of German had different dialects that were as far apart as Dutch and German are today. Indeed, these dialects persist. Germans who grow up outside of cities often learn their own local dialect and standard German as two different languages. Everyone understands standard German, but many villagers cannot speak it. They often live their entire lives within a hundred kilometers of where they grew up because if they go too far away, people can no longer understand them.

The various German dialects, sub-dialects, and accents often had consistent language shifts, where the same sound is pronounced differently across many words. For example, words that in English have a ‘p’ will in German have ‘pf” instead, like the word penny becoming Pfennig, or pepper becoming Pfeffer.

Kallstadt is located in the Pfalz region of Germany, or as they pronounce it in the local dialect, Palz. You see what I’m getting at, what is ‘pf’ in German is ‘p’ (like English) in the local dialect. Thus, you’d say “Trump” if you were speak Pfalz dialect, or “Trumpf” if you were speaking standard German.

It’s like the word for stocking, which in standard German is Strumpf. In documents written around that time in the Pfalz region, you’d find spellings like StrumpStrumpf, StrumpffStrimp, and Stromp. Both the vowels and the last consonant would change (according to a Pfalz dictionary I found online).

Friederich Trump was born in 1869, in a time when Germany was split into numerous smaller countries. The German Empire that unified Germany was created in 1871. The counsel to standardize the language and spellings was 1876. Friederich emigrated to America in 1885. In other words, his birth predates the era in which they would’ve standardized the spelling of names.

From the records we have, “Trump” was on his baptism record, and “Trump” is how he spelled his name in America, but “Trumpf”, with an ‘f’ was on his immigration form. That’s perfectly reasonable. The immigration officer was probably a German speaker, who asked his name, and spelled it according to his version of German, with an ‘f’.

This idea of an official spelling/pronunciation of a name is a modern invention, with the invention of the modern “state” and “government officials”. It didn’t exist back when Friederich was born. His only birth record is actually his baptismal record at the local church.

Thus, Trump’s name is spelled “Trump”. It was never officially spelled any other way in the past. It was never “changed”. Sure, you’ll see church documents and stuff with different spellings, but just how all words and names were handled back then. Insisting that he’s “Drumpf” is ignorant — it’s not now the German language works.


Update: Somebody named Gwenda Blair wrote book on Trump’s family, which claims the name comes from Hanns Drumpf, who settled in Kallstadt in 1608. But they can’t connect the dots. That’s because right after, the 30 years war happened. It’s a famous event in Germany because it burnt most of the church records. Most all German family trees can be traced back to the 30 years war — but no further.

It’s probable they were related. It’s possible that Hanns was even an ancestor of Trump who at one time spelled his name “Drumpf”. But that’s still not the official spelling, because that’s not how German worked at that time.


Update: According to Snopes, it’s true that “Donald Trump’s ancestors changed their surname from Drumpf to Trump“. Snopes is wrong, because they are morons. The correct answer is “there’s no record of a name change”. The fact that difference sources make conflicting claims should’ve been proof enough for Snopes that there’s no evidence to support the claim.


BTW, my great-grandmother is “Pennsylvania Dutch”, most of who came from that same region. I may be distantly related to Trump.

Also BTW, isn’t weird that we are talking about his grandfather born 150 years ago? His grandfather was 36 when his son was born, and his father was 41 when Trump was born. Three generations back, and we are already in a pre-historical era — that is to say, the era where we had writing, but not standardized spelling.

Nothing says "establishment" as Vox’s attack on Trump

Post Syndicated from Robert Graham original http://blog.erratasec.com/2016/02/vox-is-wrong-about-trump.html

I keep seeing this Ezra Klein Vox article attacking Donald Trump. It’s wrong in every way something can be wrong. Trump is an easy target, but the Vox piece has almost no substance.Yes, it’s true that Trump proposes several unreasonable policies, such as banning Muslims from coming into this country. I’ll be the first to chime in and call Trump a racist, Nazi bastard for these things.But I’m not sure the other candidates are any better. Sure, they aren’t Nazis, but their politics are just as full of hate and impracticality. For example, Hillary wants to force Silicon Valley into censoring content, brushing aside complaints from those people overly concerned with “freedom of speech”. No candidate, not even Trump, is as radical as Bernie Sanders, who would dramatically reshape the economy. Trump hates Mexican works inside our country, Bernie hates Mexican workers in their own countries, championing punishing trade restrictions.Most of substantive criticisms Vox gives Trump also applies to Bernie. For example, Vox says:His view of the economy is entirely zero-sum — for Americans to win, others must lose. … His message isn’t so much that he’ll help you as he’ll hurt them… That’s Bernie’s view of the economy as well. He imagines that economy is a zero-sum game, and that for the 1% rich to prosper, they must take from the 99% of everyone else. Bernie’s entire message rests on punishing the 1% for the sin of being rich.It’s the basis of all demagoguery that you find some enemy to blame. Trump’s enemies are foreigners, whereas Bernie’s enemies are those of the wrong class. Trump is one step in the direction of the horrors of the Nazi Holocaust. Bernie is one step in the direction of the horrors of old-style Soviet and Red Chinese totalitarian states.About Trump’s dishonesty, Vox says:He lies so constantly and so fluently that it’s hard to know if he even realizes he’s lying.Not true. Trump just lies badly. He’s not the standard slick politician, who lie so fluently that we don’t even realize they are lying. Whether we find a politician’s lying to be objectionable isn’t based on any principle except whether that politician is on our side.I gave $10 to all 23 presidential candidates, and get a constant stream of emails from the candidates pumping for more money. They all sound the same, regardless of political party, as if they all read the same book “How To Run A Presidential Campaign”. For example, before New Years, they all sent essentially the same message “Help us meet this important deadline!”, as if the end of the year is some important fund-raising deadline that must be met. It isn’t, that’s a lie, but such a fluent one that you can’t precisely identify it as a lie. If I were to judge candidate honesty, based on donor e-mails, Bernie would be near the top on honesty, and Hillary would be near the bottom, with Trump unexceptionally in the middle.Vox’s biggest problem is that their attack focuses on Trump’s style more than substance. It’s a well-known logical fallacy that serious people avoid. Style is irrelevant. Trump’s substance provides us enough fodder to attack him, we don’t need to stoop to this low level. The Vox piece is great creative fiction about how nasty Trump is, missing only the standard dig about his hair, but there’s no details as to exactly why Trump’s policies are bad, such as the impractical cost of building a 2000 mile long wall between us and Mexico, or the necessity of suspending the 6th Amendment right to “due process” when deporting 20 million immigrants.Vox’s complaint about Trump’s style is mostly that he doesn’t obey the mainstream media. All politicians misspeak. There’s no way to spend that many hours a day talking to the public without making the most egregious of mistakes. The mainstream media has a way of dealing with this, forcing the politician to grovel. They resent how Trump just ignores the problem and barrels on to the next thing. That the press can’t make his mistakes stick makes them very upset.Imagine a situation where more than half the country believes in an idea, but nobody stands up and publicly acknowledges this. That’s a symptom of repressed speech. You’d think that the only suppressor of speech is the government, but that’s not true. The mainstream media is part of the establishment, and they regularly suppress speech they don’t like.I point this out because half the country, both Democrats and Republicans, support Trump’s idea of preventing Muslims from coming into our country. Sure, it’s both logically stupid and evilly racist, but that doesn’t matter, half the country supports it. Yet, nobody admits supporting the idea publicly, because as soon as they do, they’ll be punished by the mass media.Thus, the idea continues to fester, because it can’t openly be debated. People continue to believe in this bad idea because they are unpersuaded by the ad hominem that “you are such a racist”. The bedrock principle of journalism is that there are two sides to every debate. When half the country believes in a wrong idea, we have to accept that they are all probably reasonable people, and that we can change their minds if we honestly engage them in debate.This sounds like I’m repeating the “media bias” trope, which politicians like Trump use to deflect even fair media coverage they happen not to like. But it’s not left-wing bias that is the problem here.Instead, it’s that the media has become part of the establishment, with their own seat of power. Ezra Klein’s biggest achievement before Vox was JournoList, designed to help the established press wield their power at the top of the media hierarchy. Ezra Klein is the quintessential press insider. His post attacking Trump is just a typical example of how insiders attack outsiders who don’t conform. Yes, Trump deserves criticism, but based upon substance — not because he challenges how the press establishment has defined how politics should work in America.

Nothing says "establishment" as Vox’s attack on Trump

Post Syndicated from Robert Graham original http://blog.erratasec.com/2016/02/vox-is-wrong-about-trump.html

I keep seeing this Ezra Klein Vox article attacking Donald Trump. It’s wrong in every way something can be wrong. Trump is an easy target, but the Vox piece has almost no substance.Yes, it’s true that Trump proposes several unreasonable policies, such as banning Muslims from coming into this country. I’ll be the first to chime in and call Trump a racist, Nazi bastard for these things.But I’m not sure the other candidates are any better. Sure, they aren’t Nazis, but their politics are just as full of hate and impracticality. For example, Hillary wants to force Silicon Valley into censoring content, brushing aside complaints from those people overly concerned with “freedom of speech”. No candidate, not even Trump, is as radical as Bernie Sanders, who would dramatically reshape the economy. Trump hates Mexican works inside our country, Bernie hates Mexican workers in their own countries, championing punishing trade restrictions.Most of substantive criticisms Vox gives Trump also applies to Bernie. For example, Vox says:His view of the economy is entirely zero-sum — for Americans to win, others must lose. … His message isn’t so much that he’ll help you as he’ll hurt them… That’s Bernie’s view of the economy as well. He imagines that economy is a zero-sum game, and that for the 1% rich to prosper, they must take from the 99% of everyone else. Bernie’s entire message rests on punishing the 1% for the sin of being rich.It’s the basis of all demagoguery that you find some enemy to blame. Trump’s enemies are foreigners, whereas Bernie’s enemies are those of the wrong class. Trump is one step in the direction of the horrors of the Nazi Holocaust. Bernie is one step in the direction of the horrors of old-style Soviet and Red Chinese totalitarian states.About Trump’s dishonesty, Vox says:He lies so constantly and so fluently that it’s hard to know if he even realizes he’s lying.Not true. Trump just lies badly. He’s not the standard slick politician, who lie so fluently that we don’t even realize they are lying. Whether we find a politician’s lying to be objectionable isn’t based on any principle except whether that politician is on our side.I gave $10 to all 23 presidential candidates, and get a constant stream of emails from the candidates pumping for more money. They all sound the same, regardless of political party, as if they all read the same book “How To Run A Presidential Campaign”. For example, before New Years, they all sent essentially the same message “Help us meet this important deadline!”, as if the end of the year is some important fund-raising deadline that must be met. It isn’t, that’s a lie, but such a fluent one that you can’t precisely identify it as a lie. If I were to judge candidate honesty, based on donor e-mails, Bernie would be near the top on honesty, and Hillary would be near the bottom, with Trump unexceptionally in the middle.Vox’s biggest problem is that their attack focuses on Trump’s style more than substance. It’s a well-known logical fallacy that serious people avoid. Style is irrelevant. Trump’s substance provides us enough fodder to attack him, we don’t need to stoop to this low level. The Vox piece is great creative fiction about how nasty Trump is, missing only the standard dig about his hair, but there’s no details as to exactly why Trump’s policies are bad, such as the impractical cost of building a 2000 mile long wall between us and Mexico, or the necessity of suspending the 6th Amendment right to “due process” when deporting 20 million immigrants.Vox’s complaint about Trump’s style is mostly that he doesn’t obey the mainstream media. All politicians misspeak. There’s no way to spend that many hours a day talking to the public without making the most egregious of mistakes. The mainstream media has a way of dealing with this, forcing the politician to grovel. They resent how Trump just ignores the problem and barrels on to the next thing. That the press can’t make his mistakes stick makes them very upset.Imagine a situation where more than half the country believes in an idea, but nobody stands up and publicly acknowledges this. That’s a symptom of repressed speech. You’d think that the only suppressor of speech is the government, but that’s not true. The mainstream media is part of the establishment, and they regularly suppress speech they don’t like.I point this out because half the country, both Democrats and Republicans, support Trump’s idea of preventing Muslims from coming into our country. Sure, it’s both logically stupid and evilly racist, but that doesn’t matter, half the country supports it. Yet, nobody admits supporting the idea publicly, because as soon as they do, they’ll be punished by the mass media.Thus, the idea continues to fester, because it can’t openly be debated. People continue to believe in this bad idea because they are unpersuaded by the ad hominem that “you are such a racist”. The bedrock principle of journalism is that there are two sides to every debate. When half the country believes in a wrong idea, we have to accept that they are all probably reasonable people, and that we can change their minds if we honestly engage them in debate.This sounds like I’m repeating the “media bias” trope, which politicians like Trump use to deflect even fair media coverage they happen not to like. But it’s not left-wing bias that is the problem here.Instead, it’s that the media has become part of the establishment, with their own seat of power. Ezra Klein’s biggest achievement before Vox was JournoList, designed to help the established press wield their power at the top of the media hierarchy. Ezra Klein is the quintessential press insider. His post attacking Trump is just a typical example of how insiders attack outsiders who don’t conform. Yes, Trump deserves criticism, but based upon substance — not because he challenges how the press establishment has defined how politics should work in America.

Building a Near Real-Time Discovery Platform with AWS

Post Syndicated from Assaf Mentzer original https://blogs.aws.amazon.com/bigdata/post/Tx1Z6IF7NA8ELQ9/Building-a-Near-Real-Time-Discovery-Platform-with-AWS

Assaf Mentzer is a Senior Consultant for AWS Professional Services

In the spirit of the U.S presidential election of 2016, in this post I use Twitter public streams to analyze the candidates’ performance, both Republican and Democrat, in a near real-time fashion. I show you how to integrate AWS managed services—Amazon Kinesis Firehose, AWS Lambda (Python function), and Amazon Elasticsearch Service—to create an end-to-end, near real-time discovery platform.

The following screenshot is an example of a Kibana dashboard on top of geo-tagged tweet data. This screenshot was taken during the fourth republican presidential debate (November 10th, 2015).

Kibana dashboard on top of geotagged tweet data

The dashboard shows tweet data related to the presidential candidates (only tweets that contain a candidate’s name):

Top 10 Twitter mentions (@username) – you can see that Donald Trump is the most mentioned candidate

Sentiment analysis

Map visualization – Washington DC is the most active area

The dashboard has drill-down capabilities; choosing one of the sentiments in the pie chart or one of the @mentions in the bar chart changes the view of the entire dashboard accordingly. For example, you can see the sentiment analysis and geographic distribution for a specific candidate. The dashboard shows data from the last hour, and is configured to refresh the data every 30 seconds.

Because the platform built in this post collects all geo-tagged public Twitter data and filters data only in the dashboard layer, you can use the same solution for other use cases by just changing the filter search terms.

Use same solution for other use cases by changing filter search terms

Architecture

This platform has the following architecture:

A producer device (in this case, the Twitter feed) puts data into Amazon Kinesis Firehose.

Firehose automatically buffers the data (in this case, 5MB size or 5 minutes interval, whichever condition is satisfied first) and delivers the data to Amazon S3.

A Python Lambda function is triggered when a new file is created on S3 and indexes the S3 file content to Amazon Elasticsearch Service.

The Kibana application runs on top of the Elasticsearch index to provide a visual display of the data.

Platform architecture

Important: Streaming data can be pushed directly to Amazon Elasticsearch Service. The architecture described in this post is recommended when data has to be persisted on S3 for further batch/advanced analysis (lambda architecture,not related to AWS Lambda)  in addition to the near-real-time analysis on top of Elasticsearch Service, which might retain only “hot data” (last x hours).

Prerequisites

To create this platform, you’ll need an AWS account and a Twitter application. Sign in with your Twitter account and create a new application at https://apps.twitter.com/. Make sure your application is set for ‘read-only’ access and then choose Create My Access Token at the bottom of the Keys and Access Tokens tab. By this point, you should have four Twitter application keys: consumer key (API key), consumer secret (API secret), access token, and access token secret. Write down these keys.

Create Amazon Elasticsearch Service cluster

Start by creating an Amazon Elasticsearch Service cluster that will hold your data for near real-time analysis. Elasticsearch Service includes built-in support for Kibana, which is used for visualization on top of Elasticsearch Service.

Sign in to the Amazon Elasticsearch Service console.

Choose Create a new domain (or Get Started, if this is your first time in the console).

Name your domain “es-twitter-demo” and choose Next.

Keep the default selections and choose Next.

Choose the Allow open access to the domain template for the access policy and click Next.

Note: This is not a recommended approach, and should only be used for this demo. Please read the documentation for how to setup the proper permissions.

Choose Confirm and create.

Within ~10 minutes, your domain is ready. When the creation process has reached a status of Active, the domain should be associated with both an Elasticsearch Service endpoint and a Kibana URL, which you need to store for later steps.

Your domain is ready

Create an IAM role for Firehose

Use a Firehose delivery stream to ingest the Twitter streaming data and put it to Amazon S3. Before you can ingest the data into Firehose, you need to set up an IAM role to allow Firehose to call AWS services on your behalf. In this example, the Twitter feed which is your producer application creates the Firehose delivery stream based on the IAM role.

Create a new IAM role named “firehose_delivery_role” based on the following policy (please replace A_BUCKET_YOU_SETUP_FOR_THIS_DEMO with your S3 bucket):

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::A_BUCKET_YOU_SETUP_FOR_THIS_DEMO/*"
]
}
]
}

Create a Lambda function

For this example, use a Python function (lambda_function.py) that is triggered when a new file is created on S3. The function does the following:

Reads the file content

Parses the content to JSON format (Elasticsearch Service stores documents in JSON format).

Analyzes Twitter data (tweet_utils.py):

Extracts Twitter mentions (@username) from the tweet text.

Extracts sentiment based on emoticons. If there’s no emoticon in the text the function uses textblob sentiment analysis.

Loads the data to Elasticsearch Service (twitter_to_es.py) using the elasticsearch-py library.

The Python code is available in aws-big-data-blog repository.

Download the deployment package and unzip to the s3-twitter-to-es-python folder.

Modify the s3-twitter-to-es-python/config.py file by changing the value of es_host to the Elasticsearch Service endpoint of your domain.

Zip the folder content on your local environment as my-s3-twitter-to-es-python.zip (important: zip the folder content, not the folder itself).

Sign in to the Lambda console.

Choose Create a Lambda function (or Get started now if this is your first time using the service).

Choose Skip in the blueprints screen.

Name your function (e.g., s3-twitter-to-es-python).

Choose Python 2.7 runtime and upload the zip file my-s3-twitter-to-es-python.zip.

Make sure the Handler field value is lambda_function.lambda_handler.

Choose lambda_s3_exec_role (if this value does not exist, choose Create new role S3 execution role).

Keep memory at 128MB and choose a 2min timeout.

Choose Next and Create function, then wait until the function is created.

On the Event sources tab, choose Add event source.

Choose the event source type S3, select the bucket, and choose the event type Object Created (All).

Enter a value for S3 Prefix (e.g., twitter/raw-data/) to ensure the function doesn’t trigger when data is uploaded elsewhere in the bucket.

Make sure that the event source is enabled and click Submit.

Feed the producer with Twitter streaming data

Your producer is a Node.js application that connects to the Twitter feed via the Twitter stream API and puts the streaming data into Firehose. The code is available aws-big-data-blog repository.

To use the producer application, you have to install Node.js (go to https://nodejs.org to install it on your local machine). Alternatively, you can launch a t2.micro EC2 instance based on the Amazon Linux AMI and run the following command:

sudo yum -y install nodejs npm –enablerepo=epel

Download the application, unzip the file, and run npm install from the twitter-streaming-firehose-nodejs  folder.

Modify the Config.js file with your settings (change <YOUR PARAMETERS> as follows:

firehose

DeliveryStreamName – Name your stream. The app creates the delivery stream if it does not exist.

BucketARN: Use the bucket matched to the Lambda function.

RoleARN: Get your account ID from the IAM dashboard users sign-in link https://Your_AWS_Account_ID.signin.aws.amazon.com/console/. Use the Firehose role you created earlier (“firehose_delivery_role”).

Prefix: Use the same s3 prefix that you used in your Lambda function event source (e.g., twitter/raw-data/).

twitter – Enter your twitter application keys.

region – Enter your Firehose region (e.g., us-east-1, us-west-2, eu-west-1).

Make sure your aws credentials are configured under <HOME FOLDER>/.aws/credentials as follows:

[default]
aws_access_key_id=
aws_secret_access_key=

Now that your Config.js file is modified, you can open a console window and initiate execution of your program by running the following command:

node twitter_stream_producer_app

Wait a few seconds until the delivery stream is active, and then you should see Twitter data on your screen. The app collect tweets from the US but you can modify the locations in Config.js file. For more information, go to twitter geolocation.

Discover and analyze data

Wait a few minutes until Firehose has time to deliver enough files to Amazon S3 to make it interesting to review. The files should appear under the following bucket:

s3://<bucket>/<prefix>/<year>/<month>/<day>/<hour>/

Open Kibana in your browser using your Kibana URL. To start discovering the data stored in Elasticsearch Service, you need to create an index pattern pointing to your Elasticsearch index, which is like a ‘database’ in a relational database. For more information, go to What is an Elasticsearch Index?.

Create an index pattern as follows:

Create an index pattern

On the Discover tab, choose Add near the text field on the left sidebar. You should get the following result:

Start exploring the data by choosing any field in the left sidebar and filter. You can search for a specific term by replacing the asterisk (*) in the search field with your terms. You can also filter by time by choosing the Time Filter icon at the top right.

For example, you can search for the term “Trump” to discover and understand the data related to one of the candidates.

search for the term Trump to discover and understand the data related to one of the candidates

In this 2016 election discovery platform, you can analyze the performance of the presidential candidates: How many tweets they got, the sentiment of those tweets (positive/negative/neutral/confused), and how the tweets are geographically distributed (identifying politically active areas).

Because this is a near real-time discovery platform, you can measure the immediate impact of political events on the candidates’ popularity (for example, during a political debate).

Create a dashboard

To visualize candidates’ popularity in Twitter (in how many tweets a candidate was mentioned), create a top mentions bar chart.

On the Discover tab, choose the mentions field on the left sidebar.

Choose Visualize (ignore the warning).

Choose Visualize

On the X-Axis tab, change the size from 20 to 10 and choose Apply.

Choose the Save Visualization icon at the top right.

Enter a name and choose Save.

To analyze how tweets related to the 2016 election are geographically distributed in order to identify politically active areas, create a tile map.

On the Discover tab, choose the coordinates.coordinates field.

Choose Visualize.

Note: By default, in the Node.js app, tweets are collected only from the U.S.

To center the map, choose the crop  icon.

Choose Save Visualization.

To identify candidates’ popularity (or unpopularity), visualize the sentiments field. Because there are only 4 potential values (positive/negative/neutral/confused), you can use a pie chart visualization.

On the Visualize tab, choose the New Visualization icon ().

Choose Pie chart.

Choose new search, Split Slices, Terms aggregation, and the sentiments field.

Choose Apply and Save Visualization.

Combine all the visualizations into a single dashboard.

On the Dashboard tab, choose Add Visualization () at the top right corner, and select a visualization.

Repeat the previous step for all other visualizations.

Choose Save Dashboard, enter a name for your dashboard, and choose Save.

Now you can search for the presidential candidates in the data. Put the following search terms in the search filter field:

realDonaldTrump,realBenCarson,JebBush,tedcruz,ChrisChristie,JohnKasich,
GovMikeHuckabee,RandPaul,MarcoRubio,CarlyFiorina,JebBush,HillaryClinton,
MartinOMalley,BernieSanders

Search for candidates in the data

You’ve got yourself a dashboard! Select your preferred candidate in the bar chart to drill down to performance.

Conclusion

AWS managed services, like Amazon Kinesis Firehose, AWS Lambda, and Amazon Elasticsearch Service, take care of provisioning and maintaining the infrastructure components when building near real time applications and enable you to focus on your business logic.

You can quickly and easily tie these services together to create a near real-time discovery platform. For this post, we analyzed the performance of the 2016 presidential candidates, but this type of platform can be used for a variety of other use cases.

If you have questions or suggestions, please leave a comment below.

————————–

Related

Getting Started with Elasticsearch and Kibana on Amazon EMR

 

Building a Near Real-Time Discovery Platform with AWS

Post Syndicated from Assaf Mentzer original https://blogs.aws.amazon.com/bigdata/post/Tx1Z6IF7NA8ELQ9/Building-a-Near-Real-Time-Discovery-Platform-with-AWS

Assaf Mentzer is a Senior Consultant for AWS Professional Services

In the spirit of the U.S presidential election of 2016, in this post I use Twitter public streams to analyze the candidates’ performance, both Republican and Democrat, in a near real-time fashion. I show you how to integrate AWS managed services—Amazon Kinesis Firehose, AWS Lambda (Python function), and Amazon Elasticsearch Service—to create an end-to-end, near real-time discovery platform.

The following screenshot is an example of a Kibana dashboard on top of geo-tagged tweet data. This screenshot was taken during the fourth republican presidential debate (November 10th, 2015).

Kibana dashboard on top of geotagged tweet data

The dashboard shows tweet data related to the presidential candidates (only tweets that contain a candidate’s name):

Top 10 Twitter mentions (@username) – you can see that Donald Trump is the most mentioned candidate

Sentiment analysis

Map visualization – Washington DC is the most active area

The dashboard has drill-down capabilities; choosing one of the sentiments in the pie chart or one of the @mentions in the bar chart changes the view of the entire dashboard accordingly. For example, you can see the sentiment analysis and geographic distribution for a specific candidate. The dashboard shows data from the last hour, and is configured to refresh the data every 30 seconds.

Because the platform built in this post collects all geo-tagged public Twitter data and filters data only in the dashboard layer, you can use the same solution for other use cases by just changing the filter search terms.

Use same solution for other use cases by changing filter search terms

Architecture

This platform has the following architecture:

A producer device (in this case, the Twitter feed) puts data into Amazon Kinesis Firehose.

Firehose automatically buffers the data (in this case, 5MB size or 5 minutes interval, whichever condition is satisfied first) and delivers the data to Amazon S3.

A Python Lambda function is triggered when a new file is created on S3 and indexes the S3 file content to Amazon Elasticsearch Service.

The Kibana application runs on top of the Elasticsearch index to provide a visual display of the data.

Platform architecture

Important: Streaming data can be pushed directly to Amazon Elasticsearch Service. The architecture described in this post is recommended when data has to be persisted on S3 for further batch/advanced analysis (lambda architecture,not related to AWS Lambda)  in addition to the near-real-time analysis on top of Elasticsearch Service, which might retain only “hot data” (last x hours).

Prerequisites

To create this platform, you’ll need an AWS account and a Twitter application. Sign in with your Twitter account and create a new application at https://apps.twitter.com/. Make sure your application is set for ‘read-only’ access and then choose Create My Access Token at the bottom of the Keys and Access Tokens tab. By this point, you should have four Twitter application keys: consumer key (API key), consumer secret (API secret), access token, and access token secret. Write down these keys.

Create Amazon Elasticsearch Service cluster

Start by creating an Amazon Elasticsearch Service cluster that will hold your data for near real-time analysis. Elasticsearch Service includes built-in support for Kibana, which is used for visualization on top of Elasticsearch Service.

Sign in to the Amazon Elasticsearch Service console.

Choose Create a new domain (or Get Started, if this is your first time in the console).

Name your domain “es-twitter-demo” and choose Next.

Keep the default selections and choose Next.

Choose the Allow open access to the domain template for the access policy and click Next.

Note: This is not a recommended approach, and should only be used for this demo. Please read the documentation for how to setup the proper permissions.

Choose Confirm and create.

Within ~10 minutes, your domain is ready. When the creation process has reached a status of Active, the domain should be associated with both an Elasticsearch Service endpoint and a Kibana URL, which you need to store for later steps.

Your domain is ready

Create an IAM role for Firehose

Use a Firehose delivery stream to ingest the Twitter streaming data and put it to Amazon S3. Before you can ingest the data into Firehose, you need to set up an IAM role to allow Firehose to call AWS services on your behalf. In this example, the Twitter feed which is your producer application creates the Firehose delivery stream based on the IAM role.

Create a new IAM role named “firehose_delivery_role” based on the following policy (please replace A_BUCKET_YOU_SETUP_FOR_THIS_DEMO with your S3 bucket):

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::A_BUCKET_YOU_SETUP_FOR_THIS_DEMO/*"
]
}
]
}

Create a Lambda function

For this example, use a Python function (lambda_function.py) that is triggered when a new file is created on S3. The function does the following:

Reads the file content

Parses the content to JSON format (Elasticsearch Service stores documents in JSON format).

Analyzes Twitter data (tweet_utils.py):

Extracts Twitter mentions (@username) from the tweet text.

Extracts sentiment based on emoticons. If there’s no emoticon in the text the function uses textblob sentiment analysis.

Loads the data to Elasticsearch Service (twitter_to_es.py) using the elasticsearch-py library.

The Python code is available in aws-big-data-blog repository.

Download the deployment package and unzip to the s3-twitter-to-es-python folder.

Modify the s3-twitter-to-es-python/config.py file by changing the value of es_host to the Elasticsearch Service endpoint of your domain.

Zip the folder content on your local environment as my-s3-twitter-to-es-python.zip (important: zip the folder content, not the folder itself).

Sign in to the Lambda console.

Choose Create a Lambda function (or Get started now if this is your first time using the service).

Choose Skip in the blueprints screen.

Name your function (e.g., s3-twitter-to-es-python).

Choose Python 2.7 runtime and upload the zip file my-s3-twitter-to-es-python.zip.

Make sure the Handler field value is lambda_function.lambda_handler.

Choose lambda_s3_exec_role (if this value does not exist, choose Create new role S3 execution role).

Keep memory at 128MB and choose a 2min timeout.

Choose Next and Create function, then wait until the function is created.

On the Event sources tab, choose Add event source.

Choose the event source type S3, select the bucket, and choose the event type Object Created (All).

Enter a value for S3 Prefix (e.g., twitter/raw-data/) to ensure the function doesn’t trigger when data is uploaded elsewhere in the bucket.

Make sure that the event source is enabled and click Submit.

Feed the producer with Twitter streaming data

Your producer is a Node.js application that connects to the Twitter feed via the Twitter stream API and puts the streaming data into Firehose. The code is available aws-big-data-blog repository.

To use the producer application, you have to install Node.js (go to https://nodejs.org to install it on your local machine). Alternatively, you can launch a t2.micro EC2 instance based on the Amazon Linux AMI and run the following command:

sudo yum -y install nodejs npm –enablerepo=epel

Download the application, unzip the file, and run npm install from the twitter-streaming-firehose-nodejs  folder.

Modify the Config.js file with your settings (change <YOUR PARAMETERS> as follows:

firehose

DeliveryStreamName – Name your stream. The app creates the delivery stream if it does not exist.

BucketARN: Use the bucket matched to the Lambda function.

RoleARN: Get your account ID from the IAM dashboard users sign-in link https://Your_AWS_Account_ID.signin.aws.amazon.com/console/. Use the Firehose role you created earlier (“firehose_delivery_role”).

Prefix: Use the same s3 prefix that you used in your Lambda function event source (e.g., twitter/raw-data/).

twitter – Enter your twitter application keys.

region – Enter your Firehose region (e.g., us-east-1, us-west-2, eu-west-1).

Make sure your aws credentials are configured under <HOME FOLDER>/.aws/credentials as follows:

[default]
aws_access_key_id=
aws_secret_access_key=

Now that your Config.js file is modified, you can open a console window and initiate execution of your program by running the following command:

node twitter_stream_producer_app

Wait a few seconds until the delivery stream is active, and then you should see Twitter data on your screen. The app collect tweets from the US but you can modify the locations in Config.js file. For more information, go to twitter geolocation.

Discover and analyze data

Wait a few minutes until Firehose has time to deliver enough files to Amazon S3 to make it interesting to review. The files should appear under the following bucket:

s3://<bucket>/<prefix>/<year>/<month>/<day>/<hour>/

Open Kibana in your browser using your Kibana URL. To start discovering the data stored in Elasticsearch Service, you need to create an index pattern pointing to your Elasticsearch index, which is like a ‘database’ in a relational database. For more information, go to What is an Elasticsearch Index?.

Create an index pattern as follows:

Create an index pattern

On the Discover tab, choose Add near the text field on the left sidebar. You should get the following result:

Start exploring the data by choosing any field in the left sidebar and filter. You can search for a specific term by replacing the asterisk (*) in the search field with your terms. You can also filter by time by choosing the Time Filter icon at the top right.

For example, you can search for the term “Trump” to discover and understand the data related to one of the candidates.

search for the term Trump to discover and understand the data related to one of the candidates

In this 2016 election discovery platform, you can analyze the performance of the presidential candidates: How many tweets they got, the sentiment of those tweets (positive/negative/neutral/confused), and how the tweets are geographically distributed (identifying politically active areas).

Because this is a near real-time discovery platform, you can measure the immediate impact of political events on the candidates’ popularity (for example, during a political debate).

Create a dashboard

To visualize candidates’ popularity in Twitter (in how many tweets a candidate was mentioned), create a top mentions bar chart.

On the Discover tab, choose the mentions field on the left sidebar.

Choose Visualize (ignore the warning).

Choose Visualize

On the X-Axis tab, change the size from 20 to 10 and choose Apply.

Choose the Save Visualization icon at the top right.

Enter a name and choose Save.

To analyze how tweets related to the 2016 election are geographically distributed in order to identify politically active areas, create a tile map.

On the Discover tab, choose the coordinates.coordinates field.

Choose Visualize.

Note: By default, in the Node.js app, tweets are collected only from the U.S.

To center the map, choose the crop  icon.

Choose Save Visualization.

To identify candidates’ popularity (or unpopularity), visualize the sentiments field. Because there are only 4 potential values (positive/negative/neutral/confused), you can use a pie chart visualization.

On the Visualize tab, choose the New Visualization icon ().

Choose Pie chart.

Choose new search, Split Slices, Terms aggregation, and the sentiments field.

Choose Apply and Save Visualization.

Combine all the visualizations into a single dashboard.

On the Dashboard tab, choose Add Visualization () at the top right corner, and select a visualization.

Repeat the previous step for all other visualizations.

Choose Save Dashboard, enter a name for your dashboard, and choose Save.

Now you can search for the presidential candidates in the data. Put the following search terms in the search filter field:

realDonaldTrump,realBenCarson,JebBush,tedcruz,ChrisChristie,JohnKasich,
GovMikeHuckabee,RandPaul,MarcoRubio,CarlyFiorina,JebBush,HillaryClinton,
MartinOMalley,BernieSanders

Search for candidates in the data

You’ve got yourself a dashboard! Select your preferred candidate in the bar chart to drill down to performance.

Conclusion

AWS managed services, like Amazon Kinesis Firehose, AWS Lambda, and Amazon Elasticsearch Service, take care of provisioning and maintaining the infrastructure components when building near real time applications and enable you to focus on your business logic.

You can quickly and easily tie these services together to create a near real-time discovery platform. For this post, we analyzed the performance of the 2016 presidential candidates, but this type of platform can be used for a variety of other use cases.

If you have questions or suggestions, please leave a comment below.

————————–

Related

Getting Started with Elasticsearch and Kibana on Amazon EMR