Leaders from across the tech industry and beyond recently joined us for Cloudflare’s Birthday Week, helping us celebrate Cloudflare’s 10th birthday. Many of them touched on the importance of diversity and making the Internet accessible to everyone.
Here are some of the highlights.
On the value of soliciting feedback
Selina Tobaccowala Chief Digital Officer at Openfit, Co-Founder of Gixo Former President & CTO of SurveyMonkey
When you think about diversity and inclusion, unfortunately, it’s often only the loudest voice, the squeakiest wheel [who gets heard]. And what a survey allows you to do is let people’s voices be heard who are not always willing to raise their hand or speak the loudest.
So at SurveyMonkey, we always made sure that when we were thinking about user testing and we were thinking about usability testing — that it was that broad swath of the customer because you wanted people across all different segments to submit their opinion.
I think that collecting data in a way that can be anonymized, collecting data in a way that lets people have a thoughtful versus always off the cuff conversation is really important. And what we also provided was a benchmarking product, because if you don’t know how you rank and stack against other people, you don’t know if you’re doing well or not.
Bonita Stewart Vice President, Global Partnerships & Americas Partnerships Solutions of Google
It’s been part of my mission to make sure that technology is introduced particularly into the African-American community, so that people see it as a viable career and not something that’s on a path that requires a different risk profile or certain level of education. It should be accessible. So one of the things that I did — I was doing some research and I found that close to 25% of the STEM grads come from historically Black colleges. And there are many education programs we [Google] work with, but there was never anything for the students to have an immersive experience.
And the thought was, what if we had Howard West at Google? So we had a partnership with Howard University, and worked with Dr. Frederick (President of Howard University) and said: what if your students could actually spend time in the valley so that they could have an immersive experience? So they brought their faculty, along with their students. And there was just an outpouring from Google of volunteers saying, “I’d love to teach the students, is there a role for me that I can play?”
And that was in 2017. Now we have over ten schools — historically Black colleges, as well as historically Hispanic colleges and universities.
On making the Internet accessible to those who can’t afford the expense
Erik Hersman Co-founder and CEO of BRCK
BRCK makes rugged, portable devices that provide free Wi-Fi access to areas throughout Kenya and Rwanda.
We install our devices in buses and public transportation in Kenya and Rwanda. We also put them in fixed locations across the two countries. And we have a platform on it that’s much like what you’d see at an airport, where you get you get a dashboard that pops up, you watch an ad, you do a survey, you do something to earn your time and get online — which in East Africa is really important because people have time, but they don’t have money.
And so if you want to hit this demographic and allow them to have equal access to that kind of global digital ecosystem that’s out there, that we all take part in, you need to find a way that they can do so without going into their wallet. And this is the only way we found that we could do that. And so we have businesses who end up paying us [to serve advertisements, surveys, and microwork tasks] and that’s what subsidizes that cost.
Shellye Archambeau Former CEO of MetricStream Board member for Verizon, Okta, Nordstrom, and Roper Technologies
When I first came to Silicon Valley, I was shocked. I was shocked because I’m thinking, OK, I’m going to Silicon Valley — the place with innovation, new ideas, creativity, et cetera — I just knew it had to be diverse and… [it wasn’t]. And so that part was really a shock. And you know, I’m sure some things were more challenging for me. I wasn’t in anybody else’s shoes, so I don’t know if it was easier for them, but…
I’ve been in tech my entire career so I always approach things the same way. I assume that people are going to think that I’m not quite capable. Not quite competent, not quite… Just that little — I know people are going to think that.
So I try to go in the same way each time. It’s like I have to prove myself both to the people who I’m working for and to the people who are working for me. And I’ve always found that using a servant leader approach is the most effective way. To really go in and focus on the team. If I can help the team be successful, then I will be successful. So that has worked for me over and over again.
Be intentional about expanding your networks. So get out there and meet a Black investor, get out there and meet a Black founder, get out there and meet a female founder, get out there and be intentional. Don’t sit in your chair. They’re not going to come to you. Somebody gave me a beautiful analogy once and said, “It’s like fishing in the forest. There are plenty of fish there over there in the lake.”
So if you’re fishing in the forest and you’re shocked and surprised to find that there’s no fish on your hook, well, get yourself over to the lake. And you’re going to have to get up out of your chair and walk over — especially if your company or your firm doesn’t look diverse, because it’s not welcoming. And so you have to be intentional about expanding your network.
And you’re not going to get there if you just think you’ll do it. You need to treat it like OKRs, you need to make it a strategic imperative. You need to tie executive compensation to it, and do what you need to do in order to keep the focus and make sure it is appropriately resourced.
Over the past 8 months, it’s become more important for us all to stay in contact with peers around the globe. Today, I’m proud to bring you the second episode of our new video series, Verified: Presented by AWS re:Inforce. Even though we couldn’t be together this year at re:Inforce, our annual security conference, we still wanted to share some of the conversations with security leaders that would have taken place at the conference. The series showcases conversations with security leaders around the globe. In episode two, I’m talking to Emma Smith, Vodafone’s Global Cyber Security Director.
Vodafone is a global technology communications company with an optimistic culture. Their focus is connecting people and building the digital future for society. During our conversation, Emma detailed how the core values of the Global Cyber Security team were inspired by the company. “We’ve got a team of people who are ultimately passionate about protecting customers, protecting society, protecting Vodafone, protecting all of our services and our employees.” Emma shared experiences about the evolution of the security organization during her past 5 years with the company.
We were also able to touch on one of Emma’s passions, diversity and inclusion. Emma has worked to implement diversity and drive a policy of inclusion at Vodafone. In June, she was named Diversity Champion in the SC Awards Europe. In her own words: “It makes me realize that my job is to smooth the way for everybody else and to try and remove some of those obstacles or barriers that were put in their way… it means that I’m really passionate about trying to get a very diverse team in security, but also in Vodafone, so that we reflect our customer base, so that we’ve got diversity of thinking, of backgrounds, of experience, and people who genuinely feel comfortable being themselves at work—which is easy to say but really hard to create that culture of safety and belonging.”
The world is waking up to the fight against racism and I couldn’t be happier!
But let’s be clear: learning about anti-racism and being anti-racist are not the same things. Learning is a good first step and a necessary one. But if you don’t apply the knowledge you acquire, then you are not helping to move the needle.
Since the murder of George Floyd at the hands/knees of the Minneapolis police, people all over the world have been focused on Black Lives Matter and anti-racism. At Cloudflare, we’ve seen an increase in cyberattacks, we’ve heard from the leadership of Afroflare, our Employee Resource Group for employees of African descent, and we held our first ever Day On, held on June 18, Cloudflare’s employee day of learning about bias, the history and psychological effects of racism,, and how racism can get baked into algorithms.
By way of this blog post, I want to share my thoughts about where I think we go from here and how I believe we can truly embody Diversity Equity and Inclusion (DEI) in our workplace.
Is diversity recruiting the answer to anti-racism in the workplace?
Many Cloudflarians said we should increase our diversity recruiting efforts as part of the feedback we received after our Day On event. But recruiting more diverse candidates only solves one part of the problem. There are still two major hurdles to overcome:
Employees need to feel welcome and have a sense of belonging
Employee Resource Groups (ERGs) offer opportunities to foster community and a sense of belonging. But it is beyond the scope of an ERG to ensure all employees have equal opportunities for advancement. And honestly, this is where a lot of companies fall short. It’s the reason you see people sharing pictures and calling out management teams or boards of directors all over social media. Because there is a lack of visible signs of diversity at senior levels. Numbers can be misleading. A company might state, “We have 11% employees of this group or 8% of that group.” That’s great, but how many of these employees are thriving in their current roles and getting promoted at the same pace as their white counterparts? Or being compensated at the same rate as their male counterparts? The answers to those questions are much more telling, yet seldom shared.
Folks, if we are going to see meaningful change, we all need to get onboard with Diversity, Equity and Inclusion. It’s really not the type of thing that people can opt-in or out of. It won’t work. And even if, and when, everyone opts in to make DEI a priority, that won’t be enough. We won’t start to see real change until we are all living and breathing DEI day in and day out.
What does committing to DEI every day look like?
Doing something (anything) every day that flexes our DEI muscles and gets us closer to meaningful outcomes.
Mentoring a person from an underrepresented group or asking someone from an underrepresented group to mentor you.
Scheduling coffee meetings with underrepresented people around the company and finding out how you can help to amplify their voices.
Providing candid, timely coaching to underrepresented employees to help them grow in their field or area of expertise.
Learning to value the different approaches and styles that people from underrepresented groups bring to the workplace.
Watching Cloudflare TV segments like, “Everyone at the Table” which airs weekly and promotes an open dialogue about everyday topics from the perspective of people with different perspectives.
Hosting office-wide or team-wide “listening circles” where employees can share what a just and equitable workplace looks like to them.
Requesting educational opportunities for your team or whole company such as implicit bias workshops or allyship workshops. Asking if your company’s leaders have attended similar workshops.
Asking your manager/team leadership how you may help increase the diversity of your team.
Suggesting ideas for building a more inclusive culture within your team such as running meetings in a manner where everyone has an equal opportunity to speak, keeping meetings and work social activities within working hours, and regularly hosting conversations about how the team can be more inclusive.
And finally – asking the opinion of someone from an under-represented group. This one is especially important since so many of us are not present when critical decisions are being made.
Why is committing to DEI on a daily basis important?
Because it’s easier for us to do nothing. Keeping the status quo is easy. Coming together to change the system is hard work. Especially if everyone is not on board.
Because having a company full of underrepresented people who are not being heard, seen, celebrated, or promoted is not going to get us the outcomes we want. And trust me, it doesn’t take long to realize that you are not going to make it at a company. Racism, discrimination, and unfair treatment can be very subtle but under-represented people can tell when they are valued and appreciated. And when they are being set up to fail.
Because we know too much. The system is broken. Underrepresented groups have always known this. But now that it is a fact most people acknowledge and accept, we can’t ignore it. A wise woman once said, “Do the best you can until you know better. Then when you know better, do better.” (Maya Angelou)
I’ll end my commentary with this: I view DEI as a journey that we must commit to every day. Here at Cloudflare. Across the tech industry. And in our world.
Notice I used the word journey. It’s not a destination in the sense that we do these 10 things and we have “arrived”. Instead, I believe it is a journey that we will always be on with milestones and achievements to be celebrated along the way.
To help you start flexing your DEI muscle, I’m kicking off a 21-Day DEI Challenge starting today! Every day, for the next 21 days, I challenge you to share in a public forum (bonus points for doing it on LinkedIn) how you are helping to move DEI forward. You can take a small step or a really big one. What matters is that you are flexing that muscle and challenging yourself (and others) to start the journey. #21DayDEIChallenge #BeAntiRacist #MoveTheNeedle
I hope you are up for the challenge that DEI offers us because the future of our company, industry, and society depends on it.
Postscript: This blog post is dedicated to the memory of the late Congressman John Lewis, a great civil rights leader and so much more, who challenged all of us to be brave enough to make noise and get into “good trouble” for the sake of justice and equality. Rest in Power, Mr. Lewis.
I came to the United States chasing the love of my life, today my wife, in 2015.
A Spanish native speaker, Portuguese as my second language and born in the Argentine city of Córdoba more than 6,000 miles from San Francisco, there is no doubt that the definition of “Latino” fits me very well and with pride.
Cloudflare was not my first job in this country but it has been the organization in which I have learned many of the things that have allowed me to understand the corporate culture of a society totally alien to the one which I come from.
I was hired in January 2018 as the first Business Development Representative for the Latin America (LATAM) region based in San Francisco. This was long before the company went public in September 2019. The organization was looking for a specialist in Latin American markets with not only good experience and knowledge beyond languages (Spanish/Portuguese), but understanding of the economy, politics, culture, history, go-to-market strategies, etc.—I was lucky enough to be chosen as “that person”. Cloudflare invested in me to a great extent and I was amazed at the freedom I had to propose ideas and bring them to reality. I have been able to experience far beyond my role as a sales representative: I have translated marketing materials, helped with campaigns, participated in various trainings, traveled to different countries to attend conferences and visit clients, and on.
Later, I was promoted as a sales executive for the North America (NAMER) region.
I have been very fortunate to be able to closely observe the growth and maturity of the organization throughout my time here.
Today, Cloudflare has three times more employees than when I started, and I can say that much of what makes this organization unique has remained intact: Cloudflare’s core mission is to help build a better Internet, to be transparent, to protect vulnerable yet important voices online through its Project Galileo, our open door policy, the importance of investing in people, among many others.
In recent weeks I have participated in conversations around “how do we recruit more under-represented groups and avoid bias in the selection process” – This has really filled me with joy but is certainly not the first initiative of its kind at Cloudflare. The company takes pride in having several Employee Resource Groups (ERGs) created and led by employees and executive sponsors—and highly encouraged by the organization: Afroflare, Desiflare, Nativeflare, Latinflare, Proudflare, Soberflare and Vetflare are just some of those groups (we have over 16 ERGs to-date!).
At Cloudflare I have found a space where I can develop professionally, where my ideas count, and where I am allowed to make mistakes—this is not something that I have experienced in my previous roles with other employers. I am not afraid to admit that in other organizations I have felt the stigma of being a person of color and that the working conditions were unfair compared to my colleagues.
Being an immigrant (a person of color) it is a challenge to make the decision to work for organizations that don’t fully understand the value of adding more diversity to their workforce. Cloudflare is a company that does value diversity in its workforce and has demonstrated a genuine interest in recruiting as well as retaining under-represented groups and creating a collective learning environment for them and the rest of the teams within the organization.
The company is committed to increasing the diversity within our teams and we want more diverse candidates in our selection processes. To achieve this we want to invite you (or please encourage others) to visit our careers page for more information on full-time positions and internship roles at our locations across the globe and apply.
And if you have questions, I will leave you my email: [email protected] It would be a pleasure to be able to guide you and put you in touch with the right people within Cloudflare to better understand our technology and where we are going. Your experience and skills are what we need to continue improving the Internet. Come join me at Cloudflare!
Seven months ago, I joined Cloudflare to work on the Public Policy Team focusing on our democracy projects such as Project Galileo,Athenian Project and Cloudflare for Campaigns. Since I joined the team, I have learned a lot about how important cybersecurity protections are for organizations that are the target of sophisticated cyberattacks, while also learning about the complex election security environment in the United States and abroad.
It seems fitting that on International Women’s Day, a day people throughout the world are celebrating the achievements of women, we also celebrate the Centennial Anniversary of the Women’s Suffrage Movement which was the tipping point that gave many women voting rights in the United States.
Since I have been working on Cloudflare’s election security projects, this day means something extra special to me and many of my colleagues who believe that voting is the cornerstone of democracy and that having access to information regarding voting and elections is essential.
Here are five reflections that I want to share on International Women’s Day and the Centennial Anniversary of the Nineteenth Amendment which granted women the right to vote in the United States:
1. The Women’s Suffrage Movement in the United States was a decades-long battle
The Women’s Suffrage movement burst into view in the United States in 1848 at the Seneca Falls Convention, where participants introduced the notion that women deserved their own political identity and that a righteous government cannot exist without equal rights for all. These organizers passed the torch to the Congressional Union for Woman Suffrage, founded in 1913, which raised awareness through distributing pamphlets at street meetings, organizing parades, speaking tours, and petitioning Congress to pass legislation on the movement. In 1919, the Senate passed the Nineteenth Amendment and it was officially ratified on August 26, 1920.
2. Due to racial inequality, many women of color in the United States were not granted the right to vote until 1965
With the ratification of the Nineteenth Amendment in 1920, it technically granted women the right to vote. However, due to widespread inequality within the ranks of the women’s suffrage movement who primarily focused on white middle-class interests, many African Americans, Asian Americans, Hispanics, and American Indian women did not receive the right to vote until later in the century. African American women were not guaranteed the right to vote until the Voting Right Acts of 1965. During the height of the civil rights movement, The Act was signed into law by President Lyndon Johnson to prohibit racial discrimination in voting.
3. There has been a historical, global increase of women in political power
There has also been a historical increase of women in elected offices around the world. This is evident with the highest number of women ever elected to the U.S. Congress in 2018, Slovakia electing the first female president, the United Kingdom electing 220 female MPs to the House of Commons, women making up 49% of Senate of the Republic of Mexico and female Prime Ministers in Denmark, Norway, and Finland. Foundationally, the right to vote is a nonpartisan issue that benefits the interest of the country, strengthens our democracy, and with more women in office, it promotes diversity of thought and experience.
4. The spread of voting and election information has changed
The way we share information has evolved dramatically from distributing pamphlets in 1913 to millions of people sharing information on the Internet across the world in 2020. State and local governments now use their election websites as the primary source to provide up to date announcements and information on how to register to vote, find designated polling stations, and access election results. Political campaigns use their digital infrastructure to release information about their policies, accept donations, recruit volunteers and give updates on the campaign to increase supporters’ engagement.
5. Access to election information is essential to voter turnout and democracy.
Voting is a crucial tenet of our democratic system and regardless of circumstance, individuals should have access to the information necessary to exercise their rights without outside interference. At Cloudflare, our mission is to build a better Internet and part of that is ensuring that users have access to accurate, trusted information, in a safe environment. With many upcoming elections in 2020, it is important that we have confidence in the democratic processes and that starts with ensuring their website infrastructure and internal teams are secure against malicious efforts to take them offline and shake voter’s faith in democracy.
Cloudflare has made election security a priority, investing our time in the Athenian Project and Cloudflare for Campaigns as political campaigns and state and local government election websites are the first line of defense in election security. In 2016, it was reported by the Department of Homeland Security that state and local government election infrastructure in all 50 states were targeted during the Presidential election. Fast forward to 2020, we are protecting more than 170 state and local government election websites and providing services to 18 of the 32 U.S. Presidential campaigns.
Therefore, it seems fitting that we celebrate the Centennial anniversary of the Nineteenth Amendment and International Women’s Day, highlighting the achievement of women throughout history and the importance of voter confidence in the democratic institutions that many fought to participate and have their voices heard.
Working at Cloudflare has allowed me to learn how important access to information is to Internet users, and voters across the world, and I am proud to work for a company that supports strengthening democracy.
When we started at Cloudflare in the summer of 2018, we joined a small security team intent on helping it grow quickly. Cloudflare was already a successful “unicorn” startup and its profile was changing fast, providing cyber security protection for millions of Internet-facing properties and moving towards becoming a public company. We were excited to help build the team that would ensure the security of Cloudflare’s systems and the sensitive customer data that flows through them.
Competing for security talent in the tech industry – where every company is investing heavily on security – isn’t easy. But, in 18 months, we have grown our team 400% from under 10 people to almost 50 (and still hiring). We are proud that 40% of our team are women and 25% are from an under-represented minority. We believe from experience, and the research shows, that more diverse teams drive better business results and can be a better place to work.
In honor of International Women’s Day this Sunday, we wanted to share some of our lessons learned on how to build a diverse team and inclusive culture on a modern security team.
Lessons Learned Building a Diverse Team
Our effort to build a diverse team starts from the moment we draft a job posting. We try to choose language that will resonate with a broad set of candidates, and question proposed “prerequisites” for a role such as college degrees or a minimum or maximum set of experience. For example, we choose language that invites people looking to grow, and avoid militaristic terms often seen in security job descriptions.
We are open to considering multiple locations where a role can be based. Cloudflare has 13 offices around the world. We have been flexible in which office our team members can join.
We don’t rely on one hiring source. We strive for multiple hiring sources. We appreciate employee referrals and do company-wide presentations frequently to keep our team’s open positions top of mind across our 1200-person company. We love candidates who apply through Cloudflare’s online careers site because they read a Cloudflare blog post and find it interesting, or are a happy Cloudflare customer in some way. We help fuel this source of candidates by writing blog posts on a wide range of topics like here and here. We also believe in proactively reaching out to potential candidates (see more in the next point). Having three strong channels in which we are meeting candidates makes hiring a bit easier.
Proactively reaching out to passive candidates can be hard for some hiring managers. We work hard to make everyone on our team better at this. We partnered with our recruiting team to train our security team on how to use LinkedIn and Eightfold to find potential people to reach out to, and we encourage our leaders to go to meetups and the networking components of conferences and to ask respected industry peers for referrals. Our hiring managers then reach out directly with a personalized message. Our response rate is over 10% when we take the time to personalize the messaging to fit the particular possible candidate.
We think long-term about team-building and know that it might take six months to a year to close promising passive candidates. We build a relationship by sharing updates on the company as well as new problems we are trying to solve, and over time we have seen these candidates come to appreciate the company and work and then join our team.
We do proactive engagement at a number of conferences and events such as the Grace Hopper conference, AfroTech, and the International Association of Minority Cybersecurity Professionals events. We also look to build relationships and hire through organizations dedicated to placing minority candidates such as Path Forward.
We leverage our internship program to broaden our candidate pool and change perception about viable backgrounds for roles. It is easier to convince people to consider candidates from less “pedigreed” schools or with skills developed outside traditional educational paths through direct exposure to those who’ve taken different routes but share the same passion for security. We’ve found some amazing interns who’ve proven themselves on short intern stints with us, and already progressed into full-time roles.
We make sure we put together the right interview panel for the candidate: that means not only evaluating the candidate thoroughly but also giving the candidate the opportunity to look across the table at someone they feel comfortable asking “can someone like us succeed here?” You are not just using the interview process to evaluate the candidate, you are showing the candidate who you are as a team.
We hold ourselves accountable by reviewing metrics on hiring and retention. Our company leadership team gathers once a week to review data on how the entire company is doing, including looking at how we are doing at building a diverse workforce and what we can do to improve. And we don’t just look at diversity in general, we look at diversity across management, and for those in management, we also consider things like span of control.
We also get great support from our co-founders and other executives directly in our hiring process. They are always willing to spend extra time introducing people to the company, our mission, and our values. One of them will always be the last person to meet the candidate on their final interview. You can’t beat a welcoming message from the top.
Lessons Learned Creating an Inclusive Culture
The work doesn’t stop with getting a great set of people with complementary skills to come work at Cloudflare. To us, diversity is a means to the end of developing a highly productive team, not an end in itself. And, it turns out that hiring a diverse team is not a moment to celebrate success, it is a moment where leadership responsibility increases. A diverse team – made up of people from various backgrounds who don’t automatically feel at ease with one another – is not a guarantee of success. To cultivate a truly productive team requires a culture of openness to differences and a willingness for people to share their unique perspectives with people who are different.
We obsess over making sure all these great people who decided to join will also decide to stay for the long-term. We identified a number of ways we could build a community that welcomes people from different backgrounds and celebrates open debate.
We’ve moved on from the media-favored image of security professionals as “hackers” and instead focus on innovation and empathy as our core values. We believe our role is more akin to a scientist designing a cure for a disease, a teacher helping a student solve a hard problem, or a nurse responding to a person in need of treatment. While we still need the skill to be able to break things and consider the attacker mindset we are responsible for combating, we will not succeed if we cannot stand in the shoes of our customers and empathize with their plight when we roll out painful security requirements.
We talk regularly about how team members must have a stronger than usual commitment to developing the “psychological safety” necessary for everyone to believe their opinions are welcome and valued and will contribute to the greater good.
We counter the risk that security work can become very reactive by promoting a spirit of innovation. That has led to us already open sourcing multiple solutions, contributing to development of Cloudflare products, and presenting at security conferences. We are strategic about what solutions we should build ourselves and what we should buy from other vendors, always staying current on what’s new.
Our team decided to pick a logo, and we ended up choosing an orange-to-pink hued phoenix because they represent resilience and optimism: A phoenix never dies; instead, she always rises from the ashes and becomes more majestic each time around. This embodies the security mindset — we help Cloudflare bounce back from attacks and security incidents, reemerging stronger and more secure than ever. It’s easy to feel like you never “win” against constantly evolving adversaries. Knowing that we are the phoenix, destined to bounce back from whatever setbacks we face, helps us stay optimistic no matter what we face. And of course, the image of a phoenix also fits well with the core Cloudflare name and brand. Not your typical security imagery, but something that we are proud to wear on our t-shirts because it represents our team.
We encourage every member of our organization to work on something that is outside their sub-team’s subject area so they interact with the broader team and also have a sense of personal career development.
We take our work very seriously and know when to say “Let’s get down to business” like Mulan in the Disney movie (which we’ve heard team members sing), but don’t take ourselves too seriously. We keep it light around the office.
We change our seating arrangements regularly to encourage expanding relationship circles.
We ask team members across the organization to lead meetings and give presentations to the whole group.
We promote from within. Five team members have been promoted into first-time manager roles.
We have open-ended manager round-tables to discuss vulnerable topics relating to growing a diverse team.
We support our team members playing active roles in company Employee Resource Groups such as here and speaking up on topics outside our core areas of expertise.
We take time for team-building activities. Some of our best practices are to keep the events during business hours and limit those that include alcohol.
We celebrate success. In the security world, external recognition is more often given for failure than success. Most companies don’t celebrate the prevention of harm, they celebrate new products and new business. If you are not careful, a security team can feel isolated from the rest of the company because its work is not directly tied to generating revenue and even worse can be perceived as blocking progress.
One of our favorite meetings was an informal risk review session we had with our engineers during which we white-boarded what we all thought were our biggest risk areas. It was great in the moment because it was such a collaborative session where everyone felt comfortable speaking up about their fears. No two people saw things the same way, but all were open to hearing other perspectives and many of us in the moment changed how we thought about priorities. And what made it an all-time experience was how even though we may have left the meeting a bit discouraged about all we needed to do, within a week every team member had stepped forward and volunteered to work on one of the hardest challenges. Looking back a bit over a year later, we have made strong progress in reducing all the risks identified in that meeting, and we did it together as a team.
Security is hard work, and the work is never done. But bringing together a diverse team with a positive culture has helped our team get a lot of hard and stressful work done well. There is a lot more we can do to keep things moving in the right direction for our team members and company and we welcome additional suggestions for improvements in our approaches.
Around this time of year in the United States, African-Americans are often tasked with explaining why we spend 28 (or in the case of a leap year 29) days celebrating the contributions our ancestors made to this country. It may come in the form of responding to ignorant questions posed in learning environments or expressed in well-crafted articles lauding the relevancy of Black history in our modern time.
Black history is not only relevant, it is how we ensure that our heroes are not forgotten and that we have a viable future in our respective industries. As Carter G. Woodson famously said, “If a race has no history, if it has no worthwhile tradition, it becomes a negligible factor in the thought of the world, and it stands in danger of being exterminated.”
As the US leaders of Afroflare, Cloudflare’s employee resource group (ERG) for employees of African descent, we made a personal commitment this month and beyond to effectively represent, build, and grow at Cloudflare and in the tech industry.
To honor that commitment, we decided to tackle some commonly asked questions about the state of African-Americans in tech.
How many African-Americans work in tech?
The latest report on diversity in high tech from the U.S. Equal Employment Opportunity Commission (EEOC) in May 2016, indicated African-Americans made up 7.4% of the high tech workforce, with less than 1% in Manager or Executive roles.
An updated report hasn’t been released, but according to USA Today, Wired, and Fortune, Black workers made up between 1% and 6% of Black of the tech workforce from 2018-2019.
What are the barriers to increasing those numbers?
According to the EEOC, some factors driving the lack of diversity in high tech include:
The “pipeline” problem – traditional recruiting efforts depend heavily on individuals’ personal networks, which in the US, are typically not diverse.
The inhospitable culture in relevant industries and occupations forcing women and minorities to tolerate the environment or leave the field.
The reluctance of high tech companies to train new employees.
The fast-changing nature of the industry.
How can I work to create more inclusion in tech?
The future of African-Americans in tech is dependent on the concerted and consistent effort of all high tech employees and departments.
Recruiters can build a more diverse pipeline by building relationships with Historically Black Colleges and Universities (HBCUs), attending events like Afrotech, or partnering with organizations whose mission is aligned with increasing diversity in tech. We have highlighted a few notable organizations below.
Black Girls Code, founded in April 2011, focuses on teaching young African-American girls how to code in several programming languages. They hope to “bridge the digital divide” in a society that pits underrepresented, young, aspiring, girls against more privileged individuals. They aim to “provide African-American youth with the skills to occupy some of the 1.4 million computing job openings expected to be available in the U.S. by 2020, and to train 1 million girls by 2040.”
/dev/color is a non-profit foundation led by supporters of inclusion in the tech industry with a mission to “empower Black software engineers to help one another grow into industry leaders.” /dev/color does this by focusing on helping individuals find new jobs, assist with start-ups, and most importantly, ensure that engineers find a sense of purpose in their careers and in tech.
Project Include uses data and advocacy to push diversity and inclusion initiatives in high tech. They work with companies to implement diversity initiatives that focus on three core concepts: inclusion, comprehensiveness, and accountability. Project Include shares a powerful message about what it takes to ‘walk the talk’ when it comes to diversity:
“Change is hard, especially around a multidimensional issue like diversity. It is easy for all of us to become defensive and emotional, to shift the blame to others, and to feel fundamentally unheard or misunderstood. It is so uncomfortable for us to talk about the diversity problem that we have not been able to acknowledge it in full.”
These are a few of the many tech events and organizations working to solve this problem. However, doing this work takes more than just money. It involves having difficult conversations, training employees on ally skills, and supporting ERGs to celebrate and educate tech companies on different experiences, which is what we do here at Cloudflare.
As Cloudflarians, we come to work every day to build a better Internet. As Afroflarians, we want to acknowledge the current industry problems around inclusion and work tirelessly to build a better tech industry that welcomes and supports everyone. Not just during Black History Month, but always.
“Diversity leads to better outcomes… better decisions, increased innovation, stronger financial returns, and a great place to work for everyone” said Janet Van Huysse, Head of People at Cloudflare during our Q1-2020 kickoff. Veterans, people who have served in the military, are a vital element of a diverse workforce. We come in diverse shapes, sizes, colors, genders, and orientations. We bring diverse skillsets, experiences, and perspectives.
If you haven’t served in the military and haven’t worked with many veterans, here are some of the things that you can expect from your colleagues or direct reports that are veterans.
Veterans know what it means to SERVE. Indeed, it is a truism that living in service to others is a life well-lived, and that service to others is a foundation of esprit de corps. Though relatively few of us have seen combat, we have all signed a blank check to our nation made payable for any amount, up to and including our lives. This is what it means to become part of something bigger than oneself. This translates to putting our common shared interests ahead of our personal interests even when that means becoming an instrument of a foreign policy we might not agree with.
Veterans know what it means to be part of a TEAM. The phrase “I’ve got your back” means a lot when it comes from a veteran because they’re referring to the blank check. Just about every veteran you ask will tell you they really miss being part of something bigger than themselves. Companies and organizations in the civilian world that can connect the dots in this way, like Cloudflare’s mission to help build a better internet, unlock the magic that accomplishes the seemingly impossible. We see this at Cloudflare in the incredible pace of product releases AND product improvements. We see this at Cloudflare when people go to the mat for their customers and when people come together to fix a problem.
Veterans know what it means to focus on a MISSION. When people have bought into the mission, everything and everyone aligns to achieve it. We know that together, as part of a team, with solid leadership, strategy, and tactics we can accomplish the mission. Veterans will help you drop things that are extraneous to the mission and help you focus on the things that will get the mission accomplished. When a veteran on your team asks, “What problem are we trying to solve?” or “Why are we doing this?” you can bet a paycheck that they’re trying to draw a straight line to the goal of the mission.
Veterans know what it means to COMMIT. Most people view the military as a top-down, hierarchical organization because, well… it is. But most people don’t realize the level of consensus-driven decision-making that happens prior to an order being given. “Because I told you so” is just not enough of a reason for people to risk their lives or for them to effectively execute their part of a mission. So the military involves their people in mission planning where alternatives are thrashed out, often with great conviction. But when time is up and the mission commander makes their call on how the mission will be carried out, veterans know it’s time to put aside their personal opinions, get onboard, and do whatever it takes to make the plan successful. Jeff Bezos famously calls this “disagree and commit” and veterans are well-practiced in this skill.
Veterans know the importance of MORALE. We’ve seen the unit with everything going for it fail, and we’ve seen the underdog come out on top. We’ve seen troubled units turn themselves around, seemingly overnight. Veterans know how the days drag on endlessly when morale is low, and we know the joy that comes from playing their part in a group that is proud to be doing what they’re doing.
Veterans know how to make DIVERSITY work. We had to because we had no choice in who we worked with in the military. Every year one-third of the people in our units left and new people showed up out of the blue. They were selected by someone else and we couldn’t fire them. So veterans get good at onboarding themselves into new organizations and onboarding new people to their teams. Veterans get good at figuring out what people have to offer and where they have gaps so the team can reshape itself to maximize performance.
If you’re a veteran reading this, know that Cloudflare has a seat at the table for you. This can be your opportunity to transition into the civilian world, transition into tech, or accelerate your career in tech at a rocket-ship that appreciates what you have to offer.
Supporting veterans is distinct from supporting their country’s foreign policy. Most Americans recognize the mistake we made in not welcoming home veterans of the Vietnam War because we didn’t support the war at-large. Nowadays, “thank you for your service” is a meaningful phrase most veterans hear with some regularity and I’m here to tell you that it means a lot. And it especially means a lot to those veterans who carry the lifelong burden of combat action.
So we Cloudflarians that are also veterans also want to say thank you to all of YOU for welcoming us into this company, this culture, and this team that is doing so much more than helping to build a better internet. We are proud and grateful to serve alongside you at CLOUDFLARE.
Technology should be for everyone, but it has to be built by everyone to be for everyone. At Raspberry Pi, we work to empower everyone to become a tech creator and shape our collective digital future, and we hope that our work will help to increase the tech sector’s diversity.
I asked Carrie Anne Philbin, our Director of Educator Support, and Vanessa Vallely OBE, Managing Director at WeAreTheCity, about their thoughts on how we can make the tech sector more diverse, and what part role models, education, and professional development play in this.
Vanessa, WeAreTheCity helps organisations foster a strong female workforce, and provides opportunities for women to network and develop their skills. Why do you think it’s important for women and people from minority backgrounds to support each other in the professional world?
Vanessa Vallely: I believe it is important for everyone to support each other. It is important that we work as a collective and collaborate, as at the end of the day we are all trying to achieve the same goal. 17% women in tech [in the UK] is not enough.
“We want more women in tech, and we want them to represent all aspects of society.” – Vanessa Vallely OBE
We cannot be what we cannot see, therefore asking women who are already working in tech to stand up and own their role model status is a great start.
What can individuals do to address the lack of diversity in the tech sector?
Carrie Anne Philbin: Firstly, let’s recognise that we need the tech sector to be more representative of the population of the world. It’s problematic to have a small subsection of society be the controllers of a growing digital world.
Then, we need to be the change we want to see in the industry. Let’s try different avenues and then let’s be open about our challenges and successes.
VV: I believe every woman in the tech sector is a role model to future generations. There are a number of things individuals can do, for example go back to their schools and tell their tech stories, or contribute/write blogs. This doesn’t just raise their profile, it puts their story out there for others to aspire to. I think this is really important, especially if the individual is from a background where role models are less visible. There are lots of different organisations and networks that facilitate individuals getting involved in their school or early career initiatives which has made it easier to get involved and give back.
CAP: As a woman in the computing field, I think it is important that I hold the door open for other women coming through in my wake, and that I highlight where I can, great work by others.
Ever since I realised that my skills and knowledge in computing were useful and allowed me to be creative in a whole new way, I’ve championed computer science as a subject that everyone should experience. Once you’ve created your first computer program or built your first network, you’ll never want to stop.
Carrie Anne, how does your coding session at WATC’s WeAreTechWomen conference today tie into this?
CAP: At the Raspberry Pi Foundation, I spend a lot of time thinking about how to teach computing well, and about how young people can have great learning experiences so they can become the makers and creators of tomorrow.
“Technology is not a mystery, nor is it hard to learn. I want to dispel this myth for everyone regardless of gender, ethnicity, or economic status.” – Carrie Anne Philbin
During my session at WeAreTechWomen, I hope to support attendees to write their first creative python program, based on a project I wrote for Code Club to create a virtual pet. It is my hope that the session will be the spark of inspiration that gets more women and men from diverse backgrounds excited about being creators of technology.
You’ve built a career in tech education as a teacher, YouTuber, and Director at Raspberry Pi. How can beginners get comfortable creating with tech?
CAP: There isn’t anything magical about technology, and once you know this, you can start to explore with confidence, much like our ancestors when they learned that the earth was round and not flat.
“Phrases like ‘I’m not good with technology’ or ‘It’s all too complicated for me’ are reassuring to say in a society where the accepted view is that maths and science are hard, and where this view is reinforced by our media. But it is OK to be a beginner, it is OK to learn something new, and it is OK to play, explore, fail, and succeed on the journey.” – Carrie Anne Philbin
However you like to learn, be it on your own or with others, there is a way that suits you! I’ve always been quite project-minded: I have ideas about things I want to make, and then go and see if I can. This is how I stumbled across the Raspberry Pi in 2012 — it seemed like an accessible and cheap way to make my automation dreams come true. It also wasn’t too bad at randomly generating poems.
Aside from teacher-led instruction or independent exploration, another way is to learn with others in a relaxed and informal setting. If you’re a young person, then clubs like Code Club and CoderDojo are perfect. If you’re an adult, then attending a Raspberry Jam or conferences like WeAreTechWomen can provide a supportive environment.
“By being kinder to ourselves and seeing ourselves as life-long learners, it is easier to overcome insecurity and build confidence.” – Carrie Anne Philbin
A great way to approach new learning is at your own pace, and thanks to technology, we have access to online training courses with great videos, exercises, and discussion — many of these are completely free and let you connect with a community of learners as well.
How do you think educating the next generation about computing will change the makeup of the tech sector?
CAP: We’re in an exciting phase for computing education. The world has woken up to the importance of equipping our young people with the knowledge and skills for an ever increasing digital landscape. This means computer science is gaining more prominence in school curricula and giving all children the opportunity to discover the subject.
“Education can be democratising, and I expect to see the makeup of the tech sector reflect this movement in the next five to twenty years.” – Carrie Anne Philbin
Unlike physics or music, computing is still a relatively young field, so we need to do more research into what is encouraging and what isn’t, particularly when we work with young people in schools or clubs.
We’re still learning how to teach computing, and particularly programming, well to encourage greater diversity, so it’s great to see such a vast Gender Balance in Computing research project underway as part of the National Centre for Computing Education here in England. It’s not too late for schools in England to get involved in this project either…
Today is the 31st Anniversary of National Coming Out Day. I wanted to highlight the importance of this day, share coming out resources, and publish some stories of what it’s like to come out in the workplace.
About National Coming Out Day
Thirty-one years ago, on the anniversary of the National March on Washington for Lesbian and Gay Rights, we first observed National Coming Out Day as a reminder that one of our most basic tools is the power of coming out. One out of every two Americans has someone close to them who is gay or lesbian. For transgender people, that number is only one in 10.
Coming out – whether it is as lesbian, gay, bisexual, transgender or queer – STILL MATTERS. When people know someone who is LGBTQ, they are far more likely to support equality under the law. Beyond that, our stories can be powerful to each other.
Each year on October 11th, National Coming Out Day continues to promote a safe world for LGBTQ individuals to live truthfully and openly. Every person who speaks up changes more hearts and minds, and creates new advocates for equality.
Last National Coming Out Day, I shared some stories from Proudflare members in this blog post. This year, I wanted to shift our focus to the experience and challenges of coming out in the workplace. I wanted to share what it was like for some of us to come out at Cloudflare, at our first companies, and point out some of the stresses, challenges, and risks involved.
Check out these five examples below and share your own in the comments section and/or to the people around you if you’d like!
“Coming out twice” from Lily – Cloudflare Austin
While my first experience of coming out professionally was at my previous company, I thought I’d share some of the differences between my experiences at Cloudflare and this other company.
Reflecting retrospectively, coming out was so immensely liberating. I’ve never been happier, but at the time I was a mess. LGBTQIA+ people still have little to no legal protection, and having been initially largely rejected by my parents and several of my friends after coming out to them, I felt like I was at sea, floating without a raft. This feeling of unease was compounded by my particular coming out being a two part series: I wasn’t only coming out as transgender, but now also as a lesbian.
Eventually, after the physical changes became too noticeable to ignore (around 7 months ago), I worked up the courage to come out at work. The company I was working for was awful in many ways; bad culture, horrible project manager, and rampant nepotism. Despite this, I was pleasantly surprised that what I told them was almost immediately accepted. Surely this was finally a win for me? However, that initial optimism didn’t last. As time went on, it became clear that saying you accept it and actually internalizing it are completely different. I started being questioned about needed medical appointments, and I wasn’t really being treated any different than before. I still have no idea if it played into the reason they fired me for “performance” despite never bringing it up before.
As I started applying for new jobs, one thing was always on my mind: will this job be different? Thankfully the answer was yes; my experience at Cloudflare has been completely different. Through the entire hiring process, I never once had to out myself. Finally when I had to come out to use my legal name on the offer letter, Cloudflare handled it with such grace. One such example was that they went so far as to put my preferred name in quotes next to my legal one on the document. These little nuggets of kindness are visible all over the company – you can tell people are accepting and genuinely care. However, the biggest difference was that Cloudflare supports and celebrates the LGBTQIA+ community but doesn’t emphasize it. If you don’t want it to be part of your identity it doesn’t have to be. Looking to the future I hope I can just be a woman that loves women, not a trans-woman that loves women, and I think Cloudflare will be supportive of that.
A story from Mark – Cloudflare London
My coming out story? It involves an awful lot of tears in a hotel room in Peru, about three and a half thousand miles away from anyone I knew.
That probably sounds more dramatic than the reality. I’d been visiting some friends in Minnesota and I was due to head to Peru to hike the Machu Picchu trail, but a missed flight connection saw me stranded in Atlanta overnight.
A couple of months earlier, I’d kind of came out to myself. This was less a case of admitting my sexuality, but more finally learning exactly what it is. I’d only just turned 40 and, months later, I was still trying to come to terms with what it all meant; reappraising your sexuality in your 40s is not a journey for the faint of heart! I hadn’t shared it with anyone yet, but while sitting in a thuddingly dull hotel room in Atlanta, it just felt like time. So I penned my coming out letter.
The next day I boarded a plane, posted my letter to Facebook, turned off my phone, and then experienced what was, without question, The. Longest. Flight. Of. My. Life. This was followed, perhaps unsurprisingly, by the longest taxi ride of my life.
Eventually, after an eternity or two had passed, I reached my hotel room, connected to the hotel wifi and read through the messages that had accumulated over the past 8 hours or so. Messages from my friends, and family, and even my Mum. The love and support I got from all of them just about broke me. I practically dissolved in a puddle of tears as I read through everything. Decades of pent up confusion and pain washed away in those tears.
I’ll never forget the sense of acceptance I felt after all that.
As for coming out at work, well, let’s see how it goes: Hi, I’m Mark, and I’m asexual.
A story from Jacob – Cloudflare San Francisco
I started my career working in consulting in a conservative environment where I was afraid that coming out would cause me to be taken less seriously by my male coworkers. I remember casually mentioning my partner at the time to a couple of close coworkers to gauge their response. They surprised me and turned out to be very accepting and insisted that I bring him to our Holiday Party later that year. That event was the first time I came out to my entire office and I remember feeling very nervous before stepping into the room.
My anxiety was soon quelled with a warm welcome from my office leadership and from then on I didn’t feel like I was dancing around the elephant in the room. After this experience being out at work is not something I think greatly about, I have been very fortunate to work in accepting environments including at Cloudflare!
A story from Malavika – Cloudflare London
Nearly a decade has passed since I first came out in a professional setting, when I first started working at a global investment bank in Manhattan. The financial services industry was, and continues to be, known for its machismo, and at the time, gay marriage was still illegal in the United States. Despite being out in my personal life, the thought of being out at work terrified me. I already felt so profoundly different from my coworkers as a woman and a person of colour, and thus I feared that my LGBTQIA+ identity would further reduce my chances of career advancement. I had no professional role models to signal that is was okay to be LGBTQIA+ in my career.
Soon after starting this job, a close friend and university classmate invited me to a dinner for LGBTQIA+ young professionals in financial services and management consulting. I had never attended an event targeted at LGBTQIA+ professionals, let alone met an out LGBTQIA+ individual working outside of the arts, academia or nonprofit sectors. Looking around the dining room, I felt as though I had spotted a unicorn: a handful of out senior leaders at top investment banks and consulting firms sat among nearly 40 ambitious young professionals, sharing their coming out stories and providing invaluable career advice. Before this event, I would have never believed that there were so many people “like me” within the industry, and most certainly not in executive positions. For the first time, I felt a strong sense of belonging, as I finally had LGBTQIA+ role models to look up to professionally, and I no longer felt afraid of being open about my sexuality professionally.
After this event, I felt inspired and energised. Over the subsequent weeks, my authentic self began to show. My confidence and enthusiasm at work dramatically increased. I was able to build trust with my colleagues more easily, and my managers lauded me for my ability to incorporate constructive feedback quickly.
As I reflect on my career trajectory, I have not succeeded in spite of my sexuality, but rather, because of being out as a bisexual woman. Over the course of my career, I have developed strong professional relationships with senior LGBTQIA+ mentors, held leadership positions in a variety of diversity networks and organisations, and attended a number of inspiring conferences and events. Without the anxiety of having to hide an important part of my identity, I am able to be the confident, intelligent woman I truly am. And that is precisely why I am actively involved in Proudflare, Cloudflare’s employee resource group for LGBTQIA+ individuals. I strongly believe that by creating an inclusive workplace – for anyone who feels different or out of place – all employees will have the support and confidence to shine in their professional and personal lives.
A story from Chase – Cloudflare San Francisco
I really discovered my sexuality in college. Growing up, there weren’t many queer people in my life. I always had a loving family that would presumably accept me for who I was, but the lack of any queer role models in my life made me think that I was straight for quite some time. I just didn’t know what being gay was.
I always had a best friend – someone that I would end up spending all my time with. This friend wouldn’t always be the same person, but inevitably I would latch on one person and focus most of my emotional energy on our friendship. In college this friend was Daniel. We met while pledging a business fraternity our freshman year and quickly became close friends. Daniel made me feel different. I thought about him when I wasn’t with him, I wanted to be with him all the time, and most of all I would get jealous when he would date women. He saw right through me and eventually got me to open up about being gay. Our long emotional text conversation ended with me asking if he had anything he wanted to share with me (fingers crossed). His answer – “I don’t know why everyone assumes I’m gay, I’m not.” Heart = Broken.
Fast forward 6 months and we decide to live together our Junior year. I slowly started becoming more comfortable with my sexuality and began coming out. I started with my close friends, then my brother, then slightly less close friends, but kept getting hung up on my parents. Luckily, Daniel made that easier. That text from Daniel about not being gay ended up being not as set in stone as I thought. We started secretly dating for almost a year and I was the happiest I have ever been. The thrills of a secret relationship can only last so long and eventually we knew we needed to tell the world. We came out to our parents together, as a couple. We were there for each other for the good conversations, the tough conversations, the “Facebook Official” post, and coming out at our first corporate jobs (A never ending cycle). We were so fortunate to both work at warm, welcoming companies when we came out and continue to work at such companies today.
Coming out wasn’t easy but knowing I didn’t have to do it alone made it a whole heck of a lot easier. Happy four-year anniversary, Dan.
Resources for living openly
To find resources about living openly, visit the Human Rights Campaign’s Coming Out Center. I hope you’ll be true to yourselves and always be loud and proud.
To read more about Proudflare and why Cloudflare cares about inclusion in the workplace, read Proudflare’s first pride blog post.
Abstract: We review the salient evidence consistent with or predicted by the Hoyle-Wickramasinghe (H-W) thesis of Cometary (Cosmic) Biology. Much of this physical and biological evidence is multifactorial. One particular focus are the recent studies which date the emergence of the complex retroviruses of vertebrate lines at or just before the Cambrian Explosion of ~500 Ma. Such viruses are known to be plausibly associated with major evolutionary genomic processes. We believe this coincidence is not fortuitous but is consistent with a key prediction of H-W theory whereby major extinction-diversification evolutionary boundaries coincide with virus-bearing cometary-bolide bombardment events. A second focus is the remarkable evolution of intelligent complexity (Cephalopods) culminating in the emergence of the Octopus. A third focus concerns the micro-organism fossil evidence contained within meteorites as well as the detection in the upper atmosphere of apparent incoming life-bearing particles from space. In our view the totality of the multifactorial data and critical analyses assembled by Fred Hoyle, Chandra Wickramasinghe and their many colleagues since the 1960s leads to a very plausible conclusion — life may have been seeded here on Earth by life-bearing comets as soon as conditions on Earth allowed it to flourish (about or just before 4.1 Billion years ago); and living organisms such as space-resistant and space-hardy bacteria, viruses, more complex eukaryotic cells, fertilised ova and seeds have been continuously delivered ever since to Earth so being one important driver of further terrestrial evolution which has resulted in considerable genetic diversity and which has led to the emergence of mankind.
We’re usually averse to buzzwords at HackSpace magazine, but not this month: in issue 7, we’re taking a deep dive into the Internet of Things.
Internet of Things (IoT)
To many people, IoT is a shady term used by companies to sell you something you already own, but this time with WiFi; to us, it’s a way to make our builds smarter, more useful, and more connected. In HackSpace magazine #7, you can join us on a tour of the boards that power IoT projects, marvel at the ways in which other makers are using IoT, and get started with your first IoT project!
DIY retro computing: this issue, we’re taking our collective hat off to Spencer Owen. He stuck his home-brew computer on Tindie thinking he might make a bit of beer money — now he’s paying the mortgage with his making skills and inviting others to build modules for his machine. And if that tickles your fancy, why not take a crack at our Z80 tutorial? Get out your breadboard, assemble your jumper wires, and prepare to build a real-life computer!
Shameless patriotism: combine Lego, Arduino, and the car of choice for 1960 gold bullion thieves, and you’ve got yourself a groovy weekend project. We proudly present to you one man’s epic quest to add LED lights (controllable via a smartphone!) to his daughter’s LEGO Mini Cooper.
Patriotism intensifies: for the last 200-odd years, the Black Country has been a hotbed of making. Urban Hax, based in Walsall, is the latest makerspace to show off its riches in the coveted Space of the Month pages. Every space has its own way of doing things, but not every space has a portrait of Rob Halford on the wall. All hail!
Diversity: advice on diversity often boils down to ‘Be nice to people’, which might feel more vague than actionable. This is where we come in to help: it is truly worth making the effort to give people of all backgrounds access to your makerspace, so we take a look at why it’s nice to be nice, and at the ways in which one makerspace has put niceness into practice — with great results.
And there’s more!
We also show you how to easily calculate the size and radius of laser-cut gears, use a bank of LEDs to etch PCBs in your own mini factory, and use chemistry to mess with your lunch menu.
All this plus much, much more waits for you in HackSpace magazine issue 7!
Get your copy of HackSpace magazine
If you like the sound of that, you can find HackSpace magazine in WHSmith, Tesco, Sainsbury’s, and independent newsagents in the UK. If you live in the US, check out your local Barnes & Noble, Fry’s, or Micro Center next week. We’re also shipping to stores in Australia, Hong Kong, Canada, Singapore, Belgium, and Brazil, so be sure to ask your local newsagent whether they’ll be getting HackSpace magazine.
As you can see from my EC2 Instance History post, we add new instance types on a regular and frequent basis. Driven by increasingly powerful processors and designed to address an ever-widening set of use cases, the size and diversity of this list reflects the equally diverse group of EC2 customers!
Near the bottom of that list you will find the new compute-intensive C5 instances. With a 25% to 50% improvement in price-performance over the C4 instances, the C5 instances are designed for applications like batch and log processing, distributed and or real-time analytics, high-performance computing (HPC), ad serving, highly scalable multiplayer gaming, and video encoding. Some of these applications can benefit from access to high-speed, ultra-low latency local storage. For example, video encoding, image manipulation, and other forms of media processing often necessitates large amounts of I/O to temporary storage. While the input and output files are valuable assets and are typically stored as Amazon Simple Storage Service (S3) objects, the intermediate files are expendable. Similarly, batch and log processing runs in a race-to-idle model, flushing volatile data to disk as fast as possible in order to make full use of compute resources.
New C5d Instances with Local Storage In order to meet this need, we are introducing C5 instances equipped with local NVMe storage. Available for immediate use in 5 regions, these instances are a great fit for the applications that I described above, as well as others that you will undoubtedly dream up! Here are the specs:
1 x 50 GB NVMe SSD
Up to 2.25 Gbps
Up to 10 Gbps
1 x 100 GB NVMe SSD
Up to 2.25 Gbps
Up to 10 Gbps
1 x 225 GB NVMe SSD
Up to 2.25 Gbps
Up to 10 Gbps
1 x 450 GB NVMe SSD
Up to 10 Gbps
1 x 900 GB NVMe SSD
2 x 900 GB NVMe SSD
Other than the addition of local storage, the C5 and C5d share the same specs. Both are powered by 3.0 GHz Intel Xeon Platinum 8000-series processors, optimized for EC2 and with full control over C-states on the two largest sizes, giving you the ability to run two cores at up to 3.5 GHz using Intel Turbo Boost Technology.
You can use any AMI that includes drivers for the Elastic Network Adapter (ENA) and NVMe; this includes the latest Amazon Linux, Microsoft Windows (Server 2008 R2, Server 2012, Server 2012 R2 and Server 2016), Ubuntu, RHEL, SUSE, and CentOS AMIs.
Here are a couple of things to keep in mind about the local NVMe storage:
Naming – You don’t have to specify a block device mapping in your AMI or during the instance launch; the local storage will show up as one or more devices (/dev/nvme*1 on Linux) after the guest operating system has booted.
Encryption – Each local NVMe device is hardware encrypted using the XTS-AES-256 block cipher and a unique key. Each key is destroyed when the instance is stopped or terminated.
Lifetime – Local NVMe devices have the same lifetime as the instance they are attached to, and do not stick around after the instance has been stopped or terminated.
Available Now C5d instances are available in On-Demand, Reserved Instance, and Spot form in the US East (N. Virginia), US West (Oregon), EU (Ireland), US East (Ohio), and Canada (Central) Regions. Prices vary by Region, and are just a bit higher than for the equivalent C5 instances.
Elections serve two purposes. The first, and obvious, purpose is to accurately choose the winner. But the second is equally important: to convince the loser. To the extent that an election system is not transparently and auditably accurate, it fails in that second purpose. Our election systems are failing, and we need to fix them.
Today, we conduct our elections on computers. Our registration lists are in computer databases. We vote on computerized voting machines. And our tabulation and reporting is done on computers. We do this for a lot of good reasons, but a side effect is that elections now have all the insecurities inherent in computers. The only way to reliably protect elections from both malice and accident is to use something that is not hackable or unreliable at scale; the best way to do that is to back up as much of the system as possible with paper.
Recently, there have been two graphic demonstrations of how bad our computerized voting system is. In 2007, the states of California and Ohio conducted audits of their electronic voting machines. Expert review teams found exploitable vulnerabilities in almost every component they examined. The researchers were able to undetectably alter vote tallies, erase audit logs, and load malware on to the systems. Some of their attacks could be implemented by a single individual with no greater access than a normal poll worker; others could be done remotely.
Last year, the Defcon hackers’ conference sponsored a Voting Village. Organizers collected 25 pieces of voting equipment, including voting machines and electronic poll books. By the end of the weekend, conference attendees had found ways to compromise every piece of test equipment: to load malicious software, compromise vote tallies and audit logs, or cause equipment to fail.
It’s important to understand that these were not well-funded nation-state attackers. These were not even academics who had been studying the problem for weeks. These were bored hackers, with no experience with voting machines, playing around between parties one weekend.
It shouldn’t be any surprise that voting equipment, including voting machines, voter registration databases, and vote tabulation systems, are that hackable. They’re computers — often ancient computers running operating systems no longer supported by the manufacturers — and they don’t have any magical security technology that the rest of the industry isn’t privy to. If anything, they’re less secure than the computers we generally use, because their manufacturers hide any flaws behind the proprietary nature of their equipment.
We’re not just worried about altering the vote. Sometimes causing widespread failures, or even just sowing mistrust in the system, is enough. And an election whose results are not trusted or believed is a failed election.
Voting systems have another requirement that makes security even harder to achieve: the requirement for a secret ballot. Because we have to securely separate the election-roll system that determines who can vote from the system that collects and tabulates the votes, we can’t use the security systems available to banking and other high-value applications.
We can securely bank online, but can’t securely vote online. If we could do away with anonymity — if everyone could check that their vote was counted correctly — then it would be easy to secure the vote. But that would lead to other problems. Before the US had the secret ballot, voter coercion and vote-buying were widespread.
We can’t, so we need to accept that our voting systems are insecure. We need an election system that is resilient to the threats. And for many parts of the system, that means paper.
Let’s start with the voter rolls. We know they’ve already been targeted. In 2016, someone changed the party affiliation of hundreds of voters before the Republican primary. That’s just one possibility. A well-executed attack that deletes, for example, one in five voters at random — or changes their addresses — would cause chaos on election day.
Yes, we need to shore up the security of these systems. We need better computer, network, and database security for the various state voter organizations. We also need to better secure the voterregistration websites, with better design and better internet security. We need better security for the companies that build and sell all this equipment.
Multiple, unchangeable backups are essential. A record of every addition, deletion, and change needs to be stored on a separate system, on write-only media like a DVD. Copies of that DVD, or — even better — a paper printout of the voter rolls, should be available at every polling place on election day. We need to be ready for anything.
Next, the voting machines themselves. Security researchers agree that the gold standard is a voter-verified paper ballot. The easiest (and cheapest) way to achieve this is through optical-scan voting. Voters mark paper ballots by hand; they are fed into a machine and counted automatically. That paper ballot is saved, and serves as a final true record in a recount in case of problems. Touch-screen machines that print a paper ballot to drop in a ballot box can also work for voters with disabilities, as long as the ballot can be easily read and verified by the voter.
Finally, the tabulation and reporting systems. Here again we need more security in the process, but we must always use those paper ballots as checks on the computers. A manual, post-election, risk-limiting audit varies the number of ballots examined according to the margin of victory. Conducting this audit after every election, before the results are certified, gives us confidence that the election outcome is correct, even if the voting machines and tabulation computers have been tampered with. Additionally, we need better coordination and communications when incidents occur.
It’s vital to agree on these procedures and policies before an election. Before the fact, when anyone can win and no one knows whose votes might be changed, it’s easy to agree on strong security. But after the vote, someone is the presumptive winner — and then everything changes. Half of the country wants the result to stand, and half wants it reversed. At that point, it’s too late to agree on anything.
The politicians running in the election shouldn’t have to argue their challenges in court. Getting elections right is in the interest of all citizens. Many countries have independent election commissions that are charged with conducting elections and ensuring their security. We don’t do that in the US.
Instead, we have representatives from each of our two parties in the room, keeping an eye on each other. That provided acceptable security against 20th-century threats, but is totally inadequate to secure our elections in the 21st century. And the belief that the diversity of voting systems in the US provides a measure of security is a dangerous myth, because few districts can be decisive and there are so few voting-machine vendors.
We candobetter. In 2017, the Department of Homeland Security declared elections to be critical infrastructure, allowing the department to focus on securing them. On 23 March, Congress allocated $380m to states to upgrade election security.
These are good starts, but don’t go nearly far enough. The constitution delegates elections to the states but allows Congress to “make or alter such Regulations”. In 1845, Congress set a nationwide election day. Today, we need it to set uniform and strict election standards.
There’s often tension between distributed and centralized control, especially in larger organizations. While a distributed control model allows teams to move fast and to respond to specialized local needs, a central model can provide the right level of oversight for global initiatives and challenges that span all teams.
We’ve seen this challenge arise first-hand when AWS customers grow to the point where their application footprint encompasses a plethora of AWS regions, AWS accounts, development teams, and applications. They love the fact that AWS increases their agility and responsiveness, while letting them deploy resources in the most appropriate location. This diversity and scale brings new challenges when it comes to security and compliance. The freedom to innovate must be balanced by the need to protect important data and to respond quickly when threats emerge.
Over the last couple of years we have provided our customers with an increasingly broad set of options for protection including AWS WAF and AWS Shield. Our customers are making great use of all of these options, and have asked for the ability to manage them from a single, central location.
Meet AWS Firewall Manager AWS Firewall Manager is designed to help these customers! It gives them the freedom to use multiple AWS accounts and to host applications in any desired region while maintaining centralized control over their organization’s security settings and profile. Developers can develop and innovators can innovate, while the security team gains the ability to respond quickly, uniformly, and globally to potential threats and actual attacks.
With automated policy enforcement across accounts & applications, your security team can be confident that new and existing applications comply with organization-wide security policies when they use Firewall Manager. They can find applications and AWS resources that don’t measure up, and bring them into compliance in minutes.
Firewall Manager is built around named policies that contain WAF rule sets and optional AWS Shield advanced protection. Each policy applies to a specific set of AWS resources, specified by account, resource type, resource identifier, or tag. Policies can be applied automatically to all matching resources, or to a subset that you select. Policies can include WAF rules drawn from within the organization, and also those created by AWS Partners such as Imperva, F5, Trend Micro, and other AWS Marketplace vendors. This gives your security team the power to duplicate their existing on-premises security posture in the cloud.
Take the Tour Firewall Manager has three prerequisites:
Firewall Administrator – You must designate one of the AWS accounts in your organization as the administrator for Firewall Manager. This gives the account permission to deploy AWS WAF rules across the organization.
Since I don’t own an enterprise, my colleagues were kind enough to create some test accounts for me! When I open the Firewall Manager Console in the master account, I can see where I stand with respect to the first two prerequisites:
The Learn more about… button reveals the Account ID of the administrator:
I switch to that account (in a a real-world situation it is unlikely that I would have access to the master account and this one), open the console, and see that I now meet the prerequisites. I click Create policy to move ahead:
The console outlines the process for me. I need to create rules and a rule group, define a policy with the rule group, define the scope of the policy, and then actually create the policy.
At the bottom of the page I choose to create a new policy and rule group, for resources in the US East (N. Virginia) Region, and click Next:
Then I specify the conditions for my rule, choosing from the following options:
IP address or range
String or regular expression
For example, I can create a condition that blocks malicious IP addresses (this AWS Solution shows you how to use a third-party reputation list with WAF, and may be helpful):
I’ll keep this one simple, but a rule can include multiple conditions. After I have added all of them, I click Next to proceed. Now I am ready to create my rule, and I click Createrule (I can add more conditions to it later if I want):
I give my rule a name (BlockExcludedIPs), enter a CloudWatch metric name, and add my condition (ExcludeIPs), then click Create:
I can create more rules, and include them in the same rule group. Again, I’ll keep this one simple, and click Next to move ahead:
I enter a name for my group, choose the rules that will make up the group, and click Create:
I now have two rule groups (testRuleGroup was already present in the account). I name my policy and click Next to proceed:
Now I define the scope of my policy. I choose the type of resource to be protected, and indicate when the policy should be applied:
I can also use tags to include or exclude resources:
Once I have defined the scope of my policy I click Next and review it, then click Create policy:
Now that the policy is in force, the ALBs within its scope are initially noncompliant:
Within minutes, Firewall Manager applies the policy and provides me with a status report:
Start Using AWS Firewall Manager Today You can start using AWS Firewall Manager today!
If you are using AWS Shield Advanced, you have access to AWS Firewall Manager and AWS WAF at no extra charge. If not, you are charged a monthly fee for each policy in each region, along with the usual charges for WAF WebACLs, WAF Rules, and AWS Config Rules.
At AWS, our customers have always been the motivation for our innovation. In turn, we’re committed to helping them accelerate the pace of their own innovation. It was in the spirit of helping our customers achieve their objectives faster that we launched AWS Lambda in 2014, eliminating the burden of server management and enabling AWS developers to focus on business logic instead of the challenges of provisioning and managing infrastructure.
In the years since, our customers have built amazing things using Lambda and other serverless offerings, such as Amazon API Gateway, Amazon Cognito, and Amazon DynamoDB. Together, these services make it easy to build entire applications without the need to provision, manage, monitor, or patch servers. By removing much of the operational drudgery of infrastructure management, we’ve helped our customers become more agile and achieve faster time-to-market for their applications and services. By eliminating cold servers and cold containers with request-based pricing, we’ve also eliminated the high cost of idle capacity and helped our customers achieve dramatically higher utilization and better economics.
After we launched Lambda, though, we quickly learned an important lesson: A single Lambda function rarely exists in isolation. Rather, many functions are part of serverless applications that collectively deliver customer value. Whether it’s the combination of event sources and event handlers, as serverless web apps that combine APIs with functions for dynamic content with static content repositories, or collections of functions that together provide a microservice architecture, our customers were building and delivering serverless architectures for every conceivable problem. Despite the economic and agility benefits that hundreds of thousands of AWS customers were enjoying with Lambda, we realized there was still more we could do.
How Customer Feedback Inspired Us to Innovate
We heard from our customers that getting started—either from scratch or when augmenting their implementation with new techniques or technologies—remained a challenge. When we looked for serverless assets to share, we found stellar examples built by serverless pioneers that represented a multitude of solutions across industries.
There were apps to facilitate monitoring and logging, to process image and audio files, to create Alexa skills, and to integrate with notification and location services. These apps ranged from “getting started” examples to complete, ready-to-run assets. What was missing, however, was a unified place for customers to discover this diversity of serverless applications and a step-by-step interface to help them configure and deploy them.
We also heard from customers and partners that building their own ecosystems—ecosystems increasingly composed of functions, APIs, and serverless applications—remained a challenge. They wanted a simple way to share samples, create extensibility, and grow consumer relationships on top of serverless approaches.
We built the AWS Serverless Application Repository to help solve both of these challenges by offering publishers and consumers of serverless apps a simple, fast, and effective way to share applications and grow user communities around them. Now, developers can easily learn how to apply serverless approaches to their implementation and business challenges by discovering, customizing, and deploying serverless applications directly from the Serverless Application Repository. They can also find libraries, components, patterns, and best practices that augment their existing knowledge, helping them bring services and applications to market faster than ever before.
How the AWS Serverless Application Repository Inspires Innovation for All Customers
Companies that want to create ecosystems, share samples, deliver extensibility and customization options, and complement their existing SaaS services use the Serverless Application Repository as a distribution channel, producing apps that can be easily discovered and consumed by their customers. AWS partners like HERE have introduced their location and transit services to thousands of companies and developers. Partners like Datadog, Splunk, and TensorIoT have showcased monitoring, logging, and IoT applications to the serverless community.
Individual developers are also publishing serverless applications that push the boundaries of innovation—some have published applications that leverage machine learning to predict the quality of wine while others have published applications that monitor crypto-currencies, instantly build beautiful image galleries, or create fast and simple surveys. All of these publishers are using serverless apps, and the Serverless Application Repository, as the easiest way to share what they’ve built. Best of all, their customers and fellow community members can find and deploy these applications with just a few clicks in the Lambda console. Apps in the Serverless Application Repository are free of charge, making it easy to explore new solutions or learn new technologies.
Finally, we at AWS continue to publish apps for the community to use. From apps that leverage Amazon Cognito to sync user data across applications to our latest collection of serverless apps that enable users to quickly execute common financial calculations, we’re constantly looking for opportunities to contribute to community growth and innovation.
At AWS, we’re more excited than ever by the growing adoption of serverless architectures and the innovation that services like AWS Lambda make possible. Helping our customers create and deliver new ideas drives us to keep inventing ways to make building and sharing serverless apps even easier. As the number of applications in the Serverless Application Repository grows, so too will the innovation that it fuels for both the owners and the consumers of those apps. With the general availability of the Serverless Application Repository, our customers become more than the engine of our innovation—they become the engine of innovation for one another.
Data that describe processes in a spatial context are everywhere in our day-to-day lives and they dominate big data problems. Map data, for instance, whether describing networks of roads or remote sensing data from satellites, get us where we need to go. Atmospheric data from simulations and sensors underlie our weather forecasts and climate models. Devices and sensors with GPS can provide a spatial context to nearly all mobile data.
In this post, we introduce the WIND toolkit, a huge (500 TB), open weather model dataset that’s available to the world on Amazon’s cloud services. We walk through how to access this data and some of the open-source software developed to make it easily accessible. Our solution considers a subset of geospatial data that exist on a grid (raster) and explores ways to provide access to large-scale raster data from weather models. The solution uses foundational AWS services and the Hierarchical Data Format (HDF), a well adopted format for scientific data.
The approach developed here can be extended to any data that fit in an HDF5 file, which can describe sparse and dense vectors and matrices of arbitrary dimensions. This format is already popular within the physical sciences for both experimental and simulation data. We discuss solutions to gridded data storage for a massive dataset of public weather model outputs called the Wind Integration National Dataset (WIND) toolkit. We also highlight strategies that are general to other large geospatial data management problems.
Wind Integration National Dataset
As variable renewable power penetration levels increase in power systems worldwide, the importance of renewable integration studies to ensure continued economic and reliable operation of the power grid is also increasing. The WIND toolkit is the largest freely available grid integration dataset to date.
The WIND toolkit was developed by 3TIER by Vaisala. They were under a subcontract to the National Renewable Energy Laboratory (NREL) to support studies on integration of wind energy into the existing US grid. NREL is a part of a network of national laboratories for the US Department of Energy and has a mission to advance the science and engineering of energy efficiency, sustainable transportation, and renewable power technologies.
The toolkit has been used by consultants, research groups, and universities worldwide to support grid integration studies. Less traditional uses also include resource assessments for wind plants (such as those powering Amazon data centers), and studying the effects of weather on California condor migrations in the Baja peninsula.
The diversity of applications highlights the value of accessible, open public data. Yet, there’s a catch: the dataset is huge. The WIND toolkit provides simulated atmospheric (weather) data at a two-km spatial resolution and five-minute temporal resolution at multiple heights for seven years. The entire dataset is half a petabyte (500 TB) in size and is stored in the NREL High Performance Computing data center in Golden, Colorado. Making this dataset publicly available easily and in a cost-effective manner is a major challenge.
As other laboratories and public institutions work to release their data to the world, they may face similar challenges to those that we experienced. Some prior, well-intentioned efforts to release huge datasets as-is have resulted in data resources that are technically available but fundamentally unusable. They may be stored in an unintuitive format or indexed and organized to support only a subset of potential uses. Downloading hundreds of terabytes of data is often impractical. Most users don’t have access to a big data cluster (or super computer) to slice and dice the data as they need after it’s downloaded.
We aim to provide a large amount of data (50 terabytes) to the public in a way that is efficient, scalable, and easy to use. In many cases, researchers can access these huge cloud-located datasets using the same software and algorithms they have developed for smaller datasets stored locally. Only the pieces of data they need for their individual analysis must be downloaded. To make this work in practice, we worked with the HDF Group and have built upon their forthcoming Highly Scalable Data Service.
In the rest of this post, we discuss how the HSDS software was developed to use Amazon EC2 and Amazon S3 resources to provide convenient and scalable access to these huge geospatial datasets. We describe how the HSDS service has been put to work for the WIND Toolkit dataset and demonstrate how to access it using the h5pyd Python library and the REST API. We conclude with information about our ongoing work to release more ‘open’ datasets to the public using AWS services, and ways to improve and extend the HSDS with newer Amazon services like Amazon ECS and AWS Lambda.
Developing a scalable service for big geospatial data
The HDF5 file format and API have been used for many years and is an effective means of storing large scientific datasets. For example, NASA’s Earth Observing System (EOS) satellites collect more than 16 TBs of data per day using HDF5.
With the rise of the cloud, there are new challenges and opportunities to rethink how HDF5 can be enhanced to work effectively as a component in a cloud-native architecture. For the HDF Group, working with NREL has been a great opportunity to put ideas into practice with a production-size dataset.
An HDF5 file consists of a directed graph of group and dataset objects. Datasets can be thought of as a multidimensional array with support for user-defined metadata tags and compression. Typical operations on datasets would be reading or writing data to a regular subregion (a hyperslab) or reading and writing individual elements (a point selection). Also, group and dataset objects may each contain an arbitrary number of the user-defined metadata elements known as attributes.
Many people have used the HDF library in applications developed or ported to run on EC2 instances, but there are a number of constraints that often prove problematic:
The HDF5 library can’t read directly from HDF5 files stored as S3 objects. The entire file (often many GB in size) would need to be copied to local storage before the first byte can be read. Also, the instance must be configured with the appropriately sized EBS volume)
The HDF library only has access to the computational resources of the instance itself (as opposed to a cluster of instances), so many operations are bottlenecked by the library.
Any modifications to the HDF5 file would somehow have to be synchronized with changes that other instances have made to same file before writing back to S3.
Using a pattern common to many offerings from AWS, the solution to these constraints is to develop a service framework around the HDF data model. Using this model, the HDF Group has created the Highly Scalable Data Service (HSDS) that provides all the functionality that traditionally was provided by the HDF5 library. By using the service, you don’t need to manage your own file volumes, but can just read and write whatever data that you need.
Because the service manages the actual data persistence to a durable medium (S3, in this case), you don’t need to worry about disk management. Simply stream the data you need from the service as you need it. Secondly, putting the functionality behind a service allows some tricks to increase performance (described in more detail later). And lastly, HSDS allows any number of clients to access the data at the same time, enabling HDF5 to be used as a coordination mechanism for multiple readers and writers.
In designing the HSDS architecture, we gave much thought to how to achieve scalability of the HSDS service. For accessing HDF5 data, there are two different types of scaling to consider:
Multiple clients making many requests to the service
Single requests that require a significant amount of data processing
To deal with the first scaling challenge, as with most services, we considered how the service responds as the request rate increases. AWS provides some great tools that help in this regard:
Auto Scaling groups
Elastic Load Balancing load balancers
The ability of S3 to handle large aggregate throughput rates
By using a cluster of EC2 instances behind a load balancer, you can handle different client loads in a cost-effective manner.
The second scaling challenge concerns single requests that would take significant processing time with just one compute node. One example of this from the WIND toolkit would be extracting all the values in the seven-year time span for a given geographic point and dataset.
In HDF5, large datasets are typically stored as “chunks”; that is, a regular partition of the array. In HSDS, each chunk is stored as a binary object in S3. The sequential approach to retrieving the time series values would be for the service to read each chunk needed from S3, extract the needed elements, and go on to the next chunk. In this case, that would involve processing 2557 chunks, and would be quite slow.
Fortunately, with HSDS, you can speed this up quite a bit by exploiting the compute and I/O capabilities of the cluster. Upon receiving the request, the receiving node can use other nodes in the cluster to read different portions of the selection. With multiple nodes reading from S3 in parallel, performance improves as the cluster size increases.
The diagram below illustrates how this works in simplified case of four chunks and four nodes.
This architecture has worked in well in practice. In testing with the WIND toolkit and time series extraction, we observed a request latency of ~60 seconds using four nodes vs. ~5 seconds with 40 nodes. Performance roughly scales with the size of the cluster.
A planned enhancement to this is to use AWS Lambda for the worker processing. This enables 1000-way parallel reads at a reasonable cost, as you only pay for the milliseconds of CPU time used with AWS Lambda.
Public access to atmospheric data using HSDS and AWS
An early challenge in releasing the WIND toolkit data was in deciding how to subset the data for different use cases. In general, few researchers need access to the entire 0.5 PB of data and a great deal of efficiency and cost reduction can be gained by making directed constituent datasets.
NREL grid integration researchers initially extracted a 2-TB subset by selecting 120,000 points where the wind resource seemed appropriate for development. They also chose only those data important for wind applications (100-m wind speed, converted to power), the most interesting locations for those performing grid studies. To support the remaining users who needed more data resolution, we down-sampled the data to a 60-minute temporal resolution, keeping all the other variables and spatial resolution intact. This reduced dataset is 50 TB of data describing 30+ atmospheric variables of data for 7 years at a 60-minute temporal resolution.
Programmatic access is possible using the h5pyd Python library, a distributed analog to the widely used h5py library. Users interact with the datasets (variables) and slice the data from its (time x longitude x latitude) cube form as they see fit.
Examples and use cases are described in a set of Jupyter notebooks and available on GitHub:
Now you have a Jupyter notebook server running on your EC2 server.
From your laptop, create an SSH tunnel:
$ ssh –L 8888:localhost:8888 (IP address of the EC2 server)
Now, you can browse to localhost:8888 using the correct token, and interact with the notebooks as if they were local. Within the directory, there are examples for accessing the HSDS API and plotting wind and weather data using matplotlib.
Controlling access and defraying costs
A final concern is rate limiting and access control. Although the HSDS service is scalable and relatively robust, we had a few practical concerns:
How can we protect from malicious or accidental use that may lead to high egress fees (for example, someone who attempts to repeatedly download the entire dataset from S3)?
How can we keep track of who is using the data both to document the value of the data resource and to justify the costs?
If costs become too high, can we charge for some or all API use to help cover the costs?
To approach these problems, we investigated using Amazon API Gateway and its simplified integration with the AWS Marketplace for SaaS monetization as well as third-party API proxies.
In the end, we chose to use API Umbrella due to its close involvement with http://data.gov. While AWS Marketplace is a compelling option for future datasets, the decision was made to keep this dataset entirely open, at least for now. As community use and associated costs grow, we’ll likely revisit Marketplace. Meanwhile, API Umbrella provides controls for rate limiting and API key registration out of the box and was simple to implement as a front-end proxy to HSDS. Those applications that may want to charge for API use can accomplish a similar strategy using Amazon API Gateway and AWS Marketplace.
Ongoing work and other resources
As NREL and other government research labs, municipalities, and organizations try to share data with the public, we expect many of you will face similar challenges to those we have tried to approach with the architecture described in this post. Providing large datasets is one challenge. Doing so in a way that is affordable and convenient for users is an entirely more difficult goal. Using AWS cloud-native services and the existing foundation of the HDF file format has allowed us to tackle that challenge in a meaningful way.
Dr. Caleb Phillips is a senior scientist with the Data Analysis and Visualization Group within the Computational Sciences Center at the National Renewable Energy Laboratory. Caleb comes from a background in computer science systems, applied statistics, computational modeling, and optimization. His work at NREL spans the breadth of renewable energy technologies and focuses on applying modern data science techniques to data problems at scale.
Dr. Caroline Draxl is a senior scientist at NREL. She supports the research and modeling activities of the US Department of Energy from mesoscale to wind plant scale. Caroline uses mesoscale models to research wind resources in various countries, and participates in on- and offshore boundary layer research and in the coupling of the mesoscale flow features (kilometer scale) to the microscale (tens of meters). She holds a M.S. degree in Meteorology and Geophysics from the University of Innsbruck, Austria, and a PhD in Meteorology from the Technical University of Denmark.
John Readey has been a Senior Architect at The HDF Group since he joined in June 2014. His interests include web services related to HDF, applications that support the use of HDF and data visualization.Before joining The HDF Group, John worked at Amazon.com from 2006–2014 where he developed service-based systems for eCommerce and AWS.
Jordan Perr-Sauer is an RPP intern with the Data Analysis and Visualization Group within the Computational Sciences Center at the National Renewable Energy Laboratory. Jordan hopes to use his professional background in software engineering and his academic training in applied mathematics to solve the challenging problems facing America and the world.
Responding to the lack of diversity at the RSA Conference, a group of security experts have announced a competing one-day conference: OUR Security Advocates, or OURSA. It’s in San Francisco, and it’s during RSA, so you can attend both.
Use the Join button above to receive notification of new posts in this series.
In 2009, Google disclosed that they had 400 recruiters on staff working to hire nearly 10,000 people. Someday, that might be your challenge, but most companies in their early days are looking to hire a handful of people — the right people — each year. Assuming you are closer to startup stage than Google stage, let’s look at who you need to hire, when to hire them, where to find them (and how to help them find you), and how to get them to join your company.
Who Should Be Your First Hires
In later stage companies, the roles in the company have been well fleshed out, don’t change often, and each role can be segmented to focus on a specific area. A large company may have an entire department focused on just cubicle layout; at a smaller company you may not have a single person whose actual job encompasses all of facilities. At Backblaze, our CTO has a passion and knack for facilities and mostly led that charge. Also, the needs of a smaller company are quick to change. One of our first hires was a QA person, Sean, who ended up being 100% focused on data center infrastructure. In the early stage, things can shift quite a bit and you need people that are broadly capable, flexible, and most of all willing to pitch in where needed.
That said, there are times you may need an expert. At a previous company we hired Jon, a PhD in Bayesian statistics, because we needed algorithmic analysis for spam fighting. However, even that person was not only able and willing to do the math, but also code, and to not only focus on Bayesian statistics but explore a plethora of spam fighting options.
When To Hire
If you’ve raised a lot of cash and are willing to burn it with mistakes, you can guess at all the roles you might need and start hiring for them. No judgement: that’s a reasonable strategy if you’re cash-rich and time-poor.
If your cash is limited, try to see what you and your team are already doing and then hire people to take those jobs. It may sound counterintuitive, but if you’re already doing it presumably it needs to be done, you have a good sense of the type of skills required to do it, and you can bring someone on-board and get them up to speed quickly. That then frees you up to focus on tasks that can’t be done by someone else. At Backblaze, I ran marketing internally for years before hiring a VP of Marketing, making it easier for me to know what we needed. Once I was hiring, my primary goal was to find someone I could trust to take that role completely off of me so I could focus solely on my CEO duties
Where To Find the Right People
Finding great people is always difficult, particularly when the skillsets you’re looking for are highly in-demand by larger companies with lots of cash and cachet. You, however, have one massive advantage: you need to hire 5 people, not 5,000.
People You Worked With
The absolutely best people to hire are ones you’ve worked with before that you already know are good in a work situation. Consider your last job, the one before, and the one before that. A significant number of the people we recruited at Backblaze came from our previous startup MailFrontier. We knew what they could do and how they would fit into the culture, and they knew us and thus could quickly meld into the environment. If you didn’t have a previous job, consider people you went to school with or perhaps individuals with whom you’ve done projects previously.
People You Know
Hiring friends, family, and others can be risky, but should be considered. Sometimes a friend can be a “great buddy,” but is not able to do the job or isn’t a good fit for the organization. Having to let go of someone who is a friend or family member can be rough. Have the conversation up front with them about that possibility, so you have the ability to stay friends if the position doesn’t work out. Having said that, if you get along with someone as a friend, that’s one critical component of succeeding together at work. At Backblaze we’ve hired a number of people successfully that were friends of someone in the organization.
Friends Of People You Know
Your network is likely larger than you imagine. Your employees, investors, advisors, spouses, friends, and other folks all know people who might be a great fit for you. Make sure they know the roles you’re hiring for and ask them if they know anyone that would fit. Search LinkedIn for the titles you’re looking for and see who comes up; if they’re a 2nd degree connection, ask your connection for an introduction.
People You Know About
Sometimes the person you want isn’t someone anyone knows, but you may have read something they wrote, used a product they’ve built, or seen a video of a presentation they gave. Reach out. You may get a great hire: worst case, you’ll let them know they were appreciated, and make them aware of your organization.
Other Places to Find People
There are a million other places to find people, including job sites, community groups, Facebook/Twitter, GitHub, and more. Consider where the people you’re looking for are likely to congregate online and in person.
A Comment on Diversity
Hiring “People You Know” can often result in “Hiring People Like You” with the same workplace experiences, culture, background, and perceptions. Some studies have shown [1, 2, 3, 4] that homogeneous groups deliver faster, while heterogeneous groups are more creative. Also, “Hiring People Like You” often propagates the lack of women and minorities in tech and leadership positions in general. When looking for people you know, keep an eye to not discount people you know who don’t have the same cultural background as you.
Helping People To Find You
Reaching out proactively to people is the most direct way to find someone, but you want potential hires coming to you as well. To do this, they have to a) be aware of you, b) know you have a role they’re interested in, and c) think they would want to work there. Let’s tackle a) and b) first below.
I started writing our blog before we launched the product and talked about anything I found interesting related to our space. For several years now our team has owned the content on the blog and in 2017 over 1.5 million people read it. Each time we have a position open it’s published to the blog. If someone finds reading about backup and storage interesting, perhaps they’d want to dig in deeper from the inside. Many of the people we’ve recruited have mentioned reading the blog as either how they found us or as a factor in why they wanted to work here. [BTW, this is Gleb’s 200th post on Backblaze’s blog. The first was in 2008. — Editor]
Your Email List
In addition to the emails our blog subscribers receive, we send regular emails to our customers, partners, and prospects. These are largely focused on content we think is directly useful or interesting for them. However, once every few months we include a small mention that we’re hiring, and the positions we’re looking for. Often a small blurb is all you need to capture people’s imaginations whether they might find the jobs interesting or can think of someone that might fit the bill.
Your Social Involvement
Whether it’s Twitter or Facebook, Hacker News or Slashdot, your potential hires are engaging in various communities. Being socially involved helps make people aware of you, reminds them of you when they’re considering a job, and paints a picture of what working with you and your company would be like. Adam was in a Reddit thread where we were discussing our Storage Pods, and that interaction was ultimately part of the reason he left Apple to come to Backblaze.
Convincing People To Join
Once you’ve found someone or they’ve found you, how do you convince them to join? They may be currently employed, have other offers, or have to relocate. Again, while the biggest companies have a number of advantages, you might have more unique advantages than you realize.
Why Should They Join You
Here are a set of items that you may be able to offer which larger organizations might not:
Role: Consider the strengths of the role. Perhaps it will have broader scope? More visibility at the executive level? No micromanagement? Ability to take risks? Option to create their own role?
Compensation: In addition to salary, will their options potentially be worth more since they’re getting in early? Can they trade-off salary for more options? Do they get option refreshes?
Benefits: In addition to healthcare, food, and 401(k) plans, are there unique benefits of your company? One company I knew took the entire team for a one-month working retreat abroad each year.
Location: Most people prefer to work close to home. If you’re located outside of the San Francisco Bay Area, you might be at a disadvantage for not being in the heart of tech. But if you find employees close to you you’ve got a huge advantage. Sometimes it’s micro; even in the Bay Area the difference of 5 miles can save 20 minutes each way every day. We located the Backblaze headquarters in San Mateo, a middle-ground that made it accessible to those coming from San Jose and San Francisco. We also chose a downtown location near a train, restaurants, and cafes: all to make it easier and more pleasant. Also, are you flexible in letting your employees work remotely? Our systems administrator Elliott is about to embark on a long-term cross-country journey working from an RV.
Environment: Open office, cubicle, cafe, work-from-home? Loud/quiet? Social or focused? 24×7 or work-life balance? Different environments appeal to different people.
Team: Who will they be working with? A company with 100,000 people might have 100 brilliant ones you’d want to work with, but ultimately we work with our core team. Who will your prospective hires be working with?
Market: Some people are passionate about gaming, others biotech, still others food. The market you’re targeting will get different people excited.
Product: Have an amazing product people love? Highlight that. If you’re lucky, your potential hire is already a fan.
Mission: Curing cancer, making people happy, and other company missions inspire people to strive to be part of the journey. Our mission is to make storing data astonishingly easy and low-cost. If you care about data, information, knowledge, and progress, our mission helps drive all of them.
Culture: I left this for last, but believe it’s the most important. What is the culture of your company? Finding people who want to work in the culture of your organization is critical. If they like the culture, they’ll fit and continue it. We’ve worked hard to build a culture that’s collaborative, friendly, supportive, and open; one in which people like coming to work. For example, the five founders started with (and still have) the same compensation and equity. That started a culture of “we’re all in this together.” Build a culture that will attract the people you want, and convey what the culture is.
Writing The Job Description
Most job descriptions focus on the all the requirements the candidate must meet. While important to communicate, the job description should first sell the job. Why would the appropriate candidate want the job? Then share some of the requirements you think are critical. Remember that people read not just what you say but how you say it. Try to write in a way that conveys what it is like to actually be at the company. Ahin, our VP of Marketing, said the job description itself was one of the things that attracted him to the company.
Much can be said about interviewing well. I’m just going to say this: make sure that everyone who is interviewing knows that their job is not only to evaluate the candidate, but give them a sense of the culture, and sell them on the company. At Backblaze, we often have one person interview core prospects solely for company/culture fit.
Hiring success shouldn’t be defined by finding and hiring the right person, but instead by the right person being successful and happy within the organization. Ensure someone (usually their manager) provides them guidance on what they should be concentrating on doing during their first day, first week, and thereafter. Giving new employees opportunities and guidance so that they can achieve early wins and feel socially integrated into the company does wonders for bringing people on board smoothly
Our Director of Production Systems, Chris, said to me the other day that he looks for companies where he can work on “interesting problems with nice people.” I’m hoping you’ll find your own version of that and find this post useful in looking for your early and critical hires.
Of course, I’d be remiss if I didn’t say, if you know of anyone looking for a place with “interesting problems with nice people,” Backblaze is hiring. 😉
Cerberus Technologies, in their own words: Cerberus is a company founded in 2017 by a team of visionary iGaming veterans. Our mission is simple – to offer the best tech solutions through a data-driven and a customer-first approach, delivering innovative solutions that go against traditional forms of working and process. This mission is based on the solid foundations of reliability, flexibility and security, and we intend to fundamentally change the way iGaming and other industries interact with technology.
Over the years, I have developed and created a number of data warehouses from scratch. Recently, I built a data warehouse for the iGaming industry single-handedly. To do it, I used the power and flexibility of Amazon Redshift and the wider AWS data management ecosystem. In this post, I explain how I was able to build a robust and scalable data warehouse without the large team of experts typically needed.
In two of my recent projects, I ran into challenges when scaling our data warehouse using on-premises infrastructure. Data was growing at many tens of gigabytes per day, and query performance was suffering. Scaling required major capital investment for hardware and software licenses, and also significant operational costs for maintenance and technical staff to keep it running and performing well. Unfortunately, I couldn’t get the resources needed to scale the infrastructure with data growth, and these projects were abandoned. Thanks to cloud data warehousing, the bottleneck of infrastructure resources, capital expense, and operational costs have been significantly reduced or have totally gone away. There is no more excuse for allowing obstacles of the past to delay delivering timely insights to decision makers, no matter how much data you have.
With Amazon Redshift and AWS, I delivered a cloud data warehouse to the business very quickly, and with a small team: me. I didn’t have to order hardware or software, and I no longer needed to install, configure, tune, or keep up with patches and version updates. Instead, I easily set up a robust data processing pipeline and we were quickly ingesting and analyzing data. Now, my data warehouse team can be extremely lean, and focus more time on bringing in new data and delivering insights. In this post, I show you the AWS services and the architecture that I used.
Handling data feeds
I have several different data sources that provide everything needed to run the business. The data includes activity from our iGaming platform, social media posts, clickstream data, marketing and campaign performance, and customer support engagements.
To handle the diversity of data feeds, I developed abstract integration applications using Docker that run on Amazon EC2 Container Service (Amazon ECS) and feed data to Amazon Kinesis Data Streams. These data streams can be used for real time analytics. In my system, each record in Kinesis is preprocessed by an AWS Lambda function to cleanse and aggregate information. My system then routes it to be stored where I need on Amazon S3 by Amazon Kinesis Data Firehose. Suppose that you used an on-premises architecture to accomplish the same task. A team of data engineers would be required to maintain and monitor a Kafka cluster, develop applications to stream data, and maintain a Hadoop cluster and the infrastructure underneath it for data storage. With my stream processing architecture, there are no servers to manage, no disk drives to replace, and no service monitoring to write.
Setting up a Kinesis stream can be done with a few clicks, and the same for Kinesis Firehose. Firehose can be configured to automatically consume data from a Kinesis Data Stream, and then write compressed data every N minutes to Amazon S3. When I want to process a Kinesis data stream, it’s very easy to set up a Lambda function to be executed on each message received. I can just set a trigger from the AWS Lambda Management Console, as shown following.
Regardless of the format I receive the data from our partners, I can send it to Kinesis as JSON data using my own formatters. After Firehose writes this to Amazon S3, I have everything in nearly the same structure I received but compressed, encrypted, and optimized for reading.
This data is automatically crawled by AWS Glue and placed into the AWS Glue Data Catalog. This means that I can immediately query the data directly on S3 using Amazon Athena or through Amazon Redshift Spectrum. Previously, I used Amazon EMR and an Amazon RDS–based metastore in Apache Hive for catalog management. Now I can avoid the complexity of maintaining Hive Metastore catalogs. Glue takes care of high availability and the operations side so that I know that end users can always be productive.
Working with Amazon Athena and Amazon Redshift for analysis
I found Amazon Athena extremely useful out of the box for ad hoc analysis. Our engineers (me) use Athena to understand new datasets that we receive and to understand what transformations will be needed for long-term query efficiency.
For our data analysts and data scientists, we’ve selected Amazon Redshift. Amazon Redshift has proven to be the right tool for us over and over again. It easily processes 20+ million transactions per day, regardless of the footprint of the tables and the type of analytics required by the business. Latency is low and query performance expectations have been more than met. We use Redshift Spectrum for long-term data retention, which enables me to extend the analytic power of Amazon Redshift beyond local data to anything stored in S3, and without requiring me to load any data. Redshift Spectrum gives me the freedom to store data where I want, in the format I want, and have it available for processing when I need it.
To load data directly into Amazon Redshift, I use AWS Data Pipeline to orchestrate data workflows. I create Amazon EMR clusters on an intra-day basis, which I can easily adjust to run more or less frequently as needed throughout the day. EMR clusters are used together with Amazon RDS, Apache Spark 2.0, and S3 storage. The data pipeline application loads ETL configurations from Spring RESTful services hosted on AWS Elastic Beanstalk. The application then loads data from S3 into memory, aggregates and cleans the data, and then writes the final version of the data to Amazon Redshift. This data is then ready to use for analysis. Spark on EMR also helps with recommendations and personalization use cases for various business users, and I find this easy to set up and deliver what users want. Finally, business users use Amazon QuickSight for self-service BI to slice, dice, and visualize the data depending on their requirements.
Each AWS service in this architecture plays its part in saving precious time that’s crucial for delivery and getting different departments in the business on board. I found the services easy to set up and use, and all have proven to be highly reliable for our use as our production environments. When the architecture was in place, scaling out was either completely handled by the service, or a matter of a simple API call, and crucially doesn’t require me to change one line of code. Increasing shards for Kinesis can be done in a minute by editing a stream. Increasing capacity for Lambda functions can be accomplished by editing the megabytes allocated for processing, and concurrency is handled automatically. EMR cluster capacity can easily be increased by changing the master and slave node types in Data Pipeline, or by using Auto Scaling. Lastly, RDS and Amazon Redshift can be easily upgraded without any major tasks to be performed by our team (again, me).
In the end, using AWS services including Kinesis, Lambda, Data Pipeline, and Amazon Redshift allows me to keep my team lean and highly productive. I eliminated the cost and delays of capital infrastructure, as well as the late night and weekend calls for support. I can now give maximum value to the business while keeping operational costs down. My team pushed out an agile and highly responsive data warehouse solution in record time and we can handle changing business requirements rapidly, and quickly adapt to new data and new user requests.
Stephen Borg is the Head of Big Data and BI at Cerberus Technologies. He has a background in platform software engineering, and first became involved in data warehousing using the typical RDBMS, SQL, ETL, and BI tools. He quickly became passionate about providing insight to help others optimize the business and add personalization to products. He is now the Head of Big Data and BI at Cerberus Technologies.
The collective thoughts of the interwebz
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.