Every day, all across the Internet, something bad but entirely normal happens: thousands of origin servers go down, resulting in connection errors and frustrated users. Cloudflare’s users collectively spend over four and a half years each day waiting for unreachable origin servers to respond with error messages. But visitors don’t want to see error pages, they want to see content!
Today is exciting for all those who want the Internet to be stronger, more resilient, and have important redundancies: Cloudflare is pleased to announce a partnership with the Internet Archive to bring new functionality to our Always Online service.
Always Online serves as insurance for our customers’ websites. Should a customer’s origin go offline, timeout, or otherwise break, Always Online is there to step in and serve archived copies of webpages to visitors. The Internet Archive is a nonprofit organization that runs the Wayback Machine, a service which saves snapshots of billions of websites across the Internet. By partnering with the Internet Archive, Cloudflare is able to seamlessly deliver responses for unreachable websites from the Internet Archive, while the Internet Archive can continue their mission of archiving the web to provide access to all knowledge.
Enabling Always Online in the Cloudflare dashboard allows us to work with the Internet Archive to save a copy of a website to the Wayback Machine. When a website’s origin is down, Cloudflare will go to the Internet Archive to retrieve the most recently archived version of the site, so that visitors will still be able to view the site’s content.
Trying to reach a busted origin
When a person visits a Cloudflare website, a request is made from their laptop/phone/tablet/smart fridge to Cloudflare’s edge. Our edge first looks to see if we can respond with cached content; if the requested content is not in cache, or is determined to be expired, we then obtain a fresh copy from the origin. As part of fulfilling an uncached/expired origin fetch, we also update our cache to allow subsequent requests to be served to visitors faster and more securely.If we are unable to reach the origin, our edge tries a few more times to connect before marking the origin as being down and serving an error page to the visitor. Receiving an error page is not ideal for anyone, so we try really hard to ensure that visitors to websites using Cloudflare can get some content, even if an origin is struggling.
A brief history of Always Online
When Cloudflare started 10 years ago, most of our customers were small and running on hosts that were subject to frequent downtime. These early customers feared that their host may go down at the same time a search engine was indexing their site. The search engine’s crawler would report the downed site as non-responsive and the site would drop in their search ranking. Always Online was born from that concern.
Through operating Always Online over the past 10 years, we’ve learned that fighting Internet downtime with simple, unobtrusive tools was something that our customers and their users deeply value. Though some features have undergone rewrite upon rewrite, other parts of our code have remained relatively untouched by the sands of time, a testament to their robustness. For example, Always Online clearly shows a banner indicating that it is serving an archived version of the page due to the origin being unreachable, and this transparency is well-received by both website owners and visitors.
We recently set out to make Always Online even better. We wanted to preserve what customers loved — as seamless an experience as possible for their users when their origin servers are down — while increasing the amount of content available through Always Online, ensuring it is as fresh as possible, and performing this archiving in a way that helps make the Internet a better place.
Enter the Internet Archive
Partnering with the Internet Archive’s Wayback Machine to power the next generation of Always Online accomplishes all of these goals. The Internet Archive’s mission is to provide universal access to all knowledge. Since 1996, the Internet Archive’s Wayback Machine has been archiving much of the public Web: preserving and making available millions of websites and pages that would otherwise be lost. In pursuit of that mission, they have archived more than 468 billion web pages, amounting to more than 45 petabytes of information.
Always Online’s integration with the Internet Archive will help the Archive expand their record of the Internet; many of the domains that opt-in to Always Online functionality may not have been otherwise discovered by the Archive’s crawler. And for Cloudflare customers, the Archive will seamlessly provide visitors access to content that would otherwise be errors.
In other words, Cloudflare partnering with the Internet Archive makes the Internet better, stronger, and more available to everyone.
“Through our partnership with Cloudflare, we are learning about, and archiving, webpages we might not have otherwise known about, and by integrating with Cloudflare’s Always Online service, archives of those pages are available to people trying to access them if they become unavailable via the live web” —Mark Graham, Director of the Internet Archive’s Wayback Machine
“We are excited to work with Cloudflare and expect this partnership to bring important redundancy to the Internet and allow for us to advance our ongoing efforts to make the Internet more useful and reliable.” —Brewster Kahle, Founder and Digital Librarian of the Internet Archive
How does the new Always Online work behind the scenes?
Upgrading to the new Always Online in the Cloudflare dashboard allows us to share some basic information about your website with the Internet Archive (like hostname and popular URLs), so they can begin to crawl and archive your website at regular intervals. This information sharing and crawling ensures content is available to Always Online and also serves to deepen the library of content available directly through the Archive.
If your origin goes down or is unreachable, Cloudflare’s edge will return a status code in the 520 to 527 range, indicating an issue connecting to the origin. When this happens, Cloudflare will first look to the local edge datacenter to see if there is a stale or expired version of content we can serve to the website visitor. If there isn’t a version in the local cache, Cloudflare will then go to the Internet Archive and fetch the most recently archived version of the site to serve to your visitors. When that happens, Always Online serves the archived content with a banner to let your visitors know that your origin is having problems. The banner allows for your visitors to check and see if your origin is back online with a single click. While dynamic content that requires communication with an origin server will still show an error to visitors (e.g. web applications or shopping carts), basic content will often be available with Always Online.
Enabling the new Always Online
For now, the old Always Online service will still be available, but we plan to fully transition to the Internet Archive-backed version soon.
Cloudflare customers can enable Always Online in the dashboard:
For more about Always Online, and how it works, please check out our documentation.
To get started using Always Online, please log into your Cloudflare dashboard and toggle it on.
Please see the Internet Archive’s announcement of our partnership here.
To help improve Always Online, or other parts of our slice of the Internet, drop us a line.
Remember 2016? Pokemon Go was all the rage, we lost Prince, and there were surprising election results in both the UK and US. Back in 2016, Blackbird Technologies was notorious in the world of patent litigation. It was a boutique law firm that was one of the top ten most active patent trolls, filing lawsuits against more than 50 different defendants in a single year.
In October 2016, Blackbird was looking to acquire additional patents for their portfolio when they found an incredibly broad software patent with the ambiguous title, “PROVIDING AN INTERNET THIRD PARTY DATA CHANNEL.” They acquired this patent from its owner for $1 plus “other good and valuable consideration.” A little later, in March 2017, Blackbird decided to assert that patent against Cloudflare.
As we have explained previously, patent trolls benefit from a problematic incentive structure that allows them to take vague or abstract patents that they have no intention of developing and assert them as broadly as possible. Instead, these trolls collect licensing fees or settlements from companies who are otherwise trying to start a business, produce useful products, and create good jobs. Companies facing such claims usually convince themselves that settlements in the tens or hundreds of thousands of dollars are quicker and cheaper outcomes than facing years of litigation and millions of dollars in attorneys fees.
The following is how we worked to upend this asymmetric incentive structure.
The Game Plan
After we were sued by Blackbird, we decided that we wouldn’t roll over. We decided we would do our best to turn the incentive structure on its head and make patent trolls think twice before attempting to take advantage of the system. We created Project Jengo in an effort to remove this economic asymmetry from the litigation. In our initial blog post we suggested we could level the playing field by: (i) defending ourselves vigorously against the patent lawsuit instead of rolling over and paying a licensing fee or settling, (ii) funding awards for crowdsourced prior art that could be used to invalidate any of Blackbird’s patents, not just the one asserted against Cloudflare, and (iii) asking the relevant bar associations to investigate what we considered to be Blackbird’s violations of the rules of professional conduct for attorneys.
How’d we do?
As promised, we fought the lawsuit vigorously. And as explained in a blog post earlier this year, we won as convincing a victory as one could in federal litigation at both the trial and appellate levels. In early 2018, the District Court for the Northern District of California dismissed the case Blackbird brought against us on subject matter eligibility grounds in response to an Alice motion. In a mere two-page order, Judge Vince Chhabria held that “[a]bstract ideas are not patentable” and Blackbird’s assertion of the patent “attempts to monopolize the abstract idea of monitoring a preexisting data stream between a server and a client.” Essentially, the case was rejected before it ever really started because the court found Blackbird’s patent to be invalid.
Blackbird appealed that decision to the Court of Appeals for the Federal Circuit, which unceremoniously affirmed the lower court decision dismissing the appeal just three days after the appellate argument was heard. Following this ruling, we celebrated.
As noted in our earlier blog post, although we won the litigation as quickly and easily as possible, the federal litigation process still lasted nearly two years, involved combined legal filings of more than 1,500 pages, and ran up considerable legal expenses. Blackbird’s right to seek review of the decision by the US Supreme Court expired this summer, so the case is now officially over. As we’ve said from the start, we only intended to pursue Project Jengo as long as the case remained active.
Even though we won decisively in court, that alone is not enough to change the incentive structure around patent troll suits. Patent trolls are repeat players who don’t have significant operations, so the costs of litigation and discovery are much less for them.
Funding Crowdsourced Prior Art to Invalidate Blackbird Patents
An integral part of our strategy against Blackbird was to engage our community to help us locate prior art that we could use to invalidate all of Blackbird’s patents. One of the most powerful legal arguments against the validity of a patent is that the invention claimed in the patent was already known or made public somewhere else (“prior art”). A collection of prior art on all the Blackbird patents could be used by anyone facing a lawsuit from Blackbird to defend themselves. The existence of an organized and accessible library of prior art would diminish the overall value of the Blackbird patent portfolio. That sort of risk to the patent portfolio was the kind of thing that would nudge the incentive structure in the other direction. Although the financial incentives made possible by the US legal system may support patent trolls, we knew our secret weapon was a very smart, very motivated community that loathed the extortionary activities of patent trolls and wanted to fight back.
And boy, were we right! We established a prior art bounty to pay cash rewards for prior art submissions that read on the patent Blackbird asserted against Cloudflare, as well as any of Blackbird’s other patents.
We received hundreds of submissions across Blackbird’s portfolio of patents. We were very impressed with the quality of those submissions and think they call the validity of a number of those patents into question. All the relevant submissions we collected can be found here sorted by patent number, and we hope they are put to good use by other parties sued by Blackbird. Additionally, we’ve already forwarded prior art from the collection to a handful of companies and organizations that reached out to us because they were facing cases from Blackbird.
A high-level breakdown of the submissions:
We received 275 total unique submissions from 155 individuals on 49 separate patents, and we received multiple submissions on 26 patents.
40.1% of the total submissions related to the ’335 patent asserted against Cloudflare.
The second highest concentration of prior art submissions (14.9% of total) relate to PUB20140200078 titled “Video Game Including User Determined Location Information.” The vast majority of these submissions note the similarity between the patent’s claims and the Niantic game Ingress.
A few interesting examples of prior art that were submitted that we think are particularly damaging to some of the Blackbird patents:
Internet based resource retrieval system (No. 8996546) The first two sentences of this 2004 patent’s abstract summarize the patent as a “resource retrieval system compris[ing] a server having a searchable database wherein users can readily access region-based publications similar to, but not necessarily limited to, printed telephone directories. The resource retrieval system communicates with at least one user system, preferably via the Internet.”
The Project Jengo community reviewed the incredibly broad language in the patent claims and submitted a reference to an online phone book that allowed for the searching of local results from an online AT&T database. The submission is a link to an archive of a webpage from the year 2000, potentially calling into question the Blackbird patent on eligibility grounds.
Illuminated product packaging (No. 7086751) This patent seeks protection for packaging “intended to hold a product for sale. The product package includes one or more light sources disposed therein and configured to direct light through one or more openings in the exterior of the product package, in order to entice customers to purchase the product.”
In one of the more interesting Project Jengo submissions we received, the following information was provided: The CD packaging for Pink Floyd’s ‘Pulse’ included a blinking LED within the cardboard box that was active and visible on store shelves. We felt that this also spoke to the heart of this broad and seemingly obvious patented product.
Sports Bra (No. 7867058) This Blackbird patent involves a “sports bra having an integral storage pouch.”
The Project Jengo community found that a submission on a public discussion forum that pre-dates the ’058 patent and disclosed an idea of modifying a bra by creating an incision in the inner lining and applying a velcro strip so as to form a resealable pocket within the bra… Or essentially the same invention.
As a Bonus – an Ex Parte Victory
Almost immediately after we announced Jengo, we received an anonymous donation from someone who shared our frustration with patent trolls. As we announced, this gift allowed us to expand Jengo by using some of the prior art to directly challenge other Blackbird patents in administrative proceedings.
We initiated an administrative challenge against Blackbird Patent 7,797,448 (“GPS-internet Linkage”). The patent describes in broad and generic terms “[a]n integrated system comprising the Global Positioning System and the Internet wherein the integrated system can identify the precise geographic location of both sender and receiver communicating computer terminals.” You don’t have to be particularly technical to realize how largely obvious and widely applicable such a concept would be, as many modern Internet applications attempt to integrate some sort of location services using GPS. This was a dangerous patent in the hands of a patent troll.
Based on the strength of the prior art we received from the Project Jengo community and the number of times Blackbird had asserted the ’448 Patent to elicit a settlement from startups, we filed for an ex parte reexamination (EPR) of the ’448 Patent by the US Patent & Trademark Office (USPTO). The EPR is an administrative proceeding that can be used to challenge obviously deficient patents in a less complex, lengthy, or costly exercise than federal litigation.
We submitted our EPR challenge in November 2017. Blackbird responded to the ex parte by attempting to amend their patent’s claims to make them more narrow in an effort to make their patent more defensible and avoid the challenge. In March 2018, the USPTO issued a Non-Final Office Action that proposed rejecting the ’448 Patent’s claims altogether because the claims were found to be preempted by prior art submitted by Project Jengo. Blackbird did not respond to the Office Action. And a few months later, in August 2018, the USPTO issued a final order in line with the office action, which cancelled the ’448 Patent’s claims. The USPTO’s decision means the ‘448 patent is invalid and no one can assert the incredibly broad terms of the ‘448 patent again.
Rewarding the Crowd
As promised, Cloudflare distributed more than $50,000 in cash awards to eighteen people who submitted prior art as part of the crowdsourced effort. We gave out more than $25,000 to people in support of their submissions related to the ’335 patent asserted against Cloudflare. Additionally we awarded more than $30,000 to submitters in support of our efforts to invalidate the other patents in Blackbird’s portfolio.
In general, we awarded bounties based on whether we incorporated the art found by the community into our legal filings, the analysis of the art as provided in the submission, whether someone else had previously submitted the art, and the strength and number of claims the art challenged in the specified Blackbird patent.
We asked many of the recent bounty winners why they decided to submit prior art to Project Jengo and received some of the following responses:
"Over the years I’ve been disappointed and angered by a number of patent cases where I feel that the patent system has been abused by so-called ‘patent trolls’ in order to stifle innovation and profit from litigation. With Jengo in particular, I was a fan of what Cloudflare had done previously with Universal SSL. When the opportunity arose to potentially make a difference with a real patent troll case, I was happy to try and help."
— Adam, Security Engineer
"I read the ’335 patent and thought it basically described a fundamental design principle of the world wide web (proxy servers). I was pretty sure such software was in widespread use by the priority date of the patent (1998). At that point I was curious if that was true so I did some Googling."
– David, Software Developer
"Personally, I believe the vast majority of software patents are obvious and trivial. They should have never been granted. At the same time, fighting a patent claim is costly and time consuming regardless of the patent’s merit, while filing the claim is relatively cheap. Patent trolls exploit this imbalance and, in turn, they stifle innovation. Project Jengo was a great opportunity to use my knowledge of prior academic work for a good cause."
– Kevin, Postdoctoral Research Scientist
"I’m pretty excited, I’ve never won a single thing in my life before. And to do it in service of taking down evil patent trolls? This is one of the best days of my life, no joke. I submitted because software patents are garbage and clearly designed to extort money from productive innovators for vague and obvious claims. Also, I was homeless at the time I submitted and was spending all day at the library anyway."
— Garrett, San Francisco
What was the Impact?
The whole point of Project Jengo was to flip the incentive structure around patent trolls, who assume they can buy broad patents, spend a little money to initiate litigation, and then sit back and expect that a great percentage of defendants will send them a check. Under a proper incentive structure, they should have to expend some effort to prove their claims have merit, and we wanted to make available information that would support other potential defendants who may want to push back against claims under Blackbird patents.
One very simple measure of the impact is to review the number of new lawsuits Blackbird is bringing with its patent portfolio, which is a public record. So what does Blackbird’s activity look like on that point?
In the one-year period immediately preceding Project Jengo, (Q2’16-Q2’17) Blackbird filed more than 65 cases. Since Project Jengo launched more than 2.5 years ago, the number of cases Blackbird has filed has fallen to an average rate of 10 per year.
Not only are they filing fewer cases, but Blackbird as an organization seems to be operating with fewer resources than they did at their peak. When we launched Project Jengo in May 2017, the Blackbird website identified a total team of 12: six lawyers, including two co-founders, four litigation counsel, as well as a patent analysis group of 6. Today, based on a review of the website and LinkedIn, it appears only three staff remain: one co-founder, one litigation counsel, and one member of the patent analysis group.
Ethics Complaints (sectionsubmitted by Cloudflare’s General Counsel, Doug Kramer)
We filed ethics complaints against both of Blackbird’s co-founders before the bar associations in Massachusetts, Illinois, and the USPTO based on their self-described “new model” of pursuing intellectual property claims. Our complaints were based on rules of professional conduct prohibiting lawyers from acquiring a cause of action to assert on their own behalf, or in the alternative, rules prohibiting attorneys to split contingency fees with a non-attorney.
We did not file such complaints lightly, as we take ethical standards seriously and don’t think such proceedings should be used merely to harass. In this case, we think the public perception of patent trolls, who are seen as lawyers chasing an easy buck by taking advantage of distortions in the litigation process, has damaged the public perception of attorneys and respect for the legal profession–the exact sort of values the ethical rules and bar associations are meant to protect.
We based our complaints on the assignment agreement we found filed with the USPTO, where Blackbird purchased the ’335 patent from an inventor in October 2016 for $1. It seemed apparent that the actual but undisclosed compensation between the parties was considerably more than $1, so Blackbird may have simply acquired the cause of action or the agreement involved an arrangement where Blackbird would split a portion of any recovered fees with the inventor. Such agreements are generally prohibited by the ethical rules.
In public statements, Blackbird’s defense to these allegations was that it (i) was not a law firm (despite the fact it is led exclusively by lawyers who are actively engaged in the litigation it pursues) and (ii) does not use contingency fee arrangements for the patents it acquires, but does use something “similar.” Both defenses were rather surprising to us. Isn’t an organization led and staffed exclusively by lawyers who are drafting complaints, filing papers with courts, and arguing before judges amount to a “law firm”? In fact, we found pleadings in other Blackbird cases where the Blackbird leadership asked to be treated as lawyers so they could have access to sensitive technical evidence in those cases that is usually off-limits to anyone but the lawyers. And what does it mean for an agreement to be merely “similar” to a contingency agreement?
The disciplinary proceedings in front of bar associations are generally confidential, so we are limited in our ability to report out developments in those cases. But regardless of the outcome, we’ve only approached bar associations in two states. Getting this back on the right track will require more than successful adjudications in front of such committees. Instead, it will take a broader change in orientation by these professional associations across the country to view such matters as more than mere political disputes or arguments between active litigants.
Our questions go to the very heart of ensuring an ethical legal profession, they are meant to determine what safeguards should be put in place to make sure that attorneys who take the oath are held to a standard beyond mere greed or base opportunism. They go to the question of whether being an attorney is merely a job or if there are higher standards they should be held to, making sure their monopoly over the ability to bring lawsuits as officers of the court (and all the implications, costs, and power that represents) is only wielded by people who can be trusted to do so responsibly. Otherwise, what’s the point of ethical standards?
That’s all … for now
We’ve said from the beginning that Project Jengo was a response to the patent troll litigation and we would end it as soon as the case was over. And now it is. Although we are proud of our work on this issue, we need to turn our focus back to the company’s mission — to help build a better Internet. But we may be back at some point. Patent trolls remain a risk to growing companies like Cloudflare and nothing in this experience has persuaded us that settling a patent lawsuit is ever the right answer. We don’t plan to settle, and if brought into such litigation again at some point in the future, we think we have a pretty good blueprint for how to respond.
The Blackbird prior art will remain available here, and we remain available to consult with our colleagues at other companies who face these issues, as we have done many times over the past few years.
Finally, we would like to express our sincere gratitude to the community who researched the Blackbird patent portfolio and helped us fight this troll. It was our confidence in all of you that inspired the idea of Project Jengo in the first place, so its success belongs to you.
The collective thoughts of the interwebz
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.