An overview of Cloudflare’s logging pipeline

Post Syndicated from Colin Douch http://blog.cloudflare.com/author/colin/ original https://blog.cloudflare.com/an-overview-of-cloudflares-logging-pipeline


One of the roles of Cloudflare’s Observability Platform team is managing the operation, improvement, and maintenance of our internal logging pipelines. These pipelines are used to ship debugging logs from every service across Cloudflare’s infrastructure into a centralised location, allowing our engineers to operate and debug their services in near real time. In this post, we’re going to go over what that looks like, how we achieve high availability, and how we meet our Service Level Objectives (SLOs) while shipping close to a million log lines per second.

Logging itself is a simple concept. Virtually every programmer has written a Hello, World! program at some point. Printing something to the console like that is logging, whether intentional or not.

Logging pipelines have been around since the beginning of computing itself. Starting with putting string lines in a file, or simply in memory, our industry quickly outgrew the concept of each machine in the network having its own logs. To centralise logging, and to provide scaling beyond a single machine, we invented protocols such as the BSD Syslog Protocol to provide a method for individual machines to send logs over the network to a collector, providing a single pane of glass for logs over an entire set of machines.

Our logging infrastructure at Cloudflare is a bit more complicated, but still builds on these foundational principles.

The beginning

Logs at Cloudflare start the same as any other, with a println. Generally systems don’t call println directly however, they outsource that logic to a logging library. Systems at Cloudflare use various logging libraries such as Go’s zerolog, C++’s KJ_LOG, or Rusts log, however anything that is able to print lines to a program’s stdout/stderr streams is compatible with our pipeline. This offers our engineers the greatest flexibility in choosing tools that work for them and their teams.

Because we use systemd for most of our service management, these stdout/stderr streams are generally piped into systemd-journald which handles the local machine logs. With its RateLimitBurst and RateLimitInterval configurations, this gives us a simple knob to control the output of any given service on a machine. This has given our logging pipeline the colloquial name of the “journal pipeline”, however as we will see, our pipeline has expanded far beyond just journald logs.

Syslog-NG

While journald provides us a method to collect logs on every machine, logging onto each machine individually is impractical for debugging large scale services. To this end, the next step of our pipeline is syslog-ng. Syslog-ng is a daemon that implements the aforementioned BSD syslog protocol. In our case, it reads logs from journald, and applies another layer of rate limiting. It then applies rewriting rules to add common fields, such as the name of the machine that emitted the log, the name of the data center the machine is in, and the state of the data center that the machine is in. It then wraps the log in a JSON wrapper and forwards it to our Core data centers.

journald itself has an interesting feature that makes it difficult for some of our use cases – it guarantees a global ordering of every log on a machine. While this is convenient for the single node case, it imposes the limitation that journald is single-threaded. This means that for our heavier workloads, where every millisecond of delay counts, we provide a more direct path into our pipeline. In particular, we offer a Unix Domain Socket that syslog-ng listens on. This socket operates as a separate source of logs into the same pipeline that the journald logs follow, but allows greater throughput by eschewing the need for a global ordering that journald enforces. Logging in this manner is a bit more involved than outputting logs to the stdout streams, as services have to have a pipe created for them and then manually open that socket to write to. As such, this is generally reserved for services that need it, and don’t mind the management overhead it requires.

log-x

Our logging pipeline is a critical service at Cloudflare. Any potential delays or missing data can cause downstream effects that may hinder or even prevent the resolving of customer facing incidents. Because of this strict requirement, we have to offer redundancy in our pipeline. This is where the operation we call “log-x” comes into play.

We operate two main core data centers. One in the United States, and one in Europe. From each machine, we ship logs to both of these data centers. We call these endpoints log-a, and log-b. The log-a and log-b receivers will insert the logs into a Kafka topic for later consumption. By duplicating the data to two different locations, we achieve a level of redundancy that can handle the failure of either data center.

The next problem we encounter is that we have many data centers all around the world, which at any time due to changing Internet conditions may become disconnected from one, or both core data centers. If the data center is disconnected for long enough we may end up in a situation where we drop logs to either the log-a or log-b receivers. This would result in an incomplete view of logs from one data center and is unacceptable; Log-x was designed to alleviate this problem. In the event that syslog-ng fails to send logs to either log-a or log-b, it will actually send the log twice to the available receiver. This second copy will be marked as actually destined for the other log-x receiver. When a log-x receiver receives such a log, it will insert it into a different Kafka queue, known as the Dead Letter Queue (DLQ). We then use Kafka Mirror Maker to sync this DLQ across to the data center that was inaccessible. With this logic log-x allows us to maintain a full copy of all the logs in each core data center, regardless of any transient failures from any of our data centers.

Kafka

When logs arrive in the core data centers, we buffer them in a Kafka queue. This provides a few benefits. Firstly, it means that any consumers of the logs can be added without any changes – they only need to register with Kafka as a consumer group on the logs topic. Secondly, it allows us to tolerate transient failures of the consumers without losing any data. Because the Kafka clusters in the core data centers are much larger than any single machine, Kafka allows us to tolerate up to eight hours of total outage for our consumers without losing any data. This has proven to be enough to recover without data loss from all but the largest of incidents.

When it comes to partitioning our Kafka data, we have an interesting dilemma. Rather arbitrarily, the syslog protocol only supports timestamps up to microseconds. For our faster log emitters, this means that the syslog protocol cannot guarantee ordering with timestamps alone. To work around this limitation, we partition our logs using a key made up of both the host, and the service name. Because Kafka guarantees ordering within a partition, this means that any logs from a service on a machine are guaranteed to be ordered between themselves. Unfortunately, because logs from a service can have vastly different rates between different machines, this can result in unbalanced Kafka partitions. We have an ongoing project to move towards Open Telemetry Logs to combat this.

Onward to storage

With the logs in Kafka, we can proceed to insert them into a more long term storage. For storage, we operate two backends. An ElasticSearch/Logstash/Kibana (ELK) stack, and a Clickhouse cluster.

For ElasticSearch, we split our cluster of 90 nodes into a few types. The first being “master”

nodes. These nodes act as the ElasticSearch masters, and coordinate insertions into the cluster. We then have “data” nodes that handle the actual insertion and storage. Finally, we have the “HTTP” nodes that handle HTTP queries. Traditionally in an ElasticSearch cluster, all the data nodes will also handle HTTP queries, however because of the size of our cluster and shards we have found that designating only a few nodes to handle HTTP requests greatly reduces our query times by allowing us to take advantage of aggressive caching.

On the Clickhouse side, we operate a ten node Clickhouse cluster that stores our service logs. We are in the process of migrating this to be our primary storage, but at the moment it provides an alternative interface into the same logs that ElasticSearch provides, allowing our Engineers to use either Lucene through the ELK stack, or SQL and Bash scripts through the Clickhouse interface.

What’s next?

As Cloudflare continues to grow, our demands on our Observability systems, and our logging pipeline in particular continue to grow with it. This means that we’re always thinking ahead to what will allow us to scale and improve the experience for our engineers. On the horizon, we have a number of projects to further that goal including:

  • Increasing our multi-tenancy capabilities with better resource insights for our engineers
  • Migrating our syslog-ng pipeline towards Open Telemetry
  • Tail sampling rather than our probabilistic head sampling we have at the moment
  • Better balancing for our Kafka clusters

If you’re interested in working with logging at Cloudflare, then reach out – we’re hiring!

Building a Partner Program: The Zabbix Advantage

Post Syndicated from Michael Kammer original https://blog.zabbix.com/building-a-partner-program-the-zabbix-advantage/27164/

At Zabbix, our emphasis on high performance, functionality, and reliability has led to the creation of one of the most popular monitoring solutions on the market. It’s so popular, in fact, that we get near-constant requests for Zabbix professional consulting, advice, support, and training from almost every corner of the world.

That’s why we created the Zabbix Partner Program. Our partner program was designed with one goal in mind – to get our services to the widest possible audience of qualified buyers by allowing customers to purchase them through a network of verified Zabbix partners as well as from Zabbix directly.

Our partners create high value for thousands of customers who would not otherwise enjoy access to Zabbix services by providing complete localization in terms of linguistic and cultural compatibility, availability across time zones, in-person access, and flexibility around currencies and payments.

To do that as effectively as possible, we’ve divided our partners into 3 categories:

Resellers. These are companies that promote and resell Zabbix services. Their job is to locate leads, present and promote Zabbix products and services, consult the leads regarding their ideal solutions, and arrange the contracts. At that point, Zabbix steps in and provides the services. Resellers are a great resource for customers who are limited by local regulations when it comes to buying Zabbix services in their local currency or from companies registered in their own country.

Certified Partners. Certified partners can also promote and resell Zabbix services, but they’re also officially authorized to deliver selected Zabbix services and solutions in their local languages. The ease of access and a common language allows certified partners to stay in close contact with customers. They can also sell their own value-adding services alongside Zabbix services.

Premium Partners. A premium partner has the same authorization as certified partners, but premium partner status is reserved for partners with the highest expertise and experience. Premium partners can participate in highly sophisticated Zabbix implementation, integration, and support projects.

Building a winning partner program has taught us a few things about the process, so without further ado, we’d like to share 6 best practices that we adhere to when it comes to cultivating and expanding our network of partners.

Set realistic goals

Years of running a partner program have taught us that success is impossible without clearly defined goals and success metrics. Setting firm, realistic goals for a program is the only way to measure its effectiveness and ROI. After a few quarters, it should be possible to compare performance to goals and see whether changes need to be made.

Accordingly, we make sure that Zabbix executives, sales teams, and partners are aware that getting a new program up and running (or making changes to an existing program) takes time. Expecting instant results is not realistic – we’ve learned that a ramp-up period of a few months is usually reasonable.

Make expectations clear

Nothing kills momentum faster than confusion. That’s why it’s important to make sure that partners have a solid understanding of everything that’s being asked of them. We’ve learned to give partners concise goals and objectives so that everyone is on the same page. We also create annual business plans for all three partnership programs, review them quarterly, and reward success.

Having the same KPIs as partners is also important. When different metrics for success exist, we run the risk of our partners being less enthusiastic about taking actions that will increase the success of Zabbix but may do less for them. In our experience, it’s better to build partnerships around a joint success target so that when partners win, we win.

Support your partners

At Zabbix, supporting our partners means providing outstanding sales, marketing, and technical support, all of which shows that we’re invested in their success as much as our own. Our partnership team helps partners with all presales-related questions, organizes demo calls, manages the deal registration to protect partner deals, patriciates in joint calls with customers, and helps with all possible legal questions and certifications.

Apart from day-to-day pre-sales support, we organize and participate in joint Zabbix marketing events of different formats together with our partners. These meetups, meetings, conferences, and external events organized by other vendors around the globe are designed to spread the word about Zabbix solutions and services while helping our partners generate new leads. During these events, our partners demonstrate their recent use-cases and serve as experts for the rest of the partner network and the wider Zabbix community.

Build Trust

Trust is the foundation of all partnerships, and we find that our partners trust us because we deliver the support and tools they need to be successful. It’s why we work hard to keep our partners updated with product developments and industry trends, and we continuously educate them on how to sell and overcome roadblocks.

We even allow some of our partners to conduct official Zabbix trainings, provided they have a certified trainer available. When an existing partner wants to become a training partner, we discuss their needs and plan their training certification together.

Measure and monitor

Whether launching a new program or scaling up an existing one, measuring the right key performance indicators (KPIs) can mean the difference between growth and chaos. If a business doesn’t know what to measure and optimize for their partner program, they won’t know what to improve if growth stalls out, and you’ll struggle to explain how partnerships contribute value.

It’s impossible to get far on the road to success without measuring progress along the way. That’s why we review goals and metrics with our partners every quarter, assess what’s working well and what’s missing the mark, and adapt and adjust if needed. We’ve learned not to change things up too often, but we’re always open to making tweaks that will amplify success.

Communicate effectively

One of the most important ingredients of any successful partner program is communication. It’s essential to keep partners informed about new products, promotions, and other important updates. That involves knowing the audience and understanding what each partner type and their respective employees are interested in and when.

A cornerstone of the Zabbix Partner Program is our ability to actively listen to our partners’ feedback. Our experience is that getting ahead of issues and concerns strengthens relationships, maintains trust, and guarantees that our partners feel supported and valued.

Conclusion

Becoming a Zabbix Partner is an ideal way to get recognized by potential customers and increase the visibility of your business, while also getting a leg up on your competitors by using technical support according to a professional service-level agreement.

In addition, you can count on discounts on all Zabbix services, the ability to access pre-sale consulting services, and participation in joint marketing events.

To find out more about our partner program and sign up, visit the Zabbix Partners page.

The post Building a Partner Program: The Zabbix Advantage appeared first on Zabbix Blog.

Second Interdisciplinary Workshop on Reimagining Democracy

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/second-interdisciplinary-workshop-on-reimagining-democracy.html

Last month, I convened the Second Interdisciplinary Workshop on Reimagining Democracy (IWORD 2023) at the Harvard Kennedy School Ash Center. As with IWORD 2022, the goal was to bring together a diverse set of thinkers and practitioners to talk about how democracy might be reimagined for the twenty-first century.

My thinking is very broad here. Modern democracy was invented in the mid-eighteenth century, using mid-eighteenth-century technology. Were democracy to be invented from scratch today, with today’s technologies, it would look very different. Representation would look different. Adjudication would look different. Resource allocation and reallocation would look different. Everything would look different, because we would have much more powerful technology to build on and no legacy systems to worry about.

Such speculation is not realistic, of course, but it’s still valuable. Everyone seems to be talking about ways to reform our existing systems. That’s critically important, but it’s also myopic. It represents a hill-climbing strategy of continuous improvements. We also need to think about discontinuous changes that you can’t easily get to from here; otherwise, we’ll be forever stuck at local maxima.

I wrote about the philosophy more in this essay about IWORD 2022. IWORD 2023 was equally fantastic, easily the most intellectually stimulating two days of my year. The event is like that; the format results in a firehose of interesting.

Summaries of all the talks are in the first set of comments below. (You can read a similar summary of IWORD 2022 here.) Thank you to the Ash Center and the Belfer Center at Harvard Kennedy School, and the Knight Foundation, for the funding to make this possible.

Next year, I hope to take the workshop out of Harvard and somewhere else. I would like it to live on for as long as it is valuable.

Now, I really want to explain the format in detail, because it works so well.

I used a workshop format I and others invented for another interdisciplinary workshop: Security and Human Behavior, or SHB. It’s a two-day event. Each day has four ninety-minute panels. Each panel has six speakers, each of whom presents for ten minutes. Then there are thirty minutes of questions and comments from the audience. Breaks and meals round out the day.

The workshop is limited to forty-eight attendees, which means that everyone is on a panel. This is important: every attendee is a speaker. And attendees commit to being there for the whole workshop; no giving your talk and then leaving. This makes for a very collaborative environment. The short presentations means that no one can get too deep into details or jargon. This is important for an interdisciplinary event. Everyone is interesting for ten minutes.

The final piece of the workshop is the social events. We have a night-before opening reception, a conference dinner after the first day, and a final closing reception after the second day. Good food is essential.

Honestly, it’s great but it’s also it’s exhausting. Everybody is interesting for ten minutes. There’s no down time to zone out or check email. And even though a shorter event would be easier to deal with, the numbers all fit together in a way that’s hard to change. A one-day event means only twenty-four attendees/speakers, and that’s not a critical mass. More people per panel doesn’t work. Not everyone speaking creates a speaker/audience hierarchy, which I want to avoid. And a three-day, slower-paced event is too long. I’ve thought about it long and hard; the format I’m using is optimal.

The 6.7 kernel has been released

Post Syndicated from corbet original https://lwn.net/Articles/957098/

Linus has released the 6.7 kernel.

End result: 6.7 is (in number of commits: over 17k non-merge
commits, with 1k+ merges) one of the largest kernel releases we’ve
ever had, but the extra rc8 week was purely due to timing with the
holidays, not about any difficulties with the larger release.

Some of the headline features in this release are:
the removal of support for the Itanium
architecture,
the first part of the futex2 API,
futex
support in io_uring,
the BPF exceptions mechanism,
the bcachefs filesystem,
the TCP
authentication option,
the kernel samepage merging smart
scan mode, and
networking
support for the Landlock security module.
See the LWN merge-window summaries (part 1,
part 2) and the (in-progress) KernelNewbies 6.7 page for
more information.

Friday Squid Blogging—18th Anniversary Post: New Species of Pygmy Squid Discovered

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/friday-squid-blogging-new-species-of-pygmy-squid-discovered.html

They’re Ryukyuan pygmy squid (Idiosepius kijimuna) and Hannan’s pygmy squid (Kodama jujutsu). The second one represents an entire new genus.

As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.

And, yes, this is the eighteenth anniversary of Friday Squid Blogging. The first squid post is from January 6, 2006, and I have been posting them weekly since then. Never did I believe there would be so much to write about squid—but the links never seem to end.

Read my blog posting guidelines here.

Metasploit Weekly Wrap-Up 1/05/2024

Post Syndicated from Jacquie Harris original https://blog.rapid7.com/2024/01/05/metasploit-weekly-wrap-up-40/

New module content (2)

Splunk __raw Server Info Disclosure

Metasploit Weekly Wrap-Up 1/05/2024

Authors: KOF2002, h00die, and n00bhaxor
Type: Auxiliary
Pull request: #18635 contributed by n00bhaxor
Path: gather/splunk_raw_server_info

Description: This PR adds a module for an authenticated Splunk information disclosure vulnerability. This module gathers information about the host machine and the Splunk install including OS version, build, CPU arch, Splunk license keys, etc.

[msf](Jobs:0 Agents:0) > use auxiliary/gather/splunk_raw_server_info 
[msf](Jobs:0 Agents:0) auxiliary(gather/splunk_raw_server_info) > set rhosts 127.0.0.1
rhosts => 127.0.0.1
[msf](Jobs:0 Agents:0) auxiliary(gather/splunk_raw_server_info) > set username admin
username => admin
[msf](Jobs:0 Agents:0) auxiliary(gather/splunk_raw_server_info) > set password splunksplunk
password => splunksplunk
[msf](Jobs:0 Agents:0) auxiliary(gather/splunk_raw_server_info) > set verbose true
verbose => true
[msf](Jobs:0 Agents:0) auxiliary(gather/splunk_raw_server_info) > run
[*] Running module against 127.0.0.1
[+] Output saved to /root/.msf4/loot/20231220204049_default_127.0.0.1_splunk.system.st_943292.json
[+] Hostname: 523a845e8652
[+] CPU Architecture: x86_64
[+] Operating System: Linux
[+] OS Build: #1 SMP PREEMPT_DYNAMIC Debian 6.5.6-1kali1 (2023-10-09)
[+] OS Version: 6.5.0-kali3-amd64
[+] Splunk Version: 7.1.0
[+] Trial Version?: false
[+] Splunk Forwarder?: false
[+] Splunk Product Type: splunk
[+] License State: OK
[+] License Key(s): ["FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"]
[+] Splunk Server Roles: ["indexer", "license_master"]
[+] Splunk Server Startup Time: 2023-12-21 01:40:02
[*] Auxiliary module execution completed

Craft CMS unauthenticated Remote Code Execution (RCE)

Authors: Thanh, chybeta, and h00die-gr3y [email protected]
Type: Exploit
Pull request: #18612 contributed by h00die-gr3y
Path: linux/http/craftcms_unauth_rce_cve_2023_41892

Description: This adds an exploit module that leverages a remote code execution vulnerability in CraftCMS versions between 4.0.0-RC1 and 4.4.14. This vulnerability is identified as CVE-2023-41892 and allows an unauthenticated attacker to execute arbitrary code remotely.

Enhancements and features (2)

  • #18610 from sjanusz-r7 – This PR enables the Metasploit Payload Warnings feature by default. When enabled Metasploit will output warnings about missing Metasploit payloads, for instance if they were removed by antivirus.
  • #18632 from jvoisin – This PR adds improvements to the Glibc Tunables Privilege Escalation module. In the event the file command is not present on the target the module will try to use the readelf command to get the ld.so build ID and determine whether or not the target is compatible with the exploit.

Documentation

You can find the latest Metasploit documentation on our docsite at docs.metasploit.com.

Get it

As always, you can update to the latest Metasploit Framework with msfupdate
and you can get more details on the changes since the last blog post from
GitHub:

If you are a git user, you can clone the Metasploit Framework repo (master branch) for the latest.
To install fresh without using git, you can use the open-source-only Nightly Installers or the
commercial edition Metasploit Pro

New Open Source Tool for Consistency in Cassandra Migrations

Post Syndicated from Elliott Sims original https://www.backblaze.com/blog/new-open-source-tool-for-consistency-in-cassandra-migrations/

A decorative image showing the Cassandra logo with a function represented by two servers on either side of the logo.

Sometimes you find a problem because something breaks, and other times you find a problem the good way—by thinking it through before things break. This is a story about one of those bright, shining, lightbulb moments when you find a problem the good way.

On the Backblaze Site Reliability Engineering (SRE) team, we were thinking through an upcoming datacenter migration in Cassandra. We were running through all of the various types of queries we would have to do when we had the proverbial “aha” moment. We discovered an inconsistency in the way Cassandra handles lightweight transactions (LWTs).

If you’ve ever tried to do a datacenter migration in Cassandra and something got corrupted in the process but you couldn’t figure out why or how—this might be why. I’m going to walk through a short intro on Cassandra, how we use it, and the issue we ran into. Then, I’ll explain the workaround, which we open sourced. 

Get the Open Source Code

You can download the open source code from our Git repository. We’d love to know how you’re using it and how it’s working for you—let us know in the comments.

How We Use Cassandra

First, if you’re not a Cassandra dev, I should mention that when we say “datacenter migration” it means something slightly different in Cassandra than what it sounds like. It doesn’t mean a data center migration in the physical sense (although you can use datacenter migrations in Cassandra when you’re moving data from one physical data center to another). In the simplest terms, it involves moving data between two Cassandra or Cassandra-compatible database replica sets within a cluster.

And, if you’re not familiar with Cassandra at all, it’s an open-source, NoSQL, distributed database management system. It was created to handle large amounts of data across many commodity servers, so it fits our use case—lots of data, lots of servers. 

At Backblaze, we use Cassandra to index filename to location for data stored in Backblaze B2, for example. Because it’s customer data and not just analytics, we care more about durability and consistency than some other applications of Cassandra. We run with three replicas in a single datacenter and “batch” mode to require writes to be committed to disk before acknowledgement rather than the default “periodic.”

Datacenter migrations are an important aspect of running Cassandra, especially on bare metal. We do a few datacenter migrations per year either for physical data moves, hardware refresh, or to change certain cluster layout parameters like tokens per host that are otherwise static. 

What Are LWTs and Why Do They Matter for Datacenter Migrations in Cassandra?

First of all, LWTs are neither lightweight nor transactions, but that’s neither here nor there. They are an important feature in Cassandra. Here’s why. 

Cassandra is great at scaling. In something like a replicated SQL cluster, you can add additional replicas for read throughput, but not writes. Cassandra scales writes (as well as reads) nearly linearly with the number of hosts—into the hundreds. Adding nodes is a fairly straightforward and “automagic” process as well, with no need to do something like manual token range splits. It also handles individual down nodes with little to no impact on queries. Unfortunately, these properties come with a trade-off: a complex and often nonintuitive consistency model that engineers and operators need to understand well.

In a distributed database like Cassandra, data is replicated across multiple nodes for durability and availability. 

Although databases generally allow multiple reads and writes to be submitted at once, they make it look to the outside world like all the operations are happening in order, one at a time. This property is known as serializability, and Cassandra is not serializable. Although it does have a “last write wins” system, there’s no transaction isolation and timestamps can be identical. 

It’s possible, for example, to have a row that has some columns from one write and other columns from another write. It’s safe if you’re only appending additional rows, but mutating existing rows safely requires careful design. Put another way, you can have two transactions with different data that, to the system, appear to have equal priority. 

How Do LWTs Solve This Problem?

As a solution for cases where stronger consistency is needed, Cassandra has a feature called “Lightweight Transactions” or LWTs. These are not really identical to traditional database transactions, but provide a sort of “compare and set” operation that also guarantees pending writes are completed before answering a read. This means if you’re trying to change a row’s value from “A” to “B”, a simultaneous attempt to change that row from “A” to “C” will return a failure. This is accomplished by doing a full—not at all lightweight—Paxos round complete with multiple round trips and slow expensive retries in the event of a conflict.

In Cassandra, the minimum consistency level for read and write operations is ONE, meaning that only a single replica needs to acknowledge the operation for it to be considered successful. This is fast, but in a situation where you have one down host, it could mean data loss, and later reads may or may not show the newest write depending on which replicas are involved and whether they’re received the previous write. For better durability and consistency, Cassandra also provides various quorum levels that require a response from multiple replicas, as well as an ALL consistency that requires responses from every replica.

Cassandra Is My Type of Database

Curious to know more about consistency limitations and LWTs in Cassandra? Christopher Batey’s presentation at the 2016 Cassandra Summit does a good job of explaining the details.

The Problem We Found With LWTs During Datacenter Migrations

Usually we use one datacenter in Cassandra, but there are circumstances where we sometimes stand up a second datacenter in the cluster and migrate to it, then tear down the original. We typically do this either to change num_tokens, to move data when we’re refreshing hardware, or to physically move to another nearby data center.

The TL:DR

We reasoned through the interaction between LWTs/serial and datacenter migrations and found a hole—there’s no guarantee of LWT correctness during a topology change (that is, a change to the number of replicas) large enough to change the number of replicas needed to satisfy quorum. It turns out that combining LWTs and datacenter migrations can violate consistency guarantees in subtle ways without some specific steps and tools to work around it.

The Long Version

Let’s say you are standing up a new datacenter, and you need to copy an existing datacenter to it. So, you have two datacenters—datacenter A, the existing datacenter, and datacenter B, the new datacenter. Let’s say datacenter A has three replicas you need to copy for simplicity’s sake, and you’re using quorum writes to ensure consistency.

Refresher: What is Quorum-Based Consistency in Cassandra?

Quorum consistency in Cassandra is based on the concept that a specific number of replicas must participate in a read or write operation to ensure consistency and availability—a majority (n/2 +1) of the nodes must respond before considering the operation as successful. This ensures that the data is durably stored and available even if a minority of replicas are unavailable.

You have different types of quorum you can choose from, and here’s how those defaults make a decision: 

  • Local quorum: Two out of the three replicas in the datacenter I’m talking to must respond in order to return success. I don’t care about the other datacenter.
  • Global quorum: Four out of the six total replicas must respond in order to return success, and it doesn’t matter which datacenter they come from.
  • Each quorum: Two out of the three replicas in each datacenter must respond in order to return success.

Most of these quorum types also have a serial equivalent for LWTs.

Type of Quorum Serial Regular
Local LOCAL_SERIAL LOCAL_QUORUM
Each unsupported EACH_QUORUM
Global SERIAL QUORUM

The problem you might run into, however, is that LWTs do not have an each_serial mode. They only have local and global. There’s no way to tell the LWT you want quorum in each datacenter. 

local_serial is good for performance, but transactions on different datacenters could overlap and be inconsistent. serial is more expensive, but normally guarantees correctness as long as all queries agree on cluster size. But what if a query straddles a topology change that changes quorum size? 

Let’s use global quorum to show how this plays out. If a LWT starts when RF=3, at least two hosts must process it. 

While it’s running, the topology changes to two datacenters (A and B) each with RF=3 (so six replicas total) with a quorum of four. There’s a chance that a query affecting the same partition could then run without overlapping nodes, which means consistency guarantees are not maintained for those queries. For that query, quorum is four out of six where those four could be the three replicas in datacenter B and the remaining replica in datacenter A. 

Those two queries are on the same partition, but they’re not overlapping any hosts, so they don’t know about each other. It violates the LWT guarantees.

The Solution to LWT Inconsistency

What we needed was a way to make sure that the definition of “quorum” didn’t change too much in the middle of a LWT running. Some change is okay, as long as old and new are guaranteed to overlap.

To account for this, you need to change the replication factor one level at a time and make sure there are no transactions still running that started before the previous topology change before you make the next. Three replicas with a quorum of two can only change to four replicas with a quorum of three. That way, at least one replica must overlap. The same thing happens when you go from four to five replicas or five to six replicas. This also applies when reducing the replication factor, such as when tearing down the old datacenter after everything has moved to the new one.

Then, you just need to make sure no LWT overlaps multiple changes. You could just wait long enough that they’ve timed out, but it’s better to be sure. This requires querying the internal-only system.paxos table on each host in the cluster between topology changes.

We built a tool that checks to see whether there are still transactions running from before we made a topology change. It reads system.paxos on each host, ignoring any rows with proposal_ballot=null, and records them. Then after a short delay, it re-reads system.paxos, ignoring any rows that weren’t present in the previous run, or any with proposal_ballot=null in either read, or any where in_progress_ballot has changed. Any remaining rows are potentially active transactions. 

This worked well the first few times that we used it, on 3.11.6. To our surprise, when we tried to migrate a cluster running 3.11.10 the tool reported hundreds of thousands of long-running LWTs. After a lot of digging, we found a small (but fortunately well-commented) performance optimization added as part of a correctness fix (CASSANDRA-12126), which means proposal_ballot does not get set to null if the proposal is empty/noop. To work around this, we had to actually parse the proposal field. Fortunately all we need is the is_empty flag in the third field, so no need to reimplement the full parsing code. A big impact to us for a seemingly small and innocuous change piggy-backed onto a correctness fix, but that’s the risk of directly reading internal-only tables. 

We’ve used the tool several times now for migrations with good results, but it’s still relatively basic and limited. It requires running repeatedly until all transactions are complete, and sometimes manual intervention to deal with incomplete transactions. In some cases we’ve been able to force-commit a long-pending LWT by doing a SERIAL read of the partition affected, but in a couple of cases we actually ended up running across LWTs that still didn’t seem to complete. Fortunately in every case so far it was in a temporary table and a little work allowed us to confirm that we no longer needed the partition at all and could just delete it.

Most people who use Cassandra may never run across this problem, and most of those who do will likely never track down what caused the small mystery inconsistency around the time they did a datacenter migration. If you rely on LWTs and are doing a datacenter migration, we definitely recommend going through the extra steps to guarantee consistency until and unless Cassandra implements an EACH_SERIAL consistency level.

Using the Tool

If you want to use the tool for yourself to help maintain consistency through datacenter migrations, you can find it here. Drop a note in the comments to let us know how it’s working for you and if you think of any other ways around this problem—we’re all ears!

If You’ve Made It This Far

You might be interested in signing up for our Developer Newsletter where our resident Chief Technical Evangelist, Pat Patterson, shares the latest and greatest ways you can use B2 Cloud Storage in your applications.

The post New Open Source Tool for Consistency in Cassandra Migrations appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

The collective thoughts of the interwebz